This is the multi-page printable view of this section. Click here to print.
Documentation
- 1: Using a Project Hummingbird container image
- 1.1: Using a Project Hummingbird container image
- 1.2: Custom CA Certificates (OpenSSL)
- 1.3: Custom CA Certificates (Java)
- 1.4: Custom CA Certificates (Python)
- 2: Contributing to a Project Hummingbird container image
- 2.1: Quickstart Guide
- 2.2: Development Workflow
- 2.3: Resolving Merge Conflicts in Generated Files
- 2.4: Adding New Images
- 2.5: Adding FIPS Variants
- 2.6: Testing Guide
- 2.7: Adding Kubernetes Tests
- 2.8: Image Configuration Reference
- 2.9: Test Configuration Reference
- 3: Operating Hummingbird infrastructure
- 3.1: Excluding Packages from Images
- 3.2: Managing Konflux Comments
- 3.3: Running Conforma Checks Locally
- 3.4: Upgrading Docsy
- 3.5: Version Constraints for Multi-Version Images
- 3.6: Retrying Konflux Checks
- 3.7: Switching Fedora Streams
- 3.8: Disabling Rawhide for Images
- 3.9: MR Artifact Promotion
- 3.10: Red Hat Catalog Environment Promotion
- 3.11: Setting Up a Konflux Cluster
- 3.12: Deleting RPMs from Hummingbird Repos
- 3.13: Red Hat Catalog UAT Program
- 3.14: Updating Dist-git Packages
- 3.15: Writing Documentation
- 3.16: Adding Independent Packages
- 3.17: Pulp Access
- 3.18: Measuring Documentation Quality
- 3.19: Rebuilding Packages
- 3.20: Setting Up Pulp Repositories
- 3.21: Package Metadata Fields
- 3.22: Debugging Build Failures
- 3.23: Package Modification Tracking
- 3.24: Private RPM Repositories
- 3.25: Reporting CVE Data Issues
- 3.26: Lookaside Cache Access
- 3.27: Upstream Diff Analysis
- 3.28: GPG Source Verification
- 3.29: Rebasing the Buildroot to a New Fedora Release
- 4: Lot's of background information
- 4.1: RPMs repository
- 4.1.1: RPM Pipeline
- 4.1.2: Konflux Resource Deployment
- 4.1.3: Source Pipeline Tool
- 4.2: Agent-Friendly Documentation
- 4.3: Containers repository
- 4.3.1: Image Pipeline
- 4.3.2: Global Variables Reference
- 4.3.3: Konflux Resource Deployment
- 4.3.4: Security Labels and Metadata
- 4.3.5: Container Image Labels
- 4.3.6: CI Scripts
- 4.3.6.1: build_images.sh
- 4.3.6.2: run_tests_container.sh
- 4.3.6.3: run_tests_k8s.sh
- 4.3.6.4: retrigger_failed_checks.py
- 4.3.6.5: gitlab_sync.py
- 4.4: K8s Test Pipeline
- 4.4.1: Pipeline Design
- 4.4.2: Test Format
- 4.4.3: EaaS and Debugging
- 4.5: Tools repository
- 4.5.1: Message Bus Architecture
- 4.5.2: Alloy CloudWatch
- 4.5.3: Deployment Bot
- 4.5.4: Grafana CloudWatch
- 4.5.5: Hummingbird AI
- 4.5.6: Hummingbird Events Topic
- 4.5.7: Vertex AI Cost Metrics
- 4.5.8: Hummingbird Sandbox
- 4.5.9: SNS S3 Archiver
- 4.5.10: Hummingbird Tools
- 4.5.11: ProdSec RPM Catalog
- 4.5.12: Chainguard Comparison
- 4.5.13: CVE Analysis
- 4.5.13.1: CVE Analysis Reference
- 4.5.14: Dashboard MR Linker
- 4.5.15: GitLab Event Forwarder
- 4.5.16: Hummingbird MR Human Tracker
- 4.5.17: MR Auto-Approver
- 4.5.18: Jira Event Forwarder
- 4.5.19: Container Catalog
- 4.5.19.1: Container Catalog Design Decisions
- 4.5.20: Hummingbird Agent
- 4.5.20.1: Hummingbird Agent Model Loop
- 4.5.20.2: Hummingbird Agent Design
- 4.5.20.3: Hummingbird Agent Design — Engine
- 4.5.20.4: Hummingbird Agent Evals
- 4.5.20.5: Documentation Quality Evaluation
- 4.5.20.6: Hummingbird Agent Design — Runtime
- 4.5.20.7: Hummingbird Agent Design — Decisions
- 4.5.21: Red Hat Catalog
- 4.5.22: Hummingbird Data Flow
- 4.5.23: Kubernetes Event Forwarder
- 4.5.24: MR Artifact Promotion
- 4.5.25: Jira Image Requests
- 4.5.26: PAC Trigger
- 4.5.27: Hummingbird Status
- 4.5.28: Lambda S3 Cache
- 4.5.29: RPM CVE Count
- 4.5.30: Hummingbird Dashboard
- 4.5.31: Pipeline Timing
- 4.5.31.1: RPM pipeline timing design
- 4.5.31.2: Pipeline timing implementation plan
- 4.5.32: RPM Tracker
- 4.5.33: Work Queue Service
- 4.5.33.1: Work Queue Service Design
- 4.5.33.2: Work Queue Service Design Decisions
- 4.5.34: Service metrics
- 4.5.35: VEX Checker
- 4.5.36: Hummingbird CVE agent
- 4.5.37: Upstream Monitor
- 4.5.38: Hummingbird OSV Generator
- 4.5.38.1: Hummingbird OSV Generator Workflow
- 4.5.39: S3 Lookaside Cache
- 4.5.40: Docs Content Negotiation
- 4.5.41: Hummingbird MR Collaboration
- 4.5.42: K8s Integration Tests
- 4.5.43: Trigger Manifests
- 5: Agentic SDLC
- 6: Architecture Decision Records
- 6.1: ADR-0001: Unified Observability Stack
- 6.2: ADR-0002: AI-Assisted SDLC Workqueue
- 6.3: ADR-0003: RPM metadata and versioning
- 6.4: ADR-0004: Move private_product into package metadata
- 7: Specifications
1 - Using a Project Hummingbird container image
1.1 - Using a Project Hummingbird container image
Project Hummingbird builds a collection of minimal, hardened, and secure container images with a significantly reduced attack surface. This strong focus on security combined with a highly automated update workflow aims to minimize CVE counts, targeting near-zero vulnerabilities. All images support amd64 and arm64 architectures.
Quick Start
All images are available from the Red Hat Hardened Images registry and work directly with Podman, Docker, or Kubernetes:
# Run a command directly with the curl image
podman run registry.access.redhat.com/hi/curl:latest -v https://example.com
# Start a PostgreSQL database
podman run -e POSTGRES_PASSWORD=mysecret -p 5432:5432 registry.access.redhat.com/hi/postgresql:latest
Example container build using the Hummingbird Python image:
FROM registry.access.redhat.com/hi/python:latest
COPY myapp.py /app/
WORKDIR /app
CMD ["python3", "myapp.py"]
Documentation for each image is available on the Red Hat Hardened Images Catalog.
Contents
- Available Images
- Hardened for Security
- Distroless Containers
- Understanding Image Variants
- Referencing Images in Production
- Image Verification
- Vulnerability Scanning
- Sharing Host Data
- Custom CA Certificates
- Compatibility
- Reproducible Builds
- Content-Based Layers
- Source Containers
- Roadmap
- Relationship to Fedora
- Contributing
Available Images
Available images span language runtimes, databases, web servers, CLI tools, and base runtime images.
Browse the complete catalog at the Red Hat Hardened Images Catalog.
Hardened for Security
Hummingbird applies several measures to harden container images:
- Minimal Software Footprint: Images include only essential packages required for the workload, significantly reducing the attack surface and CVE count.
- Rapid Update Deployment: Package updates ship as quickly as possible, ensuring fixes are consumed early.
- Non-Root User Default: Containers default to a non-root user (UID 65532) where technically possible, reducing privileges within the container.
- Hermetic Build Environment: All containers are built in a hermetic environment without network access, preventing unintended package drift and maintaining full control over software versions.
- Distroless Security: Shipping only what is strictly necessary for the workload reduces the attack surface and makes certain types of attacks impossible.
- FIPS-Validated Cryptography: FIPS variants provide NIST-validated cryptographic modules for compliance-sensitive workloads.
Distroless Containers
Hummingbird builds distroless containers — images that do not ship with a package manager and most do not even provide a shell.
The distroless design makes the bundled application the container’s entrypoint, offering a streamlined experience. For example, with the curl image, arguments can be passed directly: podman run registry.access.redhat.com/hi/curl:latest -v https://www.redhat.com/en.
Purpose-built containers reduce the burden on users. Instead of building custom container images and managing their vulnerabilities, a Hummingbird image with the needed application avoids CVE management overhead entirely.
Understanding Image Variants
Hummingbird provides different variants to support various use cases while maintaining security by default.
Default Variant (:latest)
- Distroless: no package manager, no shell
- Minimal attack surface
- Recommended for production
- Example:
registry.access.redhat.com/hi/python:latest
Builder Variant (:latest-builder)
- Includes
dnfpackage manager andbash - For installing additional dependencies
- Intended for multi-stage builds and development
- Example:
registry.access.redhat.com/hi/python:latest-builder
Some images provide additional variants such as PHP’s FPM variant.
FIPS Variants (:latest-fips, :latest-fips-builder)
FIPS variants ship FIPS 140-3 validated cryptographic modules for workloads that require certified cryptography. Validation applies only when running on RHEL systems installed in FIPS mode.
Two cryptographic stacks are available depending on the image:
| Stack | Images | FIPS packages |
|---|---|---|
| OpenSSL | Go, Nginx, Node.js, Python, Ruby, curl | openssl-config-fips, crypto-policies-config-fips |
| NSS | OpenJDK | nss-softokn-fips, nss-softokn-freebl-fips |
FIPS is available in both distroless (:latest-fips) and builder (:latest-fips-builder) variants. FIPS variants are available for the Hummingbird distro only (not Rawhide). For implementation details, see the FIPS Variant Guide.
Support Levels
Images are published under two support levels:
- Red Hat supported — the majority of images. Built, tested, and maintained by the Hummingbird team with full CI/CD coverage, security updates, and vulnerability tracking. Available at
registry.access.redhat.com/hi/with official Red Hat signing keys — this is the recommended registry for all users. - Community images include projects with nonstandard open-source release practices, such as MinIO, and experimental images such as
os-minimal. Build and test infrastructure is shared with Red Hat supported images, but support commitments differ. Community images can be evaluated for promotion to Red Hat supported status. Published toquay.io/hummingbird-community/.
Registry Organizations
| Registry | Description |
|---|---|
registry.access.redhat.com/hi/ |
Red Hat Hardened Container Images (recommended) |
quay.io/hummingbird/ |
Mirror of Red Hat supported images without official signing |
quay.io/hummingbird-community/ |
Community-supported images (e.g., MinIO, os-minimal) |
quay.io/hummingbird-rawhide/ |
Upstream Fedora Rawhide packages directly |
quay.io/hummingbird-ci/ |
Build infrastructure images |
Tool images used by pipelines (e.g., quay.io/hummingbird-ci/gitlab-ci) are published from
the tools repository.
Tagging Strategy
Images follow a version-based tagging scheme:
:latest— most recent version (may change):<version>— specific version (e.g.,:3.11,:16):<version>-builder— builder variant of a specific version:<version>-fips— FIPS variant of a specific version:<version>-fips-builder— FIPS builder variant of a specific version
Use versioned tags in production to stay within a supported version stream. The :latest tag is convenient for development but may introduce unexpected changes.
For general background on tag selection and digest pinning, see How to name, version, and reference container images.
Multi-version images:
For languages with multiple supported version families (Python, Node.js, .NET, etc.), separate images exist for each major version:
registry.access.redhat.com/hi/python:latest— latest Python (currently 3.14)registry.access.redhat.com/hi/python:3.11— Python 3.11.x (stays on 3.11)registry.access.redhat.com/hi/python:3.13— Python 3.13.x (stays on 3.13)
Version constraints prevent these images from accidentally upgrading to incompatible versions. See version constraints for details.
Multi-Stage Build Pattern
The recommended pattern for compiled languages:
# Build stage: use builder variant to install dependencies and compile
FROM registry.access.redhat.com/hi/go:latest-builder AS builder
RUN dnf install -y <build dependencies>
COPY . /src
WORKDIR /src
RUN go build -o /app .
# Runtime stage: use minimal base image for the compiled binary
FROM registry.access.redhat.com/hi/core-runtime:latest
COPY --from=builder /app /app
ENTRYPOINT ["/app"]
This approach provides build-time flexibility while maintaining a minimal production image.
Referencing Images in Production
Version tags like :3.11 track the main package version but not its
dependencies. When packages such as glibc or openssl receive security
patches, the image is rebuilt and pushed under the same tag with a new
digest. Using the tag alone means silently receiving different content
on each pull. To pin the exact image while still communicating the
intended version stream, combine the tag with a digest:
FROM registry.access.redhat.com/hi/python:3.11@sha256:<digest>
When both a tag and a digest are present, the container engine uses the digest. The tag serves as documentation — it makes clear which version stream the image belongs to. Retrieve the current digest with:
skopeo inspect --format '{{.Digest}}' \
docker://registry.access.redhat.com/hi/python:3.11
Because tags float on every rebuild, the pinned digest will fall behind as new security updates ship. A dependency-management bot such as Renovate can watch the tag and open merge requests whenever the digest changes, keeping images current without manual tracking.
Image Verification
Hummingbird images are signed and can be verified using the Hummingbird cosign image:
podman run --rm registry.access.redhat.com/hi/cosign:latest verify \
--key "https://security.access.redhat.com/data/63405576.txt" \
--insecure-ignore-tlog \
registry.access.redhat.com/hi/<image>:<tag>
Vulnerability Scanning
Use Syft and Grype to locally inspect and scan images. For details on how production SBOMs are generated, see Software Bill of Materials.
To scan an image for vulnerabilities:
grype registry.access.redhat.com/hi/<image>:<tag>
For current vulnerability information across all variants and versions, see the Red Hat Hardened Images Catalog.
Sharing Host Data
By default, containers do not have access to host filesystem content. Volume mounts must be added explicitly:
podman run -v /path/on/host:/path/in/container registry.access.redhat.com/hi/curl:latest ...
SELinux Relabeling
On systems with SELinux enabled (such as Fedora and RHEL), mounted volumes must be relabeled to allow container access. Use the :z or :Z option:
podman run -v /path/on/host:/path/in/container:z registry.access.redhat.com/hi/curl:latest ...
The :z option relabels content to be accessible by any container sharing the mount. The :Z option relabels content to be uniquely accessible to one container.
File Permissions
Most Hummingbird images default to a non-root user. When mounting host directories that the container needs to write to, either make the directory world-writable (within a private directory):
mkdir -m 700 /path/on/host
mkdir -m 777 /path/on/host/mnt
podman run -v /path/on/host/mnt:/path/in/container:z ...
Or run as the root user (recommended only with rootless Podman, where the root user in the container maps to the calling user on the host):
podman run --user root -v /path/on/host:/path/in/container:z ...
For read-only access, no permission changes are needed as long as files are world-readable.
Custom CA Certificates
Hummingbird images can be configured to trust custom Certificate Authority (CA) certificates for TLS connections. The approach depends on the image’s TLS stack.
OpenSSL-Based Images (curl, Nginx, Python, etc.)
Mount a CA bundle to replace or extend the system trust store:
podman run --rm \
-v /path/to/ca.crt:/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:ro,Z \
registry.access.redhat.com/hi/curl:latest https://internal-server/
For details on building derived images with merged trust stores, see the Custom CA (OpenSSL) guide.
Java-Based Images (OpenJDK, Tomcat)
Java uses its own PKCS12 truststore format. A custom truststore can be created using keytool and mounted at runtime:
podman run --rm \
-v /path/to/cacerts:/etc/pki/ca-trust/extracted/java/cacerts:ro,Z \
registry.access.redhat.com/hi/openjdk:latest java -jar app.jar
For step-by-step truststore creation, see the Custom CA (Java) guide.
Compatibility
Hummingbird images are designed for compatibility with popular images from Docker Hub, Red Hat UBI, and other registries, enabling straightforward migration of existing workloads.
Key difference: Hummingbird images default to a non-root user (UID 65532) where technically possible, while most traditional images run as root. This may require adjusting file permissions on mounted volumes:
# Ensure correct ownership for mounted data
chown -R 65532:65532 /path/to/data
For detailed comparisons (environment variables, ports, sizes, default users), see the compatibility report, also available in machine-readable form.
Reproducible Builds
Reproducible builds ensure that the same inputs and build environment always produce bit-for-bit identical output. This allows independent verification that a published image corresponds to its claimed source materials, making it straightforward to detect if malware was injected during the build process.
Hummingbird images are fully reproducible. Given the signed SLSA provenance attestation that accompanies each image, anyone can rebuild the image from its inputs (this git repo and the RPMs it describes) to verify it matches the published version exactly.
For further background, see Reproducible builds: Project Hummingbird.
Verifying Reproducibility
The podman tool is required. First, download and verify the SLSA provenance attestation using cosign (see Image Verification for key details):
IMAGE=registry.access.redhat.com/hi/curl:latest
podman run --rm registry.access.redhat.com/hi/cosign:latest verify-attestation \
--key "https://gitlab.com/redhat/hummingbird/containers/-/raw/main/ci/key.pub?ref_type=heads" \
--insecure-ignore-tlog \
--type slsaprovenance $IMAGE > attestation.json
Then, feed the attestation into the rebuild tool (capture the image ID for comparison):
iid=$(podman run -i --rm --privileged -v /mnt \
quay.io/hummingbird-ci/hummingbird-builder rebuild < attestation.json)
The --privileged flag is required because the build process uses nested containerization. However, this command is expected to be run rootless. A rootless container cannot gain more privileges than the calling user.
To verify reproducibility, pull the published image and compare image IDs:
iid2=$(podman pull $IMAGE)
[ $iid = $iid2 ] && echo "Identical"
The containers-storage image ID (a hash over the manifest and uncompressed content) is used here, not the repo digest (a hash over compressed content). Comparing uncompressed content avoids depending on the exact compression algorithm or registry format.
To keep the rebuilt image for further inspection, use the DUMP_OCIARCHIVE environment variable:
podman run -i --rm --privileged -e DUMP_OCIARCHIVE=1 -v /mnt \
quay.io/hummingbird-ci/hummingbird-builder rebuild < attestation.json | podman load
Content-Based Layers
Most container images have layers that mirror the Containerfile structure: each RUN or COPY instruction creates a layer. A single package update can invalidate a layer much larger than the package itself, requiring clients to re-pull content that has not changed.
Hummingbird images use chunkah to split images into content-based layers. Instead of layers reflecting the build process, files are grouped by the packages they belong to. This benefits both the network level (only new layers need downloading) and the storage level (common layers are stored once on disk).
Inspect which packages live in which layers using podman history (or docker history):
podman pull registry.access.redhat.com/hi/jq
podman history registry.access.redhat.com/hi/jq
Source Containers
The source code for all Hummingbird containers is available as source containers. A source container includes RPM and non-RPM content shipped in an image, pushed alongside each available tag with the -source suffix.
Inspect source container contents with skopeo:
IMAGE=registry.access.redhat.com/hi/curl:8.19.0-source
cd $(mktemp -d)
mkdir source
skopeo copy --override-os=linux docker://$IMAGE dir:source
cd source
mv version manifest.json ..
for f in $(ls); do tar xvf $f; done
Roadmap
- Image catalog expansion: The catalog of available images will continue to expand significantly.
- Image size optimization: Container-specific package configurations and dependency management to further reduce image sizes.
- Enhanced CVE and SBOM tooling: Improved vulnerability metadata and scanning integration, including methods like cargo-auditable for Rust supply chain transparency.
- Bootable containers: Expanding support for image-based OS deployments and updates.
Relationship to Fedora
Hummingbird builds and maintains its own RPM packages independently. Spec files are derived from Fedora but built in Hummingbird’s own infrastructure, allowing project-specific modifications and optimizations. Key differences from Fedora:
- Use of monorepos — one monorepo for packages and one for containers, simplifying management and automation.
- Heavy reliance on CI/CD — dependency management, testing, and releasing are automated.
- Secure supply chain requirements — container images and packages are built using Konflux, providing SLSA level 3 compliance.
- Greater emphasis on upstream tracking — software closely tracks upstream projects’ lifecycles, including parallel stable versions where offered.
There is a natural affinity between the two projects, and opportunities exist to contribute work from Hummingbird back into Fedora.
Contributing
Interested in contributing? The following resources cover the development workflow:
- Quickstart Guide — building and testing images locally, project structure, CI/CD pipeline
- Adding Images — step-by-step guide for new images and versioned variants
- FIPS Variant Guide — adding FIPS-validated cryptography to an image
- Testing Images — running tests locally and understanding automatic infrastructure retries
- Development Workflow — local setup and workflow
License
This project is licensed under the MIT License - see the LICENSE.txt file for details.
1.2 - Custom CA Certificates (OpenSSL)
Overview
You can configure OpenSSL-based container images (curl, nginx, etc.) to trust custom Certificate Authority (CA) certificates for TLS connections.
There are two approaches for custom CAs depending on your needs:
- Custom bundle file - Replace system CAs entirely with your own bundle
- Derived image - Build a new image with merged trust store
Approach 1: Custom Bundle File
Use this when you want to trust only your custom CAs and block all default CAs. This is a common case with OpenShift’s custom PKI bundle.
Mount your CA bundle to /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:
Custom bundle with Podman
podman run --rm \
-v /path/to/ca.crt:/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:ro,Z \
registry.access.redhat.com/hi/curl https://your-server/
Custom bundle with Kubernetes
apiVersion: v1
kind: ConfigMap
metadata:
name: custom-ca-bundle
data:
tls-ca-bundle.pem: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
... more certificates if needed ...
---
apiVersion: v1
kind: Pod
spec:
containers:
- name: app
image: registry.access.redhat.com/hi/curl
volumeMounts:
- name: custom-ca
mountPath: /etc/pki/ca-trust/extracted/pem
readOnly: true
volumes:
- name: custom-ca
configMap:
name: custom-ca-bundle
Custom bundle on OpenShift with CA injection
On OpenShift, the cluster admin may have already added your organization’s CA certificates to the cluster-wide trust store. You can use OpenShift’s automatic CA injection feature to make these certificates available to your pods.
Create a ConfigMap with the config.openshift.io/inject-trusted-cabundle=true label. OpenShift will automatically populate it with the cluster CA bundle as a key named ca-bundle.crt:
# OpenShift will inject a "ca-bundle.crt" into this automatically
apiVersion: v1
kind: ConfigMap
metadata:
name: trusted-ca
labels:
config.openshift.io/inject-trusted-cabundle: "true"
---
apiVersion: v1
kind: Pod
spec:
containers:
- name: app
image: registry.access.redhat.com/hi/curl
volumeMounts:
- name: trusted-ca
mountPath: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
subPath: ca-bundle.crt
readOnly: true
volumes:
- name: trusted-ca
configMap:
name: trusted-ca
The key detail is using subPath: ca-bundle.crt to mount OpenShift’s injected file to the correct location (tls-ca-bundle.pem) that OpenSSL expects.
See the Configuring a custom PKI OpenShift documentation for details on cluster-wide CA configuration.
Approach 2: Derived Image
Use this approach when you need to trust both your custom CA and the image’s builtin default CAs. This creates a new image with a merged trust store with the trust anchor command.
Create a multi-stage Containerfile with your ca.crt in the build context. This example derives from the curl image:
FROM registry.access.redhat.com/hi/curl:latest-builder AS builder
USER root
COPY ca.crt /tmp/
RUN trust anchor /tmp/ca.crt
FROM registry.access.redhat.com/hi/curl:latest
COPY --from=builder /etc/pki /etc/pki
Build and use your custom image:
podman build -t my-curl-with-ca .
podman run --rm my-curl-with-ca https://your-server/
1.3 - Custom CA Certificates (Java)
Overview
You can configure Java-based container images (openjdk, etc.) to trust custom Certificate Authority (CA) certificates for TLS connections. Java uses its own truststore format (PKCS12) rather than PEM files.
Java keystores require a password for integrity checking. The system truststore uses the standard password changeit, which is a well-known default used for tamper detection rather than confidentiality.
Volume Mount Approach
Create a custom truststore using the keytool command. You can either trust only your specific CAs (for internal-only applications) or merge with the default public CAs (typical for applications connecting to both internal and external services).
CERTDIR=$(mktemp -d) # or use a persistent directory for repeated runs
chmod 777 "$CERTDIR" # to work with non-privileged containers
# OPTIONAL: Copy the default Java truststore to merge with public CAs
# Skip this step to trust ONLY your custom CAs
podman run --rm -v "${CERTDIR}:/work:Z" \
registry.access.redhat.com/hi/openjdk:latest \
cp /etc/pki/ca-trust/extracted/java/cacerts /work/cacerts
# Import your custom CA
podman run --rm -v "${CERTDIR}:/work:Z" \
registry.access.redhat.com/hi/openjdk:latest \
keytool -import -noprompt -trustcacerts -alias my-custom-ca \
-file /work/my-ca.crt \
-keystore /work/cacerts \
-storepass changeit -storetype PKCS12 \
-J-Dkeystore.pkcs12.certProtectionAlgorithm=NONE
The -storetype PKCS12 and -J-Dkeystore.pkcs12.certProtectionAlgorithm=NONE flags
are required for compatibility with FIPS images. The system truststore is PKCS12 format
with unencrypted certificate entries (generated by p11-kit). Without these flags,
keytool defaults to PBE-encrypted certificate bags that the FIPS security provider
cannot decrypt.
Podman Example
Mount the custom truststore to the system location:
podman run --rm \
-v "${CERTDIR}/cacerts:/etc/pki/ca-trust/extracted/java/cacerts:ro,Z" \
registry.access.redhat.com/hi/openjdk:latest \
java -jar myapp.jar
Kubernetes Example
In Kubernetes, use an init container to create the truststore, then share it via an emptyDir volume:
apiVersion: v1
kind: ConfigMap
metadata:
name: custom-ca
data:
ca.crt: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
---
apiVersion: v1
kind: Pod
spec:
initContainers:
- name: prepare-truststore
image: registry.access.redhat.com/hi/openjdk:latest
command: ["sh", "-c"]
args:
- |
# OPTIONAL: Copy the default truststore to merge with public CAs
# Skip this line to trust ONLY your custom CAs
cp /etc/pki/ca-trust/extracted/java/cacerts /truststore/cacerts
chmod 666 /truststore/cacerts
keytool -import -noprompt -trustcacerts -alias custom-ca \
-file /ca/ca.crt -keystore /truststore/cacerts \
-storepass changeit -storetype PKCS12 \
-J-Dkeystore.pkcs12.certProtectionAlgorithm=NONE
volumeMounts:
- name: custom-ca
mountPath: /ca
- name: truststore
mountPath: /truststore
containers:
- name: app
image: registry.access.redhat.com/hi/openjdk:latest
volumeMounts:
- name: truststore
mountPath: /etc/pki/ca-trust/extracted/java
readOnly: true
volumes:
- name: custom-ca
configMap:
name: custom-ca
- name: truststore
emptyDir: {}
For the trust-only-specific-CAs approach, omit the cp command and create the keystore directly with keytool -import.
Derived Image Approach
Use this approach when it’s more practical to bake the certificates into the image instead of generating the truststore on each container startup. This creates a new image with a merged trust store with the trust anchor command.
Create a multi-stage Containerfile with your ca.crt in the build context. This example derives from the openjdk image:
FROM registry.access.redhat.com/hi/openjdk:latest-builder AS builder
USER root
COPY ca.crt /tmp/
RUN trust anchor /tmp/ca.crt
FROM registry.access.redhat.com/hi/openjdk:latest
COPY --from=builder /etc/pki /etc/pki
Build and use your custom image:
podman build -t my-openjdk-with-ca .
podman run --rm my-openjdk-with-ca java -jar myapp.jar
1.4 - Custom CA Certificates (Python)
Overview
You can configure Python container images to trust custom Certificate Authority (CA) certificates for TLS connections. Python libraries have different trust store behaviors compared to other OpenSSL-based images:
- urllib (built-in) reads CAs from both of these locations:
/etc/pki/tls/cert.pem- the main CA bundle file/etc/pki/tls/certs/- directory with hashed certificate symlinks
- requests (third-party) ignores
/etc/pkientirely and requires theREQUESTS_CA_BUNDLEenvironment variable to use system CAs
Note: The requests library is not included in the standard Python image. To use it, you need to build a derived image:
FROM registry.access.redhat.com/hi/python:latest
RUN ["pip3", "install", "requests"]
podman build -t localhost/my-python-with-requests .
There are two approaches for custom CAs depending on your needs:
- Custom bundle file - Replace system CAs entirely with your own bundle
- Add custom CAs - Merge your custom CAs with the system defaults
Approach 1: Custom Bundle File
Use this when you want to trust only your custom CAs and block all default CAs. This is a common case with OpenShift’s custom PKI bundle.
Mount your CA bundle to /etc/pki/tls/cert.pem, set $REQUESTS_CA_BUNDLE to that, and block the hashed directory with an empty directory:
Custom bundle with Podman
podman run --rm \
-v /path/to/ca.crt:/etc/pki/tls/cert.pem:ro,Z \
--tmpfs /etc/pki/tls/certs:notmpcopyup \
-e REQUESTS_CA_BUNDLE=/etc/pki/tls/cert.pem \
localhost/my-python-with-requests python3 -c "
import urllib.request
print(urllib.request.urlopen('https://your-server/').status)
import requests
print(requests.get('https://your-server/').status_code)
"
Note: Podman requires the :notmpcopyup option as mounting tmpfs behaves like an overlay. Docker’s tmpfs creates an empty directory by default, and does not know that option, so drop that option with Docker.
Custom bundle with Kubernetes
apiVersion: v1
kind: ConfigMap
metadata:
name: custom-ca-bundle
data:
cert.pem: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
... more certificates if needed ...
---
apiVersion: v1
kind: Pod
spec:
containers:
- name: app
# Use registry.access.redhat.com/hi/python for urllib only
# Use your derived image with requests installed for both libraries
image: registry.example.com/my-python-with-requests:latest
env:
- name: REQUESTS_CA_BUNDLE
value: /etc/pki/tls/cert.pem
volumeMounts:
- name: custom-ca
mountPath: /etc/pki/tls/cert.pem
subPath: cert.pem
readOnly: true
- name: empty-certs
mountPath: /etc/pki/tls/certs
readOnly: true
volumes:
- name: custom-ca
configMap:
name: custom-ca-bundle
- name: empty-certs
emptyDir: {}
Custom bundle on OpenShift with CA injection
On OpenShift, the cluster admin may have already added your organization’s CA certificates to the cluster-wide trust store. You can use OpenShift’s automatic CA injection feature to make these certificates available to your pods.
Create a ConfigMap with the config.openshift.io/inject-trusted-cabundle=true label. OpenShift will automatically populate it with the cluster CA bundle as a key named ca-bundle.crt:
# OpenShift will inject a "ca-bundle.crt" into this automatically
apiVersion: v1
kind: ConfigMap
metadata:
name: trusted-ca
labels:
config.openshift.io/inject-trusted-cabundle: "true"
---
apiVersion: v1
kind: Pod
spec:
containers:
- name: app
# Use registry.access.redhat.com/hi/python for urllib only
# Use your derived image with requests installed for both libraries
image: registry.example.com/my-python-with-requests:latest
env:
- name: REQUESTS_CA_BUNDLE
value: /etc/pki/tls/cert.pem
volumeMounts:
- name: trusted-ca
mountPath: /etc/pki/tls/cert.pem
subPath: ca-bundle.crt
readOnly: true
- name: empty-certs
mountPath: /etc/pki/tls/certs
readOnly: true
volumes:
- name: trusted-ca
configMap:
name: trusted-ca
- name: empty-certs
emptyDir: {}
The key detail is using subPath: ca-bundle.crt to mount OpenShift’s injected file to the correct location (/etc/pki/tls/cert.pem) that Python expects.
See the Configuring a custom PKI OpenShift documentation for details on cluster-wide CA configuration.
Approach 2: Add Custom CAs
Use this when you want to merge your custom CAs with the system defaults.
Mount your CA certificate as an individual file in /etc/pki/tls/certs/ with a properly hashed filename. The filename must be the output of openssl x509 -noout -subject_hash -in ca.crt followed by .0.
Also set $REQUESTS_CA_BUNDLE to that directory:
Custom CA file with Podman
# Generate the hashed filename
CA_HASH=$(openssl x509 -noout -subject_hash -in ca.crt)
echo "Hashed filename: ${CA_HASH}.0"
# Mount the CA certificate as an individual file in the certs directory
podman run --rm \
-v ./ca.crt:/etc/pki/tls/certs/${CA_HASH}.0:ro,Z \
-e REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ \
localhost/my-python-with-requests python3 -c "
import urllib.request
print(urllib.request.urlopen('https://your-server/').status)
print(urllib.request.urlopen('https://google.com/').status) # Default CAs still work
import requests
print(requests.get('https://your-server/').status_code)
print(requests.get('https://google.com/').status_code) # Default CAs still work
"
Custom CA file with Kubernetes
apiVersion: v1
kind: ConfigMap
metadata:
name: custom-ca
data:
ca.crt: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
---
apiVersion: v1
kind: Pod
spec:
containers:
- name: app
# Use registry.access.redhat.com/hi/python for urllib only
# Use your derived image with requests installed for both libraries
image: registry.example.com/my-python-with-requests:latest
env:
- name: REQUESTS_CA_BUNDLE
value: /etc/pki/tls/certs/
volumeMounts:
- name: custom-ca
# REQUIRED: replace example hash with: `openssl x509 -noout -subject_hash -in ca.crt` and append .0
mountPath: /etc/pki/tls/certs/12abc456.0
subPath: ca.crt
readOnly: true
volumes:
- name: custom-ca
configMap:
name: custom-ca
2 - Contributing to a Project Hummingbird container image
Welcome to the Project Hummingbird contributor documentation. This section provides guides and references for contributing to container images.
Getting Started
New to contributing? Start here:
- Quickstart Guide - Get your first contribution done in 5 minutes
- Development Workflow - Detailed development environment and workflow
Guides
Step-by-step guides for common tasks:
- Adding Images - How to add a new container image
- Testing - How to run and write tests locally
Reference
Detailed reference documentation:
- Image Configuration Reference - Complete
properties.ymlreference - Test Configuration Reference - Complete test definition reference
2.1 - Quickstart Guide
Welcome! This guide will help you make your first contribution to Project Hummingbird container images.
For more detailed information, see the full contributor documentation.
Prerequisites
- Container tools: Podman or Docker
- Build tools:
buildah,make,git - Python: Python 3.11+ with dependencies from
requirements.txt
For macOS-specific setup, see macOS setup in the Development Workflow.
Install on Fedora/RHEL:
sudo dnf install podman buildah make git python3 python3-pip
pip install -r requirements.txt
Quick Start Workflow
1. Fork and Clone
- Fork the repository on GitLab
- Clone your fork:
git clone --recurse-submodules https://gitlab.com/<your-username>/containers.git
cd containers
git remote add upstream https://gitlab.com/redhat/hummingbird/containers.git
2. Make Your Changes
Edit files in the images/<image-name>/ directory:
properties.yml- Image configurationContainerfile.j2- Container build templatetests-container.yml- Integration tests
3. Generate and Build
# Update dependent files
make
# Build the image
ci/build_images.sh <image-name>
4. Test Your Changes
# Run integration tests
ci/run_tests_container.sh <image-name>
# Run linters and checks
make check
5. Submit a Merge Request
- Create a branch:
git checkout -b fix-nginx-config - Commit your changes:
git commit -am "Fix nginx configuration" - Push to your fork:
git push origin fix-nginx-config - Open a merge request from your fork to the upstream repository on GitLab
Note for external contributors: If you’re not a project member, the CI pipeline won’t run automatically. Please ping one of the project maintainers in your merge request to trigger the build and test pipeline.
License
By contributing to this project, you agree that your contributions will be licensed under the MIT License. See LICENSE.txt for the full license text.
Common Tasks
Adding a New Image
# Create image directory
mkdir images/myapp
# Copy templates
cp images/Containerfile.j2 images/myapp/Containerfile.j2
cp images/properties.yml images/myapp/properties.yml
# Edit the template, configuration, and tests
vim images/myapp/properties.yml # Configure packages and variants
vim images/myapp/Containerfile.j2 # Customize the build template
vim images/myapp/tests-container.yml # Add integration tests
# Update dependent files
make
# Build and test
ci/build_images.sh myapp
ci/run_tests_container.sh myapp
Testing with Docker
# Automatic Docker-in-Docker setup
ci/build_images.sh --engine docker --setup <image-name>
ci/run_tests_container.sh --engine docker --setup <image-name>
Building Specific Variants
# Build only the rawhide builder variant
ci/build_images.sh nginx/rawhide/builder
# Test only the rawhide default variant
ci/run_tests_container.sh nginx/rawhide/default
# Build/test all variants for a specific distro
ci/build_images.sh nginx/rawhide
ci/run_tests_container.sh nginx/rawhide
Next Steps
- Development Workflow - Detailed development environment and workflow
- Adding Images - Step-by-step guide for adding new container images
- Testing - How to run and write tests locally
Getting Help
- Documentation: hummingbird-project.io/l/contributing-images/
- Code: gitlab.com/redhat/hummingbird
2.2 - Development Workflow
Prerequisites
Ensure the required tools are installed:
- Container tools: Podman or Docker
- Build tools:
buildah,make,git - Python: Python 3.11+ with dependencies from
requirements.txt
Install on Fedora/RHEL:
sudo dnf install podman buildah make git python3 python3-pip
macOS Setup
macOS requires Python 3.11+, Bash 5+, and GNU command-line tools. Install via Homebrew:
brew install podman bash grep coreutils python
podman machine init && podman machine start
# Add to ~/.zshrc for persistence, then source it or open a new shell
export PATH="/opt/homebrew/bin:/opt/homebrew/opt/grep/libexec/gnubin:/opt/homebrew/opt/coreutils/libexec/gnubin:$PATH"
Build scripts use Bash 5+, GNU grep and date, and the Python selected by PATH.
Note: On Intel Macs, use
/usr/local/instead of/opt/homebrew/.
Repository Setup
Fork and Clone
- Fork the containers repository on GitLab
- Clone your fork with submodules:
git clone --recurse-submodules https://gitlab.com/<your-username>/containers.git
cd containers
- Add the upstream remote:
git remote add upstream https://gitlab.com/redhat/hummingbird/containers.git
Install Python dependencies
Create and activate a virtual environment, then install the project requirements:
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install -r requirements.txt
Keep the virtual environment active when running make or CI scripts so their Python commands use these dependencies.
Development Workflow
1. Create a Branch
Create a descriptive branch starting from upstream main:
git fetch upstream
git checkout -b feature-add-redis-image upstream/main
2. Make Changes
Edit the relevant files in images/<image-name>/:
properties.yml- Image configuration (packages, variants, tags)Containerfile.j2- Container build templatetests-container.yml- Container teststests-k8s.yml- K8s testsREADME.md.j2- Image documentation template
3. Generate Files
Regenerate derived files:
make
4. Build Locally
Build the image to verify changes:
ci/build_images.sh <image-name>
# Build specific variant
ci/build_images.sh <image-name>/rawhide/builder
# Build with verbose output
ci/build_images.sh --verbose <image-name>
5. Test Locally
Container tests:
ci/run_tests_container.sh <image-name>
# Test specific variant
ci/run_tests_container.sh <image-name>/rawhide/default
# Test with verbose output
ci/run_tests_container.sh --verbose <image-name>
By default, tests use Podman. To test with Docker:
# Automatic Docker-in-Docker setup
ci/run_tests_container.sh --engine docker --setup <image-name>
K8s tests:
ci/run_tests_k8s.sh --context <context> <image-name>
# Test specific variant
ci/run_tests_k8s.sh --context <context> <image-name>/rawhide/default
# Test with verbose output
ci/run_tests_k8s.sh --verbose --context <context> <image-name>
See the Testing Guide for K8s test prerequisites and local development workflow.
Testing base images:
When modifying base images (like core-runtime), test dependent images:
ci/build_images.sh core-runtime
ci/build_images.sh --build-deps core-runtime
ci/run_tests_container.sh --include-reverse-deps core-runtime
ci/run_tests_k8s.sh --include-reverse-deps --context <context> core-runtime
Troubleshooting test failures:
Use --pause flag to inspect resources before cleanup:
ci/run_tests_container.sh --pause <image-name>
ci/run_tests_k8s.sh --pause --context <context> <image-name>
6. Run Linters
Ensure code quality:
make check
7. Commit and Push
Commit changes with a descriptive message and push to the fork:
git add .
git commit -m "feat: add Redis container image
- Add Redis 7.x image with default and builder variants
- Include basic health check tests
- Add compatibility notes for official Redis image"
git push origin feature-add-redis-image
8. Open a Merge Request
Open a merge request on GitLab:
- Go to the containers repository
- Select your fork and branch as the source,
mainas the target - Fill in the description with what changed, why, and what tests were added
9. CI Pipeline
The CI pipeline automatically:
- Builds images for multiple architectures
- Runs integration tests
- Checks for linting issues
Note for external contributors: If not a project member, the CI pipeline won’t run automatically. Ping one of the project maintainers in the merge request to trigger the pipeline.
Next Steps
- Adding Images - Step-by-step guide for adding new images
- Testing Guide - Comprehensive testing documentation
- Image Pipeline - How the complete pipeline works
2.3 - Resolving Merge Conflicts in Generated Files
Overview
During rebases of branches in this repository, you can commonly trigger merge conflicts in files that are generated by the various toolings and Makefile targets.
Rules
- This documentation is only for generated files.
- If there is not a section below for the conflicted file, you may need to check
make helpor search the Makefile for how to regenerate it (e.g.,grep -r "target_file" Makefile).
Procedures for generated files
The following sections describe how to rebase given merge conflicts in specific paths.
Regenerating rpms.lock.yaml
When rebasing a branch, you may encounter conflicts in rpms.lock.yaml files. You should
first delete the conflicting file, then run make container. Once in the container,
rerun make and pass the relative path of the target. The file will be regenerated.
Below shows an interactive example. If attempting to do this non-interactively, remove the file as described and then run something like:
podman run --pull=newer --rm -v "$PWD:$PWD:z" -w "$PWD" \
quay.io/hummingbird-ci/gitlab-ci:latest \
make <target_file>
# 1. Delete the conflicted lockfile (replace with your actual conflicting path)
rm <path/to/conflicting/rpms.lock.yaml>
# 2. Enter the CI container
make container
# 3. Inside container, regenerate the lockfile
make <path/to/conflicting/rpms.lock.yaml>
# 4. Exit container
exit
# 5. Stage the regenerated file and continue the rebase
git add <path/to/conflicting/rpms.lock.yaml>
git rebase --continue
# 6. After rebase completes, validate the result
make -j$(nproc) check
Regenerating Containerfile
If the merge conflict is in one or more of the generated Containerfiles, then always accept
the upstream (rebasing-onto) branch as the source of truth
(i.e., git checkout --theirs -- <Containerfile> and git add <Containerfile>).
When you have committed the upstream versions and are done with the rebase, you must then
regenerate the Containerfiles so they pick up your changes. Containerfiles can be generated
in two ways:
To rebuild Containerfiles for all images that are changed by your PR:
make -j$(nproc)
To rebuild a specific Containerfile, you can also use make and pass the path of the
generated Containerfile as an argument.
make generated/images/nginx/rawhide/default/Containerfile
After regenerating Containerfiles:
# Stage regenerated Containerfiles
git add <path/to/regenerated/Containerfile>
# If still in rebase, continue
git rebase --continue
# After rebase completes, validate the result
make -j$(nproc) check
2.4 - Adding New Images
This guide covers two scenarios:
- Adding a new image - Creating an entirely new image (e.g., adding
nginxfor the first time) - Adding a versioned image - Adding a new version of an existing image (e.g.,
go-1-25whengoalready exists)
Choose the appropriate section below based on your use case.
Adding Completely New Images
Follow these steps to add an entirely new container image to the project.
1. Create Image Directory
Create a directory for the new image:
mkdir images/your-service
2. Copy Base Templates
Copy the template files:
cp images/properties.yml images/your-service/properties.yml
cp images/Containerfile.j2 images/your-service/Containerfile.j2
3. Configure Image Properties
Edit images/your-service/properties.yml to configure:
- Variants: Define variants (default:
[default, builder]) - Packages: List RPM packages needed in
rpm_packages.all - Main package: Set
main_packagefor version labeling - Repository and stream: Set
repositoryandstream(both required). Add a YAML comment abovestreamdocumenting the reasoning (see Choosing a Stream Value) - Tags: Configure image tags
- Release metadata: Set
application_category,summary,description, andurl - Support level: New images start with
support_level: community
Example minimal configuration:
---
rpm_packages:
all:
- coreutils-single
- your-main-package
default:
- ca-certificates-bundle # smaller than ca-certificates; sufficient for non-builder variants
builder:
- ca-certificates # full package needed for builder toolchains
main_package: your-main-package
repository: your-service
# <reasoning for stream choice> -- see "Choosing a Stream Value" section
stream: "latest"
# deprecated: true # Optional: mark the image stream as deprecated
support_level: community
application_category: "Other"
summary: "One-line description of the image"
description: >-
Description of the software, its purpose, and its main capabilities.
url: "https://upstream-project.example.com/"
tags:
- value: latest
Prefer ca-certificates-bundle over ca-certificates in non-builder variants — it is smaller and
sufficient for TLS trust roots. Builder variants need the full ca-certificates package for
toolchain compatibility.
See the Image Configuration Reference for complete
properties.yml options.
4. Customize Containerfile Template
Edit images/your-service/Containerfile.j2:
- Add service-specific configuration (entrypoint, exposed ports, volumes, environment variables)
- The template automatically includes packages from
properties.ymlvia the{{ main_packages_arg() }}macro
The base template macros handle the standard container setup automatically:
| Macro | What it provides |
|---|---|
{{ final_stage() }} |
ENV HOME=/tmp, WORKDIR /tmp, labels, OCI archive extraction |
{{ set_user() }} |
USER 65532 (when user: default in properties.yml) |
The user field in properties.yml defaults to default (UID 65532) if not specified. No
passwd entry is needed — Linux allows running as any UID. See the
Container Configuration reference for per-variant and
custom user options.
When to override the defaults
The macros provide sensible defaults, but the image should match what the upstream image does.
If the upstream image uses a custom user, WORKDIR, or HOME, the Hummingbird image should
match for compatibility. Check the upstream image’s Dockerfile or inspect it with
podman inspect before deciding.
When the defaults are sufficient, avoid duplicating what the macros already provide:
{# The macros already handle HOME=/tmp, WORKDIR=/tmp, USER 65532. #}
{# Only add useradd/groupadd, WORKDIR, HOME, or USER when the #}
{# upstream image does something different that users depend on. #}
When the upstream image requires a specific user or working directory, set user: in
properties.yml and add WORKDIR in Containerfile.j2 as needed. Use {{ default_user }}
instead of hardcoding 65532.
Docker/registry auth in README templates
For images that interact with container registries, the auth config must be mounted to
$HOME/.docker/config.json inside the container. Check what HOME is set to — for default-user
images it is /tmp:
-v ${XDG_RUNTIME_DIR}/containers/auth.json:/tmp/.docker/config.json:ro,Z
Multi-Architecture Support
When Containerfiles or build scripts need to reference the target architecture (e.g., downloading architecture-specific binaries), use the correct naming convention for each context.
| Context | OCI architecture | RPM / Linux name |
|---|---|---|
| Intel | amd64 |
x86_64 |
| ARM | arm64 |
aarch64 |
| IBM Z | s390x |
s390x |
| POWER | ppc64le |
ppc64le |
| Containerfile | $TARGETARCH ARG variable |
$(arch) output |
- In Containerfiles or Go builds: Use the
$TARGETARCHARG variable provided by buildah (e.g.,amd64,arm64,s390x) - In shell scripts or RPM builds: Use the arch or uname -m commands (values:
x86_64,aarch64,s390x)
Set an image’s architectures property to override the distro defaults for that
image or a specific variant. See Architecture configuration.
5. Add Comparison Report
Create the required images/your-service/report.yml. Its first comparison supplies the upstream
image and documentation URL used by compatibility reports and generated README sections:
---
compare:
- pull: docker.io/upstream/your-service:latest
url: https://upstream-project.example.com/docs/
Add ignore entries only for intentional differences from the upstream image. See existing
images with similar entrypoints and runtime behavior for examples.
6. Add Integration Tests
Create test files in images/your-service/:
tests-container.yml- Required container tests (Podman/Docker)tests-k8s.yml- Optional Kubernetes tests for workload or cluster integration behavior
Container test example:
---
version-check:
command: |
test_engine_run --rm "${TEST_IMAGE}" your-service --version
K8s test example:
---
run-as-pod:
command: |
name="test-pod-${TEST_RUN_ID}"
kubectl run "${name}" --image="${TEST_IMAGE}" --restart=Never --labels="${TEST_RUN_LABEL}"
kubectl wait --for=jsonpath='{.status.phase}'=Succeeded "pod/${name}" --timeout=60s
See the Testing Guide for how to write and run tests.
7. Add Image Documentation
Create images/your-service/README.md.j2 with a high-level description, usage instructions, and
compatibility notes. The template generates multiple README files for different target audiences
(defined in images/variables.yml under readme_targets).
Important notes:
- The
readme_heading()macro automatically generates the heading with the correct product name - The
readme_description()macro renders the image description fromdescriptioninproperties.yml; use it after the heading instead of hardcoding introductory text - Use
{{ readme_targets[target].registry }}for registry references (not hardcodedquay.io/hummingbird) - The
readme_trailing_sections()macro appends compatibility, verification, vulnerability, and about sections
Example template:
{{ readme_heading() }}
{{ readme_description() }}
## Usage
podman run -d -p 8080:8080 {{ readme_targets[target].registry }}/your-service:latest
{{ readme_trailing_sections() }}
For more details, see README Generation.
8. Generate, Validate, and Submit
Follow the standard contribution workflow from the Development Workflow guide:
- Generate files (
make); commit all generated changes - Build locally (
ci/build_images.sh your-service) - Run container tests (
ci/run_tests_container.sh your-service) - Run Kubernetes tests when
tests-k8s.ymlexists (ci/run_tests_k8s.sh your-service) - Run linters (
make check) - Commit and push changes
- Open a merge request
- Wait for CI pipeline
9. Add Konflux Components and Trigger Testing
When generated konflux-templates/rendered.yml changes, the merge request pipeline runs
add_components_to_konflux automatically. This adds only the new components. After the job
completes, add a /retest comment to the merge request so Konflux starts the new image pipelines.
The manual deploy_components_to_konflux job redeploys all components and is not needed for normal
image onboarding.
Adding Versioned Images
When adding a new version of an existing image (e.g., go-1-25, helm-4, or
rabbitmq-4-3), follow this streamlined workflow.
1. Check Package Availability
Versioned packages may be available in different repositories:
- Hummingbird repository - Custom packages built by the Hummingbird project (e.g.,
golang1.25,nodejs24,dotnet-sdk-10.0) - Fedora repositories - Standard Fedora packages (e.g.,
golang,python3)
The project uses two distros. Their current repository files are defined in
images/variables.yml:
rawhide- Uses only Fedora Rawhide packages (fedora-44.repo)hummingbird- Uses Fedora 44 + Hummingbird packages (fedora-44.repo+hummingbird.repo)
To check if a package is available in Hummingbird:
# Check the Hummingbird RPMs repository
ls ../rpms/rpms/ | grep <package-name>
Important: If the package is only available in the Hummingbird repository, add distros: [hummingbird] to properties.yml (see step 3).
2. Create Image Directory and Copy Files
Follow the naming pattern <language>-<version> with dashes (e.g., go-1-25, nodejs-24, dotnet-sdk-10-0):
# Create directory
mkdir images/go-1-25
# Copy templates from the closest existing version, not the generic templates
cp images/go-1-26/Containerfile.j2 images/go-1-25/
cp -P images/go-1-26/tests-container.yml images/go-1-25/
cp images/go-1-26/report.yml images/go-1-25/
Tests shared by a version family may instead live under ci/shared-tests/, with each image’s
tests-container.yml and tests-k8s.yml symlinked to the shared files. Preserve the sibling
image’s arrangement.
Only the version that publishes latest owns README.md.j2 and the generated README files for
the shared repository. When a new version becomes the latest, move or copy the template to that
version and remove it from the previous version. Older versions without latest do not need a
README template. README files are generated from a template in the same image directory; they are
not shared automatically.
3. Configure properties.yml
Create images/go-1-25/properties.yml with the versioned package:
---
main_package: golang1.25
repository: go
# endoflife.date: parallel at major.minor (Docker Hub + Wolfi)
stream: "1.25"
application_category: "Programming Languages & Runtimes"
summary: "Statically-compiled language with concurrency and a large standard library"
description: >-
Statically typed, compiled programming language with garbage collection,
memory safety, and a large standard library. It includes the complete Go
toolchain.
url: "https://go.dev"
distros:
- hummingbird # Only if package is unavailable in Fedora
user: root
version_constraints: '1.25.*'
tags:
# NOTE: Do NOT include 'latest' or major-version tags for older versions
# Only the newest version (e.g., go-1-26) should have those tags
- value: '{{ package_major_minor_version("golang1.25") }}'
label: io.hummingbird-project.major-minor-version
- value: '{{ package_version("golang1.25") }}'
label: org.opencontainers.image.version
rpm_packages:
all:
- coreutils-single
- git-core
- glibc-devel
- golang1.25
- make
Key differences from a new image:
- Set
main_packageto the versioned package name - Set
repositoryto match the base image family (for registry organization) - Set
streamto the version this image tracks (e.g.,"1.25"for go-1-25), with a YAML comment above it documenting the reasoning (copy from an existing sibling image) - Set
version_constraintsso lockfile updates cannot cross the image’s version boundary - Copy
application_category,summary,description, andurlfrom the sibling image, then update any version-specific text - Add
distros: [hummingbird]if the package is only in the Hummingbird repository - Preserve sibling settings such as variants, FIPS packages, user, and support level unless the new upstream version requires a deliberate change
- Tag strategy for multiple versions:
- Only the newest version includes
latest - Include a major-version tag on each stream when it is unique (e.g., Helm 3 and Helm 4)
- If several streams share a major version (e.g., Go 1.25 and 1.26), only the newest includes that shared major-version tag
- Every version includes its non-conflicting stream tag and full version tag
- Only the newest version includes
4. Update Containerfile Template
Edit images/go-1-25/Containerfile.j2 to reference the versioned package in any environment variables or version-specific commands:
ENV GOPATH=/go \
GOTOOLCHAIN=local \
- GOLANG_VERSION={{ package_version('golang') }} \
+ GOLANG_VERSION={{ package_version('golang1.25') }} \
PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
5. Update Comparison Images
Edit images/go-1-25/report.yml to compare against version-specific upstream images:
compare:
- pull: docker.io/golang:1.25
url: https://hub.docker.com/_/golang
- pull: cgr.dev/chainguard/go:latest-dev
url: https://images.chainguard.dev/directory/image/go/overview
6. Tag Strategy for Multiple Versions
When multiple versions share a repository, coordinate tags to avoid conflicts. The following Go
versions both have major version 1, so only the newest can publish the 1 tag.
Newest version (go-1-26):
tags:
- value: latest # Points to newest
- value: '{{ package_major_version("golang1.26") }}' # e.g., "1"
label: io.hummingbird-project.major-version
- value: '{{ package_major_minor_version("golang1.26") }}' # e.g., "1.26"
label: io.hummingbird-project.major-minor-version
- value: '{{ package_version("golang1.26") }}' # Full version
label: org.opencontainers.image.version
Older versions (go-1-25):
tags:
# NO 'latest' tag
# NO major-version tag (would conflict with go-1-26's "1" tag)
- value: '{{ package_major_minor_version("golang1.25") }}' # e.g., "1.25"
label: io.hummingbird-project.major-minor-version
- value: '{{ package_version("golang1.25") }}' # Full version
label: org.opencontainers.image.version
Why: This ensures users can reference:
registry.access.redhat.com/hi/go:latest→ go-1-26registry.access.redhat.com/hi/go:1→ go-1-26 (latest 1.x)registry.access.redhat.com/hi/go:1.25→ go-1-25 (specific minor version)registry.access.redhat.com/hi/go:1.26→ go-1-26 (specific minor version)
When to update: If you add go-1-27, update go-1-26 to remove its latest and major-version tags, and add them to go-1-27.
7. Generate Files and Test
# Generate distro directories, lockfiles, VERSION, TAGS, etc.
make
# Build the image
ci/build_images.sh go-1-25/hummingbird/default
# Run tests
ci/run_tests_container.sh go-1-25/hummingbird/default
# Run linters
make check
Common Issues
Error: No match for argument: golang1.25
The package is not available in the configured distro repositories. If it is available only in
the Hummingbird repository, add distros: [hummingbird] to properties.yml, then regenerate files:
make
Generation removes stale distro directories. Do not edit or remove generated files individually.
Choosing a Stream Value
Every image must have a stream field in properties.yml. The stream is a version series
identity (e.g., "24", "3.11", "10.1", "latest") that is emitted as the
io.hummingbird-project.stream label on the container image. It enables the image catalog to
provide consistent structured data about multi-stream repositories.
Why Stream Matters
The image catalog must provide consistent structured data across time, including for images published before a second stream exists. Labels on published images cannot be changed retroactively. If a repository gains a new stream, all previously published images must already carry the correct version-based stream value — otherwise the catalog has inconsistent data that requires fragile repository-specific fixups.
Decision Process
flowchart TD
Start["New image: choose stream value"] --> Q1{"Does endoflife.date show\nmultiple concurrently\nsupported branches?"}
Q1 -->|Yes| Assess["Assess ecosystem granularity:\nupstream project, Docker Hub,\nDebian/Ubuntu, Wolfi/Chainguard,\nFedora, endoflife.date"]
Assess --> Version["Use version-based stream\n(match ecosystem granularity)"]
Q1 -->|No| Q2{"Does the software have a\nmeaningful version series\nidentity?"}
Q2 -->|Yes| Current["Use version-based stream\n(current major version)"]
Q2 -->|No| Latest["stream: latest"]
Version --> Doc["Document reasoning\nas YAML comment"]
Current --> Doc
Latest --> Doc
Document the reasoning for the stream choice as a YAML comment above the stream field in
properties.yml. This ensures future maintainers understand why a particular granularity was
chosen:
# endoflife.date: parallel at major.minor (2.8, 3.0, 3.2)
stream: "3.0"
# rolling release with date-based versions, no version branches
stream: "latest"
Step 1: Does endoflife.date show multiple concurrently supported branches?
Check endoflife.date for the upstream project. It provides structured data on upstream release branches, security support timelines, and versioning schemes for most projects. If it shows multiple concurrently supported branches, use a version-based stream at the granularity of those branches. If the product is not on endoflife.date, that is itself evidence that parallel version lifecycle management is not a concern for that software — proceed to step 2.
The stream granularity must match the level at which upstream maintains parallel security-supported branches. Check:
- Upstream project: At what version granularity are parallel branches maintained?
- Docker Hub official images: Are there parallel version tags? At what granularity?
- Debian/Ubuntu: Are there parallel versioned packages?
- Wolfi/Chainguard: Are there versioned package variants?
- Fedora: Are there parallel versioned packages? Single-version availability is not evidence against versioning, but multi-version availability is strong evidence for it.
Important: the image’s directory name is not evidence for granularity. The directory may use a coarser version or no version at all. The stream must reflect upstream’s actual branch structure.
Similarly, the purpose of upstream’s branches (LTS vs stable, stable vs mainline, STS vs LTS) is irrelevant to granularity. If they are parallel branches receiving security updates, the stream granularity must match.
Match the upstream project’s version naming convention. If the project and ecosystem consistently
identify branches using a format that includes a minor component (e.g., .NET uses 8.0, 9.0 —
never just 8 or 9), use that format for the stream value. This applies even when
endoflife.date abbreviates to just the major number.
Step 2: Does the software have a meaningful version series identity?
If the software uses numbered versioning where the major (or major.minor) number identifies a
version series, use a version-based stream at the current major version. The risk is asymmetric:
using a version-based stream when a new major version never ships is harmless (the label is still
correct), while using "latest" when a new version does ship breaks catalog consistency.
Note: if endoflife.date lists minor versions (e.g., memcached 1.4, 1.5, 1.6) but each immediately supersedes the previous with no overlap in support periods, the minor number is a sequential release counter, not a branch identity. Use the major version as the stream.
Reserve "latest" only for images where no meaningful version series identity exists. This is a
narrow criterion with two cases:
- Rolling/date-based releases with no version series (e.g., minio uses
RELEASE.2024-01-18...) - Base images that track the distro rather than independent software (e.g., core-runtime)
Step 3: For unversioned images, track the current version
For images without a version in their directory name or version constraints, the stream value
reflects the version currently shipped and must be updated when the upstream version changes.
This is analogous to version_constraints — an explicit declaration that is updated periodically.
Field Reference
See the Image Configuration Reference for the
stream field definition and examples.
Next Steps
- Testing Guide - How to run and write tests
- Image Configuration Reference - Complete
properties.ymlreference - FIPS Variant Guide - How to add FIPS variants to images
- Image Pipeline - How the complete pipeline works
2.5 - Adding FIPS Variants
Overview
FIPS variants provide container images that use only
FIPS 140-3 validated cryptographic modules. FIPS is implemented as a
cross-cutting modifier variant (like builder), not a separate image — the
FIPS variant uses the same base image structure but layers FIPS configuration
on top via additional RPM packages.
FIPS variants are restricted to the Hummingbird distro only, because FIPS-validated packages are not available in Fedora Rawhide.
FIPS images must not contain any non-validated cryptographic libraries. The
presence of an unvalidated crypto library (such as libgcrypt or gnutls)
would undermine the FIPS compliance guarantee, even if the application itself
does not use that library. Global tests enforce this constraint.
FIPS Validation Scope
The cryptographic modules shipped in FIPS images are FIPS 140-3 validated through the NIST Cryptographic Module Validation Program (CMVP). This validation applies only when running on RHEL systems installed in FIPS mode. Running FIPS images outside this environment (e.g., on Fedora, on RHEL not in FIPS mode, or on other Linux distributions) is outside the validated configuration.
As a best-effort goal, FIPS images aim to behave similarly on both FIPS and non-FIPS hosts — restricting cryptographic operations to FIPS-approved algorithms regardless of host configuration. This is not a guarantee; some images (particularly NSS-based ones like OpenJDK) may behave differently depending on host FIPS mode.
Validated Cryptographic Modules
All validated modules in Hummingbird are pre-built binaries from RHEL 9.2, submitted to NIST for FIPS 140-3 validation:
| Module | Package | Verified by global test |
|---|---|---|
| OpenSSL FIPS provider | openssl-config-fips |
fips-provider-matches-ubi9 |
| NSS softokn | nss-softokn-fips |
fips-nss-modules-match-rhel92 |
| NSS freebl | nss-softokn-freebl-fips |
fips-nss-modules-match-rhel92 |
Global tests validate each module’s checksum against the known-good RHEL 9.2 validated binaries. Checksums are architecture-specific because the binaries differ per architecture.
All FIPS variants also install crypto-policies-config-fips, which sets the
system-wide crypto policy to FIPS.
Two FIPS Stacks
Two FIPS stacks exist depending on which crypto library the image’s software uses:
OpenSSL stack — crypto-policies-config-fips + openssl-config-fips.
The OpenSSL FIPS provider is enabled via a drop-in configuration file at
/etc/pki/tls/openssl.d/fips-provider-enable.cnf. Enforcement is
container-side: non-approved algorithms are rejected even on non-FIPS hosts.
Used by: Nginx, Node.js, Python, Ruby. Go images also include these packages
for system tools (e.g., git-core), but Go itself uses its own
native FIPS 140-3 module.
NSS stack — crypto-policies-config-fips + nss-softokn-fips +
nss-softokn-freebl-fips. NSS checks the host kernel’s FIPS mode
(/proc/sys/crypto/fips_enabled), so enforcement behavior differs between
FIPS and non-FIPS hosts. The primary Java security provider switches to
SunPKCS11-NSS-FIPS. Used by: OpenJDK.
Both stacks share crypto-policies-config-fips for the system crypto policy,
but differ in which cryptographic library provides the validated
implementation.
Adding a FIPS Variant
1. Determine the FIPS Stack
Identify which crypto library the image’s software uses:
- OpenSSL — most languages and servers (Python, Node.js, Ruby, Nginx, Go)
- NSS — Java/OpenJDK (uses NSS via PKCS#11)
This determines which FIPS packages to install.
2. Update properties.yml
Add fips to additional_variants with a Hummingbird distro restriction,
and add the FIPS packages to rpm_packages.fips.
OpenSSL-based image:
additional_variants:
- name: fips
distros: [hummingbird]
rpm_packages:
fips:
- crypto-policies-config-fips
- openssl-config-fips
NSS-based image (OpenJDK):
OpenJDK has existing runtime and runtime-builder base variants. FIPS
is added as both fips (full JDK) and runtime-fips (headless JRE):
additional_variants:
- name: fips
distros: [hummingbird]
- name: runtime-fips
distros: [hummingbird]
rpm_packages:
fips:
- java-21-openjdk-devel # variant-specific packages
- crypto-policies-config-fips
- nss-softokn-fips
- nss-softokn-freebl-fips
runtime-fips:
- crypto-policies-config-fips
- nss-softokn-fips
- nss-softokn-freebl-fips
See the Image Configuration Reference for
complete properties.yml options, including
additional_variants
and rpm_packages.
3. Generate Files
Run make to generate the hummingbird/fips/ directory structure
(Containerfile, RPM lockfiles, VERSION, TAGS):
make
4. Write FIPS Tests
Add FIPS-specific tests to tests-container.yml using filters.variants to
target FIPS variants. A standard set of FIPS tests validates:
- Non-approved algorithm rejected — e.g., MD5 (for security use), Blowfish, RC2
- Approved algorithm works — e.g., SHA-256
- Non-approved cipher rejected — e.g., 3DES, CHACHA20-POLY1305
- Approved cipher works — e.g., AES-256-GCM (encrypt/decrypt roundtrip)
Example (Python):
fips-rejects-md5:
filters:
variants: ["*fips*"]
command: |
test_engine_run --rm "${TEST_IMAGE:?}" python3 -c "
import hashlib
try:
hashlib.new('md5')
print('MD5_ALLOWED')
except ValueError:
print('MD5_REJECTED')
" | grep -q MD5_REJECTED \
|| test_fail "FIPS mode should reject MD5"
fips-allows-sha256:
filters:
variants: ["*fips*"]
command: |
test_engine_run --rm "${TEST_IMAGE:?}" python3 -c "
import hashlib
print(hashlib.sha256(b'test').hexdigest())
" | grep -q 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 \
|| test_fail "FIPS mode should allow SHA-256"
Host FIPS mode considerations:
Where possible, write tests without filters.fips so they validate
behavior on both FIPS and non-FIPS hosts. This supports the best-effort goal
of consistent behavior across host types. Use filters.fips: true only for
tests that inherently require a FIPS-enabled host kernel — primarily
NSS-based images where enforcement depends on the host:
# NSS-based: rejection only works on FIPS hosts
fips-rejects-blowfish:
filters:
variants: ["*fips*"]
fips: true # requires FIPS-enabled host kernel
command: |
...
See the Test Configuration Reference for variant filters and FIPS mode selection.
5. Build and Test
# Build the FIPS variant
ci/build_images.sh <image>/hummingbird/fips
# Run tests
ci/run_tests_container.sh <image>/hummingbird/fips
# Run linters
make check
Go-Specific FIPS Requirements
Go 1.24+ includes a native FIPS 140-3 cryptographic module that has been
CMVP validated (CMVP Certificate #5247). Unlike the previous
golang-fips approach (which replaced the compiler with a fork using OpenSSL
as the crypto backend), native FIPS uses the same golang package with
build-time and runtime environment variables:
GOFIPS140=v1.0.0(build-time): Tellsgo buildto include the CMVP-validated FIPS module (v1.0.0) in the resulting binary. Set in the image environment so all builds in the container use it.GODEBUG=fips140=on(runtime): Enables FIPS 140-3 mode — uses NIST DRBG for randomness, negotiates only FIPS-approved TLS, and runs mandatory self-tests. Set in the image environment as the default for binaries run in the container.- Same package: The FIPS variant installs the standard
golang1.25package (noversion_packageoverride needed). - No cgo dependency: Native FIPS works with both
CGO_ENABLED=0andCGO_ENABLED=1binaries.
main_package: golang1.25
rpm_packages:
fips:
- crypto-policies-config-fips
- golang1.25
- openssl-config-fips
The openssl-config-fips package is retained for system tools in the image
(e.g., git-core) that use OpenSSL for TLS connections.
Go-specific FIPS tests verify:
crypto/fips140.Enabled()returnstrue(native FIPS module is active)go version -moutput includesbuild GOFIPS140=v1.0.0-c2097c7c(CMVP-validated module, per the security policy section 11.1)- FIPS works with
CGO_ENABLED=0static binaries (a key advantage over the oldgolang-fipsapproach)
OpenJDK-Specific FIPS Requirements
OpenJDK uses the NSS stack instead of OpenSSL, which has several implications:
- No
openssl-config-fips— usesnss-softokn-fips+nss-softokn-freebl-fipsinstead - Host FIPS mode dependency — NSS checks
/proc/sys/crypto/fips_enabled, so FIPS enforcement behavior differs between FIPS and non-FIPS hosts. Crypto rejection tests must usefilters.fips: true. - Compound variants — OpenJDK introduces
runtime-fipsalongsidefips, combining theruntimebase variant with the FIPS modifier. Both share the same NSS FIPS packages;fipsadditionally includesjava-*-openjdk-devel. - Security provider — in FIPS mode, the primary Java security provider
becomes
SunPKCS11-NSS-FIPSinstead of the defaultSUN/SunJCEproviders.
Global FIPS Tests
The following tests run automatically for all FIPS images via
ci/global-tests/tests-container.yml. Each test skips gracefully for images
using the other FIPS stack.
| Test | Validates | Applies to |
|---|---|---|
fips-provider-matches-ubi9 |
OpenSSL FIPS module checksum matches RHEL 9.2 binaries | OpenSSL stack |
fips-nss-modules-match-rhel92 |
NSS module checksums match RHEL 9.2 validated binaries | NSS stack |
fips-no-libgcrypt |
libgcrypt is not present (not a validated module) |
All FIPS |
openssl-fips-config-installed |
OpenSSL FIPS drop-in config exists | OpenSSL stack |
crypto-policy-is-fips |
System crypto policy is set to FIPS |
All FIPS |
Image-specific FIPS tests (algorithm rejection/acceptance) are defined in each
image’s tests-container.yml.
Testing on FIPS Hosts
The test runner detects host FIPS mode from /proc/sys/crypto/fips_enabled
and exports the TEST_FIPS environment variable (true or false). Tests
can filter on this using filters.fips:
# Runs only on FIPS-enabled hosts
fips-host-required:
filters:
fips: true
command: |
...
# Runs only on non-FIPS hosts
non-fips-only:
filters:
fips: false
command: |
...
# Runs regardless of host FIPS mode (default when filters.fips is omitted)
always-runs:
command: |
...
OpenSSL-based images: Most FIPS tests run regardless of host FIPS mode, because the OpenSSL FIPS provider enforces algorithm restrictions at the container level.
NSS-based images: Crypto rejection tests require filters.fips: true,
because NSS enforcement depends on the host kernel’s FIPS mode.
Next Steps
- Image Configuration Reference —
additional_variants,rpm_packages,version_package - Test Configuration Reference — variant filters, FIPS mode selection
- Testing Guide — how to run and write tests
2.6 - Testing Guide
Container tests validate image functionality with Docker and Podman. K8s tests validate images in real Kubernetes environments.
Running Container Tests Locally
Prerequisites
- Container Engine: Podman or Docker
- Python:
PyYAMLpackage (pip install PyYAMLordnf install python3-pyyaml)
With Podman (Recommended)
Tests work directly with Podman:
ci/run_tests_container.sh <image-name>
# Test specific distro
ci/run_tests_container.sh <image-name>/rawhide
# Test specific distro/variant
ci/run_tests_container.sh <image-name>/rawhide/default
# Verbose output (shows passing test output and bash trace)
ci/run_tests_container.sh --verbose <image-name>
With Docker
Use --setup to automatically configure Docker-in-Docker:
ci/run_tests_container.sh --engine docker --setup <image-name>
The --setup flag:
- Starts
hummingbird-docker-dindcontainer withmirror.gcr.io/docker:dind - Configures environment variables (
DOCKER_HOST,DOCKER_CERT_PATH, etc.) - Waits for Docker daemon to be ready
- Reuses existing container on subsequent runs
Prerequisites:
- Docker CLI (
dnf install docker-cli) - Podman (to run the dind container)
Building Images
Build images before testing:
# With Podman
ci/build_images.sh <image-name>
# With Docker
ci/build_images.sh --engine docker --setup <image-name>
Testing Base Images
When modifying base images, test dependent images:
ci/build_images.sh core-runtime
ci/build_images.sh --build-deps core-runtime
ci/run_tests_container.sh --include-reverse-deps core-runtime
Reproducing CI Failures
Reproduce Testing Farm failures locally:
# Single distro/variant tests
ci/run_tests_container.sh --include-reverse-deps --engine podman <image-name>/rawhide/default
ci/run_tests_container.sh --include-reverse-deps --engine docker --setup <image-name>/rawhide/default
# Group tests (all distros/variants)
ci/run_tests_container.sh --engine podman <image-name>/rawhide/group
ci/run_tests_container.sh --engine docker --setup <image-name>/rawhide/group
Troubleshooting Container Tests
Inspecting Failed Tests
Use --pause to inspect containers before cleanup:
ci/run_tests_container.sh --pause <image-name>
Viewing Test Output
By default, only failing tests show output. Use --verbose to see passing test output:
ci/run_tests_container.sh --verbose <image-name>
This also enables bash trace mode (set -x), showing each command as it executes.
Running K8s Tests Locally
K8s tests require kubectl and access to a Kubernetes cluster (via kubeconfig or context).
Basic Usage
ci/run_tests_k8s.sh --context <context> <image-name>
# Test specific distro/variant
ci/run_tests_k8s.sh --context <context> <image-name>/rawhide/default
# Verbose output
ci/run_tests_k8s.sh --verbose --context <context> <image-name>
Safety: The script requires explicit --context or --kubeconfig to prevent
accidental operations on production clusters.
Local Development Workflow
Build images locally and push to the internal registry:
# Build the image
ci/build_images.sh nginx/hummingbird/default
podman tag registry.access.redhat.com/hi/nginx:latest my-nginx:dev
# Push to internal registry and test
ci/run_tests_k8s.sh --context mpp-preprod --push-image my-nginx:dev nginx/hummingbird/default
This requires oc login to the target cluster and oc project <namespace> to set
the target namespace.
Testing Published Images
Test published images without building locally:
IMAGE_URL=registry.access.redhat.com/hi/nginx:latest \
IMAGE_NAME=nginx--hummingbird--default \
ci/run_tests_k8s.sh --context mpp-preprod nginx/hummingbird/default
Troubleshooting K8s Tests
Use --pause to inspect resources before cleanup, or --verbose to see passing
test output:
ci/run_tests_k8s.sh --pause --context <context> <image-name>
ci/run_tests_k8s.sh --verbose --context <context> <image-name>
Tests that encounter recognized transient infrastructure failures run up to three times. The
runner cleans up test resources and waits 60 seconds before each retry. Application assertion
failures are reported immediately. Recognized failures include transient registry and container
daemon errors, package-download timeouts from the Hummingbird repository, and transient Go
crypto/rand entropy-read failures on FIPS-enabled hosts.
Writing Tests
Test File Locations
- Container tests:
images/<name>/tests-container.yml - K8s tests:
images/<name>/tests-k8s.yml
Both use the same YAML format with different available environment variables.
Basic Tests
Create a test file with named tests:
---
version-check:
command: |
test_engine_run --rm "${TEST_IMAGE}" your-service --version
basic-functionality:
command: |
test_engine_run --rm "${TEST_IMAGE}" your-service --help
Container tests must use test_engine_run to start containers. When a detached container has both
a name and a network, the helper waits for Podman network DNS registration automatically before
returning. Fresh lookup processes avoid negative DNS caching in a single client process:
test_engine_run -d --network "${NETWORK_NAME}" --name "${CONTAINER_NAME}" "${TEST_IMAGE}"
For K8s tests:
---
cluster-access:
command: |
kubectl auth can-i get pods || test_fail "No pod access"
run-as-pod:
command: |
name="test-pod-${TEST_RUN_ID}"
kubectl run "${name}" --image="${TEST_IMAGE}" --restart=Never --labels="${TEST_RUN_LABEL}"
kubectl wait --for=jsonpath='{.status.phase}'=Succeeded "pod/${name}" --timeout=60s
External Test Scripts
For complex tests, use a separate shell script:
---
complex-test:
command: ./test-complex-scenario.sh
Create images/<name>/test-complex-scenario.sh:
#!/bin/bash
set -euo pipefail
# Load TEST_IMAGES array for cross-variant testing
# shellcheck disable=SC1090
source "${TEST_IMAGES_PATH:?}"
# Run test
test_engine_run --rm "${TEST_IMAGE:?}" your-service test
Variant-Specific Tests
Limit tests to specific variants:
build-tools-test:
variants: [builder]
command: |
test_engine_run --rm "${TEST_IMAGE}" make --version
Cross-Variant Tests
Test interactions between variants:
cross-variant-test:
variants: [group]
command: |
builder="${TEST_IMAGES[nginx/builder]:?}"
default="${TEST_IMAGES[nginx/default]:?}"
test_engine_run --rm "${builder}" nginx -version
test_engine_run --rm "${default}" nginx -version
Known Issues (Container Tests Only)
Mark container tests with known failures:
test-with-known-issue:
command: |
result=$(test_engine_run --rm "${TEST_IMAGE}" some-command)
[[ "$result" == "expected" ]] || test_fail "Custom error message"
known_issues:
- description: "Known configuration issue"
issue: "https://issues.redhat.com/browse/PROJ-1234"
pattern: "Custom error message"
fails: "sometimes"
See the Test Configuration Reference for complete configuration options.
Working with CI Tests
Container Tests (Testing Farm)
Container tests run on Testing Farm with RHEL-9-Nightly systems for x86_64
and aarch64. The runner skips variants unsupported by the host architecture.
Affected-test planning rejects required helper images that lack that architecture.
Native s390x coverage needs a Testing Farm scenario and s390x test helper images.
Viewing CI Test Results
- Open the merge request in GitLab
- Navigate to the Jobs page for the pipeline
- Identify the Testing Farm job (e.g.,
containers-rawhide-testing-farm-x86-64) - Click on the job name to go to the Konflux PipelineRun page
- If
wait-for-resultsis already finished, select it in the pipeline details, and then switch to the Testing Farm job via theARTIFACTS_URLlink in the right side pane; otherwise, select theschedulerjob, switch to the Testing Farm API details via thetf-requestlink in the right side pane, and then follow therun.artifactslink in the JSON data
Each Testing Farm request runs one or more FMF plans in parallel. For Hummingbird images, results
appear under two plans: /ci/run_tests_container/standard (non-FIPS) and
/ci/run_tests_container/fips (FIPS-enabled). Other applications only run /standard.
K8s Tests (Konflux Ephemeral Namespace)
K8s tests run in Konflux ephemeral namespaces provisioned via EaaS (Environment as a Service).
Viewing K8s Test Results
- Open the merge request in GitLab
- Navigate to the Jobs page for the pipeline
- Identify the K8s test job (e.g.,
containers-rawhide-k8s-test) - Click on the job name to go to the Konflux PipelineRun page
- View the
run-teststask logs for test output
Rerunning CI Tests
Retrigger a specific pipeline using slash commands in merge request comments:
/retest nginx--hummingbird--default-on-pull-request
For automated retriggers of only failed checks, see Retrying Konflux Checks.
Next Steps
- Test Configuration Reference - Complete test configuration reference
- Adding Images - How to add new images with tests
2.7 - Adding Kubernetes Tests
This guide describes how to add Kubernetes tests to container images.
Overview
Kubernetes tests validate that container images work correctly in Kubernetes and OpenShift environments, particularly:
- Directory permissions (important for OpenShift arbitrary UIDs)
- Version checks
- Application functionality (web servers, databases, etc.)
- Multi-container scenarios (sidecar patterns)
Confirm Test Design
Before implementing:
- Read
images/{image_name}/tests-container.ymlto understand existing tests - Identify which tests need K8s validation:
- Directory permissions (important for OpenShift arbitrary UIDs)
- Version check
- Application functionality (web server, database, etc.)
- Multi-container scenarios (sidecar patterns)
- Confirm test approach before creating files
Rules
- Script naming:
test-k8s-*.shprefix (e.g.,test-k8s-version.sh) - Script location:
images/{image_name}/test-scripts/ - YAML location:
images/{image_name}/tests-k8s.yml - Keep YAML lean: complex logic goes in scripts, not inline YAML
- No external downloads: avoid network flakes (exception: container images)
- OpenShift compatibility: writable dirs must use gid=0 + g+w permissions
Validation
- If
images/{image_name}/doesn’t exist → error “Image not found” - If no
tests-container.yml→ error “Add container tests first”
Create Test Scripts
If directory doesn’t exist → create images/{image_name}/test-scripts/
Test Script Template
#!/bin/bash
set -euo pipefail
# Test logic
test_result=$(command)
if [[ ! "${test_result}" =~ "expected" ]]; then
echo "FAIL: Description"
echo "Got: ${test_result}"
exit 1
fi
echo "SUCCESS: Test passed"
exit 0
Directory Permissions Test
If testing OpenShift compatibility → create test-k8s-directory-permissions.sh:
- Check writable directories exist and are writable
- Verify running as non-root (UID != 0)
- Test actual write with touch/rm
Version Check Test
If testing version → create test-k8s-version.sh:
- Run version command
- Check major version only (not major.minor.patch)
- Avoids breaking on patch updates
Application-Specific Tests
- If web server → deployment test with sidecar (curl container)
- If database → connection/persistence test
- If tool → primary functionality test
Create tests-k8s.yml
Pattern for each test:
test-name:
variants: [default]
command: |
# Create ConfigMap with script
kubectl create configmap "${TEST_GROUP}"-scripts-"${TEST_RUN_ID}" \
--from-file=test-k8s-name.sh=test-scripts/test-k8s-name.sh \
--dry-run=client -o yaml | kubectl label -f - --local -o yaml hum-k8s-test="${TEST_RUN_ID}" | kubectl apply -f -
# Create Pod
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: ${TEST_GROUP}-name-${TEST_RUN_ID}
labels:
hum-k8s-test: "${TEST_RUN_ID}"
spec:
restartPolicy: Never
containers:
- name: test
image: ${TEST_IMAGE:?}
command: ["sh", "/scripts/test-k8s-name.sh"]
volumeMounts:
- name: test-scripts
mountPath: /scripts
volumes:
- name: test-scripts
configMap:
name: ${TEST_GROUP}-scripts-${TEST_RUN_ID}
defaultMode: 0755
EOF
# Wait for completion
kubectl wait --for=jsonpath='{.status.containerStatuses[?(@.name=="test")].state.terminated.reason}'=Completed \
pod/"${TEST_GROUP}"-name-"${TEST_RUN_ID}" --timeout=120s || TEST_FAIL "Test did not complete"
# Check logs
out=$(kubectl logs "${TEST_GROUP}"-name-"${TEST_RUN_ID}" -c test)
[[ "${out}" =~ "SUCCESS" ]] || TEST_FAIL "Test failed: ${out}"
Multi-Container Tests
If test needs multiple containers (app + client):
- Use
emptyDirvolume mounted in both containers - emptyDir has 777 permissions, works with arbitrary UIDs
Readiness Probes
If testing app functionality:
- Add
readinessProbeto app container - Set
timeoutSeconds: 5for OpenShift network latency
OpenShift Compatibility
Read images/{image_name}/Containerfile.j2:
- If writable dirs use
user:userownership → change touser:0 - If writable dirs lack
g+w→ addchmod -R g+w - Why gid=0? OpenShift runs containers with arbitrary UIDs (for security) but always assigns gid=0 (root group). Directories must be group-owned by 0 with group-write permissions so the arbitrary UID can write to them (since it’s in group 0)
- IMPORTANT: Use
{{ default_user }}template variable, never hardcode UIDs:- ✅ Correct:
chown -R {{ default_user }}:0 ${NEWROOT}/dir - ❌ Wrong:
chown -R 65532:0 ${NEWROOT}/dir - Template variables follow project conventions and allow centralized config changes
- ✅ Correct:
If Containerfile changed:
touch images/{image_name}/Containerfile.j2
make generated/images/{image_name}/{distro}/default/Containerfile
ci/build_images.sh {image_name}/{distro}/default
Lint and Test
-
Run
make -j$(nproc) check→ if fails, fix errors before proceeding -
Test in minikube:
minikube start ci/run_tests_k8s.sh --context minikube {image_name} -
Monitor pods immediately:
kubectl get pods -w -
If Containerfile changed → test in OpenShift:
ci/build_images.sh {image_name}/{distro}/default # Tag with simple localhost name for --push-image podman tag registry.access.redhat.com/hi/{image_name}:latest localhost/{image_name}:test ci/run_tests_k8s.sh --context <ocp> --push-image localhost/{image_name}:test {image_name}/{distro}/defaultNote: Push images to
localhost/{image_name}:test, not toquay.io. Tests should never push to the official registry. -
Monitor pods:
oc get pods -w
Common Issues
- If tests fail OpenShift with “Permission denied” → check Containerfile uses gid=0 and g+w
- If wrong image used in OpenShift → verify
ci/run_tests_k8s.shdoesn’t override TEST_IMAGE when--push-imageset - If multi-container test can’t see files → add emptyDir volume
- If readiness probe times out in OpenShift → add
timeoutSeconds: 5
Automated Skill
For AI-assisted implementation, use the /add-k8s-tests <image-name> command which follows these patterns automatically.
2.8 - Image Configuration Reference
Complete reference for configuring container images via properties.yml.
properties.yml Overview
Each image can have custom settings via a properties.yml file. This reference documents all
available configuration options.
Minimal example:
---
rpm_packages:
all:
- nginx
main_package: nginx
tags:
- value: latest
Complete structure:
---
# Distro configuration (see Image Variants section)
# distros: [hummingbird] # Override default distros (use when package unavailable in some distros)
# Image variants (see Image Variants section)
# Use additional_variants to extend defaults, or variants to override completely
additional_variants: [fpm, fpm-builder] # Adds to default variants
# variants: [default, builder] # Alternative: completely replaces defaults
# Container runtime configuration (see Container Configuration section)
user: default # Optional: 'default' for default_user (65532), or literal user ID/name
# Package management (see Package Management section)
rpm_packages:
build-deps: [...]
all:
- package-name # All architectures
- name: arch-specific-package # Arch-specific (see Package Management)
arches:
only: x86_64
<distro>: [...]
<variant>: [...]
additional_repos: [...]
allow_fedora_repos: true # Optional: allow Fedora repos for hummingbird images (temporary workaround)
# Build configuration (see Build Configuration section)
architectures:
hummingbird: [aarch64, x86_64, s390x] # Optional: replace distro architectures
hermetic: true
# Exceptional legacy source builds only; do not use for new images:
# build_from_source: true
# prefetch_gomod_path: upstream-submodule
# Versioning (see Versioning section)
main_package: package-name
# version_package: # Optional: override package for version lookup per variant/distro
# fips: package-fips-name # Example: variant-specific package name
# Testing configuration (see Testing Configuration section)
reverse_dependency_tests: false # Optional: disable for images like curl
# Release configuration (see Release Configuration section)
repository: image-name # Required: Quay.io repository name
# <reasoning for stream choice>
stream: "latest" # Required: version series identity for the image catalog
application_category: "Storage" # Required: Pyxis application category
# support_level: community # Optional: community or experimental (default: Red Hat supported)
# deprecated: true # Optional: mark the image stream as deprecated
# Image metadata (see Image Metadata section)
summary: "Short one-liner description" # Required when adding labels
description: >- # Required when adding labels
Short paragraph describing the image.
1-2 sentences about key features.
url: "https://upstream-project.org" # Required when adding labels
tags:
- value: latest
- value: '{{ package_major_version("package-name") }}'
label: io.hummingbird-project.major-version
# Compliance scanning (enabled by default, see Compliance Scanning section)
# oscap:
# enabled: false # Optional: set to false to opt out
# profiles: # Optional: override which profiles run per variant
# cis: true
# stig:
# variants: ["*fips*"]
# exclude_rules: # Optional: exclude specific rules (concatenated with globals)
# - id: xccdf_org.ssgproject.content_rule_example
# reason: "Reason for exclusion"
# variants: ["*fips*"] # Optional: glob patterns
# profiles: [cis] # Optional: limit to specific profiles
Image Metadata
Images can declare metadata fields for container image labels:
summary
- Type: String
- Required: When adding image labels (currently caddy only, expanding to all)
- Length: ~40-80 characters (one-liner)
Short phrase describing what the image is. Used in table/list views and catalog cards.
Do not start with the image name – the name is always visible from context
(UI headings, name label, inspect output).
Example: "Web server with automatic HTTPS"
description
- Type: String (use
>-YAML scalar for multi-line readability) - Required: When adding image labels
- Length: ~100-250 characters (1-2 sentences)
Short paragraph for card views, podman inspect, and Kubernetes UIs
(io.k8s.description). Describes what the software is and its key
distinguishing features.
Use >- (folded scalar) in YAML to keep the
value readable across multiple lines – it folds to a single-line string. Do not
use | (literal block) as embedded newlines break label syntax.
Avoid embedded double quotes and backslashes – values are emitted as-is in Containerfile LABEL instructions with no escaping.
The same description value is used when generating the image README: the
readme_description() macro in README.md.j2 templates renders it as the
introductory paragraph. Do not duplicate this text in the template; use
{{ readme_description() }}.
Example:
description: >-
Extensible server platform with automatic HTTPS by default.
Provides HTTP/3, reverse proxying, load balancing, and static
file serving with minimal configuration.
url
- Type: String (URL)
- Required: When adding image labels
URL for the upstream project homepage. Used in
org.opencontainers.image.url and the Conforma url label.
Example: "https://caddyserver.com"
deprecated
- Type: Boolean
- Default: Omitted, which means non-deprecated
- Description: Marks the image stream as deprecated. The generated image carries
the
io.hummingbird-project.deprecated: truelabel so the image catalog can display deprecation status for streams it includes. - Example:
deprecated: true
The field applies to the stream represented by the image directory, including all variants. The catalog tracks deprecated streams within a repository, so active and deprecated streams can remain visible together.
Package updates, builds, tests, and releases continue, and published images remain available. Add deprecation notices and replacement guidance to the image README template separately.
The catalog currently covers Red Hat supported Hummingbird and Rawhide images. Community images carry the label but are not listed in those catalogs.
Image Variants
Images can have multiple variants (e.g., default and builder):
- default: Minimal runtime environment
- builder: Includes development tools and build dependencies
- custom: Define custom variants in
properties.yml
Each variant gets its own:
- Containerfile in
generated/images/<name>/<distro>/<variant>/Containerfile - RPM lockfiles in
generated/images/<name>/<distro>/<variant>/rpms - Build pipeline and release tags (non-default variants get
-<variant>suffix)
variants
- Type: Array of strings
- Default:
[default, builder] - Description: List of image variants to generate. Completely overrides the default variants.
Each variant gets its own Containerfile, lockfiles, build pipeline, and release tags. Non-default
variants get a
-<variant>suffix in their image tags. - Example:
[default](CI images that only need the default variant) - Note: Prefer
additional_variantswhen extending defaults; usevariantsonly when you need to exclude default variants.
additional_variants
- Type: Array of strings or objects
- Default: None
- Description: Additional variants to add to the base variants. The base is
variantsif specified, otherwise the default variants. Use this when you want to extend without repeating.
Simple format (string array):
additional_variants: [fpm, fpm-builder] # Adds FPM variants to base variants for all distros
Object format with distro restrictions:
Each variant can be an object with name and optional distros fields to restrict which distros
the variant is built for:
additional_variants:
- name: fips
distros: [hummingbird] # Only build fips variant for hummingbird distro
- name: fpm # Simple string still works in array
- name: fpm-builder
distros: [hummingbird, rawhide] # Build for specific distros
The distros field supports glob patterns:
additional_variants:
- name: fips
distros: ["*bird"] # Matches hummingbird
- name: special
distros: ["raw*"] # Matches rawhide
When to use distro restrictions:
Use distro restrictions when a variant requires packages or features only available in specific distros. For example, FIPS variants may only be available for Hummingbird where FIPS-validated packages are present
variant_descriptions
- Type: Object (string-to-string mapping)
- Required: When image has
descriptionand definesadditional_variantswith non-default base specializations - Description: Maps base variant names to human-readable descriptions. These descriptions are
emitted as the
io.hummingbird-project.variant.descriptionlabel on each container image. The global description fordefaultis defined inimages/variables.yml; image-specific entries add descriptions for bases likefpm,runtime, oropenssl. Modifier display (builder, fips) is handled by catalog consumers using the boolean variant labels, not by these descriptions.
Writing guidelines:
- Describe the variant holistically — what the image variant is, not just what packages it
adds on top of
default. Someone reading the description with no surrounding context (e.g., viapodman inspect) should understand the variant’s identity. - Use a noun phrase (2-6 words), not a full sentence.
- Do not start with the image name (the image context is already provided by the image name label).
- Avoid the word “runtime” in the
defaultdescription (the OpenJDKruntimevariant is a separate base, and OpenJDK’sdefaultincludes the full JDK). - Avoid expanding abbreviations redundantly (e.g., “JRE runtime” is redundant because JRE already means “Java Runtime Environment”).
- Prefer functional descriptions (what capability the variant provides) over repeating the variant name as a product name.
Example (PHP image with FPM base):
variant_descriptions:
fpm: "PHP FastCGI process manager"
Example (OpenJDK image with runtime base):
variant_descriptions:
runtime: "Headless Java runtime"
distros
- Type: Array of strings
- Default:
[rawhide, hummingbird](fromimages/variables.yml) - Description: List of distros to build for this image. Overrides the global
default_distros. Use this to disable a distro for images where a required package is not available in that distro. - Example:
[hummingbird]disables Rawhide builds for this image - Note: Each enabled distro creates a full set of variants, so the final combinations are distros × variants.
Container Configuration
user
- Type: String, integer, or object (per-variant dict)
- Required: No (defaults to
default) - Description: Specifies which user the container runs as. The
USERdirective is rendered via the{{ set_user() }}macro at the end of Containerfile templates. The special (and default) valuedefaultmaps to thedefault_uservariable (UID 65532). Foruser: default, the{{ final_stage() }}macro setsHOME=/tmpandWORKDIR /tmpearly in the image to ensure unprivileged users have a writable default environment. Images can overrideWORKDIRlater in their Containerfile.j2. For builder variants withuser: default, it additionally setsENV CONTAINER_DEFAULT_USER=65532.
Same user for all variants:
user: default
user: root # Root user
user: postgres # Literal username
Per-variant configuration:
Use when different variants need different users:
user:
default: root
builder: root
fpm: default
fpm-builder: default
Package Management
Packages are defined in properties.yml under rpm_packages:
rpm_packages:
build-deps: # Build-time dependencies (installed in builder layer, not in final
# image). Included in lockfiles for hermetic builds.
- golang # Example: Go compiler for building from source
- rpm-build # Example: Tools for building RPMs during image construction
all: # Included in all variants (Containerfiles and lockfiles)
- coreutils-single
- your-main-package
- name: grub2-efi-x64 # Arch-specific: only on x86_64
arches:
only: x86_64
- name: grub2-efi-aa64 # Arch-specific: only on aarch64
arches:
only: aarch64
hummingbird: # Included in all Hummingbird variants (distro-level key)
- versioned-package-1.2
rawhide: # Included in all Rawhide variants (distro-level key)
- unversioned-package
builder: # Included only in "builder" variant
- compiler-packages
- debug-tools
custom-variant: # Included only in "custom-variant" variant
- variant-specific-packages
Package entry format
Each entry in an rpm_packages list can be either a string (installed on all architectures) or
an object with architecture constraints:
String entry (all architectures):
- coreutils-single
Object entry (arch-specific):
- name: grub2-efi-x64
arches:
only: x86_64 # Install only on x86_64
- name: some-package
arches:
not: aarch64 # Install on all architectures except aarch64
The arches.only and arches.not fields accept either a single string or a list of strings.
Package constraints use RPM architecture names: aarch64, x86_64, s390x, and ppc64le.
How arch-specific packages work:
- Lockfiles: Arch-specific entries are passed to rpm-lockfile-prototype using its native per-arch package format. The lockfile resolver only includes matching packages for each architecture.
- Containerfiles: Arch-specific packages are installed via a
casestatement onTARGETARCH, so the correct packages are installed during multi-arch builds.
rpm_packages.build-deps
- Type: Array of strings or arch-specific objects
- Description: Build-time dependencies needed during image construction. Automatically installed in the builder layer (not in the final image) and included in lockfiles for hermetic builds. Use for compilers, build tools, and other packages needed only during the build process.
rpm_packages.all
- Type: Array of strings or arch-specific objects
- Description: Packages included in all variants, in both Containerfiles and lockfiles
rpm_packages.<distro>
- Type: Array of strings or arch-specific objects
- Description: Packages included in all variants for the specified distro. Useful when different distros ship different package names (e.g. versioned vs unversioned) and the packages should apply to every variant within that distro.
rpm_packages.<variant>
- Type: Array of strings or arch-specific objects
- Description: Packages included only in the specified variant, regardless of distro
additional_repos
- Type: Array of strings (repository filenames)
- Default: None
- Description: Additional yum repository files from
yum-repos/to include in the RPM lockfile generation. These repos are added to the variant-specific repos (see [Global Variables Reference- default_variant_repos]default-variant-repos). Specify only the filename, not the full path.
- Example:
[konflux-ci-rpm-lockfile-prototype-main-fedora-rawhide.repo] - Usage: Only needed for packages not available in standard Fedora repos or variant-specific repos
allow_fedora_repos
- Type: Boolean
- Default:
false - Description: Allow Fedora repositories during lockfile generation for Hummingbird production
images. By default, production Hummingbird images exclude Fedora repos to ensure packages come
only from the Hummingbird repository. Set to
trueto temporarily allow Fedora repos when required packages are not yet available in the Hummingbird repository. - Example:
true - Usage: Typically used temporarily for packages not yet built for Hummingbird (e.g., .NET runtime on x86_64, OpenJDK packages). Should be removed once packages are available in the Hummingbird repository.
- Note: Toolchain and similar images under
images/(such ashummingbird-builder) use Fedora repositories for signature validation regardless of this setting.
Hermetic Builds
For hermetic builds (enabled by default), all packages from:
rpm_packages.build-deps- included in lockfiles and Containerfiles (builder layer only)rpm_packages.all- included in lockfiles and Containerfiles (final image)rpm_packages.<distro>- included in lockfiles and Containerfiles for all variants of a distrorpm_packages.<variant>- included in lockfiles and Containerfiles for matching variantsdefault_rpm_packages.builder- automatically added for builder variants (see Global Variables Reference - default_rpm_packages)
are added to rpms.lock.yaml and prefetched using cachi2 for offline builds.
Build-time vs Runtime Packages:
rpm_packages.build-deps: Installed in the builder layer only (not in final image)rpm_packages.all: Installed in${NEWROOT}(the final image)rpm_packages.<distro>: Installed in${NEWROOT}for all variants of the specified distrorpm_packages.<variant>: Installed in${NEWROOT}for the specified variant
Build Configuration
architectures
- Type: Mapping of distro or
distro/variantto a non-empty list of RPM architecture names - Default:
default_architectures[distro]inimages/variables.yml
Lists replace defaults. A distro/variant entry takes precedence over a distro entry.
Unknown names, duplicate entries, empty lists, and unknown selection keys are rejected.
Package arches.only and arches.not constraints use the same architecture names.
Exclusions apply to the effective architecture list for the selected variant.
architectures:
hummingbird: [aarch64, x86_64, s390x]
hummingbird/fips: [aarch64, x86_64]
Rawhide defaults to aarch64 and x86_64. Builds convert x86_64 to amd64 and
aarch64 to arm64; s390x and ppc64le keep their names. The resolved lists drive
RPM inputs and Konflux build platforms.
Before enabling another architecture, verify package repositories, builder manifests, native build workers, test workers, and test helper images. Generated files must be regenerated from their source configuration.
hermetic
- Type: Boolean
- Default:
true - Description: Controls whether the Konflux build pipeline passes
HERMETIC=trueto the buildah task, enabling hermetic (offline, network-isolated) builds with prefetched dependencies. - Important: Images with
hermetic: falseare blocked from release by the Conforma hermetic_task policy.hermetic: falseshould not be used for any image in the containers repository.
build_from_source
- Type: Boolean
- Default:
false - Description: Controls version extraction in the
package_version()macro. Whentrue, extracts version from.gitmodulesfor images built from git submodules. Whenfalse, extracts version from RPM package metadata. Does not control whether source code is compiled in the Containerfile. - Warning: Do not use for new images. Software built directly from source is not represented by RPM package metadata and cannot be scanned for CVEs by the project’s package-based scanner. This setting exists only for exceptional legacy images.
prefetch_gomod_path
- Type: String (path relative to image directory)
- Default: None
- Description: Path to a Golang project directory (typically a git submodule) for prefetching Go modules via cachi2. Enables hermetic builds for Go projects without a vendor tree. The path is relative to the image directory.
- Usage: Existing exceptional source-built images only. Do not use for new images because source-built software cannot be scanned for CVEs by the project’s package-based scanner.
Versioning
main_package
- Type: String
- Description: Specifies the main package for version labeling
For RPM-based images:
Set to the package name. Version is extracted from RPM package metadata.
main_package: nginx
For source-built images:
Existing exceptional source-built images set this to the submodule path. Version is extracted from
the branch field in .gitmodules. Do not use this pattern for new images: source-built software
cannot be scanned for CVEs by the project’s package-based scanner.
main_package: images/minio/minio-upstream
build_from_source: true
version_constraints
- Type: String (glob pattern)
- Default: None (no constraints)
- Description: Defines a version constraint pattern to prevent unwanted major or minor
version updates. The constraint applies to all distros for this image. Constraints are validated
during
make check, and builds fail if VERSION files violate the constraint. Primarily used for multi-version images (e.g., python-3-11, nodejs-20) to ensure they stay on their intended version family.
Constraint patterns:
Patterns use glob-style matching:
3.11.*- Match any 3.11.X version (recommended)20.*- Match any 20.X version8.*- Match any 8.X version1.25.*- Match any 1.25.X version (for 3-part versioning like Go)
The pattern matches against the content of VERSION files generated during the build process.
Example:
main_package: python3.11
version_constraints: "3.11.*" # Ensure python-3-11 stays on 3.11.X across all distros
main_package: dotnet-sdk-8.0
version_constraints: "8.*" # Applies to rawhide, hummingbird, and any other distros
When to use:
- Multi-version images where each image targets a specific version family
- Preventing automatic major/minor version bumps during Renovate updates
- Ensuring version tags remain accurate (e.g.,
python:3.11doesn’t becomepython:3.12)
Violation handling:
When a constraint is violated, make check fails with a detailed error message showing:
- The VERSION file path
- The actual version found
- The constraint that was violated
- The properties.yml file where the constraint is defined
Options for resolution:
- Create a new image for the new version (recommended for major/minor bumps)
- Exclude the violating version via
yum-repos/*.repoexcludepkgs - Update the constraint to allow the new version
See Version Constraints for detailed documentation.
version_package
- Type: Object (string-to-string mapping)
- Default: None
- Description: Override the package name used for version extraction on a per-distro, per-variant,
or per-distro/variant basis. When a variant or distro installs a different package that provides the
same software (e.g., a FIPS-enabled build), this field tells the version/tag macros which package
name to look up in the lockfile. The lookup order is:
distro/variant, thenvariant, thendistro, thenmain_package.
Per-distro override:
Use when different distros ship the same software under different package names:
main_package: ruby
version_package:
hummingbird: ruby4.0 # hummingbird ships ruby4.0, rawhide ships ruby
Per-variant override:
Use when a variant installs a differently-named package:
main_package: ruby
version_package:
minimal: ruby-minimal # minimal variant resolves version from ruby-minimal
Per-distro/variant override:
Use when both distro and variant affect the package name:
main_package: ruby
version_package:
hummingbird/fips: ruby4.0-fips
Usage in templates:
The resolved package name is available as package_name_for_version in Jinja2 templates:
tags:
- value: "{{ package_version(package_name_for_version) }}"
label: org.opencontainers.image.version
ENV GOLANG_VERSION={{ package_version(package_name_for_version) }}
Testing Configuration
reverse_dependency_tests
- Type: Boolean
- Default:
true - Description: Controls whether this image participates in reverse dependency workflows (both
building and testing). When
true, changes to this image trigger rebuilds and tests of all images that depend on it. Dependencies are detected by searching forTEST_IMAGES[...]references in test scripts. Set tofalsefor images that should not trigger reverse dependency workflows when changed. - Example: Set to
falseforcurlto avoid triggering unnecessary reverse dependency tests
How it works in CI:
When an image is tested, the CI pipeline performs two phases:
-
Build Phase: Builds all dependencies needed for testing
- Uses
ci/build_images.sh --build-depsto build:- Forward dependencies (images the main image’s tests depend on)
- Reverse dependencies (images that depend on the main image, filtered by
reverse_dependency_tests: true) - Forward dependencies of those reverse dependencies
- Automatically deduplicates to ensure each image is built exactly once
- Uses
-
Test Phase: Runs tests on the main image and its reverse dependencies
- Uses
ci/run_tests_container.shandci/run_tests_k8s.shwith--include-reverse-depsto test:- The main image
- All reverse dependencies (filtered by
reverse_dependency_tests: true)
- Kubernetes runs exclude reverse dependencies from the same image group, since a single-component run cannot provide their sibling images; those run in the group pipelines. See run_tests_k8s.sh.
- Uses
Why set to false?
Set reverse_dependency_tests: false for images like curl that are:
- Used pervasively across many images (too many reverse dependencies)
- Have a small, stable API that can be fully tested in their own tests
- Would cause excessive CI load if every dependent image were tested on each change
Release Configuration
The repository and stream fields together define the release identity for an image. Both are
required and validated during CI. The repository specifies the Quay.io repository name, and the
stream specifies the version series within that repository.
repository
- Type: String
- Required: Yes
- Description: The Quay.io repository name for this image. For images where the directory name
matches the desired repository name (e.g.,
images/nginx/→repository: nginx), this is redundant but still required for explicitness. For versioned image directories, this groups multiple versions under one repository (e.g.,nodejs-20andnodejs-24both userepository: nodejs). - Example:
nodejs(used bynodejs-20andnodejs-24directories)
stream
- Type: String
- Required: Yes
- Description: The version series identity for this image within its repository. Emitted as
the
io.hummingbird-project.streamlabel on the container image, enabling the image catalog to provide consistent structured data about multi-stream repositories.
The stream value must reflect upstream’s actual branch structure at the granularity where parallel security-maintained branches exist. Use endoflife.date to determine the correct granularity.
Key rules:
- The stream is never derived from the image directory name. The directory name is not evidence
for stream granularity (e.g., a directory named
foo/for an upstream with branches2.8,3.0,3.2still requires a major.minor stream like"3.0"). - The purpose of upstream branches is irrelevant to granularity. Whether upstream calls them “LTS”, “stable”, “mainline”, or “STS” does not matter. If they are parallel branches receiving security updates, the stream granularity must match.
- Match the upstream project’s version naming convention. If the project identifies branches
as
8.0,9.0(not8,9), use that format even if endoflife.date abbreviates it. - Sequential minor versions are not parallel branches. If endoflife.date lists minor versions that each immediately supersede the previous with no support overlap, the minor is a release counter. Use the major version as the stream.
- Use
"latest"only for images where no meaningful version series identity exists: rolling-release tools with date-based versioning (e.g., minio) or base images that track the distro rather than independent software (e.g., core-runtime). - For unversioned images (no version in directory name, no version constraints), the stream value reflects the version currently shipped and must be updated when the upstream version changes.
See the stream assignment guidelines for the full decision process and rationale.
Document the reasoning for the stream choice as a YAML comment above the stream field.
This ensures future maintainers understand why a particular granularity was chosen.
Examples:
# endoflife.date: parallel at major (Docker Hub + Debian + Wolfi)
stream: "24"
# endoflife.date: parallel at major.minor (Docker Hub + Debian + Wolfi)
stream: "3.11"
# endoflife.date: branches are 10.1, 11.0, not 10, 11
stream: "10.1"
# rolling release with date-based versions, no version branches
stream: "latest"
application_category
- Type: String
- Required: Yes
- Description: The Pyxis application category for the image’s delivery repository. Emitted as
the
io.hummingbird-project.application-categorylabel on the container image and used in the generated Pyxis repo config. All images sharing the samerepositorymust use the same value. Validated duringmake checkagainst the list inci/application-categories.txt.
Common values for Hummingbird images:
"Programming Languages & Runtimes"– language runtimes and SDKs (Go, Python, Node.js, .NET, Ruby, etc.)"Web Services"– web servers, reverse proxies, and load balancers (Nginx, httpd, Caddy, HAProxy, Tomcat)"Database & Data Management"– databases and caches (PostgreSQL, MariaDB, Valkey, Memcached)"Developer Tools"– CLI utilities and build tools (git, curl, jq)"Storage"– object storage (MinIO, MinIO Client)"Operating System"– base runtime images (core-runtime)
Example:
application_category: "Web Services"
tags
- Type: Array of objects
- Description: Define image tags for Quay.io releases. Each variant gets these tags, with
non-default variants receiving a
-<variant>suffix.
Tag Object Fields:
value(required): The tag value. Can be a literal string (e.g.,latest) or a Jinja2 template using helper functions.label(optional): If specified, the tag value is written as aLABELin the Containerfile, and the actual rendered value from the label is used as the image tag. This ensures version tags match the actual package versions in the image.
Tag Helper Functions:
package_version("<package-name>")- Full version (e.g.,1.2.3-4.fc42)package_major_version("<package-name>")- Major version only (e.g.,1)package_major_minor_version("<package-name>")- Major.minor version (e.g.,1.2)rpm_lockfile.content_date-YYYYMMDDUTC date when the lockfile content last changed. Available for all images with an RPM lockfile. Only advances when the resolved RPM set changes, not on every lockfile regeneration.
Example:
tags:
- value: latest # Literal tag, no label
- value: '{{ package_major_version("nginx") }}' # Template tag with label
label: io.hummingbird-project.major-version
- value: '{{ package_major_minor_version("nginx") }}' # Tag becomes actual version from label
label: io.hummingbird-project.major-minor-version
- value: '{{ package_version("nginx") }}'
label: org.opencontainers.image.version
Note: For images with multiple versions, only set latest on the latest version.
tag_suffix_aliases
- Type: Mapping of variant names to arrays of suffixes
- Description: Add aliases for every configured tag without creating another image variant. The suffix is appended directly to each base tag. Existing canonical tags remain unchanged.
- Example:
tag_suffix_aliases:
default:
- jdk25
builder:
- jdk25-builder
support_level
-
Type: String
-
Default: Omitted (image is Red Hat supported)
-
Values:
community,experimental -
Description: Controls the support level, Quay organization, and Konflux application assignment for hummingbird-distro images. The mapping from support level to Quay org is defined in
images/variables.ymlundersupport_levels; all other names (Konflux app, service account, RPA suffix) are derived from the Quay org name. When omitted, the image is Red Hat supported and assigned tocontainers-hummingbird. Rawhide images are always assigned tocontainers-rawhideregardless of this property. -
Example:
support_level: community -
Mapping:
Value Quay org Konflux app CPE Pyxis/RPA (omitted) hummingbirdcontainers-hummingbirdYes Included communityhummingbird-communitycontainers-community-hummingbirdNo Excluded experimentalhummingbird-cicontainers-ci-hummingbirdNo Excluded -
Usage: Use
communityfor images maintained by the community (e.g., minio, os-minimal). Useexperimentalfor internal build infrastructure (e.g., hummingbird-builder). These images are built and tested normally but published to their designated Quay organization, not the official Red Hat registry.
How Labels Work:
- Template is rendered and written to Containerfile:
LABEL org.opencontainers.image.version=1.27.3-1.fc42 - Build system extracts the actual value from the image label
- Image is tagged with the extracted value:
registry.access.redhat.com/hi/nginx:1.27.3-1.fc42
This ensures tags always match the actual package versions built into the image.
Upstream Comparison Configuration
The compare section configures which upstream and third-party images to track for automated
vulnerability comparison via the container catalog’s external image scanning pipeline. Place it
at the end of properties.yml, after all other sections.
Repository and stream values are whatever the image itself uses. Variant names use Hummingbird
names where an equivalent exists (e.g. builder for upstream dev images); upstream-only
variants like slim or alpine keep their upstream names.
compare
- Type: Object
- Required: No (omit for images with no upstream or third-party equivalent)
- Description: Configures which upstream and third-party images to track. When present,
the aggregation script (
ci/aggregate_compare.py) includes this image’s sources in the combined config uploaded to DynamoDB, where the external scanning pipeline picks it up.
compare.sources
- Type: Array of source objects
- Required: Yes (when
compareis present) - Description: List of external image sources to compare against. Each source object documents one upstream registry.
source registry
- Type: String
- Required: Yes
- Description: Full registry pull reference for the upstream image. Must include the full
path — use
docker.io/library/nginx, notdocker.io/nginx.
Example: docker.io/library/nginx, cgr.dev/chainguard/python
source tags
- Type: Array of tag objects
- Required: Yes (non-empty)
- Description: Tags to track on the upstream registry. Each tag object has two fields:
| Field | Required | Description |
|---|---|---|
tag |
Yes | The registry tag to track. Must be a tag that exists on the upstream registry. |
variant |
Yes | The variant this tag represents (e.g. default, alpine, slim). Use Hummingbird variant names where equivalents exist (e.g. builder for upstream dev images). |
source variants
- Type: Array of variant description objects
- Required: Yes
- Description: Describes the variants referenced in
tags. Each variant object has:
| Field | Required | Description |
|---|---|---|
name |
Yes | Variant identifier matching the variant values used in tags. |
description |
Yes | Human-readable description of this variant. |
base |
Yes | Base variant name (use the variant name, e.g. default, alpine, slim). |
builder |
Yes | Whether this is a builder variant. Use false for upstream images. |
fips |
Yes | Whether this is a FIPS variant. Use false for upstream images. |
Tag selection guidance
Use floating tags, not version-pinned tags. Floating tags (e.g. mainline, 3.14) track
the latest patch release automatically. Version-pinned tags (e.g. 1.31.7, 3.14.10) go stale
and require manual updates.
Match the upstream project’s branch or release structure. Check the upstream registry for available tags. Common patterns:
- Branch-based tags (nginx):
mainline,mainline-alpine— these float to the latest release in the mainline branch. Only track the branch that matches your image’s stream. - Minor-version tags (python, node):
3.14,3.14-slim,3.14-alpine— these float to the latest patch within the minor version. - Latest-only (Chainguard):
latest— the only publicly available tag on the free tier.
Place source entries in the directory of the stream they track. The
aggregated config derives every tag’s stream from its directory’s own
top-level stream: value, so a source belongs in the directory whose
stream matches what the upstream tag delivers:
- Upstream has named branches (e.g. nginx
mainline) → the source lives in the directory whosestream:ismainline - Upstream has
3.14.xand3.15.xrelease series → the sources live in thestream: "3.14"andstream: "3.15"directories respectively - Rolling tags like
latestmap to the numbered stream they currently deliver — e.g. a Chainguardlatesttag that is currently Python 3.15 belongs in thepython-3-15directory - Directories whose
stream:islatest(rolling releases, date-based versions) emitstream: latestautomatically
Chainguard images: Chainguard’s free tier only offers latest and latest-dev tags. Place
the Chainguard source in the image directory whose stream matches the Chainguard image’s actual
version. Check with podman run --rm cgr.dev/chainguard/<image>:latest --version (or equivalent).
Chainguard python:latest is currently Python 3.15, so it belongs in python-3-15/properties.yml.
When Chainguard moves to a newer version, move the source to the new
stream’s directory. Use variant: "builder" for latest-dev (the dev image with a shell and
package manager).
Variant naming: use Hummingbird variant names where upstream equivalents exist — default
for plain runtime images, builder for upstream dev images that add a shell and package manager
(e.g. Chainguard’s latest-dev). Upstream-only variants with no Hummingbird equivalent (e.g.
Docker Hub slim, alpine) keep their upstream names. CVE data for all tracked variants is
served the same way, so frontends can present them however fits.
Repository association: the aggregation script automatically attaches the image’s top-level
repository: value to each aggregated tag. The catalog uses it to find the external sources
belonging to an image detail page — no need to repeat it in the compare: section.
Multi-directory images: When multiple image directories compare against the same upstream
registry (e.g. python-3-11/ through python-3-15/ all compare against
docker.io/library/python), each directory’s compare.sources lists only its own stream’s tags.
The aggregation script merges entries with the same registry — duplicate tags across directories
are an error. Because the repository is attached per tag, different Hummingbird repositories may
share one upstream registry (e.g. caddy and xcaddy comparing different tags of
docker.io/library/caddy).
Examples
nginx — branch-based tags, two upstream branches, plus Chainguard:
compare:
sources:
- registry: docker.io/library/nginx
tags:
- { tag: "mainline", variant: "default" }
- { tag: "mainline-alpine", variant: "alpine" }
variants:
- {
name: "default",
description: "Debian-based image",
base: "default",
builder: false,
fips: false,
}
- {
name: "alpine",
description: "Alpine-based image",
base: "alpine",
builder: false,
fips: false,
}
- registry: cgr.dev/chainguard/nginx
tags:
- { tag: "latest", variant: "default" }
variants:
- {
name: "default",
description: "Chainguard image",
base: "default",
builder: false,
fips: false,
}
python-3-14 — minor-version tags, one stream per directory (the aggregation script merges
all python-3-XX directories into a single docker.io/library/python entry):
compare:
sources:
- registry: docker.io/library/python
tags:
- { tag: "3.14", variant: "default" }
- { tag: "3.14-slim", variant: "slim" }
- { tag: "3.14-alpine", variant: "alpine" }
variants:
- {
name: "default",
description: "Debian-based image",
base: "default",
builder: false,
fips: false,
}
- {
name: "slim",
description: "Minimal Debian-based image",
base: "slim",
builder: false,
fips: false,
}
- {
name: "alpine",
description: "Alpine-based image",
base: "alpine",
builder: false,
fips: false,
}
python-3-15 — adds a Chainguard source: rolling latest tags mapped to the current
stream, with latest-dev using the builder variant:
compare:
sources:
- registry: docker.io/library/python
tags:
- { tag: "3.15", variant: "default" }
- { tag: "3.15-slim", variant: "slim" }
- { tag: "3.15-alpine", variant: "alpine" }
variants:
- {
name: "default",
description: "Debian-based image",
base: "default",
builder: false,
fips: false,
}
- {
name: "slim",
description: "Minimal Debian-based image",
base: "slim",
builder: false,
fips: false,
}
- {
name: "alpine",
description: "Alpine-based image",
base: "alpine",
builder: false,
fips: false,
}
- registry: cgr.dev/chainguard/python
tags:
- { tag: "latest", variant: "default" }
- { tag: "latest-dev", variant: "builder" }
variants:
- {
name: "default",
description: "Chainguard image",
base: "default",
builder: false,
fips: false,
}
- {
name: "builder",
description: "Chainguard image with shell and package manager",
base: "default",
builder: true,
fips: false,
}
minio (hypothetical) — rolling release with only a latest tag and no version series on
either side:
compare:
sources:
- registry: docker.io/minio/minio
tags:
- { tag: "latest", variant: "default" }
variants:
- {
name: "default",
description: "Official MinIO image",
base: "default",
builder: false,
fips: false,
}
Compliance Scanning
OpenSCAP compliance scanning verifies CIS and STIG compliance during the
build. Scanning runs against the container rootfs using oscap-chroot and
fails the build if any enabled rules are violated.
Scanning is enabled by default for all images. It only runs for the
Hummingbird distro and is skipped for the builder base image.
oscap.enabled
- Type: Boolean
- Default:
true(fromimages/variables.yml) - Description: Controls whether OpenSCAP compliance scanning runs for this
image. When
true, averify-compliancestep is added to the generated Containerfile for all Hummingbird variants. Set tofalseto opt out.
oscap:
enabled: false
oscap.profiles
- Type: Object mapping profile names to enablement rules
- Default:
{cis: true, stig: {variants: ["*fips*"]}}(fromimages/variables.yml) - Description: Controls which compliance profiles run for each variant.
Each profile can be set to
true(all variants),false(disabled), or an object with avariantslist of glob patterns.
Default behavior (CIS for all variants, STIG only for FIPS variants):
oscap:
profiles:
cis: true
stig:
variants:
- "*fips*"
Enable STIG for all variants:
oscap:
profiles:
stig: true
Disable CIS for a specific image:
oscap:
profiles:
cis: false
The variants field supports glob patterns using fnmatch syntax (e.g.,
*fips* matches fips, fips-builder, openssl-fips).
oscap.exclude_rules
- Type: Array of rule exclusion objects
- Default: Global rule exclusions from
images/variables.yml - Description: Rules to exclude from compliance scanning. Image-level exclusions are concatenated with global exclusions (not replaced). Each exclusion object has the following fields:
| Field | Type | Required | Description |
|---|---|---|---|
id |
String | Yes | XCCDF rule ID (e.g., xccdf_org.ssgproject.content_rule_configure_crypto_policy) |
reason |
String | Yes | Explanation for excluding this rule |
variants |
Array of strings | No | Glob patterns for variants this exclusion applies to. Omit for all variants. |
profiles |
Array of strings | No | Profile names (cis, stig) this exclusion applies to. Omit for all profiles. |
Exclude a rule for all variants and profiles:
oscap:
exclude_rules:
- id: xccdf_org.ssgproject.content_rule_configure_crypto_policy
reason: "Caddy uses Go's built-in TLS stack, not system crypto-policies"
Exclude a rule only for specific variants:
oscap:
exclude_rules:
- id: xccdf_org.ssgproject.content_rule_configure_crypto_policy
reason: "crypto-policies state files not generated in installroot"
variants:
- "*fips*"
Exclude a rule only from a specific profile:
oscap:
exclude_rules:
- id: xccdf_org.ssgproject.content_rule_some_rule
reason: "Not applicable to CIS for this image"
profiles:
- stig
Combine variant and profile filters:
oscap:
exclude_rules:
- id: xccdf_org.ssgproject.content_rule_configure_crypto_policy
reason: "crypto-policies state files not generated in installroot for FIPS variants"
variants: ["*fips*"]
profiles: [cis]
How compliance scanning works
- Profile resolution:
generate_jinja2.pydetermines which profiles (CIS, STIG) are active for the current variant based onoscap.profiles. - Rule filtering: Exclusion rules are filtered by variant (glob matching) and profile, producing per-profile exclusion lists.
- Tailoring file: If any active profile has exclusions, an XCCDF tailoring
file (
oscap-tailoring.xml) is generated in the variant directory with custom profiles extending the upstream CIS/STIG profiles. - Containerfile: The
verify-compliancecommand is added with explicit--cisand/or--stigflags based on active profiles, plus a--tailoring-fileargument if exclusions exist. - Build-time scan:
oscap-chrootevaluates the rootfs against each active profile. The build fails if any non-excluded rule fails.
Next Steps
- Adding Images - Step-by-step guide for adding new images
- Image Pipeline - How the complete pipeline works
2.9 - Test Configuration Reference
Complete reference for configuring image tests via tests-container.yml and tests-k8s.yml.
Test Definition Format
Each image can have tests defined in:
images/<name>/tests-container.yml- Container tests (run with Podman/Docker)images/<name>/tests-k8s.yml- K8s tests (run in Kubernetes cluster)
Both files use the same YAML format:
---
script:
command: ./test.sh
inline:
command: |
image=${TEST_IMAGE:?}
test_engine_run --rm "${image}" --version
Environment Variables
The test runners provide environment variables to test commands. Some are common to both test types, others are specific to container or K8s tests.
Common Variables
| Variable | Description |
|---|---|
TEST_IMAGE |
The container image being tested |
TEST_IMAGES |
Associative array with all group/variant image URLs |
TEST_IMAGES_PATH |
Path to file containing serialized TEST_IMAGES array |
TEST_VERBOSE |
Show test command output (true or false) |
TEST_GROUP |
The image group being tested (nginx, python, etc.) |
TEST_DISTRO |
The distro being tested (hummingbird, rawhide) |
TEST_VARIANT |
The variant being tested (default, builder, etc.) |
TEST_GENERATED_VARIANT_DIR |
Absolute path to the generated variant directory |
Container Test Variables
| Variable | Description |
|---|---|
TEST_ENGINE |
Container engine to use (podman or docker) |
TEST_USER_ID |
Current user ID (for permission handling) |
TEST_FIPS |
Whether host is in FIPS mode (true or false) |
K8s Test Variables
| Variable | Description |
|---|---|
TEST_RUN_ID |
Unique ID for this test run (for resource naming) |
TEST_RUN_LABEL |
Label selector for cleanup (hum-k8s-test=<id>) |
Helper Functions
Common to All Tests
Both test types provide the test_fail function:
| Function | Description |
|---|---|
test_fail |
Fail the test with a custom error message |
In K8s tests, kubectl is pre-configured with the context/kubeconfig from CLI args.
Container Tests Only
Container tests provide helper functions that wrap the container engine with
appropriate flags. Always use these functions instead of calling
${TEST_ENGINE} directly for create and run commands. This ensures tests
respect globally set options such as the hermetic mode setting
(--pull=never).
| Function | Description |
|---|---|
test_engine_create |
Run ${TEST_ENGINE} create with appropriate pull flags |
test_engine_run |
Run ${TEST_ENGINE} run with appropriate pull flags |
Note: For other container engine commands (inspect, network, volume, etc.),
use ${TEST_ENGINE:?} directly.
Variant-Aware Testing
The testing system automatically includes variant-specific default tests:
- Global tests from
ci/{variant}-tests/tests-container.yml - Image-specific tests from
images/<name>/tests-container.yml
Variant Selection
Tests can specify which variants they apply to using the filters.variants field. The field
supports both exact matches and glob patterns.
# Test that only runs for builder variants
build-tools-test:
filters:
variants: [builder]
command: |
test_engine_run --rm "${TEST_IMAGE}" make --version
# Test that runs for multiple specific variants
multi-variant-test:
filters:
variants: [default, builder]
command: |
test_engine_run --rm "${TEST_IMAGE}" echo "Hello"
# Test that runs for all variants (no filters.variants field)
universal-test:
command: |
test_engine_run --rm "${TEST_IMAGE}" echo "Always runs"
Glob Patterns in Variant Filters
Use glob patterns to match multiple variants dynamically:
# Test that runs for any FIPS variant (fips, fips-builder, etc.)
fips-test:
filters:
variants: ["*fips*"]
command: |
test_engine_run --rm "${TEST_IMAGE}" check-fips-mode
# Test that runs for all builder variants (builder, fpm-builder, etc.)
builder-test:
filters:
variants: ["*-builder", "builder"]
command: |
test_engine_run --rm "${TEST_IMAGE}" dnf --version
# Mix exact matches and patterns
mixed-filter:
filters:
variants: [default, "*fips*"]
command: |
test_engine_run --rm "${TEST_IMAGE}" echo "Runs on default and all fips variants"
Supported glob syntax:
*- matches any sequence of characters?- matches any single character
Distro Selection
Tests can specify which distros they apply to using the filters.distros field. Like variant
filters, this field supports both exact matches and glob patterns.
# Test that only runs for hummingbird distro
hummingbird-only-test:
filters:
distros: [hummingbird]
command: |
test_engine_run --rm "${TEST_IMAGE}" some-hummingbird-specific-check
# Test that runs for multiple distros
multi-distro-test:
filters:
distros: [hummingbird, rawhide]
command: |
test_engine_run --rm "${TEST_IMAGE}" echo "Runs on both"
# Test that runs for all distros (no filters.distros field)
universal-test:
command: |
test_engine_run --rm "${TEST_IMAGE}" echo "Always runs"
This is useful for tests that check distro-specific features. For example, CPE labels are only present on hummingbird images (Fedora/rawhide has no official CPE).
Glob Patterns in Distro Filters
Use glob patterns to match multiple distros dynamically:
# Test that runs for any distro ending in "bird"
bird-distro-test:
filters:
distros: ["*bird"]
command: |
test_engine_run --rm "${TEST_IMAGE}" check-something
# Test that runs for any distro starting with "raw"
raw-distro-test:
filters:
distros: ["raw*"]
command: |
test_engine_run --rm "${TEST_IMAGE}" check-rawhide-feature
Supported glob syntax:
*- matches any sequence of characters?- matches any single character
FIPS Mode Selection
Tests can specify whether they should run based on the host’s FIPS mode using filters.fips:
# Test that only runs on FIPS-enabled hosts
fips-required-test:
filters:
fips: true
command: |
test_engine_run --rm "${TEST_IMAGE}" openssl list -providers
# Test that only runs on non-FIPS hosts
non-fips-test:
filters:
fips: false
command: |
test_engine_run --rm "${TEST_IMAGE}" some-non-fips-check
# Test that runs regardless of FIPS mode (no filters.fips field)
universal-test:
command: |
test_engine_run --rm "${TEST_IMAGE}" echo "Always runs"
The TEST_FIPS environment variable is set to true when running on a FIPS-enabled host
(detected from /proc/sys/crypto/fips_enabled), and false otherwise.
Group Test Mode
When running tests for all variants (without specifying a specific variant), the system supports a special “group” test mode for tests that need to work across multiple variants simultaneously:
# Test that runs only in group mode
cross-variant-compatibility-test:
filters:
variants: [group]
command: |
# Access specific group/variant combinations using TEST_GROUP
builder_image=${TEST_IMAGES[${TEST_GROUP}/builder]:?}
default_image=${TEST_IMAGES[${TEST_GROUP}/default]:?}
echo "Builder variant: ${builder_image}"
echo "Default variant: ${default_image}"
# Test that both variants have compatible APIs
test_engine_run --rm "${builder_image}" nginx -version
test_engine_run --rm "${default_image}" nginx -version
Using TEST_IMAGES in External Scripts
External shell scripts that need to reference other image variants must source the TEST_IMAGES array file at the beginning:
#!/bin/bash
set -euo pipefail
# Load TEST_IMAGES array
# shellcheck disable=SC1090
source "${TEST_IMAGES_PATH:?}"
# Now TEST_IMAGES is available - always use :? for proper error checking
test_engine_run --rm "${TEST_IMAGES[curl/default]:?}" ...
test_engine_run --rm "${TEST_IMAGES[httpd/default]:?}" ...
Note: Inline test commands in YAML files automatically have access to TEST_IMAGES and do not need this sourcing pattern.
Using TEST_GROUP for Dynamic References
The TEST_GROUP variable contains the current image group name (e.g., nginx, python,
aspnet-runtime-8-0). Use it with TEST_IMAGES to dynamically reference the current image’s
variants without hardcoding the group name:
multi-stage-build:
filters:
variants: [builder]
command: |
# Build in builder variant, run in default variant
"${TEST_ENGINE}" build -t localhost/myapp -f - . <<EOF
FROM ${TEST_IMAGES[${TEST_GROUP}/builder]:?}
# ... build steps ...
FROM ${TEST_IMAGES[${TEST_GROUP}/default]:?}
COPY --from=0 /app /app
EOF
test_engine_run --rm localhost/myapp /app/myapp
Test Helper Functions
The test runner provides helper functions available in test commands:
test_fail(message)
Immediately fails the test with a custom error message sent to stderr:
test-name:
command: |
result=$(test_engine_run --rm "${TEST_IMAGE}" some-command)
[[ "$result" == "expected" ]] || test_fail "Expected 'expected', got '$result'"
Known Issues (Container Tests Only)
Container tests can specify known log patterns that should not cause the test to report an error (for which support is still unimplemented) or failure. This helps distinguish between expected failures (tracked issues) and unexpected failures (new regressions), and detects when expected failures stop occurring.
Add known_issues to any test to specify patterns matching expected failures:
test-name:
command: |
result=$(test_engine_run --rm "${TEST_IMAGE}" some-command)
[[ "$result" == "expected" ]] || test_fail "Custom error message"
known_issues:
- description: "Known configuration issue"
issue: "https://issues.redhat.com/browse/PROJ-1234"
pattern: "Custom error message"
fails: "sometimes" # optional, defaults to "always"
Fields
- description: Human-readable explanation of the issue (required)
- issue: Full URL to the issue tracking this problem (required)
- pattern: Regular expression(s) to match against test output (required)
- fails: Frequency of failure (optional, defaults to “always”)
Pattern Support
The pattern field supports both single patterns and arrays of patterns:
known_issues:
# Single pattern
- description: "Simple failure case"
issue: "https://issues.redhat.com/browse/PROJ-1"
pattern: "Connection failed"
# Multiple patterns (any match triggers)
- description: "Network connectivity issues"
issue: "https://issues.redhat.com/browse/PROJ-2"
pattern:
- "Connection timeout"
- "Network unreachable"
- "curl: \\(28\\)"
fails: "sometimes"
Failure Frequency
- always: Known failures that consistently fail every time (default)
- sometimes: Intermittent failures that may pass on retry (flaky tests)
Unexpected Pass Detection
The system automatically detects when tests with known issues labeled as fails: "always"
suddenly start passing. This will not prevent missing known issues being masked by other (still
occurring) known issues.
Automatic Retries
The container and K8s test runners automatically retry tests that fail with certain transient infrastructure errors. This helps avoid false test failures caused by temporary registry, network, package-download, image-pull, container-startup, or K8s scheduling problems.
Retry Behavior:
- Failed tests are checked against a list of retriable error patterns
- If a match is found, the test is automatically retried
- If the test still fails after all attempts, it’s reported as a normal failure
K8s tests run up to three times, with cleanup and a 60-second delay between attempts. Retriable
conditions include image-pull and container-creation errors, as well as kubectl wait timeouts.
Application assertion failures are not retried.
Next Steps
- Testing Guide - How to run and write tests
- Adding Images - How to add new images with tests
3 - Operating Hummingbird infrastructure
Operations guides for managing Hummingbird infrastructure, including Konflux cluster setup and CI/CD operations.
3.1 - Excluding Packages from Images
AI Agent Note: When asked to exclude packages, always ask the user for a ticket link or explanation first. This is required for the comment documenting the exclusion.
Overview
When a faulty RPM build is published to Hummingbird repositories, it can be
excluded from container images by adding an excludepkgs directive to the
repository configuration.
Workflow
1. Identify the source package NVR
Identify the Name-Version-Release of the faulty source package. Example:
ncurses-6.5-8.20250614.hum1
2. Generate excludepkgs value
Query the Hummingbird repositories and generate the excludepkgs value:
podman run --rm quay.io/hummingbird-ci/hummingbird-builder:latest sh -c '
dnf5 repoquery --queryformat="%{sourcerpm} %{name}-%{evr}.*
" 2>/dev/null | sed -n "s/^ncurses-6.5-8.20250614.hum1.src.rpm //p" | tr "\n" " "
'
Replace ncurses-6.5-8.20250614.hum1 with the source NVR. This outputs the
ready-to-use excludepkgs value:
ncurses-6.5-8.20250614.hum1.* ncurses-base-6.5-8.20250614.hum1.* ...
3. Add excludepkgs to repo file
Edit yum-repos/hummingbird.repo and add the binary package NEVRs to excludepkgs
in the [hummingbird] section (the binary repo, not [hummingbird-source]).
Document the exclusion with a one-line comment containing the ticket link:
[hummingbird]
...
# HUM-1234: ncurses-6.5-8.20250614.hum1 causes segfault in terminfo parsing
excludepkgs=ncurses-base-6.5-8.20250614.hum1.* ncurses-libs-6.5-8.20250614.hum1.*
Multiple packages are space-separated. Use .* suffix to match all architectures.
4. Merge the change
Merge the repo file change to main. Renovate will automatically apply the exclusions when it next rebases existing lockfile update branches or creates new ones.
Verification
Check that the excluded package no longer appears in any rpms/rpms.lock.yaml files
in subsequent Renovate-generated merge requests.
Removing Exclusions
When a newer version is available, exclusions can be removed.
1. Verify a newer version exists
Check the latest available version (should differ from excluded 6.5-8.20250614.hum1):
podman run --rm quay.io/hummingbird-ci/hummingbird-builder:latest \
dnf5 repoquery --latest-limit=1 --queryformat="%{evr}" "ncurses-libs" 2>/dev/null
2. Remove the exclusion
Remove the excludepkgs line and its comment from yum-repos/hummingbird.repo.
3. Merge and verify
Merge the change. Renovate will pick up the newer version in subsequent lockfile updates.
3.2 - Managing Konflux Comments
When Konflux comments become excessive on merge requests, you can clean them up using the provided comment deletion tool.
Prerequisites
- COM_GITLAB_TOKEN: GitLab API token with
apiscope and Maintainer access
Usage
If Konflux comments become excessive on merge requests, you can clean them up using:
# Show help for the comment deletion tool
make delete-konflux-comments ARGS="--help"
# Example: Delete comments from a specific merge request
make delete-konflux-comments ARGS="https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/123"
# Example: Delete only failed build/test comments (dry run)
make delete-konflux-comments ARGS="--dry-run --comment-category failed -- https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/123"
# Example: Delete all Konflux comments from a merge request
make delete-konflux-comments ARGS="--all https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/123"
Comment Categories
The tool can target specific types of Konflux comments:
- failed: Only comments related to failed builds/tests
- all: All Konflux-related comments (use with
--allflag)
Safety Features
- Dry Run Mode: Use
--dry-runto preview what would be deleted without actually removing comments - Selective Deletion: Target specific comment categories to avoid removing important information
- Confirmation: The tool will show what it plans to delete before taking action
Examples
# Set your GitLab token
export COM_GITLAB_TOKEN=your_token_here
# Preview what failed comments would be deleted
make delete-konflux-comments ARGS="--dry-run --comment-category failed -- https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/789"
# Delete only failed build comments
make delete-konflux-comments ARGS="--comment-category failed -- https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/789"
# Delete all Konflux comments (use with caution)
make delete-konflux-comments ARGS="--all https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/789"
When to Use
Consider cleaning up Konflux comments when:
- Merge requests have accumulated many outdated failure comments
- Comments are making it difficult to find relevant discussion
- You want to start fresh after resolving systematic issues
- The comment thread has become cluttered with automated messages
Note: Use this tool judiciously, as it permanently removes comments that might contain useful debugging information.
Related Operations
- Retrying Konflux Checks - Retrigger failed CI pipeline runs
3.3 - Running Conforma Checks Locally
Overview
Conforma (Enterprise Contract) policy checks run automatically in the Konflux pipeline before release (see Image Pipeline — Stage 5). These checks can also be run locally against any Konflux-built image to validate compliance, test policy changes, or investigate failures.
Prerequisites
Install the Conforma CLI:
curl -sLO https://github.com/conforma/cli/releases/download/snapshot/ec_linux_amd64
chmod 755 ec_linux_amd64
mkdir -p ~/.local/bin
mv ec_linux_amd64 ~/.local/bin/ec
Running All Checks
The full set of Conforma checks (signatures, attestations, SLSA provenance, labels, etc.) can be
run against any Konflux-built image using ec validate image with the Konflux signing key.
Public Key
Testing Farm and local verification use the Konflux build signing public key
committed at ci/key.pub. The authoritative source is the Konflux
member cluster Secret at k8s://openshift-pipelines/public-key. Update this
file whenever Konflux rotates its build signing key so both checks can verify
newly signed attestations.
Obtaining the Image Reference
Combine the IMAGE_URL and IMAGE_DIGEST results from a successful build PipelineRun:
<IMAGE_URL>@<IMAGE_DIGEST>
Merge request build images follow the pattern:
quay.io/redhat-user-workloads/hummingbird-tenant/<component>:on-mr-<MR_ID>-<COMMIT_SHA>
Running the Validation
Create a policy file that matches the pipeline policy but with any modifications
needed for testing. For example, to run the pipeline policy with label checks enabled (removing
the labels.required_labels and labels.optional_labels exclusions):
cat > /tmp/policy.yaml << 'YAML'
sources:
- name: Release Policies
config:
exclude:
- test.required_tests_passed:sast-snyk-check
- test.no_skipped_tests:sast-snyk-check
- test.required_tests_passed:sast-snyk-check-oci-ta
- test.no_skipped_tests:sast-snyk-check-oci-ta
- test.no_failed_tests:ecosystem-cert-preflight-checks
- test.no_erred_tests:ecosystem-cert-preflight-checks
- test.no_failed_informative_tests
- test.no_test_warnings:deprecated-image-check
- trusted_task.current
- rpm_repos.ids_known
- buildah_build_task.privileged_nested_param
- schedule.weekday_restriction
- cve.cve_blockers
- schedule.date_restriction
include:
- '@redhat'
data:
- github.com/release-engineering/rhtap-ec-policy//data
- oci::quay.io/konflux-ci/tekton-catalog/data-acceptable-bundles:latest
- oci::quay.io/konflux-ci/konflux-vanguard/data-acceptable-bundles:latest
- oci::quay.io/konflux-ci/integration-service-catalog/data-acceptable-bundles:latest
policy:
- oci::quay.io/enterprise-contract/ec-release-policy:konflux
ruleData:
allowed_registry_prefixes:
- "quay.io/hummingbird-ci/"
YAML
Then validate:
ec validate image \
--image "quay.io/redhat-user-workloads/hummingbird-tenant/<component>@sha256:<digest>" \
--policy /tmp/policy.yaml \
--public-key ci/key.pub \
--ignore-rekor \
--strict=false \
--show-successes \
--output text
Options
| Flag | Purpose |
|---|---|
--ignore-rekor |
Required — Konflux does not use a Rekor transparency log |
--strict=false |
Return exit code 0 even on failures (useful for investigation) |
--show-successes |
Include passing checks in output (off by default) |
--output json |
Machine-readable output (also: text, yaml, summary) |
Running Label Checks Only
The label checks (labels.required_labels, labels.optional_labels) are the only Conforma
rules that can be evaluated without signatures or attestations, because they only read OCI image
metadata. This lighter-weight approach does not require the signing key and works against any
registry image, including published (non-Konflux-built) images.
Setup
Fetch the policy rego files and upstream rule data (one-time):
ec fetch policy \
--source oci::quay.io/enterprise-contract/ec-release-policy:konflux \
--dest /tmp/ec-policy
git clone --depth=1 \
https://github.com/release-engineering/rhtap-ec-policy.git \
/tmp/rhtap-ec-policy
Running Checks
ec opa eval \
--data /tmp/ec-policy/policy/*/policy \
--data /tmp/rhtap-ec-policy/data/rule_data.yml \
--input <(echo '{"image":{"ref":"quay.io/hummingbird-rawhide/caddy:latest"}}') \
'{"deny": data.labels.deny, "warn": data.labels.warn}' \
--format pretty
This works against any image accessible from a container registry — both Konflux-built and
published images. It uses the ec CLI’s embedded OPA engine with ec.oci.* built-in functions
to read image config directly from the registry.
Note: The
--dataflag (not--bundle) must be used for the rego files. Loading the policy as a bundle creates an OPA root conflict that prevents the rule data from being resolved. Theec.oci.*built-in functions are only available inec opa, not in standaloneopaorconftest.
Interpreting Results
For ec validate image:
- Successes indicate passing policy rules (visible with
--show-successes). - Violations are failures that block release. Each includes a
code(rule name) andmsg. - Warnings are non-blocking advisories.
For ec opa eval (label checks only):
denyentries are failures. Each includescode,msg, and optionallyeffective_on(a date when the rule becomes enforced — future dates mean the rule is not yet active).warnentries are non-blocking warnings for optional labels.- Empty arrays mean the image passes all label checks.
Updating Cached Data
The fetched policy and rule data are point-in-time snapshots. To pick up upstream changes:
# Re-fetch the policy rego files (for ec opa eval)
rm -rf /tmp/ec-policy
ec fetch policy \
--source oci::quay.io/enterprise-contract/ec-release-policy:konflux \
--dest /tmp/ec-policy
# Update the rule data (for ec opa eval)
git -C /tmp/rhtap-ec-policy pull
The ec validate image command fetches policy and data bundles on each run based on the
policy.yaml references, so no manual update is needed.
3.4 - Upgrading Docsy
Overview
Both Hummingbird documentation sites use Docsy as their Hugo theme, installed as a Hugo module. Docsy releases roughly quarterly. This runbook covers how to upgrade to a new Docsy version.
The public docs (documentation) own all layout overrides. The
internal docs (infrastructure/internal-docs/) only have
build scaffolding and pick up layouts from the public docs via Hugo module
import.
Layout override inventory
We override four Docsy templates: three to inject the companion page banner and
sidebar lock icons, and one to add llms-full.txt cross-linking. Each override
is a copy of the upstream template with a small patch applied.
layouts/docs/_td-content.html
Overrides Docsy’s shared layouts/_td-content.html for docs pages only.
Patch: One line added after the description lead, before <header>:
{{ partial "companion.html" . }}
layouts/docs/list.html
Overrides Docsy’s layouts/docs/list.html for docs section pages.
Patch: Same one line added after the description lead, before <header>:
{{ partial "companion.html" . }}
layouts/partials/sidebar-tree.html
Overrides Docsy’s layouts/_partials/sidebar-tree.html for the navigation
sidebar.
Patch: One snippet inserted in two places (foldable and non-foldable
branches), after the {{ $s.LinkTitle }} </span> and before </a>:
{{- if isset $s.Params "companion" }}{{ if not $s.Params.companion }} <i class="fa-solid fa-lock companion-icon internal-only" title="Internal only"></i>{{ end }}{{ end -}}
layouts/all.md
Overrides Docsy’s layouts/all.md (the catch-all markdown output template).
Patch: One line changed in the LLMS index section to add a link to
llms-full.txt alongside the existing llms.txt link.
Other overrides (no forward-porting needed)
These files are fully custom and don’t track an upstream template:
| File | Purpose |
|---|---|
layouts/shortcodes/include.html |
Custom file-include shortcode |
layouts/_default/_markup/render-link.html |
Link rewriting for hummingbird URLs and /l/ aliases |
layouts/_default/_markup/render-heading.html |
Delegates to Docsy’s td/render-heading.html |
layouts/partials/companion.html |
Environment-aware companion page banner |
layouts/partials/breadcrumb.html |
Custom breadcrumb with stable-URL display |
layouts/partials/footer.html |
Fully custom 3-column footer |
layouts/home.html |
Custom home page with cover image and feature cards |
layouts/index.llms.txt |
Custom recursive llms.txt with full doc tree |
layouts/index.llms-full.txt |
Full content dump for LLM consumption |
layouts/partials/llms-doc-tree.txt |
Recursive tree-walking partial for llms.txt |
layouts/partials/llms-full-doc-tree.txt |
Recursive content partial for llms-full.txt |
Upgrade procedure
1. Read the release notes
Check the Docsy changelog and the release upgrade guide for the target version. Note any breaking changes that affect templates we override.
2. Update the Hugo module
From the documentation repo worktree:
# Update to new version (replace vX.Y.Z with the target version)
podman run --pull=newer -it --rm -u 0 \
-v $(pwd):/src:z -w /src \
-e RUNNING_IN_CONTAINER=1 -e HUGO_CACHEDIR=/src/.hugo_cache \
quay.io/hummingbird-ci/gitlab-ci:latest sh -c \
"hugo mod get github.com/google/docsy/theme@vX.Y.Z && hugo mod tidy"
3. Update npm dependencies
Docsy v0.16.0+ sources Bootstrap and Font Awesome from npm:
podman run --pull=never -it --rm -u 0 \
-v $(pwd):/src:z -w /src \
-e RUNNING_IN_CONTAINER=1 -e HUGO_CACHEDIR=/src/.hugo_cache \
quay.io/hummingbird-ci/gitlab-ci:latest sh -c \
"hugo mod npm pack && npm install --no-package-lock"
4. Forward-port layout overrides
For each of the four tracked overrides:
-
Find the new upstream template in the Hugo module cache:
find .hugo_cache -path "*docsy/theme@vX.Y.Z*" \ -name "_td-content.html" -o \ -name "sidebar-tree.html" -o \ -name "all.md" -o \ \( -name "list.html" -path "*docs*" \) -
Compare the new upstream against our current override to see what changed upstream.
-
Start from the new upstream template and re-apply the patch documented in the inventory above.
-
For
docs/list.html, keep all upstream hooks (especially{{ .Render "_td-content-after-header" -}}).
5. Verify dark mode and code highlighting
The site uses Docsy’s light/dark mode toggle and dark-aware code syntax highlighting. After upgrading, confirm these config and SCSS settings are still in place:
config.yaml:params.ui.showLightDarkModeMenu: trueconfig.yaml:markup.highlight.noClasses: false(required for CSS-based Chroma themes to work)assets/scss/_styles_project.scss:@import 'td/code-dark';(loads light/dark Chroma themes:friendlyfor light,nativefor dark)
If a Docsy upgrade changes the dark mode mechanism, check the Look and Feel docs for updated instructions.
6. Build and validate
make build # or: make build-host (if Hugo/Go are available locally)
The build should complete with no errors and no deprecation warnings.
Validation checklist
After upgrading, verify:
- Build passes without errors or deprecation warnings
- Companion banners render on docs pages with
companion: truefront matter - Lock icons appear in the internal site sidebar for internal-only pages
- Home page cover image and search render correctly
- Light/dark mode toggle works and all page elements adapt (navbar, sidebar, footer, code blocks)
- Code blocks have dark background with readable syntax colors in dark mode
- Agent-support outputs generate, if enabled (
.mdURL variants,/llms.txt,/llms-full.txt) - Stable URL aliases (
/l/...) still resolve
For the internal site, also build infrastructure/internal-docs/ after
updating its config/internal/config.yaml and go.mod to the same Docsy
version.
Updating the internal site
The internal site only needs config and module changes (no layout work):
-
Update
internal-docs/config/internal/config.yaml: change thetheme:list to match the public site. -
Update the Hugo module and rebuild:
cd internal-docs make container # Inside container: hugo mod get github.com/google/docsy/theme@vX.Y.Z && hugo mod tidy exit make setup # npm dependencies make build
Rollback
To revert to a previous Docsy version:
podman run --pull=never -it --rm -u 0 \
-v $(pwd):/src:z -w /src \
-e RUNNING_IN_CONTAINER=1 -e HUGO_CACHEDIR=/src/.hugo_cache \
quay.io/hummingbird-ci/gitlab-ci:latest sh -c \
"hugo mod get github.com/google/docsy/theme@vPREVIOUS && hugo mod tidy"
Then restore the layout overrides from the previous commit (git checkout HEAD~1 -- layouts/).
3.5 - Version Constraints for Multi-Version Images
Overview
Version constraints prevent unwanted major or minor version updates for multi-version
container images (e.g., python-3-11, nodejs-20). Constraints are defined in
properties.yml and validated during make check by comparing VERSION files against
defined patterns.
Adding Version Constraints
Edit the image’s properties.yml to add a version constraint:
---
main_package: python3.11
version_constraints: '3.11.*' # Allow 3.11.X, block 3.12+ and 4.X
The constraint applies to all distros for the image:
---
main_package: dotnet-sdk-8.0
version_constraints: '8.*' # Applies to rawhide, hummingbird, and any other distros
Constraint Patterns
Constraints use glob-style patterns:
3.11.*- Match any 3.11.X version (recommended for version families)20.*- Match any 20.X version8.*- Match any 8.X version1.25.*- Match any 1.25.X version (for Go-style 3-part versions)
The pattern matches against the version from the VERSION file. For example, if the
VERSION file contains 3.11.14, the constraint 3.11.* will match.
How Validation Works
- Lockfile Generation: When
make allruns, lockfiles are generated with resolved package versions - VERSION File Creation:
generate_jinja2.pycreates VERSION files from resolved package versions - Constraint Checking: When
make checkruns,check_version_constraints.pyvalidates VERSION files against constraints - Failure: If a violation is detected, the build fails with a detailed error message
- Renovate Integration: Renovate MRs that violate constraints fail CI and are blocked from automerge
Handling Constraint Violations
When a build fails due to a version constraint violation, you have several options:
Option 1: Accept the new version (create new image)
If the new major/minor version is acceptable:
-
Create a new image directory (e.g.,
images/python-3-14/) -
Copy properties and configuration from the old version
-
Update the constraint in the new directory:
version_constraints: '3.14.*' -
Update
repositoryfield to group versions together -
Consider adding
latesttag to the new version if appropriate
Option 2: Block the version update (exclude package)
If the new version should be blocked entirely:
- Add the package to
yum-repos/*.repoexcludepkgs - Follow the procedure in Excluding Packages from Images
- Renovate will skip the excluded version in future updates
- When the issue is resolved, remove the exclusion
Option 3: Update the constraint (allow version bump)
If the version bump is acceptable for the existing image:
-
Update the constraint in
properties.yml:version_constraints: '3.12.*' # Allow 3.12.X now -
Consider if this changes the image’s purpose (major version change)
-
Update image documentation and tags accordingly
-
Consider renaming the image directory to reflect the new version
Examples
Python Multi-Version Images
# images/python-3-11/properties.yml
distros:
- hummingbird
main_package: python3.11
repository: python
version_constraints: '3.11.*'
# images/python-3-13/properties.yml
distros:
- hummingbird
main_package: python3.13
repository: python
version_constraints: '3.13.*'
Node.js Multi-Version Images
# images/nodejs-20/properties.yml
main_package: nodejs20
repository: nodejs
version_constraints: '20.*'
# images/nodejs-24/properties.yml
main_package: nodejs24
repository: nodejs
version_constraints: '24.*'
.NET SDK Multi-Version Images
# images/dotnet-sdk-8-0/properties.yml
main_package: dotnet-sdk-8.0
repository: dotnet-sdk
version_constraints: '8.*'
.NET Runtime Multi-Version Images
# images/dotnet-runtime-8-0/properties.yml
main_package: dotnet-runtime-8.0
repository: dotnet-runtime
version_constraints: '8.*'
ASP.NET Runtime Multi-Version Images
# images/aspnet-runtime-8-0/properties.yml
main_package: aspnetcore-runtime-8.0
repository: aspnet-runtime
version_constraints: '8.*'
OpenJDK Multi-Version Images
# images/openjdk-21/properties.yml
main_package: java-21-openjdk-headless
repository: openjdk
version_constraints: '21.*'
Go Multi-Version Images (3-part versions)
# images/go-1-25/properties.yml
main_package: golang1.25
repository: go
distros:
- hummingbird
version_constraints: '1.25.*' # Matches 1.25.0, 1.25.3, etc.
Troubleshooting
Constraint not working
- Verify package name matches exactly (case-sensitive)
- Ensure pattern syntax is correct (use
*wildcard) - Run
make allto regenerate VERSION files beforemake check
False positive violations
- Check for typos in the constraint pattern
- Verify the pattern matches the version format (e.g.,
3.11.*not3.11*) - Ensure you’re using glob patterns, not regex
Package renamed in repositories
If a package is renamed (e.g., python3.11 → python311):
- Update
main_packagefield - Update constraint with new package name (not needed - constraints check VERSION files)
- Regenerate lockfiles and VERSION files with
make all
Implementation Details
The version constraint system consists of:
- properties.yml: Optional
version_constraintsfield with a glob pattern string - ci/check_version_constraints.py: Python script that validates VERSION files
- Makefile integration: Script runs as part of
make check - CI integration: Failures block merge, including Renovate automerge
The validation process:
- Reads
properties.ymlfor each image - If
version_constraintsdefined, finds all VERSION files across all distros - Applies the same constraint pattern to all distros for the image
- Compares VERSION file content against glob pattern using Python’s
fnmatch - Reports violations with file path, version, constraint, and properties file location
- Exits with error code 1 if any violations found
See Also
- Image Configuration Reference - Complete
properties.ymlfield reference - Excluding Packages from Images - How to exclude problematic package versions
- Switching Fedora Streams - How to pin distro versions
3.6 - Retrying Konflux Checks
When Konflux CI checks fail on merge requests, you can automatically retrigger them using the provided retrigger script.
Prerequisites
- GITLAB_TOKEN: GitLab API access token with
apiscope and Maintainer access
Usage
For automated retriggers of failed Konflux pipeline runs:
# Retrigger only failed pipeline runs and wait for them to start
make retrigger-failed-checks ARGS="https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/123"
# Preview what would be retriggered without posting comments
make retrigger-failed-checks ARGS="--dry-run https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/123"
How It Works
The retrigger script will:
- Extract failed pipeline runs from the commit statuses on the merge request
- Generate
/retest pipeline-namecommands for each failed run - Post the retest commands as merge request comments (unless
--dry-run) - Wait for the pipeline runs to start (unless
--dry-run)
Manual Retrigger Commands
You can also manually retrigger specific checks using slash commands in merge request comments:
/retest gitlab-ci--default--main-on-pull-request
To retrigger a specific pipeline run as listed in .tekton/images-on-pull-request.yaml.
See the Tekton Pipelines as Code documentation for more details on available commands.
Examples
# Set your GitLab token
export GITLAB_TOKEN=your_token_here
# Retrigger failed checks for a specific merge request
make retrigger-failed-checks ARGS="https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/456"
# Dry run to see what would be retriggered
make retrigger-failed-checks ARGS="--dry-run https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/456"
The script will output which pipeline runs are being retriggered and wait for them to start before completing.
Related Operations
- Managing Konflux Comments - Clean up excessive comments on merge requests
3.7 - Switching Fedora Streams
AI Agent Note: When asked to switch Fedora streams, follow the workflow below. The repo file naming convention determines the GPG key automatically.
Overview
The rawhide distro can be pinned to a specific Fedora release branch instead
of following the rolling Rawhide stream. This provides stability during the
period when Rawhide moves to N+1 but the current stable release is still N-1.
For example, when Fedora 44 branches from Rawhide, the rawhide distro can be
pinned to Fedora 44 packages while Rawhide itself starts tracking Fedora 45.
Repo ID naming convention
All Fedora repo files use a fedora-<stream> naming convention for their
repo IDs, where <stream> matches the filename stem:
fedora-43.repouses[fedora-43],[fedora-43-source],[fedora-43-updates], etc.fedora-44.repouses[fedora-44],[fedora-44-source], etc.fedora-rawhide.repouses[fedora-rawhide],[fedora-rawhide-source], etc.
This convention ensures:
- No DNF repo ID conflicts when multiple repo files coexist in
yum-repos/ - Predictable, greppable repo IDs in lockfiles
- The GPG key name can be derived automatically from the filename
Workflow
1. Create or update the repo file
Create a new file yum-repos/fedora-<version>.repo (e.g., fedora-45.repo).
The repo file must follow these conventions:
- Repo IDs match the filename stem: Use
[fedora-45],[fedora-45-debuginfo], and[fedora-45-source]as section names - Version-specific URLs: Point to the correct Fedora version’s repository
(e.g.,
/development/45/or/releases/45/) - Version-specific GPG key: Reference the correct signing key
(e.g.,
RPM-GPG-KEY-fedora-45-$basearch)
Use an existing repo file (e.g., yum-repos/fedora-44.repo)
as a template.
Example for a development branch:
[fedora-45]
name=Fedora 45 - Developmental packages for the next Fedora release
baseurl=https://koji-s3-cache.hummingbird-project.io/download-ib01.fedoraproject.org/pub/fedora/linux/development/45/Everything/$basearch/os/
enabled=1
countme=1
metadata_expire=6h
repo_gpgcheck=0
type=rpm
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-45-$basearch
skip_if_unavailable=False
2. Update variables.yml
Edit images/variables.yml and change the
default_variant_repos.rawhide entry to reference the new repo file:
default_variant_repos:
rawhide:
- fedora-45.repo
3. Regenerate all derived files
Run regeneration to update rpms.in.yaml files, Containerfiles, and lockfiles:
make -j 16 FORCE=true
4. Verify GPG key derivation
The Jinja templates in macros/setup_newroot.yml.j2 automatically derive the
GPG key name from the repo filename:
fedora-45.repobecomesRPM-GPG-KEY-fedora-45-primaryfedora-44.repobecomesRPM-GPG-KEY-fedora-44-primaryfedora-rawhide.repobecomesRPM-GPG-KEY-fedora-rawhide-primary
No template changes are needed when switching versions. Verify by checking the
generated Containerfile for the builder image contains the correct key:
grep rpmkeys images/hummingbird-builder/rawhide/default/Containerfile
Switching back to rolling Rawhide
To switch back to the rolling Rawhide stream, update variables.yml to
reference fedora-rawhide.repo:
default_variant_repos:
rawhide:
- fedora-rawhide.repo
Then regenerate all derived files with make -j 16 FORCE=true.
3.8 - Disabling Rawhide for Images
AI Agent Note: When asked to disable Rawhide for images, follow the workflow below. Run
make allafter editing properties to clean up stale directories automatically.
Overview
When Rawhide is unstable (mass rebuild, broken critical packages, Fedora branching), Rawhide variants can be temporarily disabled for specific images or globally. This stops building and testing Rawhide variants while keeping Hummingbird variants unaffected.
Disabling Rawhide for a single image
1. Edit properties.yml
Add an explicit distros field to images/<name>/properties.yml that
excludes rawhide:
distros:
- hummingbird
This overrides the default distros (which include rawhide) for this image
only.
2. Regenerate
make all
The stale Rawhide directories are removed automatically. Generated resources
(.gitattributes, Konflux templates, Tekton pipelines) are updated to
exclude the Rawhide variants.
3. Commit and merge
Commit the properties change together with the removed generated files.
Disabling Rawhide for all images
1. Edit variables.yml
Remove rawhide from default_distros in
images/variables.yml:
default_distros:
- hummingbird
2. Edit per-image overrides
Images that explicitly list distros including rawhide in their own
properties.yml are not affected by the variables.yml change. Edit those
files to remove rawhide as well.
3. Regenerate
make all
4. Commit and merge
Commit all properties changes together with the removed generated files.
Re-enabling Rawhide
Reverse the properties change (add rawhide back to default_distros or
remove the per-image distros override), then regenerate:
make all
The Rawhide directories no longer exist, so Make creates all targets from scratch.
Verification
After running make all:
git statusshows Rawhide directories as deleted (when disabling) or new files (when re-enabling).gitattributesno longer lists Rawhide paths for affected imageskonflux-templates/rendered.ymland.tekton/pipelines no longer reference Rawhide variants for affected images
See Also
- Image Configuration Reference -
distrosfield documentation - Switching Fedora Streams - Pin Rawhide to a specific Fedora release branch
3.9 - MR Artifact Promotion
Overview
An RPM package releases the image built and tested in its merge request (MR) by default. Merging the MR starts a promotion PipelineRun, not another RPM build. The promoted image keeps the MR source revision and digest; the merge commit records why it was released.
The promotion flow, from the merge-commit parents through MR Snapshot lookup and the push-style Component Snapshot to the automatic Release, is described in MR artifact promotion. On the RPM side, the MR build produces a digest-pinned OCI image containing unsigned RPMs, required Testing Farm tests gate the merge, and Release Service adds duplicate-RPM filtering and Pulp publication to the managed Conforma, signing, and publication checks.
Prerequisites
- The package uses two-parent merge commits on
main, and its MR build and required tests passed. - The Component’s build ServiceAccount can push source Trusted Artifacts to its Quay image repository
and use the pipeline SCC. The
hummingbird-promotionServiceAccount can read build evidence and create Snapshots; it must be able to read the source Trusted Artifact. - The Hummingbird tools image contains both promotion commands and is pinned by digest. The Hummingbird Tools Task reference describes their inputs and validation.
- The MR Snapshot, linked PipelineRun, and image remain available through merge and retry. KubeArchive can supply removed Konflux objects but cannot restore an expired image; check the actual Quay expiry for the selected digest and its tags and do not assume a fixed retention period.
Choose the push pipeline
Imported packages use promotion by default. Set push_pipeline: build in
ci/package-overrides.yaml only for a documented exception, then run make generate. The generator
places each package’s push-triggered PipelineRun in .tekton/rpms-on-promotion.yaml or
.tekton/rpms-on-push.yaml accordingly. Every package remains in trigger-manifest-push.json (see
trigger manifests documentation) so
its pushes reach Pipelines as Code. Remove the build override to restore promotion.
Review the generated YAML and confirm the Hummingbird tools image digest is current. Keep the MR
build in .tekton/rpms-on-pull-request.yaml. A package merge should start one promotion run and no
push-build run.
Verify a promotion
Find these resources in the tenant’s Konflux UI or with oc. Follow their names and links rather
than treating the promotion run as a build:
oc -n hummingbird-tenant get pipelineruns -l appstudio.openshift.io/component=syft-main
oc -n hummingbird-tenant get snapshots -l appstudio.openshift.io/component=syft-main
oc -n hummingbird-tenant get releases -o custom-columns=NAME:.metadata.name,SNAPSHOT:.spec.snapshot
Select the promotion PipelineRun for merge commit M; its COMPONENT_SNAPSHOT result names the
created Snapshot. Its annotations name the tested MR Snapshot and build PipelineRun. Find the
Release whose spec.snapshot matches, then follow its managed PipelineRun.
| Resource | Check |
|---|---|
| MR build PipelineRun | type=build, MR event, successful build result, source revision B, and digest D. |
| Tested MR Snapshot | The package’s Component entry has source B and image D; required MR tests passed. |
| Promotion PipelineRun | type=promotion, merge revision M, three Tasks (clone-repository, find-tested-mr-artifact, create-promotion-component), and a COMPONENT_SNAPSHOT result. |
| Promotion Component Snapshot | One Component entry with the same source B and image D; type=component, push event and Component labels; annotations link M and the source evidence. Its start-time annotation comes from the promotion PipelineRun. |
| Automatic Release | spec.snapshot names the promotion Snapshot. Check its conditions and managed PipelineRun; the promotion Task did not create it. |
| Managed release | Confirm Conforma and Pulp Tasks succeeded. Compare signed RPM SHA256 values in the Release’s status.artifacts.rpmfiles with matching Pulp package records. Signing changes RPM bytes, so these hashes need not equal the unsigned RPM hashes inside image D. |
The Component Snapshot normally reports that no required IntegrationTestScenarios were found and testing was skipped. The promotion Task fails before Snapshot creation if a configured scenario could retest it. Verify that the MR image digest and the Component Snapshot digest are identical; this proves promotion reused the tested image. Look up the merge SHA in Konflux PipelineRuns to confirm that no push build ran for this Component.
The clone Task runs under the Component build ServiceAccount and stores the source Trusted Artifact
in the Component’s Quay repository. Lookup and Snapshot creation run under
hummingbird-promotion.
Recover a failed run
| Failure | Action |
|---|---|
| Clone cannot push or restore the source Trusted Artifact | Check the Component build ServiceAccount, its Quay access, and the hummingbird-promotion read path. The clone Task needs the pipeline SCC. |
| Lookup cannot find passed MR evidence | Check the MR Snapshot, its build PipelineRun, required test result, and KubeArchive access. If the evidence cannot be recovered, build and test again before release. |
Package source changed on main after M |
Stop the stale promotion. Use a fresh MR build and tests for the newer package inputs. |
| Tested image expired or cannot be pulled | Do not create a Snapshot with a missing image. Rebuild and run the required tests before release. |
| Snapshot creation hits quota or finds a conflict | Check the tenant quota or the existing deterministic Snapshot. A retry reuses only a matching live Snapshot; it does not reuse an archived or conflicting one. |
| Promotion Task or PipelineRun needs a fix | Start a new promotion PipelineRun for the original merge SHA using the corrected manifest and Task pins. An old PipelineRun keeps its old Task revisions, and a garbage-collected run cannot be rerun. Do not bump the RPM Release solely to retry promotion. |
| Release fails after the Component Snapshot exists | Recover the failed Release through the Release Service workflow using that Snapshot. Do not create another promotion Snapshot or rebuild RPMs to diagnose a release-service failure. |
Record the MR Snapshot, tested digest, promotion PipelineRun and Component Snapshot, automatic Release, managed PipelineRun, Conforma result, and Pulp RPM checksums in the work item. Keep a compact evidence attachment when Konflux resources may be garbage-collected. An automatic Release can fail and later succeed through a retry using the same Snapshot; link both outcomes.
Development
Run python3 ci/generate_resources.py all and pytest test/test_generate_resources.py. Review the
generated push, promotion, and MR PipelineRuns for the package.
3.10 - Red Hat Catalog Environment Promotion
Environments
| Environment | URL | Source | Deploy trigger |
|---|---|---|---|
| MR Preview | GitLab Pages (/mr-{IID}) |
MR branch | Auto on MR pipeline |
| Experimental | images.experimental.hummingbird-project.io/<branch-name> |
experiment/* branch |
Auto on push to experiment branch |
| Staging | images.staging.hummingbird-project.io |
main |
Auto on merge to main |
| Production | images.redhat.com |
main |
Auto on merge to main |
Disambiguation: The infra experimental host (
images.experimental.hummingbird-project.io,experiment/*branches) is an off-main deploy sandbox. The former in-app/experimentalproduct route was removed in HUM-2171 — use/apiinstead. See Archived surfaces.
Promotion Paths
Routine: staging to production
- MR merged to
main deploy_redhat_catalog_staginganddeploy_redhat_catalogrun automatically in parallel (both trigger on push tomainwhen catalog files change)- UAT sign-off on staging — product, design, and engineering release approvers complete the UAT checklist and file a record (UAT program runbook)
Exploration: experimental to staging to production
- Create branch
experiment/<name>frommain(must use exactlyexperiment/, notexperimental/or other prefixes — without it, no deploy triggers) - Push triggers
deploy_redhat_catalog_experimentalautomatically - Branch deploys to
images.experimental.hummingbird-project.io/<name>/ - Iterate on experimental host for days or weeks
- Light UAT (optional spot-check on experimental URL — see UAT program — experimental path)
- When ready: open MR from
experiment/<name>tomain - Normal MR review, merge, then routine promotion path (full UAT on staging)
Retiring an experiment
-
Delete the
experiment/<name>branch -
Remove the branch prefix from the experimental S3 bucket:
aws s3 rm s3://redhat-catalog-experimental-spa/<name>/ --recursive -
Invalidate the CloudFront cache:
aws cloudfront create-invalidation \ --distribution-id <EXPERIMENTAL_DISTRIBUTION_ID> \ --paths "/<name>/*"
Build Configuration Per Environment
The infrastructure pipeline sets these environment variables at build time:
| Variable | Experimental | Staging | Production |
|---|---|---|---|
ASSET_PATH |
/<branch-name>/ |
/ |
/ |
DPAL_USE_STAGING |
true |
true |
(unset, uses prod) |
TRUSTARC_DATA_DOMAIN |
images.experimental.hummingbird-project.io |
images.staging.hummingbird-project.io |
images.redhat.com |
CATALOG_API_BASE_URL |
(unset, uses app default) | (unset, uses app default) | (unset, uses prod) |
CATALOG_BASE_URL |
(unset, uses app default) | (unset, uses app default) | (unset, uses prod) |
HUMMINGBIRD_API_URL |
(unset, uses app default) | (unset, uses app default) | (unset, uses prod) |
JIRA_API_URL |
Jira API URL | Jira API URL | Jira API URL |
Infrastructure
SAM Stacks
| Stack | Template | ResourcePrefix | CatalogDomainName |
|---|---|---|---|
redhat-catalog-prod |
template.yaml |
redhat-catalog |
images.redhat.com |
redhat-catalog-staging |
template.yaml |
redhat-catalog-staging |
images.staging.hummingbird-project.io |
redhat-catalog-experimental |
template-experimental.yaml |
redhat-catalog-experimental |
images.experimental.hummingbird-project.io |
Staging uses the same template.yaml as production (single-site CloudFront distribution).
Experimental uses template-experimental.yaml which adds a CloudFront Function for
path-prefix SPA routing across multiple branches.
DNS
ALIAS records (images.staging/experimental.hummingbird-project.io → CloudFront) are
created automatically by post_deploy.sh after each deploy.
First-time bootstrap: The SAM template creates an ACM certificate with DNS
validation. The ACM validation CNAME must exist in the hummingbird-project.io
Route 53 zone before the first sam deploy, or CloudFormation will wait
30 minutes and roll back. To bootstrap:
- Deploy with
CatalogDomainName=""(uses CloudFront default domain, no cert) - Create the ACM validation CNAME in Route 53 (get the value from the ACM console
or
aws acm describe-certificate) - Redeploy with
CatalogDomainNameset to the real domain
After the first successful deploy, no manual DNS steps are needed.
CI/CD Variables
Set in GitLab project settings:
| Variable | Value |
|---|---|
REDHAT_CATALOG_STAGING_URL |
https://images.staging.hummingbird-project.io |
Pipeline Flow
experiment/* push:
redhat-catalog-experimental-check
-> deploy_redhat_catalog_experimental
MR (target main):
redhat-catalog (build + test)
-> redhat-catalog-browser-tests
-> redhat-catalog-visual-mock (Playwright mock API; MR vs merge-base diff)
-> pages (preview)
main push:
deploy_redhat_catalog_staging (auto)
deploy_redhat_catalog (auto)
Visual regression details: redhat-catalog testing guide — Visual regression and e2e/visual/README.md.
3.11 - Setting Up a Konflux Cluster
Setting Up the Cluster
- Set up the new cluster in the infrastructure repository, mirroring the existing cluster configuration
- In the Konflux UI, create a new Component for the Application with the same name as the git repository
- Verify the Pipeline as Code
Repositoryresource is created - Run two successive full pipelines in the infrastructure repository to deploy resources from konflux/rendered.yml and fix up owner references
- Delete the initial Component created from the UI
- Disable Konflux build status comments (requires elevated permissions):
kubectl patch \
--namespace hummingbird-tenant \
repository REPOSITORY_NAME \
--type merge \
--patch '{"spec":{"settings":{"gitlab":{"comment_strategy":"disable_all"}}}}'
Next Steps
- Retrying Konflux Checks - Retrigger failed CI pipeline runs
- Managing Konflux Comments - Clean up excessive comments
3.12 - Deleting RPMs from Hummingbird Repos
Overview
- This document should be used to remove RPMs from the Hummingbird repositories
Setup
- make sure you have followed the local development setup instructions in the infrastructure repository
- install pulp cli via dnf install pulp-cli
- setup the local pulp environment via:
cki_secret PULP_PUBLIC_RHEL_PRIMITIVES_CONFIG_FILE > ~/.config/pulp/cli.toml
# or
vault kv get -mount=apps -field cli.toml hummingbird/PULP_PUBLIC_RHEL_PRIMITIVES_CONFIG_FILE > ~/.config/pulp/cli.toml
Steps
- Note the names of the rpms to delete. In this example, we’ll use
systemd-stub - Obtain the relevant metadata required to delete.
- You need to execute this command once for each architecture.
pulp --domain public-hummingbird rpm content -t package list --name systemd-stub | jq -r '.[] | "\(.pulp_href)"'
or for prettier content,
pulp --domain public-hummingbird rpm content -t package list --name systemd-stub | jq -r '.[] | "\(.name) \(.version)-\(.release) \(.arch) \(.pulp_created) \(.pulp_href)"'
Example output from basic command:
/api/pulp/public-hummingbird/api/v3/content/rpm/packages/019b0e50-b89e-749b-a82b-405e00cdd1ed/
/api/pulp/public-hummingbird/api/v3/content/rpm/packages/019b0ded-4234-72b6-82e2-36803fc57a3b/
- Using the
hrefs, construct your delete commands:
pulp --domain public-hummingbird rpm repository content remove --repository aarch64 --package-href /api/pulp/public-hummingbird/api/v3/content/rpm/packages/019b0e50-b89e-749b-a82b-405e00cdd1ed/
pulp --domain public-hummingbird rpm repository content remove --repository aarch64 --package-href /api/pulp/public-hummingbird/api/v3/content/rpm/packages/019b0ded-4234-72b6-82e2-36803fc57a3b/
- Repeat for
x64_64andsourcerepositories - Verify deletion by navigating to https://console.redhat.com/api/pulp-content/public-hummingbird/
- Deletion is a serial operation that can be impacted by other automated jobs updating the repositories.
Related Operations
3.13 - Red Hat Catalog UAT Program
User Acceptance Testing (UAT) for the Red Hat Catalog SPA — human sign-off on staging before production promotion.
Epic: HUM-2179 · Story: HUM-2183 · Depends on: HUM-2060 (staging host + promotion gates)
Purpose
UAT complements automated validation (smoke, E2E, visual regression, security scans). It catches product, content, and UX issues that automation does not cover. UAT does not replace automated gates.
| Layer | Owner | Blocks prod? |
|---|---|---|
| Unit / lint / MR security | CI | MR merge |
| UAT sign-off | Product + design + eng | Human review before prod is live |
| Post-prod smoke | CI / on-call | Rollback decision |
Environments (disambiguation)
| Term | Meaning | URL pattern |
|---|---|---|
| Staging | Pre-prod validation target for UAT | https://images.staging.hummingbird-project.io |
| Experimental host | Infra sandbox for experiment/* branches |
https://images.experimental.hummingbird-project.io/<branch>/ |
Former /experimental route |
Removed in-app product route (HUM-2171) | Returns 404 — use /api instead. See archived-surfaces.md |
See Red Hat Catalog Environment Promotion for full promotion paths.
Promotion gates
Staging → production (full UAT)
Both deploy_redhat_catalog_staging and deploy_redhat_catalog trigger
automatically on merge to main (when catalog files change). UAT is a
human review layer that runs concurrently with the automated deploys —
if UAT reveals issues, the release owner must roll back production.
Required before a release is considered complete:
deploy_redhat_catalog_stagingsucceeded onmain- MR-blocking security jobs clean (or approved exceptions) — see security-scanning.md
- UAT sign-off recorded (checklist + Jira comment; git record in
redhat-catalog/docs/uat-records/)
Light UAT on experimental host
Experimental hosts are for iteration, not customer release. When opening an MR
from experiment/<name> to main, use a light UAT pass on the experimental
URL — not the full checklist:
| Check | Required? |
|---|---|
| Feature under test works on experimental host | Yes |
| No console errors on primary journey | Yes |
| Masthead/footer not broken | Yes |
| Full staging UAT checklist | No — runs on staging after merge |
| Named approver sign-off | Eng owner only (no prod gate yet) |
After merge, staging receives the full UAT before prod.
Named approvers (staging → prod)
Confirm names with the product owner during checklist review (HUM-2060 open question on approval owners). Until confirmed, use these roles:
| Role | Primary | Backup | Responsibility |
|---|---|---|---|
| Product | TBD | — | Catalog journeys, copy, feature completeness |
| Design | TBD | — | Visual polish, layout, theme, responsive UX |
| Engineering release | TBD | TBD | Pipeline green, staging commit matches intent, prod deploy authorization |
Any Block from product or design stops prod promotion until defects are fixed on staging and UAT re-run.
Runbook — request and run UAT
1. Request UAT (release owner)
When main pipeline completes staging deploy + automated gates:
- Create or reuse a Release ticket in Jira (label
redhat-catalog, link HUM-2179 epic). - Comment with pipeline URL, commit SHA, and summary of user-facing changes.
- Label ticket
uat-requested. - Notify approvers (#hummingbird or team channel — use team default).
2. Execute UAT (approvers / delegates)
- Open uat-checklist.md against staging.
- File results in
redhat-catalog/docs/uat-records/YYYY-MM-DD-<topic>.md. - Log defects as Jira bugs linked to the release ticket.
Expected turnaround: 2 business days from uat-requested for routine
releases; same-day for hotfixes when approvers are available.
3. Sign off or block
- Pass: All checklist items pass or have accepted exceptions; all three
approver rows signed; Jira comment posted (template below); label
uat-pass. - Fail: Label
uat-fail; roll back production if already deployed, fix defects on staging, and re-run UAT.
4. Record production promotion (engineering release)
- Verify Jira release ticket has
uat-passand linked sign-off record. - Production deploys automatically on merge to
main— confirmdeploy_redhat_catalogsucceeded in the GitLabmainpipeline. - Comment on release ticket with prod pipeline URL.
What blocks release
| Blocker | Who resolves |
|---|---|
| Staging smoke or visual CI failed | Engineering |
| Open Critical/High security finding without exception | Engineering + security |
| UAT checklist failure (product/design) | Engineering fixes → re-UAT |
| Missing approver sign-off | Approver |
| Staging commit ≠ intended release SHA | Release owner |
Jira sign-off template
Post on the release ticket when UAT completes:
h3. UAT sign-off — staging → prod
*Staging URL:* https://images.staging.hummingbird-project.io
*Commit:* {full SHA}
*Pipeline:* {GitLab pipeline URL}
*Checklist record:* redhat-catalog/docs/uat-records/{filename}.md
||Role||Name||Decision||
|Product|{name}|Approve / Block|
|Design|{name}|Approve / Block|
|Engineering release|{name}|Approve / Block|
*Result:* Approved for prod / Blocked
*Defects:* {HUM-xxx links or "none"}
*Tester:* {name}, {date}
Suggested Jira labels
| Label | Meaning |
|---|---|
uat-requested |
Staging ready; waiting for human validation |
uat-pass |
Full sign-off complete |
uat-fail |
Blocked; defects logged |
uat-light |
Experimental → staging spot-check only |
Optional custom field (if added in Jira project settings): UAT status
(Not started / In progress / Passed / Failed).
Release ticket template
Create a Jira Task or Release issue per prod promotion:
## Release summary
- **Target:** staging → production
- **Commit:**
- **User-facing changes:**
## Automated gates
- [ ] Staging deploy green
- [ ] Production deploy green (auto-triggered on merge)
- [ ] Security scans clean (or exceptions documented)
## UAT
- [ ] Checklist completed ([uat-checklist.md](link))
- [ ] Sign-off record filed ([uat-records/](link))
- [ ] Product approve
- [ ] Design approve
- [ ] Engineering release approve
## Prod verify
- [ ] deploy_redhat_catalog succeeded in main pipeline
- [ ] Post-deploy verification
Related documentation
Out of scope
- Automating UAT (Playwright covers regression; UAT stays human)
- Legal/compliance sign-off (separate security program)
3.14 - Updating Dist-git Packages
Packages are automatically updated from Fedora dist-git. Each update creates a separate MR that is automatically approved and merged when CI passes.
Packages configured for MR artifact promotion release the image built
and tested in that MR after it merges. The operating guide shows how to follow the promotion
PipelineRun, automatic Release, and Pulp publication, including retries. Other packages build again
on main.
Portable package filenames
Import, update, and sync escape Windows-invalid filename characters with ~ followed by their
hexadecimal character code. For example, fix:one.patch becomes fix~3aone.patch. Reserved device
names and trailing dots or spaces are escaped too. Safe filenames stay unchanged. If an escaped
name would collide with another filename, including on a case-insensitive filesystem,
sanitization fails before changing files.
Packaging specs, scripts, and configuration files use local names. Patch contents and source
archives stay unchanged. Download URLs keep their original locations. A tracked Source or Patch
declared by URL gets an RPM #/ suffix for its local filename. The sources file keeps Fedora
lookaside names.
Merges normalize both the old and new Fedora snapshots before applying local changes.
The transformation is deterministic and idempotent, so no filename map is stored in metadata.
Modification checks and ordinary diff compare sanitized upstream trees. Use
diff --raw to see the filename substitutions themselves. Sanitization does not change Release,
version, or modification status.
To migrate existing packages:
make dist-git ARGS='--dry-run sanitize-paths --all'
make dist-git ARGS='sanitize-paths patch'
# Or migrate every package:
make dist-git ARGS='sanitize-paths --all'
This command leaves changes in the working tree without committing. It includes tracked files and
new files that Git does not ignore, and skips downloaded lookaside archives. Review and commit the
renamed files and updated references together. For a clean Fedora package,
include its existing Upstream: SHA trailer in the package commit to preserve history validation.
make check runs dist_git.py check-paths on tracked paths throughout the repository. Stage the
migration before checking, so Git’s index contains the new names. The check also rejects Windows
reserved names and paths that differ only by case.
New clones can check out the corrected tree on Windows. Older commits still contain the original filenames, so checking out those commits requires a filesystem that supports them.
dist_git.py --dry-run policy
--dry-run preserves each command’s established behavior. It is not a universal
side-effect-free preview: several commands still modify package files or metadata. Place the
flag before the command, for example make dist-git ARGS='--dry-run update bash'.
Each command’s --help also describes its policy.
| Command | Behavior with --dry-run |
|---|---|
import |
Imports files, writes metadata, and regenerates resources, without staging or committing. |
update |
Updates package files, but does not save metadata, run source pipelines, commit, or write recovery state. Existing safety checks still apply. |
update --continue |
Checks for unresolved conflict markers and reports the intended commit, without source verification, spec finalization, or committing. |
sync |
Replaces package files, but does not save metadata, run source pipelines, commit, or write recovery state. With --mark, files are left unchanged only when the requested SHA already matches the recorded SHA and the branch is unchanged. If upstream has advanced or the branch changes, normal sync still replaces files and can discard local edits, even with --mark. |
rebuild |
Modifies spec releases without committing. |
rebuild-rev-deps |
Modifies reverse-dependency spec releases without committing. |
sanitize-paths |
Previews portable path changes without changing files. |
add-private-product |
Previews configuration and infrastructure changes without contacting Pulp, modifying files, or committing. |
mark-modified, set-upstream |
Still write metadata. These commands never commit, with or without --dry-run. |
rename, update-releases, list, diff, check-paths, ls-sources |
Reject the flag before executing the command. |
Compatibility decision: mark-modified and set-upstream continue accepting --dry-run
because existing callers use it for their metadata-only workflows. The flag does not turn
either command into a preview or suppress metadata writes. Other unsupported combinations,
including read-only commands where the flag would be ignored, are rejected explicitly.
A new side-effect-free preview mode is outside this policy.
Version-validation failures exit with code 1, both with and without --dry-run.
A normal update saves recovery state for correction and update --continue. A failed dry-run
does not create recovery state, so it cannot be resumed with --continue. Its package-file
edits remain available for inspection; correct the problem before starting another update.
Testing Locally
# Dry-run (check all packages, no MRs)
./ci/dist_git_update_multi_mr.sh --clone
# Dry-run (check first 5 packages, no MRs)
./ci/dist_git_update_multi_mr.sh --clone --max-packages=5
# Check only a specific package (for testing/debugging)
./ci/dist_git_update_multi_mr.sh --clone --only-package=libgcrypt
# Check only clean packages (skip modified/independent)
./ci/dist_git_update_multi_mr.sh --clone --clean-only
# Check only modified packages (skip clean/independent)
./ci/dist_git_update_multi_mr.sh --clone --modified-only
# Create up to 3 test MRs (checks all packages, stops after finding 3 updates)
export CHORE_MR_GITLAB_TOKEN="glpat-xxxxxxxxxxxxxxxxxxxx"
./ci/dist_git_update_multi_mr.sh --clone --max-updates=3 --create-mrs
# Check first 10 packages, create up to 3 MRs
export CHORE_MR_GITLAB_TOKEN="glpat-xxxxxxxxxxxxxxxxxxxx"
./ci/dist_git_update_multi_mr.sh --clone --max-packages=10 --max-updates=3 --create-mrs
# Create MRs only for clean packages (skip modified/independent)
export CHORE_MR_GITLAB_TOKEN="glpat-xxxxxxxxxxxxxxxxxxxx"
./ci/dist_git_update_multi_mr.sh --clone --clean-only --create-mrs
Environment Variables
CHORE_MR_GITLAB_TOKEN- GitLab token withapiandwrite_repositoryscopes (required for--create-mrsand resolved diagnostic-MR cleanup)CHORE_MR_APPROVAL_GITLAB_TOKEN- GitLab token withapiscope for auto-approving MRs (used by CI)GITLAB_REMOTE_URL- Target repo (default: https://gitlab.com/redhat/hummingbird/rpms.git)
Flags
--clone- Clone from GitLab to /tmp (safe for local testing, uses latest main branch)--max-packages=N- Check only the first N packages (limits input set)--max-updates=N- Stop after finding N updates (limits output MRs created)--create-mrs- Actually create MRs (requires token)--only-package=NAME- Check only the specified package (for testing/debugging specific packages)--clean-only- Skip packages withmodification_statusof ‘modified’ or ‘independent’, only process clean packages--modified-only- Skip packages withmodification_statusof ‘clean’ or ‘independent’, only process modified packages (mutually exclusive with--clean-only)
Using –clean-only
The --clean-only flag filters out packages marked as ‘modified’ or ‘independent’ before attempting
updates. This is useful for:
- Better failure detection - Exit code 1 indicates real update failures, not expected errors from modified/independent packages
- Cleaner output - No error messages for packages that can’t be auto-updated by design
- Efficient CI - Focus on packages that should update automatically
- Performance - Avoids invoking
dist_git.pyfor packages that will fail
Without --clean-only, the script attempts to update all packages. Modified/independent packages fail
with:
ERROR: Cannot auto-update <package>
Status: modified/independent
Reason: <reason>
Use 'sync' to force update or 'mark-modified --clean' to allow updates
These expected failures can mask genuine update issues. Using --clean-only prevents these false
failures.
Using –modified-only
The --modified-only flag filters to only process packages marked as ‘modified’, skipping clean and
independent packages. This is useful for checking the merge logic, as well as getting an overview of
current merge conflicts.
Auto-Merge and Auto-Approval
MRs created by this script are configured to:
- Auto-merge when pipeline succeeds (set via
merge_request.merge_when_pipeline_succeeds) - Auto-approve after 10 minutes via the
chore_mr_approvalCI job (gives Konflux time to post commit statuses)
Version-Constrained Updates
Packages with track_upstream set to a version prefix in their metadata are version-constrained.
This is used for versioned packages like golang1.26 that track a specific upstream version line.
How It Works
The track_upstream and release_monitoring_project_id metadata fields work together across two
systems:
dist_git.py update (dist-git sync from Fedora):
When track_upstream is a version prefix (e.g., "1.26"), the update command uses prefix matching:
1.26matches1.26,1.26.0,1.26.3(allowed)1.26does not match1.27.0,2.0(skipped)
Skipped packages log a warning:
WARNING: Skipping golang1.26: upstream version 1.27.0 doesn't match tracked version 1.26
check_upstream_versions.py (release-monitoring.org checks):
When release_monitoring_project_id is a string, Anitya is queried using that name instead of the
RPM package name. For example, golang1.26 with release_monitoring_project_id: "golang" queries
Anitya for golang. When it is an integer, the v2 API is queried directly by project ID. When
track_upstream is a version prefix, the list of upstream versions returned by Anitya is filtered
to only those matching the prefix. This means check_upstream_versions.py check will report
1.26.5 as the latest version for golang1.26 even if Anitya reports 1.27.0 as the latest
golang release.
When --update is used, the script updates the spec file, downloads new sources, and commits the
result. Packages that need custom update logic can provide a hooks file at
metadata/<package>.update-hooks.yaml to override the spec update, source download, or add a
post-update step. See Package Modification Tracking for details.
Setting Up Version Constraints
# Constrain golang1.26 to only receive 1.26.x updates
./ci/dist_git.py set-upstream golang1.26 --project-id golang --track-version 1.26
# Remove the version constraint
./ci/dist_git.py set-upstream golang1.26 --no-track-version
Behavior
dist_git.py: Version constraint is checked before pre-release and Koji build checksdist_git.py: Thesynccommand bypasses the version constraint (explicit force operation)check_upstream_versions.py:release_monitoring_project_iddetermines the Anitya lookup (int for project ID, string for name, absent for RPM name);track_upstreamfilters versions when set to a prefix- Batch operations continue processing other packages after skipping constrained ones
Pre-Release Version Filtering
By default, the update mechanism skips pre-release versions to prevent unstable packages from entering the repository automatically. Pre-release patterns include:
- Tilde notation:
5.3.0~rc1,2.0~beta1,1.0~alpha(RPM standard) - Suffix notation:
5.3.0-rc1,2.0.beta1,3.0-alpha,1.0.dev - Development markers:
1.5git20240101,2024.01.snapshot,1.0dev
Manual Override
To explicitly update to a pre-release version:
# Update single package to pre-release version
./ci/dist_git.py update --allow-prerelease package-name --skip-build-check
# Batch update allowing pre-releases
./ci/dist_git.py update --allow-prerelease --skip-build-check
Pre-release filtering behavior
- Pre-release detection occurs before Koji build checks (saves API calls)
- Skipped packages log a warning with the detected pattern
- Batch updates continue processing other packages
- The
synccommand bypasses this check (explicit force operation)
3.15 - Writing Documentation
Overview
New documentation is always welcome! This guide explains how to contribute to the Project Hummingbird documentation.
Documentation Sites
The Project Hummingbird documentation is open by default and available at two sites:
- Public documentation: https://hummingbird-project.io/
- Internal documentation: https://internal-documentation.hummingbird-project.io/
Documentation Repositories
Documentation content comes from six repositories:
-
Containers repository - Documentation focused on container images (cross-cutting layout):
documentation/contributing/- How to add and modify container imagesdocumentation/background/- Container image architecture and reference docsdocumentation/ci-scripts/- CI script documentationdocumentation/operating/- Operator runbooks for image maintenancedocumentation/using/- End-user guides (custom CA certificates, etc.)README.md- User guideCONTRIBUTING.md- Quickstart guide
-
RPMs repository - Documentation focused on RPM packages (cross-cutting layout):
documentation/background/- RPM pipeline architecture and Konflux deploymentdocumentation/operating/- Operator runbooks for package managementCONTRIBUTING.md- Contributing guide for RPM packages
-
Tools repository - Documentation for infrastructure tools and services (cross-cutting layout):
documentation/background/- Cross-cutting pages: architecture (message bus, data flow, artifact promotion, shared contracts) and team workflow guidesdocumentation/components/- One page per tool or service; components with design docs or DDRs are page bundles: a directory with_index.mdas the main page plus child pages likedesign.md/design-decisions.mddocumentation/operating/- Operator runbooks for tools and services
The
contributing/andusing/mounts are pre-provisioned for future use; the shared Contributing section is containers-scoped, so the team workflow guides live inbackground/. -
K8s test pipeline repository - Documentation for Kubernetes integration testing (per-component layout):
documentation/- Pipeline design, test format, and EaaS debugging
Section mounts are pre-provisioned for a future split into the standard subdirectories.
-
Public documentation repository - Site infrastructure and cross-cutting guides:
content/docs/using/- End-user guides for using container imagescontent/docs/operating/- Infrastructure operation guides (Konflux, GitLab, etc.)content/docs/background/- General architecture and concepts
-
Infrastructure repository - Internal-only documentation (cross-cutting layout):
documentation/operating/- Internal operational docs (AWS, DNS, GitLab, etc.)documentation/background/- Internal background docs (error budgets, etc.)documentation/presentations/- Team presentations and talks
How They Work Together
The repositories are layered using Hugo Modules:
- Public documentation imports specific directories from the containers, rpms, tools, and k8s-test-pipeline repositories
- Internal documentation is built from the infrastructure repository’s
internal-docs/subtree. It imports the public documentation (which transitively includes the imported repository documentation) and mounts the infrastructure repo’sdocumentation/directory as the Internal section
Hugo Module Mounts:
# In documentation/config.yaml
module:
imports:
- path: gitlab.com/redhat/hummingbird/containers
mounts:
- {source: 'README.md', target: content/snippets/readme.md}
- {source: 'CONTRIBUTING.md', target: content/snippets/contributing.md}
- {source: 'documentation/contributing', target: content/docs/contributing, files: "*.md"}
- {source: 'documentation/background', target: content/docs/background/containers, files: "*.md"}
- {source: 'documentation/ci-scripts', target: content/docs/background/containers/ci-scripts, files: "*.md"}
- {source: 'documentation/operating', target: content/docs/operating, files: "*.md"}
- {source: 'documentation/using', target: content/docs/using, files: "*.md"}
- path: gitlab.com/redhat/hummingbird/rpms
mounts:
- {source: 'CONTRIBUTING.md', target: content/snippets/rpms-contributing.md}
- {source: 'documentation/background', target: content/docs/background/rpms, files: "*.md"}
- {source: 'documentation/contributing', target: content/docs/contributing, files: "*.md"}
- {source: 'documentation/operating', target: content/docs/operating, files: "*.md"}
- {source: 'documentation/using', target: content/docs/using, files: "*.md"}
- path: gitlab.com/redhat/hummingbird/tools
mounts:
- {source: 'documentation/background', target: content/docs/background/tools, files: "*.md"}
# Per-component page bundles: multi-file components are branch
# bundles (_index.md + child pages); the recursive glob plus the
# shared target keeps each component's main-page URL flat.
- {source: 'documentation/components', target: content/docs/background/tools, files: ["*.md", "**/*.md"]}
- {source: 'documentation/contributing', target: content/docs/contributing, files: "*.md"}
- {source: 'documentation/operating', target: content/docs/operating, files: "*.md"}
- {source: 'documentation/using', target: content/docs/using, files: "*.md"}
- path: gitlab.com/redhat/hummingbird/pipelines/k8s-test-pipeline
mounts:
# Serves the current flat layout; the section mounts below are
# ready for whenever the repository adopts the standard subdirs.
- {source: 'documentation', target: content/docs/background/k8s-test-pipeline, files: "*.md"}
- {source: 'documentation/background', target: content/docs/background/k8s-test-pipeline, files: "*.md"}
- {source: 'documentation/contributing', target: content/docs/contributing, files: "*.md"}
- {source: 'documentation/operating', target: content/docs/operating, files: "*.md"}
- {source: 'documentation/using', target: content/docs/using, files: "*.md"}
For README.md and CONTRIBUTING.md, front matter is provided via wrapper files.
This creates a unified documentation site from multiple repositories without duplication.
Documentation Layout Patterns
Source repositories use one of two layout patterns depending on their content:
Cross-cutting layout (containers, rpms, tools): The repository documents cross-cutting concerns like contributing, operating, and background information. Documentation is organized into subdirectories matching the site sections:
documentation/background/- Explanation and reference materialdocumentation/contributing/- How to contributedocumentation/operating/- Operator runbooksdocumentation/using/- End-user guides
Each subdirectory is mounted into the matching site section, so operating docs from
different repositories appear together under Operating. Not every repository uses every
subdirectory: the tools repository keeps its team workflow guides in background/ (the
shared Contributing section is currently containers-scoped) and leaves contributing/
and using/ to the pre-provisioned mounts, unused for now.
The tools repository combines the cross-cutting sections with per-component page bundles
inside components/: one page per component, and components with design docs or DDRs
become bundles — a directory with _index.md as the main page plus child pages like
design.md or design-decisions.md. The recursive components mount targets the same
content/docs/background/tools path as the other background mounts, so a bundle’s main
page keeps a flat URL (/docs/background/tools/<component>/) while its child pages nest
one level below it (/docs/background/tools/<component>/design/).
Per-component layout (k8s-test-pipeline): The repository is a monorepo where each documentation page covers one tool or component. Documentation is a flat directory with one file per component:
documentation/<component-name>.md
The entire directory is mounted under background/<repo-name>/ on the site. Section
mounts for the standard subdirectories are pre-provisioned in the site configuration,
so the repository can adopt the cross-cutting layout without further site changes.
Getting Started
Quick Start
Clone, install dependencies, and start the documentation server:
git clone https://gitlab.com/redhat/hummingbird/documentation
cd documentation
make setup # Installs npm dependencies; only needed once per checkout
make serve # Available at http://localhost:1313/
For the internal documentation:
git clone https://gitlab.com/redhat/hummingbird/infrastructure
cd infrastructure/internal-docs
make serve # Available at http://localhost:1314/
Recommended Setup
For the best development experience, set up all repositories with direnv:
mkdir -p ~/git/hummingbird
cd ~/git/hummingbird
# Clone all repositories
git clone https://gitlab.com/redhat/hummingbird/containers
git clone https://gitlab.com/redhat/hummingbird/rpms
git clone https://gitlab.com/redhat/hummingbird/tools
git clone https://gitlab.com/redhat/hummingbird/pipelines/k8s-test-pipeline
git clone https://gitlab.com/redhat/hummingbird/documentation
git clone https://gitlab.com/redhat/hummingbird/infrastructure
# Set up environment variables
cat > .envrc << 'EOF'
export CONTAINERS_REPO_PATH=$(pwd)/containers
export RPMS_REPO_PATH=$(pwd)/rpms
export TOOLS_REPO_PATH=$(pwd)/tools
export K8S_TEST_PIPELINE_REPO_PATH=$(pwd)/k8s-test-pipeline
export DOCUMENTATION_REPO_PATH=$(pwd)/documentation
export INFRASTRUCTURE_REPO_PATH=$(pwd)/infrastructure
EOF
direnv allow
This automatically uses your local repositories for imported content, making changes immediately visible in the documentation preview.
Adding Documentation
Add new documentation as markdown files inside the appropriate directory under
content/. Every file should end in .md.
Preview your changes:
make serve # Starts local server with live reload
Check for problems:
make check # Runs linters and validation
After make setup, run the Docsy print-layout regression tests:
make test
These tests build a small site with Mermaid diagrams in section and child pages.
They check renderer inclusion in normal and combined print views, including nested
sections and no_print exclusions. After changing print rendering, also check a
diagram page and its “Print entire section” view in a browser, then export a PDF
and confirm that the diagrams remain visible.
Link Format
Documentation uses two types of internal links depending on the context:
Same-Directory Links and Cross-Section Links in the Documentation Repository
Use relative .md links for pages in the same directory, and for cross-section
links in the documentation repository:
[Testing Guide](testing-images.md)
[Adding Images](adding-images.md)
Cross-Section and Cross-Repository Links
Use full URLs with /l/ aliases for links across repositories or across
sections in all repositories:
[Image Pipeline][image-pipeline]
[Global Variables][global-vars]
[Retrying Checks][retrying-checks]
[image-pipeline]: https://hummingbird-project.io/l/image-pipeline
[global-vars]: https://hummingbird-project.io/l/global-variables-reference
[retrying-checks]: https://hummingbird-project.io/l/retrying-konflux-checks
Why full URLs?
- Work in standalone contexts (GitHub, text editors, when files are copied)
- Hugo’s link render hook automatically converts them to relative links in the rendered site
- Stable even when files are moved or reorganized
- Clear and explicit about which page is being referenced
URL Stability
Documentation URLs (for example, /docs/background/tools/page/) are not stable.
They change whenever content is reorganized, and a moved page’s previous URL returns a
404 by design.
Only /l/ aliases are stable and guaranteed to keep resolving. Use them for any
reference that must survive reorganization:
- Cross-repository links
- Cross-section links
- Merge request comments
- Scripts
- Bookmarks
Link Render Hook
The documentation site includes a custom link render hook
(layouts/_default/_markup/render-link.html) that:
- Detects
https://hummingbird-project.io/l/...URLs - Looks up the page with the matching alias
- Converts to a relative link in the rendered HTML
- Keeps external links as-is with
target="_blank"
This provides the best of both worlds: full URLs in markdown source files that work everywhere, and optimized relative links in the rendered site.
Stable Link Aliases
All cross-referenced pages should have stable /l/ aliases. These aliases serve two purposes:
- Incoming links from external sources (documentation, scripts, merge request comments)
- Cross-references within documentation (across sections or repositories)
Stability Policy
Only /l/ aliases are stable and guaranteed to keep resolving. Documentation URLs change
whenever content is reorganized, and a moved page’s previous URL returns a 404 by design.
Never add an alias for a page’s previous URL after a move — aliases are curated stable links, not a redirect backlog.
Adding an Alias
Add an alias in the page front matter:
---
title: "Konflux Integration"
aliases: [/l/konflux-integration]
---
Now you can link to https://hummingbird-project.io/l/konflux-integration and it
will continue working even if the page moves.
When to Add Aliases
Add /l/ aliases to pages that:
- Are referenced from other repositories (e.g., containers repo → docs repo)
- Are linked to from CI scripts or infrastructure
- Are shared in merge request comments or external documentation
- Are referenced across documentation sections
Naming Convention
- Use
/l/prefix for all stable links - Keep names short, memorable, and descriptive
- Use kebab-case (lowercase with hyphens)
- No nested paths (flat structure only)
Submitting Changes
- Fork the repository on GitLab
- Clone your fork locally
- Create a new branch for your changes
- Make your changes and test them with
make serve - Run
make checkto ensure there are no errors - Commit your changes with a descriptive commit message
- Push to your fork
- Create a merge request against the main repository
Source Repository Checks
Merge requests that change mounted documentation are checked in the source
repository’s own pipeline. The containers, rpms, tools, and
pipelines/k8s-test-pipeline repositories each have a check-afdocs job that
builds this site with the merge request’s working tree mounted in place of the
published module, then runs the same agent-friendly docs (afdocs) suite the
documentation repository runs on its own merge requests. The infrastructure
repository has the equivalent job for the internal documentation site
(check-afdocs-internal-docs), which also covers the internal documentation
mounted from that repository.
- The job runs automatically when a merge request changes a path the site mounts, and is available as an optional manual job otherwise.
- A failure means the rendered site would break. Fix it in the source repository — a broken link or oversized page is a content problem there, not a build problem here.
- Like a production build, the check resolves the other repositories'
documentation at their default branches, so a failure can also come from
another repository’s
main. Re-run the job once that repository is fixed.
# Run the same check locally from a documentation checkout:
make check-afdocs TOOLS_REPO_PATH=/path/to/tools
Documentation Structure
The public documentation is organized into several main sections:
- Using - Guides for users of Project Hummingbird container images
- Contributing - Guides for contributors to the project
- Operating - Guides for infrastructure operators
- Background - Technical background and architecture information
The internal documentation adds:
- Internal - Internal-only infrastructure documentation, organized into:
- Operating - AWS, DNS, GitLab, OpenShift, and other service runbooks
- Background - Error budgets, SLIs/SLOs
- Presentations - Team talks and lightning talks
When adding new documentation, place it in the most appropriate section.
3.16 - Adding Independent Packages
Independent packages are packages that originate in the Hummingbird repository rather than being imported from Fedora dist-git.
Steps to Add an Independent Package
1. Create Package Directory
mkdir rpms/<package-name>
cd rpms/<package-name>
2. Add Package Files
Create the following files in the package directory:
<package-name>.spec- RPM spec filesources- SHA512 checksums of source tarballs- Source tarballs (
.tar.gz,.tar.bz2,.tar.xz) - Any patches or additional configuration files
Rust specs can opt in to isolated tests by requiring cargo-nextest when checks are enabled
and replacing %cargo_test with %cargo_nextest. Keep existing feature flags and exclusions;
the macro preserves RPM compiler flags and the rpm profile, then runs doctests separately.
Use NEXTEST_PROFILE=ci to select an upstream CI profile. For serial suites, retain
RUST_TEST_THREADS=1 and also set NEXTEST_TEST_THREADS=1. Publish the runner before its consumers.
3. Generate the sources File
The sources file must contain SHA512 checksums in BSD-style format:
sha512sum --tag oras-1.3.0.tar.gz oras-1.3.0-vendor.tar.bz2 > sources
This produces the correct format:
SHA512 (oras-1.3.0.tar.gz) = fb871c0577f621f7e1f56a54f249f96c659b29c771dad529f9a9e838379b2d56bca9cef03a90071915a568e28e266326989e67359e589a87908c4fc077b14689
SHA512 (oras-1.3.0-vendor.tar.bz2) = fa23324bf3910c3dc050eb3c0ebfef3224168489679cf1195d668c656c6c53beaac94b4786e45c2cf3782f9437b0d166da9c18e608a3b474b04957ff1dec5226
Important: Use sha512sum --tag to generate the BSD-style format. Do not use the default GNU
format.
4. Create Package Metadata
Create metadata/<package-name>.json:
{
"modification_status": "independent",
"upstream_repo": "https://github.com/example/project",
"version": "1.3.0"
}
Fields:
modification_status: Must be"independent"for packages not imported from Fedora (see Package Metadata Fields)upstream_repo: Canonical upstream git repository URL (required — CI enforces this). If no upstream repo exists, usehttps://src.fedoraproject.org/rpms/<name>as a fallback.version: Package version (must match spec file)
5. Generate Konflux Resources
Generate the Konflux Component and ImageRepository resources:
make generate-host
Or directly:
python3 ci/generate_resources.py all
This updates:
konflux-templates/rendered.yml.tekton/pipeline files
6. Commit Changes
git add rpms/<package-name>/ metadata/<package-name>.json konflux-templates/ .tekton/
git commit -m "Add <package-name>-<version>-<release>
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>"
Note: Source tarballs (.tar.gz, .tar.bz2, .tar.xz) are automatically ignored by
.gitignore and should be uploaded to the lookaside cache instead.
Example: Adding oras
# 1. Create directory
mkdir rpms/oras
cd rpms/oras
# 2. Add spec file and sources
# ... create oras.spec ...
# 3. Generate sources file
sha512sum --tag oras-1.3.0.tar.gz oras-1.3.0-vendor.tar.bz2 > sources
# 4. Create metadata
cat > ../../metadata/oras.json << 'EOF'
{
"modification_status": "independent",
"upstream_repo": "https://github.com/oras-project/oras",
"version": "1.3.0"
}
EOF
# 5. Generate Konflux resources
make generate-host
# 6. Commit
git add rpms/oras/ metadata/oras.json konflux-templates/ .tekton/
git commit -m "Add oras-1.3.0-1.hum1"
See Also
- Package Metadata Fields -
modification_statusandreleaseconfiguration - Package Modification Tracking - Marking packages as modified vs clean
- Rebuilding Packages - Rebuilding existing packages
- Updating Dist-git Packages - Importing/updating from Fedora
3.17 - Pulp Access
Overview
Pulp is used to host and distribute Hummingbird RPM repositories. Access to the Pulp API requires a service account with credentials stored in the vault.
Existing credentials
The public Pulp service account credentials are stored in the vault at
rhel-primitives/PULP_PUBLIC_RHEL_PRIMITIVES_CONFIG_FILE. Retrieve the cli.toml field
and save it locally (see ci/pulp-setup/README.md for usage).
Creating a new Pulp service account
When you need a dedicated service account (e.g., for private product repositories):
- Go to https://access.redhat.com/terms-based-registry/
- Log in with your
@redhat.comaccount - Click New Service Account
- Enter a username like
hummingbird-<qualifier>-pulp-bot - Enter a description like
bot account for hummingbird <qualifier> pulp operations - Click Create
- Capture the full username (in the format
<id>|<username>) and the token
Configure the CLI
Create a cli.toml with the new credentials:
[cli]
base_url = "https://packages.redhat.com"
api_root = "/api/pulp/"
username = "<id>|hummingbird-<qualifier>-pulp-bot"
password = "<token>"
verify_ssl = true
format = "json"
dry_run = false
timeout = 0
verbose = 0
domain = "<target-domain>"
You can either save this to ~/.config/pulp/cli.toml or pass it explicitly with
--config ./cli.toml when running scripts.
Store credentials in the vault
Store the credentials in the vault at apps/hummingbird with the key
HUMMINGBIRD_<QUALIFIER>_PULP_BOT_CONFIG_FILE (e.g.,
HUMMINGBIRD_PRIVATE_PULP_BOT_CONFIG_FILE). Create two subkeys:
cli.toml— the full config file contentsvalue— the token value
Related
- Lookaside Cache Access — AWS credentials for source tarball uploads
- Private RPM Repositories — setting up private per-product Pulp repos
- ci/pulp-setup/README.md — Pulp domain and repository creation script
3.18 - Measuring Documentation Quality
The documentation quality evaluation measures whether public documentation is complete enough for an LLM with web search to answer questions correctly. Low scores indicate documentation gaps; score improvements after content changes validate the fix.
For config schema, CLI options, and scoring details, see the Documentation Quality Evaluation reference in the tools repo. For the general evaluation framework, see Hummingbird Agent Evals.
How it works
The evaluation runs in two phases:
- Ask – multiple LLM configurations answer each golden question using web search against public documentation. Models never see internal sources; the test is whether public docs contain the information.
- Judge – a separate LLM scores each response against expected facts and coherence criteria defined in the golden dataset.
Documentation quality is the variable being measured, not model quality. Models are the instrument – if scores are low across all models, the documentation is incomplete.
The evaluation is resume-friendly: existing response and scored files are skipped on re-run, so interrupted runs can be continued without repeating expensive model calls.
Golden dataset
The config and golden dataset live in evals/docs_eval.yml in this repo.
The file combines evaluation settings (models, judge, repetitions) with
scoring criteria (questions, expected facts, coherence checks, thresholds).
Running
Prerequisites
-
GCP Application Default Credentials:
gcloud auth application-default login export GOOGLE_CLOUD_PROJECT=<gcp-project> -
The hummingbird-agent package installed (
pip install -e .from thehummingbird-agent/directory in the tools repo), or the container image.
Via Makefile
make eval-docs
Manual
python -m hummingbird_agent.evals.docs_eval.run_eval \
-c evals/docs_eval.yml
Writing questions
Each question tests a documentation topic, not a model capability. When adding or editing questions:
- Facts should be verifiable from public documentation. Avoid opinions or information that requires internal knowledge.
- Assign higher weights to more important facts. A weight-2 fact contributes twice as much to the score as a weight-1 fact.
- The coherence criterion checks narrative quality, not just fact presence. A response that lists correct facts without connecting them should score lower on coherence.
- Calibrate the threshold by running a baseline: set the threshold where current documentation should fail but improved documentation should pass.
3.19 - Rebuilding Packages
AI Agent Note: When asked to rebuild packages, use the
rebuildcommand:./ci/dist_git.py rebuild <package> --reason "<reason>". For rebuilding reverse dependencies (e.g., “rebuild all Go packages”), userebuild-rev-deps <package> --reason "<reason>". Always ask the user for a ticket link or explanation first to use as the reason. If the command fails with “uses macros in Release field”, see the “Packages requiring manual rebuild” section below for instructions.
Overview
This document covers three scenarios for triggering package builds:
-
No-change rebuild: Bump the Release field to rebuild with identical sources (e.g., to fix a faulty published RPM or pick up toolchain changes).
-
Rebuilding reverse dependencies: Rebuild all packages that depend on a changed package (e.g., rebuild all Go packages when golang updates).
-
Backporting a patch: Add an upstream patch that hasn’t yet landed in Fedora to fast-track a fix or feature.
All scenarios use the .N release suffix pattern to ensure our builds sort higher than the
upstream Fedora release while remaining lower than the next upstream version.
No-Change Rebuild
Using the rebuild command (recommended)
The rebuild command automates the Release field bump:
# Rebuild a single package
./ci/dist_git.py rebuild <package> --reason "<reason>"
# Rebuild multiple packages (one commit per package)
./ci/dist_git.py rebuild <package1> <package2> ... --reason "<reason>"
# Rebuild all packages (one commit per package)
./ci/dist_git.py rebuild --all --reason "<reason>"
# Rebuild all packages except specific ones (requires --all)
./ci/dist_git.py rebuild --all --exclude <pkg1>,<pkg2> --reason "<reason>"
Examples:
# Single package
./ci/dist_git.py rebuild ncurses --reason "published multiple times with different hashes"
# Multiple packages
./ci/dist_git.py rebuild grep unzip sed --reason "fix faulty builds"
# All packages (useful after toolchain updates)
./ci/dist_git.py rebuild --all --reason "toolchain update: GCC 15"
# All packages except a few already rebuilt earlier in the same rollout
./ci/dist_git.py rebuild --all --exclude glibc,gcc,llvm --reason "toolchain update: GCC 15"
--exclude takes a comma-separated list of package names and only works together with
--all — it is rejected when combined with an explicit package list, since you can simply
omit the packages you don’t want to rebuild in that case.
The command:
- Automatically bumps the Release field using the
.Nsuffix pattern - Handles
%autoreleaseby resolving and replacing with explicit values - Preserves macros in the Release field (e.g.,
%{revision}) - Creates a properly formatted commit message (one per package with
--all) - Does not mark the package as modified (release-only changes are ephemeral)
Supports --dry-run to preview changes without committing:
./ci/dist_git.py --dry-run rebuild <package> --reason "test"
Creating MRs for rebuild commits
For packages configured for MR artifact promotion, the MR build and
required tests produce the image that is released after merge. Follow that guide to verify the
promotion PipelineRun and automatic Release or recover a failure. Other packages build again on
main.
After creating rebuild commits locally, use rebuild_multi_mr.sh to push each commit as its own
merge request (one MR per package, auto-merge enabled):
./ci/rebuild_multi_mr.sh
By default the script compares against origin/main. For local development, use --base to point
at a different ref:
# Use local main branch as base (useful when origin/main is not up to date)
./ci/rebuild_multi_mr.sh --base main
# Use a specific commit SHA as base
./ci/rebuild_multi_mr.sh --base abc1234
# Preview what would be created without pushing
./ci/rebuild_multi_mr.sh --dry-run
# Limit to at most N MRs
./ci/rebuild_multi_mr.sh --max-updates=5
The script:
- Creates a
chore/rebuild-{package}branch per commit and pushes it - Titles each MR
chore(rpms): Rebuild {package}: {reason} - Enables auto-merge on all rebuild MRs
- Leaves the current branch untouched
- Skips branches that already exist on the remote (idempotent)
- Only processes commits whose message matches
Rebuild {package}: {reason}; other commits in the range are silently skipped
Full workflow example:
# 1. Create the rebuild commits
./ci/dist_git.py rebuild grep ncurses bash --reason "HUM-1234: toolchain update"
# 2. Preview the MRs that would be created
./ci/rebuild_multi_mr.sh --base main --dry-run
# 3. Create the MRs
./ci/rebuild_multi_mr.sh --base main
Packages requiring manual rebuild
Some packages use complex macro systems that the automated rebuild command cannot handle. These require manual editing of the spec file.
Macro indirection patterns
These packages define the Release field using a macro, where the macro itself contains %{?dist}.
The rebuild command cannot detect or manipulate these without expanding all macros, which would
break the macro system.
nodejs packages (nodejs20, nodejs22, nodejs24, nodejs25):
%{load:%{_sourcedir}/nodejs.srpm.macros}
%nodejs_define_version node 1:25.8.2-%{autorelease} -p
...
Release: %{node_release}
The %{node_release} macro is defined by an external macro system loaded from nodejs.srpm.macros.
The release component is embedded in the version definition.
How to rebuild: Edit the %nodejs_define_version node line to bump the release component (e.g.,
change -%{autorelease} to -1.1 or increment existing .N).
kernel-headers:
%define specrelease 59%{?buildid}%{?dist}
...
Release: %{specrelease}
How to rebuild: Edit the %define specrelease line to add/increment the .N suffix before
%{?buildid}:
%define specrelease 59.1%{?buildid}%{?dist}
krb5:
%global krb5_release 4%{?dist}
...
Release: %{krb5_release}
How to rebuild: Edit the %global krb5_release line to add/increment the .N suffix:
%global krb5_release 4.1%{?dist}
Why these can’t be automated
The rebuild command can handle:
- ✅ Simple numeric:
Release: 5%{?dist} - ✅ Macros ending with dist:
Release: %{baserelease}%{?dist}(e.g., rpm, gcc) - ✅ Complex macros with dist:
Release: %{?snapver:0.%{snapver}.}%{baserelease}%{?dist} - ✅ Content after dist:
Release: 11.1%{?dist} %{?extra_version:-e %{extra_version}}(e.g., unbound)
The rebuild command cannot handle:
- ❌ Macros without
%{?dist}:Release: %{node_release} - ❌ Macros where dist is inside the macro definition:
%{krb5_release}contains%{?dist}
This is because detecting and manipulating macros that contain dist internally would require expanding all macros (which changes the spec file semantically) or implementing RPM’s full macro parser.
Manual rebuild process
If you need to rebuild manually or the automated command doesn’t work for your use case, follow these steps:
1. Identify the package to rebuild
Identify the source package name and locate its spec file in rpms/<package>/<package>.spec.
If you have a binary RPM name, the source package name may differ. Query the Hummingbird repos to get the source RPM name:
podman run --rm quay.io/hummingbird-ci/hummingbird-builder:latest \
dnf5 repoquery --queryformat '%{SOURCERPM}' <binary-package> 2>/dev/null
Example: ncurses-libs-6.5-8.20250614.hum1 -> SRPM ncurses-6.5-8.20250614.hum1.src.rpm -> spec
file at rpms/ncurses/ncurses.spec
2. Determine the Release bump pattern
The .N bump suffix must always appear immediately before %{?dist}. The %{?dist} suffix
should always be the final component since it identifies the build environment.
| Current Pattern | Example Before | Example After |
|---|---|---|
| Simple numeric | Release: 3%{?dist} |
Release: 3.1%{?dist} |
| Already bumped | Release: 3.1%{?dist} |
Release: 3.2%{?dist} |
| With macro | Release: 8.%{revision}%{?dist} |
Release: 8.%{revision}.1%{?dist} |
| autorelease | Release: %autorelease |
Release: 1.1%{?dist} |
For %autorelease, first resolve its value using rpmspec, then replace with the resolved value
plus .1. In the Hummingbird monorepo, %autorelease always evaluates to 1.
Note: If the Release field is missing
%{?dist}entirely or looks unusual (e.g.,1build1instead of1.1%{?dist}), flag this to the user for resolution. Check the git history to understand the original value:git log -p -S "Release:" -- rpms/<package>/<package>.specThis helps determine the correct fix when a previous bump was malformed.
3. Modify the spec file
Use sed to edit only the Release: line, avoiding any unintended whitespace changes that text
editors may introduce:
sed -i 's/^Release: 3%{?dist}$/Release: 3.1%{?dist}/' rpms/<package>/<package>.spec
Verify the change with git diff before committing:
git diff rpms/<package>/<package>.spec
The diff should show only the Release line change:
- Release: 3%{?dist}
+ Release: 3.1%{?dist}
Important: Only modify the Release line. Do not introduce any other changes such as whitespace fixes or trailing newline modifications. If the diff shows additional changes, reset and retry with
sed.Important: Do not change the
releasefield inmetadata/<package>.jsonduring local rebuilds or backports. That field records the Fedora release for the version currently shipped and is present only while Hummingbird ships Fedora’s release; see Package Metadata Fields.
4. Verify the bump is correct
Use rpm --eval to confirm the new release sorts higher than the original:
# Returns -1 if first < second (correct), 1 if first > second (wrong)
rpm --eval '%{lua:print(rpm.vercmp("3.hum1", "3.1.hum1"))}'
# Expected output: -1
5. Commit the change
Use this commit message format:
Rebuild <package>: <reason>
<ticket link or explanation>
Example:
Rebuild ncurses: published multiple times with different hashes
HUM-1234
6. Verify the commit
After committing, verify only the Release line was changed:
git show --stat HEAD
Expected output should show exactly 1 insertion and 1 deletion:
rpms/<package>/<package>.spec | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
If the commit shows more changes, amend or reset and redo the change using sed.
Important notes about rebuilds
Modification status
Rebuilds do not change a package’s modification_status. Spec Release: bumps are ephemeral
and do not make a package modified versus clean. See
Package Metadata Fields for how modification_status and metadata
release relate to rebuilds.
During updates, Release lines are normalized to avoid conflicts, and the automation ignores Release-only changes, so automatic Fedora updates continue normally after a rebuild.
If you want to explicitly prevent automatic updates (e.g., you’re investigating an issue), you can manually mark the package as modified:
./ci/dist_git.py mark-modified <package> --modified --reason "Investigating build issue"
Note: This will block automatic Fedora updates until you mark it clean again.
Rebuilding Reverse Dependencies
When a compiler, runtime, or toolchain package changes, you may need to rebuild all packages that
depend on it. The rebuild-rev-deps command automates finding and rebuilding reverse dependencies.
When to use
Common scenarios for rebuilding reverse dependencies:
- Golang/Python runtime updates: When updating
golang1.26,python3.14, etc., rebuild all Go/Python packages - Toolchain changes: When updating
gcc, rebuild packages that BuildRequire it - Library ABI changes: When a library’s ABI changes, rebuild packages that BuildRequire it
Usage
./ci/dist_git.py rebuild-rev-deps <package> --reason "<reason>"
The command:
- Finds all packages that have
BuildRequires: <package>in their spec files - Rebuilds each package (bumps Release field and commits)
- Reports a summary of successful/failed rebuilds
Examples:
# Rebuild all Go packages when golang updates
./ci/dist_git.py rebuild-rev-deps golang1.26 --reason "golang 1.26.2 update"
# Rebuild all Python packages when python updates
./ci/dist_git.py rebuild-rev-deps python3.14 --reason "python 3.14.1 update"
# Rebuild packages that depend on a specific library
./ci/dist_git.py rebuild-rev-deps openssl --reason "openssl 3.4.0 update"
Virtual BuildRequires (golang/python)
For virtual BuildRequires like golang1.25, golang1.26, python3.13, python3.14, the command
automatically handles translation to the actual BuildRequires target:
# These all work the same way:
./ci/dist_git.py rebuild-rev-deps golang1.26 --reason "..."
./ci/dist_git.py rebuild-rev-deps go-rpm-macros --reason "..."
Important: Only the latest version triggers rebuilds. This is because:
- Multiple golang versions exist:
golang1.25,golang1.26 - They all provide the same virtual package:
Provides: golang = <version> - DNF always picks the highest version to satisfy
Requires: golang
Therefore:
- ✅
rebuild-rev-deps golang1.26→ Rebuilds all Go packages (latest version) - ❌
rebuild-rev-deps golang1.25→ Error: Not the latest version
This ensures rebuilds only happen when the active runtime actually changes.
Creating MRs for reverse dependency rebuilds
After creating rebuild commits, use rebuild_multi_mr.sh to push each commit as its own MR:
./ci/rebuild_multi_mr.sh --base main
See the “Creating MRs for rebuild commits” section above for full details.
Dry-run mode
Preview which packages would be rebuilt without making changes:
./ci/dist_git.py --dry-run rebuild-rev-deps golang1.26 --reason "test"
This shows:
- Which packages have the BuildRequires dependency
- What the new Release values would be
- Does not commit any changes
Backporting a Patch
Use this workflow when you need to fast-track an upstream fix or feature that hasn’t yet been released in Fedora.
1. Obtain the patch
Fetch the patch from the upstream repository. For GitHub PRs, append .patch to the PR URL:
curl -L https://github.com/<org>/<repo>/pull/<number>.patch \
> rpms/<package>/<NNNN>-<short-description>.patch
Name the patch file with a numeric prefix matching the next available PatchN: slot in the spec
file (e.g., 0004-fix-foo.patch if Patch1-3 already exist).
Keep the upstream provenance in the patch header:
Upstream: https://github.com/<org>/<repo>/commit/<direct-commit>
Origin: https://github.com/<org>/<repo>/commit/<original-commit>
Upstream is the commit that the patch applies. Add Origin only when Upstream adapts or
backports an earlier commit.
2. Add the patch to the spec file
Add a PatchN: declaration after the existing patches:
Patch3: 0003-existing-patch.patch
Patch4: 0004-fix-foo.patch
The patch will be applied automatically if the spec uses %autosetup -p1. If the spec uses explicit
%patchN macros, add the corresponding apply line in the %prep section.
3. Check for a gorget source-pipeline
If metadata/<package>.source-pipeline.yaml exists, check whether its transform: step applies
patches or otherwise runs against patched source (e.g. resolving a lockfile with yarn install,
pnpm fetch, npm ci):
# run from the root of the rpms repo checkout
grep -n -E "patch -p[0-9]|git apply" metadata/<package>.source-pipeline.yaml
This catches the common forms, but isn’t exhaustive — if the transform applies patches some other
way (a different tool, or a script that wraps patch/git apply), read the transform: step
directly rather than trusting a grep miss.
If it does, and your new patch touches a file that step depends on (a lockfile or manifest such as
yarn.lock, package.json, pnpm-lock.yaml, go.sum, Cargo.lock), add the same
patch -p1 < "${PACKAGE_DIR}/<your-patch>.patch" line there too, in the same relative order as
its PatchN: slot in the spec. The patch file itself needs no extra fetch step — it’s already in
${PACKAGE_DIR}/ because you committed it there in step 2.
The pipeline’s transform: step and the spec’s %prep/%goprep are two independent lists of
patches to apply — nothing keeps them in sync automatically. A patch declared only in the spec is
invisible to whatever artifact the transform step generates (e.g. an offline yarn/pnpm cache built
from the lockfile), and the generated artifact silently drifts from what %build actually applies.
See “Known sharp edge: patch-list duplication” in
the source-pipeline design doc for the grafana12.4/grafana13.1 incident this rule
comes from. test/test_source_pipeline_patches.py checks for this drift, but treat it as a safety
net, not a substitute for updating the pipeline yourself here.
4. For Go packages: check go-vendor-tools.toml’s pre_commands
This applies whether or not the package uses gorget — rpms/<package>/go-vendor-tools.toml’s
[archive] pre_commands (sed edits, go get bumps, go mod tidy) only ever run against the
vendor archive’s own checkout, never against the plain source tarball (Source0), which is fetched
separately:
grep -n -A2 "pre_commands" rpms/<package>/go-vendor-tools.toml
If your new patch targets a vendored Go module (or you’re editing pre_commands directly instead
of adding a patch — don’t), and any pre_commands entry mutates go.mod/go.sum (directly, via
go get, or via go mod tidy/edit), your patch must apply the same go.mod/go.sum change to
the actual source tree, not just to go-vendor-tools.toml. pre_commands cannot substitute for
this: they require live network access to run go get, which %prep doesn’t have (Konflux builds
are hermetic). Compute the resulting go.mod/go.sum diff offline (e.g. clone upstream, apply the
same edits, run go mod tidy if pre_commands does) and commit that as the patch.
Skipping this leaves go.mod in the build tree and vendor/modules.txt in the generated vendor
archive silently requiring different versions of the same package, which go build -mod=vendor
rejects as inconsistent vendoring. See the source-pipeline design doc for the trivy
incident this rule comes from. test/test_govendortools_gomod_patch_sync.py checks for this
drift, same caveat as above.
5. Bump the Release
Follow the same .N suffix pattern as no-change rebuilds:
- Release: 3%{?dist}
+ Release: 3.1%{?dist}
6. Mark the package as modified
Mark a Fedora-imported package as modified before you commit. This prevents an automatic Fedora update from overwriting the patch:
make dist-git ARGS='mark-modified <package> --modified --reason "Backport <issue or CVE>"'
Preserve existing modification reasons. Do not change an independent package to modified.
mark-modified updates the metadata file but does not commit it.
7. Commit the patch
Use this commit message format:
Backport <short description> to <package>
Backport: <link to PR or commit>
<ticket link if applicable>
Example:
Backport reproducible build sorting fix to dnf5
Backport: https://github.com/rpm-software-management/dnf5/pull/2522
Include the patch, spec file, and metadata file in the same package commit. Do not use
dist_git.py rebuild for this step. That command describes and commits a no-change rebuild.
8. Verify patch application
Apply the patch to the exact packaged source release.
Related Operations
- Excluding Packages from Images - temporarily block faulty packages in container builds
3.20 - Setting Up Pulp Repositories
Overview
- This document and script should be used to initialize and setup a pulp domain and a set of repositories
- We choose the domain prefix of
public-to ensure that the repositories are publicly available
Setup
- make sure you have followed the local development setup instructions in the infrastructure repository
- install pulp cli via dnf install pulp-cli
- setup the local pulp environment via:
cki_secret PULP_PUBLIC_RHEL_PRIMITIVES_CONFIG_FILE > ~/.config/pulp/cli.toml
# or
vault kv get -mount=apps -field cli.toml hummingbird/PULP_PUBLIC_RHEL_PRIMITIVES_CONFIG_FILE > ~/.config/pulp/cli.toml
- Run the creation script
Create
./create-pulp-resources.sh public-hummingbird "source,x86_64,s390x,ppc64le,aarch64"
3.21 - Package Metadata Fields
Overview
Each package has a metadata file at metadata/<package>.json. This page explains metadata
configuration and tooling behavior for day-to-day package operations. The
RPM metadata and versioning specification defines the authoritative metadata
contract, including field definitions, package states, and validation rules.
Two fields control how Hummingbird treats local changes and rebuilds relative to Fedora:
| Field | Purpose |
|---|---|
modification_status |
Whether the package may be auto-updated from Fedora |
fedora.release |
Fedora release baseline when the package still ships the imported Fedora version |
Fedora provenance is stored in the fedora object, with git_url, branch, and sha identifying
the import or last sync. fedora.release is present only while the package ships that Fedora
version. A package ahead of Fedora retains provenance but has no fedora.release; an independent
package has no fedora object. There is no top-level release field.
modification_status
modification_status records whether a package still matches its Fedora import, diverges from
that import, or has no Fedora origin.
| Status | Meaning | Auto-updates from Fedora |
|---|---|---|
clean |
Unmodified Fedora import | Allowed |
modified |
Local changes or an upstream version ahead of Fedora | Blocked |
independent |
Hummingbird-independent package (not from Fedora) | Blocked |
When to use each value
clean: Default for packages imported from Fedora with no local source changes. Automatic Fedora updates are allowed. Conflicts withmodification_reason(must be absent).modified: Use after backports, custom patches, or updates ahead of Fedora that must not be overwritten by an automatic update. Requiresmodification_reason.independent: Required for packages that are not imported from Fedora. These packages have nofedoraobject. Conflicts withmodification_reason(must be absent).
Imports set the status automatically, using clean for Fedora imports and independent for
Hummingbird-independent imports. After local source changes, set status with
dist_git.py mark-modified. See Package Modification Tracking.
Rebuilds that only bump the spec Release: line do not change modification_status.
Release-only changes are ephemeral and are ignored when deciding whether a package is modified
versus clean.
modification_reason
When modification_status is modified, modification_reason is required. It should be a short
explanation of why the package cannot be auto-updated (for example, a CVE backport or a custom
spec change). Clear the reason when marking the package clean again.
modification_reason applies only to modified packages. Do not add it for clean or independent
packages. See When not to update these fields.
fedora.release
The fedora.release field in metadata is not the same as the Release: line in the package spec.
It records the imported Fedora release without a dist tag such as .fc42. It remains present
after local patches or release-only rebuilds while the package stays on the imported Fedora
version. The spec’s Release: line is authoritative for the Hummingbird build.
| Location | What it represents |
|---|---|
metadata/<package>.json → fedora.release |
Fedora release baseline for the imported version, without its dist tag |
Spec Release: |
The release used for the Hummingbird build (includes %{?dist} / .hum1, and may include local .N rebuild suffixes) |
Who writes metadata fedora.release
| Writer | When | Value stored |
|---|---|---|
dist_git.py import / update / sync |
Fedora import or refresh | Resolved Fedora release with dist suffix stripped (e.g. 5 from 5.fc42) |
check_upstream_versions.py |
Local upstream version bump (check --update) |
Removes fedora.release for Fedora-origin packages moving ahead of Fedora, retaining the other provenance fields |
Fedora-imported packages
For packages with a fedora object:
fedora.releaseis set on import and refreshed whendist_git.py update/syncbrings in a Fedora version. Dist suffixes such as.fc42are stripped.- When
check_upstream_versions.py check --updatemoves the package ahead of Fedora, it removesfedora.releaseand marks the packagemodified. The default updater resets a literal spec release to0.1%{?dist}. It preserves%autoreleaseand other package-specific release macros; anupdate_spechook can replace the default spec-update behavior. See Package Modification Tracking for update hooks. - Do not change
fedora.releaseduring local rebuilds or backports. Rebuilds bump only the spec’sRelease:line. With a Fedora baseline, the first rebuild appends.1and later rebuilds increment that suffix. Without a Fedora baseline, rebuilding0.1advances it to0.2. - When Fedora uses
%autorelease,fedora.releasestores the resolved numeric release, not the literal macro. Imports and updates resolve it through Fedora build services and replace the local spec’s%autoreleasewith that value plus%{?dist}. For rebuilds that still see%autorelease, follow Rebuilding Packages.
Independent packages
For Hummingbird-independent packages with modification_status: "independent":
- There is no Fedora origin, so metadata does not contain a
fedoraobject orfedora.release. - The
.hum1suffix comes from the specRelease:line via%{?dist}(for exampleRelease: 1%{?dist}), not from metadata. dist_git.py rebuildincrements the spec’s current release directly for independent packages.
Spec Release: vs metadata fedora.release
No-change rebuilds and reverse-dependency rebuilds change the spec Release: only. See
Rebuilding Packages. When dist_git.py update or sync imports a Fedora version,
it records that Fedora release in fedora.release. Local patches merged onto that version
retain the Fedora release baseline. When check_upstream_versions.py moves a package ahead of
Fedora, it removes fedora.release. See the specification’s
built NVR rules for the complete versioning contract.
private_product
private_product records an internal product assignment for private RPM distribution. It is
independent of the package’s Fedora relationship and versioning state. See the
metadata specification for the field’s format.
Read and write assignments through hum.meta. The library checks that a product matches an
RPA entry in ci/konflux_rpa_config.yml; resource generation uses that assignment to select the
private Konflux application and exclude the package from public RPAs. Packages without an
assignment remain public, regardless of their names.
See Private RPM Repositories for assignment and onboarding commands.
When not to update these fields
Most day-to-day package work changes the spec, sources, or patches, not metadata
modification_status, modification_reason, or fedora.release. Leave those fields alone unless the
package’s relationship to Fedora actually changed.
| Situation | Do not change | Why |
|---|---|---|
| Independent package gets a CVE patch, backport, or other source edit | modification_status, modification_reason |
Status stays independent. There is no Fedora auto-update to block, so do not switch to modified or add a modification_reason. |
No-change rebuild (spec Release: bump only) |
modification_status, modification_reason, fedora.release |
Status stays unchanged; fedora.release continues to record a Fedora baseline when one exists. |
| Fedora-imported package gets a local patch or backport | fedora.release |
Mark the package modified with a reason, but keep fedora.release if present. Only bump the spec’s Release: if needed. |
Package is already modified and you add another local change |
modification_status |
Leave status as modified. Update modification_reason only if the existing reason no longer describes why auto-updates must stay blocked. |
| Upstream Fedora update lands and automatic Fedora updates should resume | Do not leave stale fields | Use dist_git.py update to refresh version and Fedora provenance. Mark clean once the package matches Fedora; this clears modification_reason. |
Examples
Independent package + CVE fix: Edit the spec and add the patch. Keep the metadata status and version unchanged, as shown in this excerpt, not a complete metadata file:
{
"modification_status": "independent",
"version": "1.3.0"
}
Do not add modification_reason, and do not change status to modified.
Fedora package + CVE backport: After the patch, mark modified so auto-updates stay blocked:
make dist-git ARGS='mark-modified <package> --modified --reason "Backport CVE-2024-12345"'
Do not edit fedora.release as part of that change.
Rebuild only: Use dist_git.py rebuild. Leave metadata
modification_status and fedora.release unchanged.
Related documentation
- RPM metadata and versioning specification defines the metadata contract.
- Package Modification Tracking covers marking packages modified or clean, checking status, validation, and update hooks.
- Rebuilding Packages covers spec release bumps for no-change rebuilds and backports.
- Adding Independent Packages covers packages not imported from Fedora.
- Updating Dist-git Packages covers imports and Fedora updates.
3.22 - Debugging Build Failures
Overview
When ./ci/build_rpms.sh fails to build a package, you can use the --shell-after flag to drop
into an interactive debugging environment. This preserves the build state and allows you to
investigate failures, modify files, and re-run the build without starting over.
Quick Start
# Run build and drop into shell after completion (even on failure)
./ci/build_rpms.sh --shell-after <package>
After the build completes (success or failure), you’ll be dropped into a mock shell inside the build environment.
Checking Sources and Patches Only
Many build failures come from simple problems: a missing or incorrect source tarball, or a patch that
does not apply. To catch these without running a full build, use --prep-only:
./ci/build_rpms.sh --prep-only <package>
This builds the SRPM, which fails if any Source/Patch file is missing. It then installs the
BuildRequires and runs only %prep (rpmbuild -rp, equivalent to rpmbuild -bp), which unpacks
the tarballs and applies the patches. The script exits non-zero if any step fails. The %prep
output is saved to builds/<package>/results/prep.log. --prep-only cannot be combined with
--hermetic, --shell-before, or --shell-after.
Understanding the Build Environment
The build happens in two layers:
- Outer container: The rpm-build-pipeline podman container (Red Hat build environment)
- Mock shell: An isolated RPM build environment inside the container (similar to a chroot)
When you use --shell-after, you’re placed directly in the mock shell.
Key directories in the outer container
/results/build.log- Full build output with all commands that were run/sources/- Package source files (spec, patches, etc.)/config/- Mock configuration/repo/- Your local rpms repository (read-only mount)
Key directories in the mock shell
/builddir/build/BUILD/<package>-*/- Unpacked source tree (modify and re-run build/test commands here)/builddir/build/originals/<package>.spec- The spec file (modify to test spec changes)/builddir/build/SOURCES/- Source tarballs and patches/builddir/build/BUILDROOT/- Install root (where %install places files)
Debugging Workflow
1. Connect to the outer container
To get a second shell without killing your existing mock shell session, connect to the outer container:
# In another terminal, find the running container
podman ps | grep rpm-build-pipeline
# Get a shell in the outer container
podman exec -it <container-id> bash
2. Examine the build failure
From the outer container, check the build log to understand what failed:
# View the end of the build log
tail -100 /results/build.log
# Search for specific errors
grep -i error /results/build.log
3. Enter the mock shell
To investigate or modify the build environment, you can get an additional shell in the mock chroot,
particularly for agents which cannot use the initial interactive --shell-after from above.
# From the outer container, chroot into the mock build environment
chroot /var/lib/mock/local-x86_64/root /bin/bash
4. Iterate on the fix
The recommended approach for debugging is to extract and re-run specific commands directly:
Fast iteration: Extract and re-run specific commands (recommended)
For quick iteration while developing a fix, extract the exact build or test command from the build log and run it directly:
# From the outer container, find what command the failing phase runs
grep -A 30 'Executing(%check)' /results/build.log
# Look for lines starting with "+ " that show the actual commands
# Example output:
# + cd /builddir/build/BUILD/dnf5-5.4.0.0-build/dnf5-5.4.0.0
# + /usr/bin/ctest --test-dir redhat-linux-build --output-on-failure ...
# Run that command in the mock chroot
podman exec -u root <container-id> chroot /var/lib/mock/local-x86_64/root \
su mockbuild -c "cd /builddir/build/BUILD/<package>-*/... && <actual-command>"
For build failures during %build: Look for Executing(%build) in the log, find commands like
make or ninja, then re-run them:
# Example: extract the build command
grep -A 30 'Executing(%build)' /results/build.log | grep -E '^\+ (make|ninja|cmake)'
# Re-run in the build directory
podman exec -u root <container-id> chroot /var/lib/mock/local-x86_64/root \
su mockbuild -c "cd /builddir/build/BUILD/<package>-*/<build-subdir> && make -j14"
For test failures during %check: Look for Executing(%check) and extract the test command:
# Example: extract the test command
grep -A 30 'Executing(%check)' /results/build.log | grep -E '^\+ (ctest|make check|pytest)'
# Re-run tests
podman exec -u root <container-id> chroot /var/lib/mock/local-x86_64/root \
su mockbuild -c "cd /builddir/build/BUILD/<package>-*/... && ctest --output-on-failure"
This approach is fast because it skips rpmbuild overhead and goes straight to the failing command.
It works with the preserved BUILD directory from --shell-after and is ideal when you’re modifying
source files and want quick feedback.
Modifying source files: Edit files directly in /builddir/build/BUILD/<package>-*/ and re-run
the command to test your changes quickly.
Modifying the spec file: You can also test spec file changes by editing
/builddir/build/originals/<package>.spec in the mock chroot:
# From the host, edit the spec file in the mock chroot
podman exec -it -u root <container-id> chroot /var/lib/mock/local-x86_64/root \
vim /builddir/build/originals/<package>.spec
# Run rpmbuild to see the change
podman exec -u root <container-id> chroot /var/lib/mock/local-x86_64/root \
env HOME=/builddir LANG=C.UTF-8 su mockbuild -c \
'rpmbuild -bb --target x86_64 --nodeps /builddir/build/originals/<package>.spec'
Full validation: Apply fixes and rebuild with build_rpms.sh
Manual rpmbuild in the preserved environment may not work reliably for all packages due to complex
build dependencies and environment requirements.
Once you’ve identified a fix using the fast iteration approach:
-
Exit the debug environment and apply your fix to the actual source files or spec file in
rpms/<package>/ -
Test the complete build using the build script:
./ci/build_rpms.sh <package>
This ensures your fix works through the entire build process in a clean environment.
Related Operations
- Rebuilding Packages - Bump release or backport patches
- Package Modification Tracking - Mark packages as modified
3.23 - Package Modification Tracking
Overview
The RPMs repository tracks whether packages have been locally modified from their Fedora upstream source. This tracking prevents automatic updates from overwriting local changes like backported patches or custom modifications.
Modification Status
Each package metadata file (metadata/<package>.json) has a modification_status of clean,
modified, or independent. That field (and related modification_reason / release configuration)
is documented in Package Metadata Fields. This page covers the
workflows for checking status, marking packages, viewing diffs, and configuring update hooks.
An optional track_upstream string field controls whether a package is checked by
check_upstream_versions.py for new upstream releases (via release-monitoring.org). Set it to
"latest" to track the latest version, or to a version prefix like "1.26" to constrain updates to
that series. Its presence enables tracking; omit the field to disable it. The check subcommand
only checks packages with track_upstream set when no explicit package arguments are given. The
list subcommand shows all packages regardless of this field.
Checking Package Status
View a package’s modification status:
jq .modification_status metadata/<package>.json
View reason for modification (if modified):
jq .modification_reason metadata/<package>.json
List all modified packages:
for f in metadata/*.json; do
status=$(jq -r .modification_status "$f" 2>/dev/null)
if [ "$status" = "modified" ]; then
pkg=$(basename "$f" .json)
reason=$(jq -r .modification_reason "$f" 2>/dev/null)
echo "$pkg: $reason"
fi
done
Viewing Package Differences
To see what changes exist in a modified package compared to upstream Fedora:
# Show full diff for a package
./ci/dist_git.py diff bash
# Show summary statistics
./ci/dist_git.py diff bash --stat
# Show only which files changed
./ci/dist_git.py diff bash --name-only
# Show raw diff (includes Release: bumps and whitespace)
./ci/dist_git.py diff bash --raw
# Diff all modified packages
./ci/dist_git.py diff --all
What’s shown:
- By default, the diff ignores Release: number changes (no-change rebuilds)
- Trailing whitespace and blank line changes are ignored
- Use
--rawto see absolutely everything, including Release: bumps
Package types:
- Modified packages: Shows the differences
- Clean packages: Shows nothing (useful for verification)
- Independent packages: Skips with message “no upstream to diff against”
Marking Packages
Mark as Modified
Use this when you make local changes to a package (backports, custom patches, etc.):
./ci/dist_git.py mark-modified <package> --modified \
--reason "Brief explanation of why"
Examples:
# After backporting a patch
./ci/dist_git.py mark-modified gcc --modified \
--reason "Backport CVE-2024-12345 fix from upstream"
# After custom spec change
./ci/dist_git.py mark-modified systemd --modified \
--reason "Add custom service unit for Hummingbird"
The reason field is required and should be concise but descriptive. It helps future maintainers understand why the package can’t be auto-updated.
Mark as Clean
Use this to re-enable automatic updates after confirming your changes are no longer needed (e.g., the fix landed in Fedora):
./ci/dist_git.py mark-modified <package> --clean
This removes the modified status and allows the package to receive automatic updates from Fedora
again.
Configure Upstream Tracking
Use set-upstream to configure upstream tracking settings for a package. Each flag independently
sets or clears one metadata field. At least one flag is required; omitted flags leave their fields
untouched.
./ci/dist_git.py set-upstream <package> [flags]
| Flag | Sets field | Clears with |
|---|---|---|
--track-version latest |
track_upstream: "latest" |
--no-track-version |
--track-version VER |
track_upstream: "VER" |
--no-track-version |
--project-id ID |
release_monitoring_project_id (int) |
--no-project-id |
--project-id NAME |
release_monitoring_project_id (str) |
--no-project-id |
Each set/clear pair is mutually exclusive (can’t pass --track-version and --no-track-version
together).
Examples:
# Enable upstream version tracking (any version)
./ci/dist_git.py set-upstream bash --track-version latest
# Disable upstream version tracking
./ci/dist_git.py set-upstream bash --no-track-version
# Set upstream name with version constraint for versioned packages
./ci/dist_git.py set-upstream golang1.26 --project-id golang --track-version 1.26
# Set release-monitoring.org project ID (integer)
./ci/dist_git.py set-upstream python3.11 --track-version 3.11 --project-id 13254
# Remove project ID (reverts to RPM name lookup)
./ci/dist_git.py set-upstream python3.11 --no-project-id
# Combine multiple flags in one call
./ci/dist_git.py set-upstream golang1.26 \
--track-version 1.26 --project-id 13254
Metadata fields:
| Field | Description | Example |
|---|---|---|
track_upstream |
"latest" or version prefix to constrain updates |
"latest", "1.26" |
release_monitoring_project_id |
Anitya project ID (int) or upstream name (str) | 13254, "golang" |
version_suffix_strip |
Suffix to strip from Anitya-reported versions | "-RELEASE" |
upstream_version_transform |
Named transform from Anitya version to RPM scheme | "openjdk_to_rpm" |
source_availability_check |
Source probe to run before selecting a version | {"type": "http", "urls": [...]} |
These fields affect two systems:
dist_git.py update: Whentrack_upstreamis a version prefix, skips upstream versions that don’t match. For example,track_upstream: "1.26"allows1.26,1.26.0,1.26.3but rejects1.27.0.check_upstream_versions.py: Whenrelease_monitoring_project_idis an integer, queries the v2 API directly by Anitya project ID. When it is a string, queries release-monitoring.org using that name instead of the RPM package name (e.g., looks upgolanginstead ofgolang1.26). When absent, uses the RPM package name. Whentrack_upstreamis a version prefix, filters the reported upstream versions to only those matching the prefix. Only packages withtrack_upstreamset are checked bycheck_upstream_versions.py checkwhen no explicit package arguments are given.
When a successful lookup reports releases but none match the configured version prefix, the checker skips that package with a warning instead of failing the job. For example, Python 3.15 tracking can wait while Anitya only reports Python 3.14 releases. The package is not counted as up to date, no spec or source update is attempted, and a warning-only run exits successfully. Empty upstream results and HTTP, connection, or JSON parsing failures during lookup remain errors. Existing warnings for unpublished source archives remain nonfatal as well.
Find release-monitoring.org project IDs by searching on https://release-monitoring.org.
The project ID can be combined with a version prefix to filter versions returned by the project ID lookup:
{
"release_monitoring_project_id": 13254,
"track_upstream": "3.11"
}
Some upstream projects tag releases with a suffix that is not part of the RPM version (e.g.,
swift-6.3.3-RELEASE). After Anitya strips the version prefix, the reported version still contains
the suffix (6.3.3-RELEASE), which is incompatible with RPM’s Version: field (hyphens are not
allowed). Use version_suffix_strip to remove it before comparison and update:
{
"release_monitoring_project_id": 21267,
"version_suffix_strip": "-RELEASE"
}
Some upstream projects publish source tarballs independently of tag creation, and Anitya may report
a version before the tarball is available. Use source_availability_check to HEAD-probe one or more
required source URLs before selecting a version. Versions whose source returns 404 are skipped in
favour of the next available version. The skip is reported as a warning in the scheduled-job log
and summary, but does not fail the update job. Routine warnings are omitted from Slack notifications.
The general HTTP probe accepts a non-empty urls list. Every URL must be available. URL templates
may use ${VERSION}, ${UPSTREAM_REPO}, and ${TRACK_UPSTREAM}:
{
"source_availability_check": {
"type": "http",
"urls": [
"${UPSTREAM_REPO}/releases/download/v${VERSION}/libfabric-${VERSION}.tar.bz2"
]
}
}
Legacy named checkers remain available for source layouts that need package-specific logic:
openjdk_osci— probeshttps://openjdk-sources.osci.io/openjdk{feature}/openjdk-{version}.tar.xzopenjdk_gachecks for ajdk-<version>-gatag inupstream_repoand excludes build zero. It selects base releases with at most four version components. Backport fixes from extra-component emergency releases separately, as Fedora does. Use it when fetching OpenJDK archives directly from GitHub after Fedora maintenance ends.tomcat_apache— probes the Tomcat source archive ondownloads.apache.org, and its detached signature for Tomcat 10 before allowing an update
{
"release_monitoring_project_id": 369281,
"track_upstream": "21",
"upstream_version_transform": "openjdk_to_rpm",
"source_availability_check": "openjdk_osci"
}
version_from_ref (fixed-ref, no-tagged-release packages)
Most gorget source-pipeline.yaml fetch steps are version-templated (ref: "v${VERSION}"), so
dist_git.py update can invoke gorget with metadata/<package>.json’s own version directly. A
few packages instead pin a fixed git-snapshot commit with no tagged upstream release at all (gcc,
glibc, libyuv, php-patchwork-jsqueeze, vim) – their version field doesn’t change when the pinned
commit does, so it can’t drive gorget’s --version argument.
version_from_ref tells dist_git.py update how to derive gorget’s --version string from a
newly-pinned commit, instead of requiring someone to compute it by hand every time (the previous
process, documented as a manual step in each such package’s source-pipeline.yaml comments).
Currently one type is supported:
commit-date(used by gcc):"<metadata.json's version>-<commit's own YYYYMMDD date>", where the date comes fromgit log -1 --format=%cd --date=format:%Y%m%d <ref>run against the newly-pinned commit.
{
"version_from_ref": {"type": "commit-date"}
}
Currently gcc-only. dist_git.py update only fully automates a fixed-ref pin refresh for gcc’s
simple case (a full 40-char commit SHA lives directly in a spec %global). The other four
packages pin via a git describe-style string (e.g. glibc’s %{glibcsrcdir} macro,
glibc-2.43-47-gbc95068f5f) whose trailing hash is only a 10-character abbreviation – resolving
that back to a full SHA needs a local clone containing the commit object, real extra network work
with its own failure modes (ambiguous hashes, upstream unavailability), not yet implemented (see
HUM-4621’s “Remaining work”). For those four, dist_git.py update detects when the pipeline’s
pinned ref: no longer matches the merged spec and refuses to auto-commit – it’s routed to a
draft, no-test-labeled MR for manual resolution instead, the same way a real git merge conflict
is, rather than silently committing a sources file gorget was never actually run against.
Sudo Patchlevel Comparisons
For sudo, upstream version comparisons include the spec’s optional extraver macro.
Fedora’s Version: 1.9.17 with extraver p2 therefore compares as 1.9.17p2, preventing
an unnecessary update and a doubled p2 in the source URL. This comparison does not change
the spec, the RPM version/release, or the version stored in package metadata. Other packages'
extraver macros are not treated as upstream version suffixes.
Per-Package Update Hooks
When check_upstream_versions.py check --update updates a package, by default it sets Version: to
the new upstream version and Release: to 0.1%{?dist} (unless %autorelease is used), adds a
changelog entry, and downloads new sources from the URLs declared in the spec. Some packages need
custom logic (e.g. generating stripped tarballs or patching macro-based version lines). A
per-package hooks file lets you override or extend these default phases without changing
check_upstream_versions.py itself.
Hooks file location
metadata/<package>.update-hooks.yaml
For example, metadata/nodejs25.update-hooks.yaml.
Hook phases
The YAML file supports three optional keys. Each value is a shell command string executed with
bash -eo pipefail -c in the package directory as the working directory.
| Phase | Behaviour |
|---|---|
update_spec |
Replaces the default update that sets Version: to the new upstream version and Release: to 0.1%{?dist}. A changelog entry is still added automatically. |
download_sources |
Replaces the default URL-based source download. Must print one filename per line to stdout for files to upload to the lookaside cache. Redirect any other output to stderr (>&2). |
post_update |
Additive — runs after spec + sources are ready. No default equivalent. |
Omitting a phase means the default logic runs for that phase. Packages without a hooks file behave identically to before.
Unknown phase keys in the YAML cause a ValueError (fail-fast).
Environment variables
Every hook receives these environment variables:
| Variable | Example |
|---|---|
UPDATE_PACKAGE |
nodejs25 |
UPDATE_OLD_VERSION |
25.6.1 |
UPDATE_NEW_VERSION |
25.8.2 |
UPDATE_SPEC_FILE |
/home/rpms/rpms/nodejs25/nodejs25.spec |
UPDATE_PACKAGE_DIR |
/home/rpms/rpms/nodejs25 |
UPDATE_SOURCES_FILE |
/home/rpms/rpms/nodejs25/sources |
UPDATE_ROOT_DIR |
/home/rpms |
Example
See metadata/nodejs25.update-hooks.yaml for a
working example that uses all three hook phases.
How Auto-Updates Work
The ./ci/dist_git.py update command (used by automation) checks modification status before
updating packages:
- clean packages: Updated automatically when new Fedora versions are available
- modified packages: Automatically merged with upstream changes (conflicts create draft MRs)
- independent packages: Update blocked (not sourced from Fedora)
- version-constrained packages: Skipped if upstream version doesn’t match
track_upstreamprefix
To force-update a modified package (discarding local changes):
./ci/dist_git.py sync <package>
The sync command bypasses the modification check and force-updates to the latest upstream version.
After syncing, the package is automatically marked clean.
Resolving Merge Conflicts
When modified packages are updated from Fedora, dist_git.py update attempts to automatically merge
local changes with the new upstream version using git’s three-way merge. When conflicts occur, the
update still succeeds but creates a commit with conflict markers, and the automation files a draft
merge request labeled with CONFLICT: for manual resolution.
Understanding Conflict Markers
Git uses this conflict marker structure:
<<<<<<< HEAD
Fedora's version (new upstream)
=======
Hummingbird's local modifications
>>>>>>> hummingbird-local
ALL THREE markers must be removed for a clean resolution.
Update branch/MR structure
MRs are created on branches following the pattern:
chore/dist-git-update-PACKAGENAME
These branches are automatically created by the dist_git_update GitLab schedule. If they have
conflicts, they result in draft MRs with:
- Title prefix:
CONFLICT: chore(rpms): Update ... - Description listing the conflicting files
no-testlabel to skip CI tests (saves resources since conflicts need manual resolution)
Resolution Process
-
Check out the conflict branch:
git fetch origin git checkout origin/chore/dist-git-update-PACKAGENAME -
Examine the conflict:
# Find all files with conflict markers git grep -nE '^<{7} .+|^={7}$|^>{7} .+' -- rpms/PACKAGENAME/ # View the specific conflict git show HEAD:rpms/PACKAGENAME/PACKAGENAME.spec | grep -B5 -A10 "^<<<<<<< HEAD" -
Understand the local changes:
# Review commit history to understand why changes were made git log --oneline -- rpms/PACKAGENAME/ git log -p -- rpms/PACKAGENAME/ # With diffs # Check the modification reason jq -r .modification_reason metadata/PACKAGENAME.jsonUnderstand the context for correct resolution:
- What was the original purpose of the local change? Is it transient or permanent?
- Is it a workaround for a bug, a security patch, or a configuration difference?
- Does it affect other packages (e.g., nss builds nspr as a subpackage)?
- Check spec file comments (e.g., NOTE: comments) for packaging details
Decide which version to accept:
- Accept HEAD (Fedora) for: release number lags, fixed workarounds that Fedora improved or addressed differently
- Keep hummingbird-local for: security patches not in Fedora, FIPS requirements, critical fixes, and other permanent modifications
- Merge both for: test skip lists, independent changes that don’t conflict logically
- When in doubt: Accept Fedora’s version for packaging metadata (Release:, subpackage versions), keep Hummingbird’s version for functional changes (patches, dependencies, build options)
-
Resolve the conflict: Edit the file to choose the appropriate version (HEAD, hummingbird-local, or merge both). Verify no markers remain:
git grep -nE '^<{7} .+|^={7}$|^>{7} .+' -- rpms/PACKAGENAME/ -
Validate the resolution: Check that local modifications are preserved:
# Check the diff against upstream (works on working tree, staging not required) ./ci/dist_git.py diff PACKAGENAME # Compare with previous modification commits to verify git log -p -- rpms/PACKAGENAME/The diff should show only the intended local modifications (ignoring Release: bumps). This confirms the merge preserved your changes correctly. Note:
dist_git.py diffcompares the filesystem working tree against upstream, so it works before or after staging. -
Amend the commit: Record the original conflicted commit SHA, then amend:
# Record the original conflicted commit SHA ORIGINAL_SHA=$(git rev-parse HEAD) # Stage the resolved files and amend the commit git add rpms/PACKAGENAME/ git commit --amend -m "$(git log -1 --format=%B | head -n -1) Conflicted-Update: $ORIGINAL_SHA"This preserves the original commit message while adding a
Conflicted-Update:trailer that records which commit contained the conflict markers. This helps track the resolution history and can be useful for auditing or debugging later. -
Push the resolution:
git push origin HEAD:chore/dist-git-update-PACKAGENAME --force-with-lease --push-option merge_request.unlabel=no-testThis removes the
no-testlabel from the MR, which triggers CI tests to run and verifies the resolution works correctly. Some developers might haveoriginas read-only remote, and a different writable remote (e.g.originw).
Common Conflicts
nss: Subpackage Release Numbers
The nss package builds nspr as a subpackage with its own release number offset.
BACKGROUND:
nssbuilds bothnssandnsprRPMs from the same sourcenspr_releaseuses an offset (%[%baserelease+n]) to avoid NVR clashes- The spec file NOTE explains: reset to 1 when
nspr_versionchanges, increment when onlynsschanges - Fedora manages these offsets in their ecosystem to prevent conflicts
CONFLICT EXAMPLE:
<<<<<<< HEAD
%global nspr_release %[%baserelease+3]
=======
%global nspr_release %[%baserelease+1]
>>>>>>> hummingbird-local
REASONING: When updating to a new upstream nss version from Fedora:
- Accept Fedora’s
nspr_releaseoffset (HEAD) - they manage NVR clashes - Our local offset was specific to Hummingbird rebuilds
- New upstream version should reset to Fedora’s packaging values
- Don’t try to “calculate” what it should be - trust Fedora’s packaging
RESOLUTION: Accept HEAD (Fedora’s value)
Special Case: Rebuild-Only Changes
Release-only changes (no-change rebuilds) are automatically ignored by the modification detection logic. This means:
- Bumping
Release: 3%{?dist}→Release: 3.1%{?dist}does not mark the package as modified - The package can still receive automatic Fedora updates
- The Release bump will be preserved if the update doesn’t change the upstream Release field
You do not need to mark packages as modified for rebuild-only changes, unless you want to explicitly prevent automatic updates for other reasons.
CI Validation
The CI pipeline validates modification status consistency using make check, which runs:
./ci/validate_package_modifications.py --all
This validation ensures:
- All packages have a
modification_statusfield - The value is one of:
clean,modified,independent - If
track_upstreamis present, it must be a string ("latest"or a version prefix) - Modified packages have a
modification_reason - Independent packages do not have source/branch/sha fields (Hummingbird-independent only)
- Git commit history matches the declared modification status
The validation runs on every merge request and push to main, failing the build if metadata is inconsistent.
For local development, run the full validation:
./ci/validate_package_modifications.py --all
Or validate specific packages:
./ci/validate_package_modifications.py bash glibc gcc
Validation Modes
The validation script has two modes:
Fast mode (default): Checks git commit history patterns
./ci/validate_package_modifications.py --all
This validates that all commits since the last Sync follow standard patterns (have Upstream: trailers). Runs in less than a minute for all packages.
Thorough mode: Clones upstream repos and compares filesystems
./ci/validate_package_modifications.py --all --thorough
This performs full filesystem comparisons with upstream Fedora repositories. Slow and unreliable (hundreds of upstream dist-git clones) but authoritative - validates actual state regardless of git commit history.
For CI and daily development, fast mode is sufficient. Use thorough mode when:
- Debugging discrepancies between metadata and actual state
- Auditing the entire repository for hidden modifications
- Investigating why a package can’t be updated
Workflow Examples
Backporting a Patch
-
Add patch file and modify spec (see Rebuilding Packages)
-
Commit the changes
-
Mark as modified:
./ci/dist_git.py mark-modified dnf5 --modified \ --reason "Backport reproducible build fix (upstream PR#2522)" -
Package is now protected from automatic Fedora updates
Re-enabling Auto-Updates
When your backported fix lands in Fedora:
-
Verify the fix is in the latest Fedora version:
./ci/dist_git.py update dnf5 # This will fail with "modified" error -
Mark the package clean:
./ci/dist_git.py mark-modified dnf5 --clean -
Update from Fedora:
./ci/dist_git.py update dnf5 # Now succeeds
Importing New Packages
When importing packages, modification status is set automatically:
# Fedora package → marked as "clean"
./ci/dist_git.py import fedora/neofetch
# Hummingbird-independent package → marked as "independent"
./ci/dist_git.py import hummingbird/custom-tool
No manual marking needed for imports.
Troubleshooting
CI Fails: “Missing modification_status field”
This means a metadata file is missing the required field. This means that the package was not imported properly.
CI Fails: “Marked as clean but package has modifications”
The package has local changes but metadata says it’s clean. To fix:
-
Check what changed:
git log -p -- rpms/<package>/ -
Mark as modified with the appropriate reason:
./ci/dist_git.py mark-modified <package> --modified --reason "..."
CI Fails: “Marked as modified but package is actually clean”
The package has no local changes but is marked modified. To fix:
-
Verify it’s actually clean:
./ci/dist_git.py update <package> # Check if upstream matches -
If confirmed clean, remove the modified status:
./ci/dist_git.py mark-modified <package> --clean
Update Blocked: “Cannot auto-update <package>”
This is expected for modified packages. Options:
-
Wait for fix to land in Fedora, then mark clean and update
-
Force-sync to discard local changes:
./ci/dist_git.py sync <package> -
Keep blocked if the local changes are still needed
Related Documentation
- Package Metadata Fields -
modification_statusandreleaseconfiguration - Rebuilding Packages - How to rebuild and backport patches
- Updating Dist-git Packages - How automatic updates work
3.24 - Private RPM Repositories
AI Agent Note: Assign a private product through
hum.meta, which savesprivate_productinmetadata/<package>.json, then runmake generate. The product must match an existing RPA entry inci/konflux_rpa_config.yml.
Overview
Hummingbird LTS products (e.g., LTS OpenJDK, LTS DotNet) require subscription-gated RPMs that must not be publicly accessible. Each LTS product gets its own private Pulp repository with independent credentials, enabling per-product subscription enforcement.
The private RPM pipeline uses the same build infrastructure as public RPMs but routes packages to separate Pulp domains via dedicated Konflux Applications and ReleasePlanAdmissions (RPAs).
How it works
- Each private product has a dedicated Konflux Application (e.g.,
private-<product>-rpms-main) separate from the publicrpms-mainapplication. - A ReleasePlan connects the private Application to its RPA, enabling auto-releases when builds complete.
- Packages are assigned to a private product via
private_productinmetadata/<package>.json, read and written throughhum.meta. - A dedicated RPA per product routes those packages to private Pulp repositories.
- Packages with
private_productare automatically excluded from the public RPA.
Architecture
metadata/<package>.json konflux_rpa_config.yml
"private_product": "<product>" rpas:
- name: ...public...
- name: ...private-<product>...
private_product: <product>
| |
v v
Component resource ReleasePlanAdmission
application: private- targets: private-<product>-rpms-main
<product>-rpms-main pulp_signed_domain: private-hummingbird-<product>
^
|
ReleasePlan
application: private-<product>-rpms-main
releasePlanAdmission: hummingbird-rpms-private-<product>
Assigning a Package to a Private Product
Set private_product through hum.meta for an existing product:
make container <<'SH'
python3 - <<'PY'
from pathlib import Path
from hum.meta import MetadataRepository
repo = MetadataRepository(Path.cwd())
metadata = repo.load("<package>")
metadata.private_product = "<product>"
repo.save(metadata)
PY
SH
This updates metadata/<package>.json without changing the package’s state. The library checks
that the product has a matching RPA entry. Do not infer private or LTS status from a package-name
suffix, and do not add assignments to ci/package-overrides.yaml.
Then regenerate all resources:
make generate
This will:
- Set the package’s Konflux Component to use
application: private-<product>-rpms-main - Include the package in the private product’s RPA component list
- Automatically exclude the package from the public RPA
Verifying the assignment
After running make generate, verify the changes:
# Check the component's application assignment
grep -A5 '<package>-main' konflux-templates/rendered.yml | grep application
# Check the package appears in the private RPA
grep '<package>-main' releng/hummingbird-rpms-private-<product>.yaml
# Check the package is excluded from the public RPA
grep '<package>-main' releng/hummingbird-rpms-tech-preview-staging.yaml
# (should return no results)
Onboarding a New Private Product
Automated onboarding
Use the add-private-product subcommand to automate Pulp setup, RPA configuration,
package assignment, and resource regeneration in a single step:
./ci/dist_git.py add-private-product <product> \
--packages pkg1,pkg2 \
--infra-repo <path-to-infrastructure-repo> \
--pulp-config <path-to-cli.toml>
This handles Pulp infrastructure (step 2), RPA config (step 5), package assignment (step 6), infrastructure repo templates (step 4 file creation), and resource regeneration. You still need to manually complete: credentials secret deployment (step 3), merging the infrastructure repo MR created by this tool (step 4), and content guard setup (step 7).
Use --dry-run to preview changes without modifying anything.
Manual onboarding
Adding a new LTS product (beyond existing ones) requires these steps: service account, Pulp infrastructure, credentials secret, Konflux Application, RPA configuration, resource regeneration, and content guard setup.
1. Create a Pulp service account (optional)
If you want isolated credentials for private repo operations (recommended), create a dedicated service account before setting up the Pulp infrastructure. See Pulp Access for instructions on creating a service account, configuring the CLI, and storing credentials in the vault. Otherwise, you can reuse the existing public Pulp credentials.
2. Create Pulp infrastructure
Use the existing Pulp setup script to create the private domains, RPM repositories, and
file repositories (for SBOMs and attestations). See
ci/pulp-setup/README.md for the required
pulp-cli-console plugin install before running these commands:
# Unsigned (staging) - RPM repos
./ci/pulp-setup/create-pulp-resources.sh \
--domain private-hummingbird-<product>-unsigned
# Unsigned (staging) - file repos (SBOM/attestations)
./ci/pulp-setup/create-pulp-resources.sh \
--domain private-hummingbird-<product>-unsigned \
--type file
# Signed (production) - RPM repos
./ci/pulp-setup/create-pulp-resources.sh \
--domain private-hummingbird-<product>
# Signed (production) - file repos (SBOM/attestations)
./ci/pulp-setup/create-pulp-resources.sh \
--domain private-hummingbird-<product> \
--type file
If using a dedicated service account, add --config <path-to-cli.toml> to each command.
This creates per-architecture RPM repositories (source, x86_64, s390x, ppc64le,
aarch64) with distributions, and file repositories (metadata, rpm-catalog) for
SBOM and attestation storage.
3. Deploy the Pulp credentials secret
If using a dedicated service account (step 1), add a Kubernetes Secret template to the infrastructure repo so the credentials are deployed to the Konflux cluster:
-
Create
kubernetes/setup-konflux/47-pulp-private-hummingbird-config-file-secret.yml.j2:--- kind: Secret apiVersion: v1 metadata: name: pulp-private-hummingbird-config-file-secret stringData: cli.toml: «{ lookup_secret("HUMMINGBIRD_PRIVATE_PULP_BOT_CONFIG_FILE[deployed]:cli.toml") | forceescape }» type: Opaque -
Add an entry to
secrets.ymlfor the vault key:HUMMINGBIRD_PRIVATE_PULP_BOT_CONFIG_FILE: backend: hv meta: active: true created_at: "<timestamp>" deployed: trueThe secret name (
pulp-private-hummingbird-config-file-secret) must match thepulp_secret_namein the private RPA configuration (step 5). -
Add an ExternalSecret to the konflux-release-data repo (
rhtap-release-data) so the credentials are available to the release pipeline. Create the ExternalSecret in the cluster-specific directory (e.g.,tenants-config/cluster/kflux-prd-rh03/managed/rhtap-releng-tenant/):--- apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: name: hummingbird-private-pulp-bot-production-secret annotations: argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true argocd.argoproj.io/sync-wave: "0" spec: dataFrom: - extract: key: releng/konflux/rhtap-releng-tenant/public-network/hummingbird-private-pulp-bot-production refreshInterval: 1h secretStoreRef: kind: SecretStore name: releng-vault target: creationPolicy: Owner deletionPolicy: Delete name: hummingbird-pulp-credentials-private-production-secretAdd it to the cluster’s
kustomization.yamland regenerate auto-generated files withtenants-config/build-manifests.sh. -
Add the secret to the releng vault at
https://vault.devshift.netunder the pathreleng/konflux/rhtap-releng-tenant/public-network/hummingbird-private-pulp-bot-productionwith the following keys:Key Value cli.tomlFull contents of the Pulp CLI config file expires-onnaownerYour Kerberos username
4. Create the Konflux Application
The Application resource is managed in the infrastructure repo, not this repo. Create it
there following the same pattern as rpms-main:
-
Create a new directory
kubernetes/private-<product>-rpms-main/in the infrastructure repo. -
Add
00-application.yml.j2with the standard Application template:--- apiVersion: appstudio.redhat.com/v1alpha1 kind: Application metadata: name: {{ env["PROJECT_NAME"] }} spec: appModelRepository: {url: ""} displayName: {{ env["PROJECT_NAME"] }} gitOpsRepository: {url: ""} -
Add the new project to the CI matrix in
infrastructure/.gitlab-ci.yml. Find thePROJECT_NAMElist under thekonflux-rh03/hummingbird-tenantcontext and add the new application name:- PROJECT_NAME: - rpms-main - private-<product>-rpms-main # add this line PROJECT_CONTEXT: - konflux-rh03/hummingbird-tenant -
Add
01-release-plans.yml.j2with a ReleasePlan that references the private RPA. Copy fromkubernetes/rpms-main/01-release-plans.yml.j2and update thereleasePlanAdmissionlabel to match the private RPA name:metadata: name: hummingbird-rpm-release-private-<product> labels: release.appstudio.openshift.io/auto-release: "true" release.appstudio.openshift.io/standing-attribution: "true" release.appstudio.openshift.io/releasePlanAdmission: hummingbird-rpms-private-<product> spec: application: {{ env["PROJECT_NAME"] }} target: rhtap-releng-tenant # ... copy remaining spec from rpms-main/01-release-plans.yml.j2The ReleasePlan connects the private Application to its ReleasePlanAdmission. Without it, builds in the private Application will not trigger releases.
The IntegrationTestScenario for the new private application is generated automatically by this repo’s
ci/generate_resources.py— it derives the application list fromci/konflux_rpa_config.yml. Once you add the RPA entry in step 5 and runmake generate, the new application’s testing-farm ITS will appear inkonflux-templates/rendered.ymland the trigger manifest. The ITS is deployed to the cluster via the*-releasesinfrastructure sync after this MR merges to main.
Merge this MR in the infrastructure repo first — the CI pipeline will deploy the Application and ReleasePlan to the cluster. Components and RPAs in this repo reference it by name, so the Application must exist before they are applied.
5. Add RPA configuration
Add a new entry to the rpas: list in ci/konflux_rpa_config.yml:
rpas:
# Existing public RPA (private packages are automatically excluded)
- name: hummingbird-rpms-tech-preview-staging
# ... existing config ...
# New private product RPA
- name: hummingbird-rpms-private-<product>
application_prefix: private-<product>-rpms
release_org: registry.stage.redhat.io/hummingbird-tech-preview
single_component_mode: true
service_account_name: hummingbird-rpm-release-staging
pulp_unsigned_domain: private-hummingbird-<product>-unsigned
pulp_signed_domain: private-hummingbird-<product>
pulp_secret_name: hummingbird-pulp-credentials-private-production-secret
pipeline_revision: <release-pipeline-branch>
pipeline_url: https://github.com/scoheb/release-service-catalog.git
private_product: <product>
component_filter:
path_prefix: rpms/
Key fields:
| Field | Purpose |
|---|---|
name |
Kubernetes resource name for the RPA |
application_prefix |
Combined with branch to form the Konflux Application name |
private_product |
Matches the product assignment in metadata/<package>.json |
pulp_unsigned_domain |
Pulp domain for unsigned RPMs (staging) |
pulp_signed_domain |
Pulp domain for signed RPMs (production) |
pulp_secret_name |
Kubernetes secret containing Pulp publishing credentials |
6. Assign packages and regenerate
Assign each package through hum.meta as described in Assigning a package, then
regenerate:
make generate
This produces:
- Updated
konflux-templates/rendered.ymlwith per-component application assignments - A new RPA file at
releng/hummingbird-rpms-private-<product>.yaml - Updated public RPA excluding private packages
7. Set up Pulp content guard
To restrict access to the private Pulp repositories, a content guard must be configured so only customers with the correct subscription can access the content:
- Obtain a SKU for the private product’s subscription offering.
- Create a feature for that SKU in the Feature service.
- Contact the Pulp team to:
- Map the organization ID to the feature
- Set up the content guard on the private Pulp domain(s)
This step is required before customers can access the private repositories. Without it, the repositories are created but have no access control.
Pulp credentials
If using a dedicated service account (step 1), see Pulp Access for vault storage instructions.
If using the shared service account, private RPAs reuse the same Pulp publishing credentials
as the public RPA (hummingbird-pulp-credentials-production-secret). Per-product access
control for customers is handled downstream by Red Hat’s subscription entitlement system,
not at the Pulp publishing layer.
Public/Private Routing
Packages with private_product set in metadata/<package>.json are automatically excluded from
the public RPA and included only in their product’s private RPA. Set metadata.private_product
to None and save through hum.meta to return the package to public-only publishing.
Related Files
| File | Purpose |
|---|---|
metadata/<package>.json |
Per-package private_product assignment |
ci/konflux_rpa_config.yml |
RPA definitions (public + private) |
ci/generate_resources.py |
Generates Components, RPAs, and PipelineRuns |
ci/pulp-setup/create-pulp-resources.sh |
Creates Pulp domains and repositories |
konflux-templates/macros/releng/release-plan-admission.yml.j2 |
RPA template |
konflux-templates/macros/component.yml.j2 |
Component template (per-component application) |
konflux-templates/macros/image-repository.yml.j2 |
ImageRepository template (per-component application) |
3.25 - Reporting CVE Data Issues
Overview
Sometimes CVE data published on cve.org or NIST NVD is incorrect – wrong affected package, wrong severity score, wrong affected version range, or missing fix information. When this happens, the incorrect data propagates into vulnerability scanners and can cause false positives (or false negatives) for Hummingbird container images.
This document describes how to report these issues to get the data corrected at the source.
When to Report
Report a CVE data issue when you find any of the following:
- Wrong affected package – the CVE Record lists a package that is not actually affected
- Wrong version range – the affected or fixed version boundaries are incorrect
- Wrong severity – the CVSS score or severity rating does not match the actual impact
- Missing fix information – a fix exists upstream but the CVE Record does not reflect it
- Wrong CPE match – the NVD CPE configuration matches products that are not affected
How to Report
Reporting to the CVE Program (cve.org)
The CVE Program accepts corrections through the CVE Numbering Authority (CNA) that owns the record.
- Find the CVE Record at
https://www.cve.org/CVERecord?id=CVE-YYYY-XXXXX. - Identify the CNA listed in the record (shown in the “Assigning CNA” field).
- Contact the CNA directly to request a correction. Most CNAs accept reports via:
- Their security reporting email (often listed in their CNA page)
- GitHub issues if the CNA is an open source project
- If the CNA is unresponsive, use the CVE Program Request form to dispute the record.
Reporting to NIST NVD
NIST NVD enriches CVE Records with CVSS scores and CPE match data. To request corrections:
- Navigate to the NVD entry at
https://nvd.nist.gov/vuln/detail/CVE-YYYY-XXXXX. - Click the “Contact” link or email
nvd@nist.govwith:- The CVE ID
- The specific field that is incorrect
- Evidence of the correct data (links to upstream commits, release notes, etc.)
Reporting to Red Hat Product Security
If the incorrect CVE data is causing issues in Red Hat’s vulnerability tracking:
- File a Jira ticket in the HUM project with the CVE ID and a description of the data issue.
- Red Hat Product Security can update the VEX feed to reflect the correct disposition for Hummingbird, even before the upstream CVE data is corrected.
Tracking the Correction
After reporting, track the status of the correction:
- CVE Record updates are published at
https://www.cve.org/CVERecord?id=CVE-YYYY-XXXXX - NVD updates appear at
https://nvd.nist.gov/vuln/detail/CVE-YYYY-XXXXX(NVD may take days to weeks to process corrections) - Update the HUM Jira ticket with the correction status so the team is aware
Requesting information on the fix status of an existing CVE
File a support request at https://access.redhat.com/support.
3.26 - Lookaside Cache Access
Overview
Source tarballs for RPM packages are stored in an S3-based lookaside cache rather than in git. To upload files to the cache (e.g., when updating a package to a new upstream version), you need AWS credentials with the appropriate permissions.
The ci/upload-to-lookaside-cache.sh script handles uploads directly, and
ci/check_upstream_versions.py --update calls it automatically when downloading new source
archives.
Prerequisites
You must be a poweruser in the arr-cloud-aws-core group
(it-cloud-aws-727920394381-poweruser). Request access to this group if you do not already have it.
Obtaining AWS Credentials
There are two ways to authenticate.
Option A: Browser-based login
Run aws login, which opens a browser for authentication:
aws login
Option B: Kerberos-based login via container
Use the CKI tools container to obtain credentials via Kerberos:
$ podman run --rm -it \
-e KRB5CCNAME=FILE:/tmp/krb5cc_$(id -u) \
-v $HOME/.aws:/cki/.aws:U,Z \
quay.io/cki/cki-tools:latest
bash-5.2# kinit <userid>@IPA.REDHAT.COM
bash-5.2# AWS_IDP_URL=https://auth.redhat.com/auth/realms/EmployeeIDP/protocol/saml/clients/itaws \
cki_aws_login --duration 43200 \
--account 727920394381 --role poweruser
Replace <userid> with your Kerberos user ID. This writes credentials to $HOME/.aws on the host
(bind-mounted into the container).
Note: This overwrites the AWS default profile credentials. If you use the default profile for
other purposes, back up $HOME/.aws/credentials before running this command.
Uploading Files
Once credentials are configured in $HOME/.aws, you can upload files to the lookaside cache.
Manual upload
./ci/upload-to-lookaside-cache.sh -f <file> -p <package>
Example:
./ci/upload-to-lookaside-cache.sh -f rpms/tar/tar-1.35.tar.xz -p tar
Automated upload via version checker
check_upstream_versions.py --update downloads new source archives and uploads them to the
lookaside cache automatically:
./ci/check_upstream_versions.py check --update <package>
The check subcommand only processes packages with "track_upstream": true in their metadata when
no explicit package arguments are given. To see all packages and their upstream status, use
check_upstream_versions.py list. See
Package Modification Tracking for how to enable tracking.
See Also
- Adding Independent Packages - Adding new packages with source tarballs
- Rebuilding Packages - Rebuilding existing packages
3.27 - Upstream Diff Analysis
Overview
The upstream diff analysis tool classifies locally modified RPM packages by how their changes relate to upstream Fedora. This helps the team systematically decide which modifications should be proposed upstream, which are Hummingbird-specific, and which need more complex handling.
The primary interface is the /upstream-diff Claude Code skill, which orchestrates the full
workflow. The skill delegates all deterministic operations (diffing, metadata lookup, caching) to
ci/upstream_diff.py and applies its own judgment only for the classification step.
Classification Categories
Each modified package is classified into one of five categories:
| Category | Meaning | Action |
|---|---|---|
upstreamable |
Changes can be submitted to Fedora as-is | File upstream PR |
hummingbird-specific |
Changes are intentional and specific to Hummingbird | Keep locally, no upstream action needed |
complex |
Changes need investigation or partial upstreaming | Requires human decision |
mixed |
Some changes are upstreamable, others are Hummingbird-specific | Split and handle separately |
no-diff |
Package is marked modified but has no actual diff | Consider unmarking as modified |
Workflow
Invoke the skill in Claude Code with /upstream-diff. When called with no arguments, it offers
three modes:
Analyze
Prepares the next batch of unanalyzed or stale packages, reviews each diff and any upstream PR activity, then classifies each package into a category with reasoning and a recommended action. Results are cached locally so subsequent runs skip already-analyzed packages.
You can also target specific packages by name (e.g., /upstream-diff analyze bash glibc).
View Results
Displays the cached analysis as a summary table. Supports filtering by category and showing details for a single package. Use this to get a quick overview of where things stand across all modified packages.
JIRA
Creates or updates a JIRA issue for a specific package to track the upstream proposal. The skill generates a pre-filled description and analysis comment from the cached data, previews it for confirmation, and then files or updates the issue. The JIRA issue key is recorded in the cache so it appears in the view output.
Script Reference
The skill uses ci/upstream_diff.py under the hood. The script can also be called directly for
automation, scripting, or when working outside Claude Code.
Note that prepare and save are two halves of a classification pipeline: prepare gathers raw
data (diffs, metadata, upstream PRs, category definitions), but its output requires human or LLM
judgment to select a category and write the reasoning fields before calling save. The skill’s
Analyze mode bridges this gap automatically.
Prepare
Gather metadata, diffs, upstream PR status, and the classification schema — this is the data the skill’s Analyze mode feeds into its classification step:
# Specific packages
./ci/upstream_diff.py prepare bash glibc
# Next batch of unanalyzed packages
./ci/upstream_diff.py prepare --batch 10
Save
Store a classification result. The skill’s Analyze mode calls this after classifying each package:
./ci/upstream_diff.py save <package> \
--category <category> \
--changes-summary "One-line summary of changes" \
--reasoning "Why this category was chosen" \
--recommendation "Recommended next action" \
--upstream-prs <pr_id1> <pr_id2> ...
# No related upstream PRs
./ci/upstream_diff.py save <package> \
--category hummingbird-specific \
--changes-summary "Custom Hummingbird macros for FIPS" \
--reasoning "FIPS build flags are specific to Hummingbird" \
--upstream-prs ""
The --hummingbird-macros flag can be added when changes use Hummingbird-specific RPM macros.
Results are cached in .cache/upstream-diff-analysis.json.
View
Display cached results — this is what the skill’s View Results mode calls:
# Summary table
./ci/upstream_diff.py view
# Single package detail
./ci/upstream_diff.py view <package>
# Filter by category
./ci/upstream_diff.py view --category upstreamable
# Include unanalyzed modified packages
./ci/upstream_diff.py view --all
# Markdown-formatted links (for JIRA or rendered contexts)
./ci/upstream_diff.py view --markdown
# Raw JSON
./ci/upstream_diff.py view --json
Check Upstream PRs
Check whether local changes have already been submitted or merged in the upstream Fedora dist-git repository:
./ci/upstream_diff.py check-prs <package1> <package2> ...
Shows open and recently merged (last 90 days) pull requests from the upstream Fedora dist-git repo.
Update Related PRs
After initial analysis, update which upstream PRs are related to a package’s local changes:
# Set related PRs
./ci/upstream_diff.py save --set-upstream-prs <package> <pr_id1> <pr_id2> ...
# Clear related PRs
./ci/upstream_diff.py save --set-upstream-prs <package>
JIRA Templates
Generate pre-filled JIRA content from cached analysis — this is what the skill’s JIRA mode uses:
./ci/upstream_diff.py jira-template <package> [--epic HUM-1613]
Record a JIRA issue key after creating an issue:
./ci/upstream_diff.py save --set-jira <package> HUM-XXXX
Related Documentation
- Package Modification Tracking — how packages are marked modified and how diffs are computed
- Rebuilding Packages — how to rebuild and backport patches
3.28 - GPG Source Verification
Overview
Some packages verify the authenticity of their upstream source tarball with a detached OpenPGP
signature where upstream signs each release with a private key and publishes a .asc/.sig alongside
the tarball. During source fetching, the
gorget source-pipeline tool checks that signature against the
project’s public key before the bytes are ever used in a build.
The trusted public keys live centrally in metadata/gpg-keys/<project>.gpg, one keyring file
per upstream project. Centralized storage means:
- the full set of signers we trust is auditable in one directory, and
- rotating or revoking a key is a single, reviewable commit.
The directory is handed to gorget as --gpg-keys-dir metadata/gpg-keys (see
ci/check_upstream_versions.py), so a pipeline’s keyring: field is just a filename within it.
How the pieces fit together
Three things wire up verification for a package:
-
The keyring —
metadata/gpg-keys/<project>.gpg, the trusted public key(s). -
The pipeline step — a
verify:entry inmetadata/<package>.source-pipeline.yaml:fetch: - type: url url: "https://curl.se/download/curl-${VERSION}.tar.xz" - type: url url: "https://curl.se/download/curl-${VERSION}.tar.xz.asc" verify: - type: gpg-signature target: "curl-${VERSION}.tar.xz" # the artifact to verify signature: "curl-${VERSION}.tar.xz.asc" # its detached signature keyring: "curl.gpg" # filename in metadata/gpg-keys/gorget imports
keyringinto a fresh, throwaway GPG homedir per check, then runs the equivalent ofgpg --verify <signature> <target>. A bad or missing signature fails the fetch. -
CI validation —
test/test_gpg_keys.py(run bymake check) confirms every file inmetadata/gpg-keys/is a parseable public key, and that everykeyring:referenced by a pipeline actually exists.
Adding a key for a new package
Prerequisite: gpg (from the gnupg2 package). It ships in the CI image, run the
commands below inside a container, e.g.
podman run --rm -it -v "$PWD:$PWD:z" -w "$PWD" quay.io/hummingbird-ci/gitlab-ci:latest bash.
-
Obtain the upstream public key. Prefer a key you can already trust: many packages already ship the maintainer’s key next to their spec (e.g.
rpms/curl/mykey.asc,rpms/bash/chet-gpgkey.asc). Otherwise download it from the project’s official key page. -
Store it as a keyring in
metadata/gpg-keys/. The keyring may be ASCII-armored or binary; this repo standardizes on binary.gpg. Convert an armored key with--dearmor:gpg --dearmor < rpms/<package>/<upstream-key>.asc > metadata/gpg-keys/<project>.gpgName the file after the upstream project, not the RPM (so multiple versioned packages, e.g.
python3.11/python3.12, can share one keyring). -
Verify the key is what you expect. Print its fingerprints and confirm they match the fingerprints published on the upstream’s official channel:
gpg --show-keys --with-fingerprint metadata/gpg-keys/<project>.gpg -
Wire up the pipeline. Add (or extend)
metadata/<package>.source-pipeline.yamlwith thefetchsteps for the tarball + signature and theverify: [{type: gpg-signature, ...}]step shown above. -
Prove the whole chain end to end before committing import the keyring into a throwaway homedir and verify a real release signature against it:
V=<version> curl -fsSLO "https://<upstream>/<tarball>-$V.tar.xz" curl -fsSLO "https://<upstream>/<tarball>-$V.tar.xz.asc" export GNUPGHOME=$(mktemp -d) gpg --import metadata/gpg-keys/<project>.gpg gpg --verify "<tarball>-$V.tar.xz.asc" "<tarball>-$V.tar.xz" # expect "Good signature" -
Run the checks:
make check(validates the keyring and the pipeline reference).
Rotating or replacing a key
Upstream may roll to a new signing key (expiry, policy, new maintainer). Because the trusted set is just files in one directory, rotation is a single commit:
-
Obtain the new public key from the upstream’s official channel and confirm its fingerprint out of band.
-
Replace the contents of
metadata/gpg-keys/<project>.gpg(re-run the--dearmorstep). If upstream signs a transition period with both keys, you may keep both by importing them into the same keyring:gpg --dearmor < old-key.asc > metadata/gpg-keys/<project>.gpg gpg --dearmor < new-key.asc >> metadata/gpg-keys/<project>.gpg -
Re-run the end-to-end verification (step 5 above) against the latest release, then
make check. -
Commit with a message recording why the key changed and how you confirmed the new fingerprint, this file is the audit trail for what we trust.
Revoking / removing a key
- If a project drops GPG verification, delete both
metadata/gpg-keys/<project>.gpgand theverify:step from its pipeline in the same commit. (CI fails a pipeline that references a missing keyring, and if the key is orphaned the reverse is easy to spot.) - If a key is compromised, remove it immediately and replace it with the upstream’s revocation / replacement key.
Troubleshooting
gpg: no valid OpenPGP data found— the file isn’t a real keyring (empty, truncated, or you saved an HTML error page). Re-download and re---dearmor.test_gpg_keys.pycatches this in CI.gpg: Can't check signature: No public key— the signature was made with a key that isn’t in the keyring. Upstream likely rotated keys, follow Rotating or replacing a key above.BAD signature— the tarball does not match the signature. Do not paper over this. It means a corrupted download or, in the worst case, tampering. Re-fetch from the canonical source and if it persists, escalate rather than accepting the artifact.
3.29 - Rebasing the Buildroot to a New Fedora Release
AI Agent Note: This document is the procedural runbook for a Fedora buildroot rebase (e.g. F43 -> F44). It was written after executing the F44 rebase (see HUM-2018 and its child stories HUM-3369, HUM-3370, HUM-4536, HUM-4537 for the detailed history) and should be followed, and updated, the next time this operation is performed (e.g. F44 -> F45). Read this whole document before starting; the ordering between steps matters and several steps have hard dependencies on the one before it.
Overview
Periodically, Fedora ships a new stable release and Hummingbird needs to move its buildroot
(mock/mock.cfg) and the packages that track a specific Fedora branch (rather than rawhide) onto
it. This is a multi-day, multi-phase effort touching the core toolchain, CI/testing configuration,
and potentially every package in the repository (since changing the compiler/linker toolchain can
affect any build). The F44 rebase took about 2 calendar weeks.
Pre-flight checks
Before starting, confirm:
-
The target Fedora release is actually stable (not just branched) and its package repos are reachable from the build environment.
-
Which packages currently track the outgoing release branch explicitly (i.e.
branchis a versioned Fedora branch likef43, notrawhide) via:# List every package whose branch is pinned to a specific Fedora release (not rawhide) ./ci/dist_git.py list | xargs -I{} sh -c 'echo -n "{}: "; python3 -c "import json; print(json.load(open(\"metadata/{}.json\"))[\"branch\"])"' 2>/dev/null | grep -E ': f[0-9]+$'For each, decide up front whether it should move to the new branch, move to
rawhide, or stay pinned (see “Packages that intentionally stay behind” below) — don’t leave this undecided until a package fails to build. -
Whether any of those packages carry FIPS-validation-critical modifications. Packages whose
modification_reasonreferences FIPS certification should generally not be rebased to new branch content, even though they still need to be rebuilt against the new toolchain. Moving them would risk invalidating the FIPS-validated source. (See HUM-4989 for a related, more general proposal to record why a package is pinned in its metadata rather than relying on the modification reason text.) -
Whether any package whose version is the “which version is default” signal for an adjacent language/tooling ecosystem (e.g.
python-rpm-macros) needs the same branch-pinning treatment as the interpreter itself — especially if that ecosystem is mid-transition to a new major version upstream at the same time as this rebase. See “Known gotchas: python3dist / rawhide-tracked toolchain-adjacent packages” below for why this matters and what already went wrong once. -
Whether to pause the routine automated dist-git sync bot for the duration of the Step 4 mass rebuild. See “Known gotchas: automation racing a manual mass rebuild” below for the tradeoff.
Step 1 — Buildroot config + core toolchain (separate MRs)
Do not bundle the buildroot config change with the toolchain rebuilds in one MR — split them so the buildroot config lands and validates quickly, independent of the (much slower) toolchain rebuilds.
mock/mock.cfg: updatedist,releasever,bootstrap_image,description, and the[fedora]/[fedora-updates]repobaseurls and names to the new release. Merge this on its own or bundled only withfedora-repos(see next point) andbinutilsif it’s a clean (unmodified) package — do not bundleglibc/gcc/llvmhere.- Testing/CI fallback config (
rpms/ci/repos/fedora-43.repo→fedora-44.repo, andci/default-tests/tests-rpm.yml’s repo copy + GPG key import): these reference the buildroot’s Fedora version directly, and there’s exactly one of each — bump them in the same MR asmock.cfg(or immediately alongside it), not later. During the F44 rebase this was deferred and wasn’t caught untilphpfailed its install test days later in Step 4, against a fallback repo that still pointed at the outgoing release. (This is distinct from the per-package test fixture gotcha below, which recurs throughout every step — see “Known gotchas: package-specific test fixtures with hardcoded Fedora-version references.”) fedora-repos: rebase to the new release. If Fedora’s spec uses a conditional ELN-style macro forRelease:, evaluate whether to keep it or hardcode a plainRelease: N%{?dist}— this is a judgment call each time depending on what’s simplest to maintain; document whichever choice is made in the package’smodification_reason.binutils(if it carries local CVE backport patches): before assuming the patches still apply, check whether the CVEs they address have already been fixed upstream in the new Fedora version. If so:- Pull Fedora’s current spec as the new baseline (don’t try to merge patch-by-patch on top of the old spec).
- Re-add only the local patches that are not already fixed upstream.
- For a quick validation build, it’s sufficient to confirm the patch set applies and compilation begins — you don’t need to wait for a full build to complete just to validate the patch set is sane. Kill the build once compilation starts.
glibc: this is the highest-risk package in this phase because of a circular bootstrap dependency (see “Known gotcha: glibc/gcc bootstrap circularity” below). Expect this to require a temporary workaround, tracked as amodifiedpackage, to be reverted in Step 2.gcc: reapply any local modification (e.g. disabling non-essential language frontends) against the new spec. Expect this to be one of the longest-running builds in the whole rebase (multiple hours per architecture) — plan CI capacity accordingly.llvm: rebase and rebuild; this can run in parallel withgccand typically finishes faster.
Step 2 — Bootstrap cleanup
Once the new compiler (from Step 1) is published to Pulp:
- Rebuild
annobinagainst the new compiler. This must happen before the next step, or reverting the glibc bootstrap workaround will immediately re-trigger the same circular dependency. - Revert the glibc bootstrap workaround from Step 1, now that the new compiler and annobin are
both available. Clear the
modifiedstatus/reason that was recording the workaround. - Rebuild
libtoolagainst the new compiler. This is easy to miss (it was missed during the F44 rebase and had to be done out-of-band mid-mass-rebuild) but several packages (apr,audit,authselect,automake,avahi,bind,catatonit,cryptsetup,find, and likely others) will fail to build against the new toolchain without it. Do this here, not in Step 4.
Step 3 — Toolchain-adjacent, release-pinned packages
Rebase the remaining packages that were explicitly tracking the outgoing Fedora branch and don’t carry a permanent reason to stay behind (see “Packages that intentionally stay behind”). For each:
- Decide whether it should move to the new versioned branch (e.g.
f44) or torawhide— moving torawhidecan simplify build-dependency resolution for clean (unmodified) packages, but ties future updates to rawhide’s churn. - For packages with local modifications, check whether the modification is still needed (e.g. a security backport that may have landed upstream by now) before reapplying it blindly.
- Check
test/rpms/<pkg>.ymlfor a hardcoded Fedora-version reference before opening this package’s MR (see “Known gotchas: package-specific test fixtures with hardcoded Fedora-version references” below) — fix it in this same MR if present.
Step 4 — Mass rebuild of everything else
Rebuild every remaining package against the new toolchain so that linking, annobin annotations, and compiler hardening are consistent across the whole package set.
-
Build the exclusion list first: everything already rebuilt in Steps 1–3, plus anything merged in the hours immediately before starting this step (check recent merge history). Use
dist_git.py rebuild --all’s--excludeoption (HUM-5183):./ci/dist_git.py rebuild --all --exclude pkg1,pkg2,... --reason "<toolchain> toolchain rebuild"During the F44 rebase this flag didn’t exist yet, so the exclusion list had to be hand-rolled with
./ci/dist_git.py list | grep -v -E '^(pkg1|pkg2|...)$'piped intorebuild— no longer necessary now that--excludehas landed. -
Watch for packages whose
Release:field doesn’t fit the simple.Nbump pattern (e.g. packages using build-time macros forRelease:) — these will fail to commit cleanly and need to be handled in a separate pass. -
Submit in batches, not one giant MR wave — use
./ci/rebuild_multi_mr.sh --dry-runto preview, then./ci/rebuild_multi_mr.sh --max-updates=Nto submit and auto-merge-on-green in controlled increments. Plan for several days; a run of ~460 packages took about a week of elapsed time across 5 batches during the F44 rebase, gated by CI/Testing Farm throughput more than build time itself. -
Check
test/rpms/<pkg>.ymlfor a hardcoded Fedora-version reference before each package’s rebuild MR merges (see “Known gotchas: package-specific test fixtures with hardcoded Fedora-version references” below) — this is what letphp’s failure slip through undetected for days during the F44 rebase, so don’t rely on remembering it only once at the end. -
Expect a small number of packages to need bespoke fixes that a routine rebuild can’t resolve automatically — license metadata regressions surfaced by newer scanners, compiler/linker regressions in vendored dependencies (e.g. a
bindgenversion incompatible with a newer clang), confirmed upstream compiler-triggered bugs in JIT/JVM-style software, or stale version-gates in local patches that assumed an older toolchain. Budget time for a handful of these near the end of the mass rebuild.
Known gotchas
glibc/gcc bootstrap circularity
A new Fedora release’s glibc spec may use a feature of the new gcc (e.g. a new compiler flag),
while the new gcc may in turn require symbols only present in the new glibc. If Hummingbird’s
own Pulp repos still serve the old toolchain at a higher priority than the new Fedora repos during
the transition window, this becomes a real chicken-and-egg problem, not just a theoretical one.
The pragmatic fix used for F44 was to drop the new-toolchain-specific flag from glibc temporarily
(if it’s a defensive/hardening-only flag whose absence doesn’t break correctness), rebuild
glibc and gcc against each other with it disabled, then re-enable it once both are published
and annobin/libtool have also been rebuilt against the new compiler (Step 2 above).
Package-specific test fixtures with hardcoded Fedora-version references
Beyond the single global fallback config bumped in Step 1, some packages carry their own
test/rpms/<pkg>.yml test suite that independently hardcodes a Fedora repo/GPG-key reference for
setting up a test chroot (during F44 this was glib2, hummingbird-release, ca-certificates,
crypto-policies, and openssl). Don’t wait for a one-time catch-all audit to find these — fix
each one in the same MR as that package’s own rebuild/update, whenever it happens to land during
the rebase (Step 1, 3, or 4, whichever touches that specific package — see the reminders in each of
those steps above). Relying on an exhaustive upfront sweep of every test/rpms/*.yml is fragile and
is exactly what let php’s failure slip through undetected for days during the F44 rebase — its own
MR was fine, but a different package’s stale GPG key/repo reference broke shared test
infrastructure php depended on. When you open the rebuild/update MR for any package, grep
test/rpms/<pkg>.yml for the outgoing release string first.
See HUM-5184 for tracking a structural fix (single source of truth or a CI lint check) so this class of drift is caught automatically instead of relying on either of the above being remembered.
Automation racing a manual mass rebuild
The routine automated dist-git sync bot keeps running on its normal schedule regardless of an in-progress manual mass rebuild. If both touch the same packages in the same window, you’ll get a wave of merge-conflicted bot MRs that need manual cleanup — during F44 this was roughly 70 MRs at once. Decide before starting Step 4 (see “Pre-flight checks” above) whether to pause routine sync automation for the duration of the mass rebuild, or accept the cleanup cost.
External CI capacity constraints
Large rebuild waves can coincide with unrelated capacity crunches on shared infrastructure (e.g. Testing Farm). These are outside Hummingbird’s control but can stall a batch for a day; check external status pages when a batch is unexpectedly slow (for reference, the F44 rebase hit https://status.testing-farm.io/issues/2026-07-14-big-queue-in-red-hat-ranch/).
python3dist / rawhide-tracked toolchain-adjacent packages
If any packages central to a specific language ecosystem’s “what version is default” (e.g.
python-rpm-macros) are tracking rawhide rather than a pinned branch, a routine automated sync
during this window can silently pull in an experimental version bump (e.g. Fedora bootstrapping the
next language version) that corrupts auto-generated package metadata across dozens of downstream
packages without any build failing immediately. See HUM-4988 and HUM-4989 for the full incident
report and proposed guardrails from the F44 rebase. Decide during “Pre-flight checks” above whether
any “default version” declaration packages need the same branch-pinning treatment as the
interpreter itself before you start.
Packages that intentionally stay behind
Not every package needs to move to the new branch. Document (in modification_reason and/or the
epic tracking the rebase) any package that’s staying pinned to the old branch, and why — otherwise
this looks like unfinished work to the next person. Reasons seen so far:
- FIPS validation: the package carries a modification tied to FIPS certification, and moving to newer upstream content risks invalidating that certification. The package is still rebuilt against the new toolchain — it just doesn’t pick up newer Fedora source content.
- Package no longer exists upstream: Fedora sometimes drops a package entirely in a new release (e.g. an older LTS language runtime superseded by a newer one). If Hummingbird still needs to carry it, it has no newer branch to rebase to and simply stays on its last available branch, rebuilt against the new toolchain.
Reference: commands used during the F44 rebase
These are shown verbatim from the F44 rebase for concreteness. When reusing them for the next
rebase (e.g. F45), remember to substitute the actual target release everywhere F44 appears below
— including the exclusion list of already-rebuilt packages, which will differ each time.
# Preview a rebuild plan across all packages
./ci/rebuild_multi_mr.sh --dry-run
# Submit rebuild MRs in controlled batches, auto-merge on green
./ci/rebuild_multi_mr.sh --max-updates=50
# Rebuild everything except a known set of already-handled packages, using the
# --exclude flag (HUM-5183, landed after the F44 rebase — see Step 4 above)
./ci/dist_git.py rebuild --all --exclude glibc,gcc,llvm,annobin,binutils,fedora-repos,python3.13,python-gitlab --reason "F44 toolchain rebuild"
During the F44 rebase itself, --exclude didn’t exist yet, so the equivalent exclusion list had to
be hand-rolled:
./ci/dist_git.py rebuild $(./ci/dist_git.py list | grep -v -E '^(glibc|gcc|llvm|annobin|binutils|fedora-repos|python3.13|python-gitlab)$') --reason "F44 toolchain rebuild"
Related tickets
- HUM-2018 — F44 buildroot rebase epic (full history)
- HUM-3369 / HUM-3370 / HUM-4536 / HUM-4537 — Steps 1-4, each with a detailed what/why/fix writeup
- HUM-4988 / HUM-4989 — python-rpm-macros/python3dist regression incident and guardrails
- HUM-5182 —
bump_release()double-suffix bug - HUM-5183 —
dist_git.py rebuild --excludeoption (implemented) - HUM-5184 — generalize the testing-infra Fedora-version drift fix
4 - Lot's of background information
Background and explanation documentation for understanding Project Hummingbird systems and concepts.
4.1 - RPMs repository
Thank you for your interest in contributing! This repository contains RPM packaging, minimal tests, and CI plumbing used by Hummingbird to build, lint, and validate RPMs in containers and Testing Farm.
This guide is adapted from the Hummingbird containers contribution guide and aligned with our workflows and tooling. See the original for broader context: Hummingbird Containers CONTRIBUTING.
Code of Conduct
Be respectful and constructive. By participating, you agree to uphold a professional and inclusive environment.
Repository layout
rpms/<package>/– RPM dist-gits (spec, sources), mostly auto-importedci/– helper scripts and default testsbuild_rpms.sh– build RPMs using Mock in the Konflux-compatible containerrun_tests_rpm.sh– run rpmlint and install/rebuild tests in a containerdefault-tests/tests-rpm.yml– default tests included for all packagesrun_tests_rpm.fmf– tmt/Testing Farm entry point
konflux-templates/– Konflux/PAC resourcesmock/– mock configurationtest/rpms/<package>.yml– package-specific tests
Prerequisites
- Fedora or RHEL-like environment
- Podman
- rpmlint
- tmt (optional, for local TF-style runs)
- jq, python3, python3-yaml (used by
run_tests_rpm.sh)
Build locally
Build a package’s SRPM and RPMs using the Konflux-aligned environment:
./ci/build_rpms.sh <package_name>
# Results are written to: /tmp/konflux-build-<package_name>-*/results/
Building in Lima VM
When building inside a Lima VM (macOS users), use the --build-dir flag to specify a directory on
the VM’s native filesystem. This avoids permission issues with mock’s bootstrap process on macOS
mounts:
limactl shell fedora bash -c 'cd /path/to/repo && ./ci/build_rpms.sh --build-dir /tmp/rpm-build <package_name>'
The build directory must be on the VM’s native filesystem (not a macOS mount) to ensure Linux file
ownership, permissions, and symlinks work correctly with /var/lib/mock.
Interactive repository debugging
To investigate package/dependency/installability issues, you can run an interactive shell in the same environment used by the package builds:
./ci/build_rpms.sh --shell-before setup
This will show you the mock command that would be used to build the setup package, and drop you
into a shell in the same environment before executing the build, with dnf available. You can then
run the command manually and inspect the environment, repositories, and package metadata.
If you want it to actually build the local package first, so that you can include it in your
investigation, use --shell-after instead:
./ci/build_rpms.sh --shell-after setup
Test locally (containerized)
Run the default and package-specific tests against one or more built RPMs:
# Basic usage (binary rpm)
./ci/run_tests_rpm.sh --rpm /path/to/pkg-1.2-1.fcXX.x86_64.rpm <package_name>
# Include source RPM tests
./ci/run_tests_rpm.sh \
--rpm /path/to/pkg-1.2-1.fcXX.x86_64.rpm \
--src-rpm /path/to/pkg-1.2-1.fcXX.src.rpm \
<package_name>
# Test all RPMs in the build output directory after build_rpms.sh
./ci/run_tests_rpm.sh $(printf -- '--rpm %s ' builds/<package_name>/RPMS/*.rpm) \
--repo-dir builds/<package_name>/RPMS/ \
--src-rpm builds/<package_name>/SRPMS/*.src.rpm \
<package_name>
Notes:
- Tests run inside a Podman container.
- The test image defaults to
registry.access.redhat.com/hi/core-runtime:latest-builder. You can override viaTEST_IMAGE:
TEST_IMAGE=registry.access.redhat.com/hi/core-runtime:specific-tag ./ci/run_tests_rpm.sh --rpm /path/to/pkg.rpm <package_name>
- Container execution is performed as root with
HOME=/rootto avoid XDG state permission issues in dnf5.
Test in tmt/Testing Farm locally
You can drive the same FMF test locally with tmt. The FMF test expects an OCI artifact that contains
RPMs, referenced via IMAGE_URL (Testing Farm sets this automatically). For local trials you can
point to any compatible OCI artifact or skip the ORAS pull logic and directly call the script.
# If using an OCI artifact with RPMs
IMAGE_URL="oci://registry/namespace/artifact:tag_or_digest" tmt run -a
# Or run the script directly with built RPMs (bypassing the FMF wrapper)
./ci/run_tests_rpm.sh --rpm /path/to/pkg.rpm --src-rpm /path/to/pkg.src.rpm <package_name>
If you need longer time in Testing Farm, the FMF test includes duration, which you can adjust in
ci/run_tests_rpm.fmf.
Dist-git Imports
The ci/dist_git.py tool imports Fedora/CentOS dist-git packages into the rpms/ directory. We
expect most packages to not have (permanent) Hummingbird specific changes, so most of them will keep
syncing with upstream dist-gits. In most cases that will be Fedora rawhide, but for some packages we
may pick a different upstream, e.g. stable Fedora or even CentOS Stream.
The status of all imports is tracked in imports.json. Active Fedora releases are tracked in upstream-releases.json, which is updated from the Bodhi API.
Note that rawhide is automatically resolved to the highest numbered Fedora release at runtime and
is not stored in the JSON file.
See ./ci/dist_git.py --help for all available options. Some examples:
- Update upstream-releases.json from Bodhi API (should be done periodically, e.g., when new Fedora versions are released):
./ci/dist_git.py update-releases
This fetches the latest Fedora releases from Bodhi. The rawhide branch automatically resolves to
the highest numbered Fedora version (e.g., if f43 and f44 are available, rawhide uses f44).
- Import a new package. This requires specifying the dist-git URL (with
fedora/being a shortcut for the Fedora dist-git URL) and optionally a branch (default: rawhide):
./ci/dist_git.py import fedora/bash
./ci/dist_git.py import --branch f42 fedora/glibc
./ci/dist_git.py import --branch c10s https://gitlab.com/redhat/centos-stream/rpms/postfix.git
- Update all or a single package:
./ci/dist_git.py update
./ci/dist_git.py update bash
This only imports changes if these were actually built in Koji, to ensure we only import changes
which are meant to be released. You can disable this check with --skip-build-check.
- Re-sync a package to upstream, discarding any local modifications. We use this after Fedora adopted our change, or it is no longer relevant:
./ci/dist_git.py sync bash
All of these commands automatically commit changes with descriptive commit messages including the
upstream SHA. To avoid that, you can use the --dry-run option.
- Enable upstream version tracking for a package (used by
check_upstream_versions.py check). These commands modify the metadata file but do not create a git commit:
./ci/dist_git.py set-upstream bash --track
./ci/dist_git.py set-upstream bash --no-track
- Check upstream version status. The
checksubcommand checks tracked packages;listshows all packages in a table:
./ci/check_upstream_versions.py check
./ci/check_upstream_versions.py list
./ci/check_upstream_versions.py list --json
Package-specific overrides
Per-package build configuration can be customized in ci/package-overrides.yaml. If a package is
not listed, it uses default settings.
Available options:
| Option | Description | Default |
|---|---|---|
timeout_hours |
Build timeout in hours | 4 |
build_platforms |
List of MPLs (instance sizes) for multi-platform builds | Pipeline defaults |
Example configuration:
# Long-running package with custom timeout
setup:
timeout_hours: 12
# Package requiring larger build instances
llvm:
timeout_hours: 12
build_platforms:
- "linux-d160-c8xlarge/arm64"
- "linux-d160-c8xlarge/amd64"
All available MPLs can be found in the Konflux multi-platform builds documentation.
After modifying overrides, regenerate the pipeline files:
make generate
This updates .tekton/rpms-on-push.yaml and .tekton/rpms-on-pull-request.yaml with the new
configuration.
Branching and pull requests
- Create feature branches from the default branch.
- Keep edits focused and small; separate unrelated changes into separate PRs.
- Include meaningful commit messages (why + what). Reference related issues if applicable.
- PRs must pass CI (build + tests). Fix lint/test failures or mark known failures properly.
Commit message conventions
- First line: short imperative summary (≤ 72 chars)
- Body (optional): context, rationale, and user/ops impact
- Reference issues using standard notation (e.g. “Fixes: #123”)
Spec guidelines
- Don’t use
%changelogfor Hummingbird specific changes. It’s just a point of conflict with Fedora imports, commit messages are good enough. - Hummingbird specific changes increase
Release:number in steps of 0.1 (e.g. 2 → 2.1 → 2.2). This avoids colliding with Fedora’s Release number namespace and allows us to sync again to a future -3.
Packaging and testing guidelines
- Spec files should be reproducible and minimal.
- Prefer pinned container digests for CI images where feasible. If a tag and digest are both present
(
name:tag@sha256:<digest>), the digest is authoritative for content selection. - Default tests in
ci/default-tests/tests-rpm.ymlmust be fast, deterministic, and safe for all packages. - Package-specific tests (
rpms/<package>/tests-rpm.yml) can override or extend defaults. Keep them bounded in runtime and dependencies. - When possible, capture flaky or environmental issues under
known_issuesin test YAML with clear matching patterns and descriptions.
Style
- Shell: bash with
set -euo pipefail, readable variable names, and early returns where reasonable. - YAML: consistent indentation and quoting; prefer explicitness over magic.
- Comments: add when necessary to explain non-obvious rationale; avoid restating the code.
Security and supply chain
- Avoid embedding credentials or secrets; use environment variables or CI secret stores.
- Prefer pulling images by digest for stability and repeatability in CI.
- Validate inputs and sanitize any paths used by scripts.
Resolving CVEs
When manually fixing a CVE (e.g., adding a patch or performing a manual
update), a Fixes: CVE-YYYY-XXXX must be added to the MR description
and/or a commit. This will indicate to the build system that a CVE fix has
been submitted for review.
Reporting issues
Open an issue describing the problem, reproduction steps, and environment. Attach logs (build/test) when possible. If the failure is intermittent, call that out explicitly.
Licensing
Ensure files include appropriate licenses and that any third-party content is compatible with the project’s license.
Thank you
Your contributions make the Hummingbird RPMs better for everyone. We appreciate your time and feedback!
4.1.1 - RPM Pipeline
Explanation of the RPM build pipeline, from spec file changes through validation, building, signing, and publishing to the Hummingbird package repository.
Overview
The RPM pipeline consists of five main stages:
- Spec Change - A package spec file is modified in
rpms/<package>/ - MR Build & Validation - Konflux builds RPMs and required tests validate them before merge
- Promotion or Push Build - After merge, packages reuse the tested MR image by default;
explicit build exceptions build again on
main - Release & Signing - Integration Service creates the Release; the managed pipeline checks Conforma and signs the RPMs
- Publishing - Signed RPMs are uploaded to the Hummingbird Pulp repository
After publishing, updated RPMs are picked up by the container image pipeline via lockfile updates.
flowchart TD
A["Spec Change<br/>rpms/<package>/"] --> B["Merge Request"]
B --> C["MR CI + Konflux build<br/>required Testing Farm tests"]
C --> D["Merge to main"]
D --> E{"Promotion enabled?"}
E -->|Yes| F["Promotion PipelineRun<br/>reuse tested image digest"]
E -->|No| G["Konflux push build<br/>mock hermetic build per arch"]
F --> H["Automatic Release<br/>Conforma, signing"]
G --> H
H --> I["Publish signed RPMs to Pulp"]
I --> J["Container lockfile updates<br/>(see Image Pipeline)"]
Stage 1: Spec Change
An RPM change begins as a commit in rpms/<package>/. The change typically
includes a modified .spec file (updated version, new patch, or Release bump)
and any associated source files or patches.
How changes originate
| Method | Description | Modification Status |
|---|---|---|
| Automated Fedora sync | ci/dist_git.py update merges upstream Fedora changes |
No independent packages |
| Upstream version update | ci/check_upstream_versions.py --update bumps to new upstream release |
No independent packages |
| Manual patch backport | Developer adds a CVE patch and references it in the .spec |
Marked modified afterward |
| No-change rebuild | ci/dist_git.py rebuild <package> bumps the Release field |
Status unchanged |
| Reverse dependency rebuild | ci/dist_git.py rebuild-rev-deps <package> rebuilds all dependents |
Status unchanged |
See Rebuilding Packages and Updating Dist-git Packages for detailed workflows.
Package metadata
Each package has a metadata file at metadata/<package>.json that tracks its
relationship to upstream:
{
"source": "https://src.fedoraproject.org/rpms/curl.git",
"branch": "rawhide",
"sha": "abc123...",
"version": "8.12.1",
"release": "1",
"modification_status": "clean",
"upstream_repo": "https://github.com/curl/curl"
}
Key fields:
| Field | Purpose |
|---|---|
modification_status |
clean (auto-updates enabled), modified (auto-updates blocked), or independent (no upstream) |
modification_reason |
Explanation of local modifications (when modified) |
release |
Fedora release at last import/update (not the spec Release:), without dist tag; present only while shipping Fedora’s release |
upstream_repo |
Canonical upstream git repository URL |
track_upstream |
Version prefix constraint (e.g., "1.26" for golang1.26) |
version_transform |
Version mapping rule for CVE analysis (e.g., dotnet_sdk_to_runtime) |
cve_product |
CVE vendor/product override (string, or a list for multiple products; e.g. "Oracle Corporation / Oracle Java SE") |
See Package Metadata Fields for modification_status and release
configuration, and Package Modification Tracking for managing modification
status day to day.
Stage 2: Merge Request & Validation
Changes reach main via merge requests. Automated MRs are created using:
ci/create_mr.sh- Single MR with configurable optionsci/rebuild_multi_mr.sh- One MR per package with auto-merge enabledci/dist_git_update_multi_mr.sh- Automated Fedora update MRs (scheduled CI job)ci/upstream_update_multi_mr.sh- Automated upstream version update MRs (scheduled CI job)
The scheduled upstream-version update job creates normal MRs for successful package updates. If a package update fails after a new version has been identified, it creates (or reuses) a draft diagnostic MR instead. The diagnostic MR includes the original error but no partial package changes, and must be resolved manually before a package update can be proposed.
Both scheduled update jobs send a Slack completion message when SLACK_WEBHOOK_URL is configured:
success sends a plain success message, including when upstream warnings were logged, while failure
includes actionable package failures and links to MRs needing attention. Routine upstream warnings
remain in the job log and summary, not in Slack.
These completion messages also act as a heartbeat. The notification runs in the job’s after_script,
so a job that never starts, or a runner failure before after_script can run, may send no message.
A missing expected message is a signal to check the schedule and runner availability; it is not an
automatic missed-run alert.
CI validation
GitLab CI runs validation jobs on every merge request:
check- Linting, type checking, and spec validationcheck_nevr_conflicts- Predicts the NEVR(A) each changed package’s spec would build (usingrpmspecwith the real dist-tag macros, without actually building) and checks it against the public Pulp repo. Catches the case where a Konflux build succeeds and the MR merges, but the resulting NEVR was already published (typically two independent Release bumps based on stalemain, e.g. tworebuildMRs), which fails the post-merge build+sign+publish pipeline since Pulp treats each NEVRA as unique and immutable. Only scheduled for MRs that touchrpms/**/*, and within that, only checks packages whoserpms/<package>/directory itself changed – matching Konflux’s own build trigger – not packages whose only change is undermetadata/, since a metadata-only change (e.g.ci/dist_git.py mark-modified) never causes a rebuild/republish. Run locally withmake check-nevr-conflicts ARGS='<package>', or against real built RPMs withmake check-nevr-conflicts ARGS='--rpms-dir builds/<package>'afterci/build_rpms.sh. Seeci/check_nevr_conflicts.py’s module docstring for full details. Known limitations: only the public signed Pulp domain is checked (packages assigned aprivate_productinmetadata/<package>.jsonthroughhum.metaare noted but not fully verified, since their domains require Pulp API credentials this tool doesn’t have); and since it reflects Pulp’s state at job-run time, it reduces but cannot fully close the race where another MR publishes the same NEVR between this check passing and this MR’s eventual merge – only a hard check inside the actual Konflux publish pipeline (a separate pipeline-bundle repository,quay.io/hummingbird-ci/rpmbuild-pipeline, not covered by this job) could close that window completely; and in spec mode someOKresults can be for a package name a real build never actually produces (e.g. a base package with no top-level%filessection, roughly 14% of packages in this repo) – expected and harmless, and never a reason to discount aCONFLICTfound elsewhere in the same run.tree_status- Repository consistency checks (metadata integrity, spec file presence)- Testing Farm - Integration tests run via an IntegrationTestScenario that exercises built RPMs on a real RHEL compose (see Testing below)
Auto-approval for chore MRs
Automated update MRs (chore/* branches) follow an accelerated path:
- MR is created with auto-merge enabled
- Konflux builds the package and posts commit statuses
- After a delay, the
chore_mr_approvalCI job auto-approves the MR - GitLab merges the MR once the pipeline succeeds
Stage 3: Promotion or Push Build
Konflux builds RPMs for merge requests. After merge, packages use
MR Artifact Promotion to release that tested image by
default. Packages explicitly configured with push_pipeline: build build again on main.
Build triggers
Each package has a Tekton MR build PipelineRun definition generated from its directory in
rpms/<package>/ (see .tekton/rpms-on-pull-request.yaml.j2 and the rendered
.tekton/rpms-on-pull-request.yaml). PipelinesAsCode triggers it for MR
changes. A push to main triggers one PipelineRun per affected package: a promotion run from
.tekton/rpms-on-promotion.yaml by default, or a build from .tekton/rpms-on-push.yaml for an
explicit push_pipeline: build exception.
The setup package acts as a canary: changes to ci/ or mock/ directories
also trigger a setup build on pull requests, validating build infrastructure
changes before they reach main.
Build process
The build pipeline (build-rpm-package) executes for each target
architecture:
- Checks out the repository at the MR head or push commit
- Calculates build dependencies
- Runs
mockin hermetic mode (network-isolated with pre-fetched dependencies) - Produces RPM artifacts and build logs
- Generates an SBOM (Software Bill of Materials)
- Stores artifacts as Trusted Artifacts in the Konflux OCI registry
Build architectures
Packages are built for the architectures specified in their PipelineRun definition. Packages build for x86_64, aarch64, and s390x by default. Per-package overrides can select other architectures or build platforms.
Build output
Build artifacts are stored in the Konflux OCI registry:
quay.io/redhat-user-workloads/hummingbird-tenant/<package>--main
Konflux resources
Build infrastructure is defined across two repositories:
- rpms repo (
konflux-templates/) - Per-package Component and ImageRepository resources, ReleasePlanAdmission, EnterpriseContractPolicy, IntegrationTestScenario - infrastructure repo (
kubernetes/rpms-main/) - Application, ReleasePlan, ServiceAccount, and Secret resources
See Konflux Resource Deployment for details on how these resources are managed and deployed.
Testing
RPM packages are validated through integration tests that run on Testing Farm infrastructure.
Tests run via Testing Farm on RHEL-9-Nightly systems for both
x86_64 and aarch64 architectures. Test results appear as external jobs in
GitLab CI pipelines, providing pass/fail status and links to the Konflux
PipelineRun.
Test triggering
Tests are only triggered on merge requests, not on main branch builds. For a
package at rpms/<package>/, tests are triggered for all changes below that
directory.
Integration Test Scenario
Tests are triggered via an IntegrationTestScenario resource generated by this
(rpms) repository:
Configuration: konflux-templates/scenarios.yml (application × template matrix) and
konflux-templates/scenarios/<type>.yml.j2 (full CR content per scenario type); both consumed
by the generator for CR rendering and manifest emission
Rendered output: konflux-templates/rendered.yml
Deploy path: via the *-releases infrastructure sync on merge to main (not the
components sync — ITS is static repo-level config not intended to change in MRs; concurrent
MR pipelines would otherwise reset the live CR). Renovate IMAGE_TAG bumps auto-deploy on
merge via the same path.
The scenario uses the upstream Testing Farm pipeline for Konflux CI and is parameterized as follows:
The generator emits two scenario types per application (instantiated from templates in
konflux-templates/scenarios/):
-
testing-farm(non-optional,pull_requestcontext) — runs on every pre-merge component snapshot. Parameters:COMPOSE: RHEL-9-Nightly- Test environment OS/composePIPELINE_MODE: rpm- Configures RPM testing mode (vs. container)ARCH: x86_64,aarch64- Architectures to test (creates separate test runs per arch)PASS_SNAPSHOT_TO_TF: false- Don’t pass full snapshot JSON (package info comes viaIMAGE_NAME/IMAGE_URLinstead)IMAGE_TAG:- Version of the tmt-via-testing-farm pipeline bundleGIT_URL:- Allowlisted upstream repo URL (publicrpms-mainapplication only; omitted for private applications)
-
testing-farm-upstream(optional,component_attr-main+component_setup-maincanary contexts) — runs only on canary component snapshots. Additional parameter:TMT_PATH_FROM_CONTEXT: "true"- Derives the tmt test path from the component context
Key characteristics:
- Single scenario per application - Not per-package; runs for every package build in that application
- Context-based triggering -
testing-farmruns onpull_requestsnapshots only;testing-farm-upstreamruns on canary component snapshots and is optional (not listed in the trigger manifest) - Package identification - Konflux automatically provides
IMAGE_NAME(e.g.,openssl-main) andIMAGE_URL(OCI image with built RPMs) as environment variables
FMF test plan
The root folder of the rpms repository is marked with a .fmf directory to
identify it as an FMF metadata tree for tmt. The test plan in
ci/run_tests_rpm.fmf sets up the test instance and runs default and
package-specific tests via ci/run_tests_rpm.sh.
Test environment
These environment variables are available to the tmt test execution:
| Variable | Description |
|---|---|
IMAGE_NAME |
Component name (e.g., openssl-main) |
IMAGE_URL |
OCI image URL with built RPMs (e.g., quay.io/.../openssl-main@sha...) |
SNAPSHOT |
Snapshot metadata (if PASS_SNAPSHOT_TO_TF is true) |
COMPOSE |
OS/compose being tested (RHEL-9-Nightly) |
ARCH |
Architectures being tested (x86_64,aarch64) |
TMT_VERSION |
tmt version running the tests |
Stage 4: Release and Signing
Integration Service creates an automatic Release for a passing push Component Snapshot. For an MR artifact promotion, the Snapshot reuses the tested digest and does not run the MR tests again. Release Service validates Conforma and signs the RPMs before publishing:
- Method: Kerberos-based signing via
konflux-release-signing-prod@IPA.REDHAT.COM - Pipeline: Signing runs as part of the release pipeline, before the Pulp upload step
- Signing image:
quay.io/konflux-ci/signing:latest(verify pinned digest in ReleasePlanAdmission)
Stage 5: Publishing to Pulp
After signing, RPMs are uploaded to the Hummingbird Pulp repository.
Release pipeline
The push-rpms-to-pulp release pipeline handles the upload. Configuration is
defined in the ReleasePlanAdmission resource
(releng/hummingbird-rpms-tech-preview-staging.yaml):
mapping:
rpm-repositories:
- name: x86_64
repository_id: public-hummingbird-x86_64-rpms
- name: aarch64
repository_id: public-hummingbird-aarch64-rpms
- name: src
repository_id: public-hummingbird-source-rpms
# See ReleasePlanAdmission for the full list of published architectures
Pulp repository
Published RPMs are available in the Pulp content index:
Pulp automatically regenerates repository metadata (repodata) after each upload, making new packages immediately resolvable by DNF/YUM clients.
What Happens Next
Once RPMs are available in Pulp, the container image pipeline picks them up via lockfile updates. See the Image Pipeline documentation in the containers repo for the full container lifecycle.
Related Documentation
- MR Artifact Promotion - Current path that releases a tested MR image without a push rebuild
- Rebuilding Packages - No-change rebuilds, reverse dependency rebuilds, and patch backports
- Updating Dist-git Packages - Automated Fedora sync workflow
- Package Metadata Fields -
modification_statusandreleaseconfiguration - Package Modification Tracking - Managing modification status and version constraints
- Konflux Resource Deployment - How Konflux resources are defined and deployed
- Image Pipeline - Container image pipeline (continues from where this document ends)
4.1.2 - Konflux Resource Deployment
Konflux resources for RPM packages are split between two repositories:
- rpms: Per-package resources generated from package directories
- infrastructure: Application-level resources and deployment to the Konflux cluster
This separation allows independent iteration on each concern.
Deployment Matrix
| Resource Type | Source | RPMs MR | RPMs Push | Infra MR | Infra Push |
|---|---|---|---|---|---|
| Component | rpms | manual | manual | - | - |
| ImageRepository | rpms | manual | manual | - | - |
| ReleasePlanAdmission | rpms | - | automatic | manual | automatic |
| EnterpriseContractPolicy | rpms | - | automatic | manual | automatic |
| IntegrationTestScenario | rpms | - | automatic | manual | automatic |
| Application | infrastructure | - | - | manual | automatic |
| ReleasePlan | infrastructure | - | - | manual | automatic |
| ServiceAccount, Secret | infrastructure | - | - | manual | automatic |
Legend:
- RPMs MR: Downstream pipeline triggered from rpms MR (deploys from MR commit)
- RPMs Push: Downstream pipeline triggered from rpms push to main
- Infra MR: Infrastructure MR pipeline (manual trigger)
- Infra Push: Infrastructure push to main or web pipeline
Resource Locations and Rationale
RPMs Repo
Resources are defined in konflux-templates/ and rendered to konflux-templates/rendered.yml.
Component and ImageRepository:
- Ownership: Components are managed by the rpms repo. The infrastructure pipeline uses
ONLY_DOWNSTREAMso only explicit downstream triggers from rpms deploy changes, giving the rpms repo full control over the component lifecycle. - Dynamic generation: Generated from
rpms/*/directories viaci/generate_konflux_resources.sh, depending on package presence and configuration. - Timing: Must be deployed early during MR review so Konflux can build and test new packages.
ReleasePlanAdmission:
- Static configuration: Unlike containers (where RPA contains per-image tag mappings), the RPMs ReleasePlanAdmission is static—it configures the Pulp publishing pipeline without per-package data. It stays in the rpms repo for consistency with the containers pattern.
- Main branch only: Deployed only on push to main to maintain deployment consistency.
EnterpriseContractPolicy:
- Consistency: Follows the containers repo pattern of keeping policy definitions alongside the resources they govern.
RPMs Repo — IntegrationTestScenario
IntegrationTestScenario resources are defined via two config files:
konflux-templates/scenarios.yml— the application × template matrix (which app gets which scenario type, with which triggers and optional per-instance vars)konflux-templates/scenarios/<type>.yml.j2— full CR content per scenario type (params, resolverRef, Renovate markers, timeout annotations)
The generator renders these into konflux-templates/rendered.yml alongside Components and
ImageRepositories. The same matrix drives the trigger-manifest integration_test_scenarios
section, so the manifest cannot drift from deployed ITS.
Deploy path: ITS is deployed by the *-releases downstream pipeline (not *-components).
The releases sync fetches rendered.yml from the main branch and runs on every merge to
main, so ITS changes deploy automatically after merge — including Renovate bundle-digest and
IMAGE_TAG bumps. The components sync is deliberately excluded: it deploys from
PARENT_COMMIT_SHA (MR heads), and ITS is static repo-level config that is not supposed to
change in an MR — concurrent or stale MR pipelines would otherwise reset the live CR to older
versions. Pre-merge ITS deployment is intentionally unavailable by design.
No-prune behaviour: The deploy uses server-side apply without pruning — resources
removed or renamed in rendered.yml are not deleted from the cluster. A renamed or
removed ITS remains live and continues producing test PipelineRuns until deleted manually:
kubectl delete integrationtestscenario <name> -n hummingbird-tenant
See the infrastructure repo’s kubernetes-rendering.md “No pruning” section for details.
The integration_test_scenarios section of .tekton/trigger-manifest-pull-request.json
is derived from the same scenarios.yml matrix, ensuring the manifest never drifts from
deployed scenarios. See the trigger manifests documentation
for the full manifest schema.
Infrastructure Repo
Application and ReleasePlan are defined in kubernetes/rpms-main/:
- Independent iteration: Changes are decoupled from rpms repo activity—they can be modified, test-deployed via manual trigger in an infrastructure MR, verified, and merged without touching the rpms repo.
- Static configuration: These resources don’t depend on per-package data.
ServiceAccount and Secret for releases are defined in kubernetes/setup-konflux/:
- Security: Secret specifications (names, structure, credential references) should not be exposed in the rpms repo.
- Independent iteration: Like other infrastructure resources, these can be modified and test-deployed without touching the rpms repo.
The service account is referenced by ReleasePlanAdmission to authorize publishing RPMs to Pulp.
4.1.3 - Source Pipeline Tool
Source Pipeline Tool
Codename: Gorget — the iridescent throat patch that makes hummingbirds distinctive. In ornithology, the gorget is the defining feature used to identify species; in this project, it represents the verification layer that distinguishes independently-sourced packages from unverified ones.
Problem Statement
RPM-based distributions that derive packages from upstream sources face a common set of supply chain challenges:
-
Supply chain trust gap. Distributions that consume source tarballs from another distribution’s lookaside cache (e.g., Fedora’s) inherit an unverified trust boundary. Package maintainers download upstream release artifacts, may modify them, and upload to a lookaside cache. Downstream consumers use those tarballs without verifying they match upstream. Any maintainer could, intentionally or unintentionally, introduce modifications that compromise packages.
-
Version dependency on upstream packagers. Downstream distributions are blocked on upstream maintainers to package new releases. If upstream ships a critical fix, downstream must wait for the intermediate distribution to update before consuming it.
-
Non-durable security transforms. When a distribution applies a security fix to vendored dependencies (e.g., patching a lockfile to bump a vulnerable transitive dependency), a future upstream update regenerates vendor artifacts from the upstream baseline. The fix silently disappears. The update succeeds, CI passes, and the vulnerable dependency is back.
-
Unverified patches. Patches (
.patchfiles) from upstream distributions flow into packages without verification. There is no check whether patches correspond to upstream commits, whether they are needed for the target build environment, or whether new patches have been introduced.
Tool Overview
A containerized pipeline tool that reads a declarative per-package YAML definition, fetches source tarballs directly from upstream, applies transforms, verifies integrity, enforces policies, and emits artifacts ready for a lookaside cache.
┌──────────────────────────────┐
│ source-pipeline tool │
│ (container image) │
│ │
┌───────────┐ │ ┌───────┐ ┌───────────┐ │ ┌───────────────┐
│ package │ │ │ fetch │───▸│ transform │ │ │ tarballs │
│ dir │──────────▸│ └───────┘ └─────┬─────┘ │───▸│ sources │
│ (spec, │ │ │ │ │ report │
│ patches, │ │ ┌───────────┐ │ │ └───────┬───────┘
│ config) │ │ │ verify │◂─────┘ │ │
│ │ │ └─────┬─────┘ │ ▼
└───────────┘ │ │ │ ┌───────────────┐
│ ┌─────▼─────┐ │ │ lookaside │
┌───────────┐ │ │ enforce │ │ │ cache │
│ pipeline │ │ │ policy │ │ └───────────────┘
│ yaml │──────────▸│ └─────┬─────┘ │
│ │ │ │ │
└───────────┘ │ ┌─────▼─────┐ │
│ │ emit │ │
│ └───────────┘ │
└──────────────────────────────┘
Design principles
- Runs anywhere. Same container image via podman — locally for development, in CI for automation, in SLSA-attested build environments for provenance.
- Declarative. Per-package behavior is defined in YAML, not imperative scripts. The YAML serves as an attestation artifact — anyone can read it to understand exactly how a package’s sources are produced.
- Fail-closed. Verification failures and policy violations fail the pipeline. No silent fallbacks. On failure, the package is skipped (stays at its current version) and the automation continues to the next package.
- Ecosystem-aware. Built-in support for Go (vendor archives), npm (node_modules), cargo (vendor), and Composer (PHP) workflows, with escape hatches for custom transforms.
Container interface
podman run --rm \
-v ./<package-dir>:/package:ro \
-v ./pipeline.yaml:/pipeline.yaml:ro \
-v ./gpg-keys:/gpg-keys:ro \
-v ./output:/output \
source-pipeline:latest \
--version <new-version> \
[--old-version <old-version>] \
[--verify-patches /patches-from-upstream] \
[--dry-run]
Inputs (mounted read-only):
/package— the package directory (spec file, patches, existingsourcesfile)/pipeline.yaml— the declarative pipeline definition/gpg-keys— centralized GPG keyring directory
Outputs (written to /output):
- Source tarballs (ready for lookaside upload)
sources— updated sources manifest with checksumsreport.json— verification and policy results (pass/fail per check, patch classifications)
Exit codes:
0— success, all checks passed1— error (download failure, tool error)2— policy violation (verification or constraint failure)
--dry-run runs all stages through Verify and Policy but skips Emit — no tarballs are written to
/output. Exit codes and report.json behave identically, so developers can preview what would
happen without producing artifacts. Useful for validating a new pipeline YAML before committing.
On any non-zero exit, report.json is still written with the failure details (stage, error type,
message). The calling automation uses this to log the failure and skip the package — no package
update commit is created, the package stays at its current version, and the automation continues
to the next package. In the scheduled upstream-version-update job, the wrapper also creates (or
reuses) a draft diagnostic MR for a per-package update failure. That MR contains the full error
and no partial package changes, so a maintainer can reproduce and resolve the problem safely.
Transient failures (exit 1) may self-heal on the next scheduled run; verification and policy
failures (exit 2) require human intervention. When a later scheduled run successfully processes the
same package, it closes and removes the still-empty diagnostic MR branch. A diagnostic MR with
maintainer changes is left open for manual handling.
Pipeline Stages
1. Fetch
Downloads source artifacts directly from upstream.
- Parses
Source:URLs from the spec file (resolving RPM macros like%{version},%{name},%{url}) - Downloads each source from the upstream URL
- Supports fetching from git repositories at a tag, branch, or commit
- For packages without a pipeline YAML, uses a default fetch-from-spec behavior (covering the “trivial” package case)
2. Transform
Applies per-package source modifications.
- Vendor archive generation — runs ecosystem-specific tooling (e.g.,
go_vendor_archivefor Go,npm packfor Node.js,cargo vendorfor Rust) - Vendor dependency pinning — modifies lockfiles (
go.mod/go.sum,package-lock.json,Cargo.lock) to bump specific dependencies to required versions before vendoring. This is the enforcement counterpart to policy’s validation: pins are applied during transform, then policy confirms the result. Solves the non-durable security transforms problem — a declarative pin is re-applied on every source generation, so upstream updates cannot silently revert a CVE fix. - Source stripping — removes content that cannot be distributed (crypto, bundled pre-built binaries, non-free assets)
- UI asset builds — builds JavaScript/TypeScript UI assets from source (for packages like Prometheus, Jaeger, Grafana that currently vendor pre-built UI)
- Custom transforms — escape hatch for arbitrary commands when built-in stages are insufficient
- Toolchain versioning — packages can declare required toolchain versions (Node.js, Go, Rust,
etc.) via a
toolchain:section. The container ships defaults; per-package overrides ensure thatbuild-ui,vendor, andrun:steps use the correct toolchain without requiring separate container images per package
Known sharp edge: patch-list duplication
Two independent mechanisms hand-apply changes outside of %prep, and neither reads its list from
the spec’s PatchN: declarations — so both can silently drift out of sync with what the spec
actually declares. This is the canonical writeup; rebuilding-packages.md and the /cve skill
point back here instead of re-telling the story.
Custom run: transforms that hand-apply patches before resolving a lockfile (patch -p1 < ...
followed by yarn install/pnpm fetch/etc., as grafana’s yarn-cache generation does) maintain
their own copy of “which patches touch this source tree” rather than reading it from the spec’s
PatchN: declarations. Nothing enforces these two lists stay in sync. A patch added to the spec
through the normal backport workflow (see
Rebuilding Packages) is invisible to the
transform unless a human also updates the pipeline YAML — and if that patch touches a file the
transform’s install command resolves against (a lockfile or manifest), the generated artifact
silently drifts from what %build actually applies. This broke grafana12.4 and grafana13.1: four
separate CVE backports bumped yarn.lock without updating the pipeline’s yarn-cache generation
step, and the mismatch didn’t surface until the next automated version bump re-ran the transform
from a pristine checkout. test/test_source_pipeline_patches.py checks for this drift across all
packages with a pipeline definition, but it’s a safety net for a design gap, not a fix for it: a
built-in transform primitive that reads PatchN: from the spec directly, instead of requiring the
YAML to hand-duplicate the list, would close this class of bug at the source rather than relying on
the check to catch it after the fact.
go-vendor-tools.toml’s [archive] pre_commands have the same shape of problem for Go
packages, whether or not they’re migrated to gorget: pre_commands (sed edits, go get bumps, go mod tidy) only ever run against the vendor archive’s own checkout, never against the plain source
tarball (Source0), which is fetched separately. A patch that bumps a vendored dependency’s
version must be mirrored into go.mod/go.sum by a spec patch — nothing keeps pre_commands and
the spec’s patches in sync. This broke trivy: a CVE backport added go get calls to pre_commands
to bump vendored dependencies (CVE-2026-15788/15792/56852) without a matching spec patch, so
go.mod in the build tree and vendor/modules.txt in the generated vendor archive ended up
requiring different versions of the same package — go build -mod=vendor rejected it as
inconsistent vendoring. The mismatch sat latent for over a week until an unrelated version bump’s
%check run finally caught it. test/test_govendortools_gomod_patch_sync.py checks for this
drift, same caveat as above.
3. Verify
Validates integrity and authenticity of fetched sources.
- GPG signature verification — downloads signature files (
.asc,.sig) declared in the spec and verifies against upstream keys stored in a centralized keyring directory. Keys are organized by upstream project (e.g.,gpg-keys/curl.gpg). Centralized storage means the full set of trusted keys is auditable in one directory, and key rotation or revocation is a single-commit operation. - Checksum verification — compares against published checksums where available
- Reproducibility check — for packages with existing tarballs from another distribution, optionally compares the independently-fetched tarball to identify divergences
- Re-publication detection — if a previously committed checksum exists in the
sourcesfile for the same version and the freshly downloaded artifact does not match, the pipeline fails (exit code 2). This catches upstream projects that silently re-publish release artifacts under the same version. The pipeline will continue to fail on automated retries until a human explicitly updates anaccepted-checksumsentry in the pipeline YAML with the new hash and a reason. This forces investigation, prevents automated retries from silently accepting changed content, and provides a committed audit trail of what changed and why.
4. Enforce policy
Validates the final artifacts. Acts as a safety net for vendor-pin (confirms pins took effect)
and catches violations in packages that don’t use vendor-pin.
- Vendor dependency constraints — ensures vendored dependencies meet version requirements
(e.g.,
sanitize-html >= 2.17.5for a CVE fix) - Ecosystem-specific checks — Go module verification, npm audit, cargo audit
- License compliance — flags vendored dependencies with incompatible licenses
- Fail-closed: any policy violation exits with code 2
5. Emit
Produces final artifacts.
- Writes tarballs to the output directory
- Generates
sourcesfile in dist-git format (SHA512 (filename) = hash) - Writes
report.jsonwith verification results, policy check results, and patch classification
Pipeline Schema
Per-package pipeline definitions are YAML files that declare the full source generation workflow.
Packages without a pipeline definition use a built-in default that fetches sources from the spec’s
Source: URLs — no transforms, no verification, no policy.
Schema definition
# Spec preparation (optional)
# Runs before fetch — fixes version macros that the specfile library cannot
# trace, so Source URLs resolve correctly.
spec-update:
# Macro substitutions applied before Source URL resolution
macros:
- name: go_patch # %global go_patch <new-value>
value: "${VERSION_PATCH}" # extracted from VERSION (e.g., 1.25.3 → 3)
- name: k8s_ver # %global k8s_ver <new-value>
value: "${VERSION}"
# Reset Release to 0.1%{?dist} on version bump (common for versioned packages)
reset-release: true
# Source fetching
fetch:
sources:
# Fetch Source0 from the URL declared in the spec
- spec-source: 0
# Fetch Source1 (e.g., a GPG signature file)
- spec-source: 1
# Or fetch from an explicit URL (for sources not in the spec)
- url: "https://example.com/extra-source-${VERSION}.tar.gz"
# Or fetch from a git repository at a tag/commit
- git:
repo: "https://github.com/example/project"
ref: "v${VERSION}" # tag, branch, or commit hash
include-history: false # include .git dir in tarball (default: false)
# Vendor archive generation (optional)
vendor:
ecosystem: go # go | npm | cargo | composer
config: go-vendor-tools.toml # ecosystem-specific config file in the package dir
source-dir: . # directory to vendor from (default: extracted source root)
submodules: # (optional) vendor multiple Go submodules independently
- server
- etcdctl
- etcdutl
# Source transforms (optional, ordered)
transform:
# Built-in transform types
- strip-tarball:
source: "node-v${VERSION}.tar.gz"
remove:
- "deps/openssl/"
- "deps/ngtcp2/ngtcp2/crypto/"
output: "node-v${VERSION}-stripped.tar.gz"
# Pin vendored dependency versions before vendor archive generation.
# Modifies lockfiles (go.mod/go.sum, package-lock.json, Cargo.lock) to
# bump specific dependencies, then re-resolves the dependency graph.
# Runs before the vendor stage so the pinned versions are included in
# the vendor archive. Re-applied on every source generation, so upstream
# updates cannot silently revert a security fix.
#
# Version constraints are always minimum versions — the version field
# means "at least this version." The tool translates to ecosystem-native
# operations:
# Go: go get <package>@v<version> (minimum version selection)
# npm: npm install <package>@">= <version>"
# Cargo: set dependency requirement to ">= <version>" in Cargo.toml
- vendor-pin:
- package: golang.org/x/crypto
ecosystem: go
version: "0.31.0"
reason: "CVE-2024-45337"
- package: sanitize-html
ecosystem: npm
version: "2.17.5"
reason: "CVE-2024-XXXXX"
- package: tokio
ecosystem: cargo
version: "1.38.1"
reason: "CVE-2024-YYYYY"
# Custom command (escape hatch)
- run: "./packaging/make-tarball.sh ${VERSION}"
outputs:
- "package-${VERSION}-stripped.tar.gz"
# UI asset build
- build-ui:
ecosystem: npm # npm | yarn
source-dir: "web/ui"
output: "${PACKAGE}-${VERSION}-ui.tar.gz"
# Toolchain requirements (optional)
# Declares the toolchain versions a package needs during source generation.
# The container ships default toolchain versions; this section overrides them
# when a package requires something different. Any primitive that invokes a
# toolchain (vendor, build-ui, run) uses the versions declared here.
#
# Packages that don't declare a toolchain section get the container defaults.
# If a required toolchain is not available in the container image, the pipeline
# fails with exit code 1 and a clear message naming the missing tool.
toolchain:
node: "20" # Node.js version for build-ui, npm vendor, and run: steps
go: "1.23" # Go version for go vendor and run: steps
rust: "1.80" # Rust version for cargo vendor and run: steps
python: "3.12" # Python version for run: steps
# Integrity verification (optional)
verify:
gpg:
signature-source: 1 # spec Source index containing the .asc/.sig
keyring: "curl.gpg" # key name in the GPG keys directory
checksums:
url: "https://example.com/SHA256SUMS"
algorithm: sha256
# Override for upstream re-publications. Required when upstream re-publishes
# a release artifact with different content under the same version. The
# pipeline refuses to accept changed content automatically — a human must
# add the new checksum here after investigation.
accepted-checksums:
- file: "example-1.2.3.tar.gz"
sha512: "abc123..."
reason: "Upstream re-published with corrected LICENSE file (verified via upstream issue #456)"
# Policy enforcement (optional)
# Validates the final artifacts. vendor-constraints acts as a safety net:
# if vendor-pin (above) is used, policy confirms the pin took effect; if
# vendor-pin is not used, policy catches violations that need manual action.
# Uses the same minimum version semantics as vendor-pin.
policy:
vendor-constraints:
- package: sanitize-html
ecosystem: npm
version: "2.17.5"
reason: "CVE-2024-XXXXX"
- package: golang.org/x/crypto
ecosystem: go
version: "0.31.0"
reason: "CVE-2024-45337"
# Patch verification (optional)
patches:
verify: true # diff-detect new/changed patches
classify: true # attempt to classify by origin
upstream-repo: "https://github.com/curl/curl"
fail-on-unverified: false # warn-only by default
# Patch lifecycle rules (optional)
# Declares version-scoped applicability for patches. The pipeline tool reads
# these rules from patch headers (preferred) or from this YAML, and enforces
# them during updates: patches outside their valid range are flagged or dropped.
lifecycle:
- file: "fix-memory-leak.patch"
applies-to: "< 1.5.0" # drop this patch at version 1.5.0+
reason: "Fixed upstream in 1.5.0 (commit abc123)"
action: drop # drop | warn (default: warn)
- file: "cve-2024-45337.patch"
applies-to: "< 0.31.0"
reason: "CVE-2024-45337 — fixed upstream in golang.org/x/crypto 0.31.0"
action: drop
- file: "distro-branding.patch"
applies-to: "*" # carry forward unconditionally
reason: "Distribution-specific branding, always required"
# Post-update spec modifications (optional)
# Runs after fetch/transform — extracts metadata from downloaded sources and
# patches it into the spec.
post:
# Extract bundled dependency versions and splice into spec between markers
- bundled-provides:
modules-txt: "vendor/modules.txt" # Go modules.txt path inside extracted source
start-marker: "# --- bundled-deps.sh ---"
end-marker: "# --- end bundled-deps.sh ---"
# Run a custom command (escape hatch for complex metadata extraction)
- run: "./packaging/fill-versions.sh ${SPEC_FILE} source-v${VERSION}-stripped.tar.gz"
Variable substitution
The following variables are available in all string values:
| Variable | Value |
|---|---|
${VERSION} |
New upstream version (e.g., 1.25.3) |
${VERSION_MAJOR} |
Major version component (e.g., 1) |
${VERSION_MINOR} |
Minor version component (e.g., 25) |
${VERSION_PATCH} |
Patch version component (e.g., 3) |
${OLD_VERSION} |
Previous version |
${PACKAGE} |
Package name (directory name) |
${SPEC_FILE} |
Path to the spec file |
Version constraint semantics
All version: fields in vendor-pin and policy.vendor-constraints use minimum-version
semantics: the value means “at least this version.” The tool translates this into
ecosystem-native operations:
- Go:
go get <package>@v<version>(minimum version selection via MVS) - npm:
npm install <package>@">= <version>" - Cargo: sets the dependency requirement to
>= <version>inCargo.toml
Default behavior (no pipeline YAML)
When no pipeline YAML is provided, the tool applies a built-in default:
- Parse all
Source:URLs from the spec - Download each from the upstream URL
- Generate
sourcesfile with SHA512 checksums - No transforms, no verification, no policy enforcement
This covers the common case where the upstream distribution’s tarball is identical to upstream’s release artifact.
Patch Verification
When packages are updated from an upstream distribution, patch files (.patch) may arrive without
inspection. The pipeline tool can optionally verify and manage patches.
Verification approach
The pipeline tool can optionally verify patches when invoked with --verify-patches:
-
Diff detection. Compare the set of
.patchfiles in the updated package directory against the previous version. Identify new patches, removed patches, and modified patches. -
Header parsing. Extract metadata from patch headers:
From:— author identitySubject:— description- Commit hash references (e.g.,
From <hash>,cherry picked from commit <hash>) Bug:/CVE:references
-
Classification. Attempt to classify each patch:
- Upstream backport — references a commit hash that exists in the upstream repo
- CVE fix — references a CVE identifier
- Build/packaging fix — modifies build system files (Makefile, configure, CMakeLists)
- Distribution-specific — modifies paths, branding, or distribution-specific integration
- Unclassified — cannot be automatically categorized
-
Upstream verification. For patches claiming to be backports, check whether the referenced commit exists in the upstream repo (via
git ls-remoteor the upstream API). -
Reporting. Output classification and verification results in
report.json. Optionally fail on unverified patches (controlled bypatches.fail-on-unverifiedin the pipeline YAML).
Patch lifecycle enforcement
Patches have version-scoped lifetimes. A CVE backport is only valid until the upstream version that includes the fix. A build system workaround may only apply to a specific major version. Without enforcement, a patch that should have been dropped at version 2.0 silently persists, and a patch that must be carried forward can be accidentally removed during an update.
The pipeline tool enforces patch lifecycle rules declared either in patch headers or in the pipeline
YAML’s patches.lifecycle section.
Header-based declaration (preferred). Patch authors add structured keywords to the patch header:
From: maintainer@example.com
Subject: Backport fix for CVE-2024-45337
Applies-To: < 0.31.0
Lifecycle-Action: drop
Lifecycle-Reason: Fixed upstream in golang.org/x/crypto 0.31.0
---
YAML-based declaration (fallback). For patches from upstream distributions that cannot have
headers modified, rules are declared in patches.lifecycle in the pipeline YAML (see schema above).
Header-based rules take precedence when both exist for the same patch.
Enforcement behavior:
- During an update to version
${VERSION}, the tool evaluates each patch’sapplies-torange against the new version. - If a patch is outside its valid range:
action: drop— the patch file is deleted from the package directory and the correspondingPatch:declaration and%patch/%autopatchapplication directives are removed from the spec. If the patch is applied inside a conditional block (%if), the tool flags it for manual intervention instead of attempting removal. The tool reports all actions inreport.json.action: warn(default) — the patch is flagged inreport.jsonbut not removed. The update proceeds.
- If a patch has
applies-to: *, it is always carried forward. - Patches without any lifecycle declaration are treated as having no version constraint (equivalent
to
applies-to: *,action: warn).
Carry-forward enforcement. The inverse case is also important: some patches (branding,
distribution-specific integration) must never be dropped. If a patch marked applies-to: * is
missing after an update (e.g., removed by an upstream distribution sync), the tool flags it as an
error.
Example report output
{
"patches": {
"new": [
{
"file": "fix-memory-leak.patch",
"classification": "upstream-backport",
"upstream_commit": "abc123def456",
"verified": true
}
],
"removed": ["old-workaround.patch"],
"unchanged": ["distro-paths.patch"],
"lifecycle": [
{
"file": "cve-2024-45337.patch",
"applies_to": "< 0.31.0",
"current_version": "0.31.0",
"action": "drop",
"result": "removed — version 0.31.0 is outside applies-to range"
},
{
"file": "cve-2024-99999.patch",
"applies_to": "< 2.0.0",
"current_version": "2.0.0",
"action": "drop",
"result": "flagged — patch is applied inside a conditional %if block; manual removal required"
},
{
"file": "distro-branding.patch",
"applies_to": "*",
"action": "carry-forward",
"result": "present"
}
]
}
}
Examples
Trivial package (curl)
No pipeline YAML needed. The default behavior fetches Source0 from the URL in the spec. If you
want GPG verification:
fetch:
sources:
- spec-source: 0
- spec-source: 1
verify:
gpg:
signature-source: 1
keyring: "curl.gpg"
Transformed package (Node.js)
Strip bundled OpenSSL, verify upstream checksums, extract component versions into spec:
spec-update:
macros:
- name: nodejs_define_version node
value: "${VERSION}"
reset-release: true
toolchain:
node: "22"
fetch:
sources:
- url: "https://nodejs.org/dist/v${VERSION}/node-v${VERSION}.tar.gz"
transform:
- strip-tarball:
source: "node-v${VERSION}.tar.gz"
remove:
- "deps/openssl/"
output: "node-v${VERSION}-stripped.tar.gz"
verify:
checksums:
url: "https://nodejs.org/dist/v${VERSION}/SHASUMS256.txt"
algorithm: sha256
post:
- run: "./packaging/fill-versions.sh ${SPEC_FILE} node-v${VERSION}-stripped.tar.gz"
Simple Go vendor (caddy)
A pattern shared by nats-server, oauth2-proxy, and similar Go projects:
fetch:
sources:
- git:
repo: "https://github.com/caddyserver/caddy"
ref: "v${VERSION}"
vendor:
ecosystem: go
Multi-submodule Go vendor (etcd)
fetch:
sources:
- url: "https://github.com/etcd-io/etcd/archive/v${VERSION}/etcd-${VERSION}.tar.gz"
vendor:
ecosystem: go
config: go-vendor-tools.toml
submodules:
- server
- etcdctl
- etcdutl
Code generation + Go vendor (opentelemetry-collector-contrib)
Built-in primitives handle fetch and vendor; the code generation step requires run::
fetch:
sources:
- url: "https://github.com/open-telemetry/opentelemetry-collector-releases/archive/v${VERSION}/opentelemetry-collector-releases-${VERSION}.tar.gz"
vendor:
ecosystem: go
config: go-vendor-tools.toml
source-dir: _build
transform:
- run: |
curl -fSL -o ocb "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2Fv${VERSION}/ocb_${VERSION}_linux_amd64"
curl -fSL -o ocb_checksums.txt "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2Fv${VERSION}/ocb_${VERSION}_checksums.txt"
sha256sum -c --ignore-missing ocb_checksums.txt
chmod +x ocb
tar -xzf "opentelemetry-collector-releases-${VERSION}.tar.gz"
cd "opentelemetry-collector-releases-${VERSION}"
if [ -f scripts/prepare-obi.sh ]; then
bash scripts/prepare-obi.sh otelcol-contrib
fi
../ocb --skip-compilation --config distributions/otelcol-contrib/manifest.yaml
outputs:
- "opentelemetry-collector-contrib-${VERSION}-generated.tar.bz2"
Git snapshot (libXtst)
A pattern shared by libX11, libXext, libXi, libXrender:
fetch:
sources:
- git:
repo: "https://gitlab.freedesktop.org/xorg/lib/libXtst"
ref: "libXtst-${VERSION}"
include-history: true
Built-in Primitives Coverage
An audit of 48 existing tarball/vendor scripts across 25+ packages identified the following coverage:
Fully declarative (no run: needed) — ~30 script files (15 unique patterns):
| Built-in primitive | Packages covered |
|---|---|
vendor: {ecosystem: go} |
caddy, nats-server×2, oauth2-proxy |
vendor: {ecosystem: go, submodules: [...]} |
etcd (3 submodules) |
strip-tarball: {remove: [...]} |
nodejs×5, cyrus-sasl, perl-libnet, java in-tree libs×2 |
fetch: {git: {repo, ref}} |
oniguruma, php-jsqueeze, libX11/Xext/Xi/Xrender/Xtst |
fetch + verify: {checksums} |
nodejs download+verify |
vendor: {ecosystem: composer} |
composer |
Partially declarative (generic primitives + 1–2 run: steps):
| Package | Generic part | Custom run: step |
|---|---|---|
| otel-collector, otel-collector-contrib | vendor: {ecosystem: go} |
OCB binary code generation |
| java-openjdk×2 | strip-tarball, fetch: {git} |
./configure + make store-source-revision (needs boot JDK) |
| selinux-policy | fetch: {git} (×3 repos) |
Multi-repo selective archiving |
Genuinely custom (run: required) — 7 packages:
| Package | Why it can’t be declarative |
|---|---|
| nss-fips | Container-based RPM download with subscription-manager credentials + QEMU cross-arch |
| openssl-fips-provider | Container-based SRPM download + nested RPM extraction |
| ca-certificates | Interactive multi-source crypto trust data merging |
| gcc | GCC-specific changelog/PR extraction from git history |
| gdb | Interactive patch management tooling (stgit) |
| python3.14 | Koji task-specific JIT stencil artifact extraction |
| erlang27 | Complex patch reformatting + spec rewriting |
The run: escape hatch exists for these ~7 packages. All other packages should use built-in
primitives to maintain the declarative contract.
Open Questions
- Implementation approach. The pipeline YAML schema (ordered transform stages,
run:escape hatches, variable substitution) resembles a bespoke Ansible without the ecosystem. The implementation choice also determines the language. Alternatives to consider:- Custom tool (Python or Go) — Python is consistent with existing tooling; Go produces a single static binary. Either way we own the full stack.
- Tekton StepActions — already in some build ecosystems. Source generation could be a parameterized Tekton pipeline rather than a custom tool. Downside: harder to run locally.
- Shared shell function library — the majority of scripts decompose into 3–4 operations
(
vendor_go,strip_tarball,fetch_git). A thin shell library called from per-package Makefiles may be more honest than YAML that serializes shell commands. - Minimal declarative YAML + driver — keep the YAML as a pure declaration of intent (what to
fetch, what to strip, what to vendor) with no
run:blocks, no ordering, no conditionals. A thin driver interprets it by calling shell functions. Packages that can’t be expressed this way keep their shell scripts. This preserves the attestation value of the YAML without building a workflow engine.
Hummingbird Integration
This section describes how the source pipeline tool integrates with Hummingbird’s existing infrastructure. The tool itself is distribution-agnostic; this section covers the Hummingbird-specific wiring.
Current Architecture
Two update systems
| System | Packages | Source of tarballs | Hook system |
|---|---|---|---|
dist_git.py update |
450 (clean + modified) | Fedora lookaside cache (via sources file copied from Fedora dist-git) |
None |
check_upstream_versions.py |
22 (independent) | Upstream URLs (via download_sources hooks or default spec URL download) |
Yes: update_spec, download_sources, post_update |
dist_git.py update flow:
- Clones Fedora dist-git for the package
- Checks version, Koji build status, pre-release filtering
- Copies entire Fedora dist-git checkout into
rpms/<package>/viashutil.copytree— this includes the spec, patches,.gitignore, andsourcesfile - For modified packages, performs a 3-way git merge to preserve local changes
- Commits everything (spec, patches,
sourcesfile, metadata)
The sources file is committed to git as a text manifest (format: SHA512 (filename) = hash).
Actual tarballs are excluded by .gitignore and stored in a lookaside cache.
check_upstream_versions.py flow:
- Queries release-monitoring.org for new upstream versions
- Runs three hook phases per package:
update_spec— updates the spec’s Version/Release (default:specfile.update_version())download_sources— downloads source tarballs (default: fetches from spec Source URLs)post_update— additional steps (default: no-op)
- Uploads downloaded tarballs to Hummingbird’s lookaside cache
- Updates the
sourcesfile with new checksums - Commits
Hooks are defined in metadata/<package>.update-hooks.yaml. 12 packages currently have hooks.
Three lookaside cache backends
Configured in mock/dist-git-client.ini:
| Backend | URL pattern | Used by |
|---|---|---|
| Fedora | src.fedoraproject.org/repo/pkgs/rpms/{name}/{filename}/{hashtype}/{hash}/{filename} |
~410 packages (default) |
| CentOS Stream | sources.stream.centos.org/sources/rpms/{name}/... |
1 package (rust-rpm-sequoia) |
| Hummingbird | d1766whheab9hg.cloudfront.net/rpms/{name}/... (S3: arr-hummingbird-prod-dist-git-cache) |
~40 packages (independent + forked) |
The forked_from field in ci/package-overrides.yaml determines which cache a package uses at
build time. Packages without forked_from default to Fedora’s cache.
How Konflux builds consume sources
Tekton PipelineRuns (generated by ci/generate_resources.py from Jinja2 templates) pass parameters
to the build-rpm-package pipeline bundle (quay.io/hummingbird-ci/rpmbuild-pipeline):
monorepo-subdir: rpms/<package>— locates the spec, patches, andsourcesfilepackage-name: <name>— upstream name for lookaside URL constructionforked-from: <url>— (optional) selects the lookaside backenddist-git-client-configdir: mock/— (optional) points to the customdist-git-client.ini
The pipeline uses dist-git-client to read the sources file and download tarballs from the
appropriate cache, then runs mock to build RPMs.
Existing source generation patterns
Some independent packages already have tarball generation scripts:
rpms/opentelemetry-collector-contrib/create-vendor-tarball.sh— downloads upstream release, runs OCB to generate source code, vendors Go dependenciesrpms/caddy/create-vendor-tarball.sh— downloads upstream tarball, generates Go vendor archiverpms/nodejs25/packaging/make-nodejs-tarball.sh— downloads upstream, strips bundled content
These scripts are wired into the update path via download_sources hooks in
metadata/<package>.update-hooks.yaml. The pattern is: script runs, prints output filenames to
stdout, automation uploads them to the Hummingbird lookaside cache.
Integration with dist_git.py update
The current flow copies everything from Fedora dist-git (including the sources file) via
shutil.copytree. The pipeline tool inserts after this step:
Current:
1. Clone Fedora dist-git
2. copytree into rpms/<package>/ ← sources file points to Fedora lookaside
3. Commit
Proposed:
1. Clone Fedora dist-git
2. copytree into rpms/<package>/ ← sources file points to Fedora lookaside
3. Run source-pipeline tool ← fetches from upstream, replaces sources file
4. Upload tarballs to Hummingbird lookaside
5. Commit ← sources file now points to Hummingbird lookaside
Implementation:
- After the
shutil.copytree(or merge for modified packages), check ifmetadata/<package>.source-pipeline.yamlexists - If it exists: invoke the pipeline tool via podman, collect outputs, upload to lookaside, replace
the
sourcesfile - If it does not exist: run the built-in default (fetch from spec URLs, upload, replace
sources) - A
--skip-pipelineflag allows falling back to the current behavior during migration - Update
ci/package-overrides.yamlto setforked_fromto hummingbird for each migrated package (so Konflux builds fetch from the Hummingbird cache)
Integration with check_upstream_versions.py
The pipeline tool replaces all three hook phases from *.update-hooks.yaml, consolidating
per-package update behavior into a single *.source-pipeline.yaml file:
Current:
1. update_spec hook/default ← *.update-hooks.yaml
2. download_sources hook/default ← *.update-hooks.yaml
3. post_update hook/default ← *.update-hooks.yaml
Proposed:
1. spec-update (pipeline YAML) ← replaces update_spec hooks
2. fetch + transform (pipeline YAML) ← replaces download_sources hooks
3. post (pipeline YAML) ← replaces post_update hooks
This reduces per-package metadata from three files (metadata/<package>.json,
*.update-hooks.yaml, *.source-pipeline.yaml) to two (metadata/<package>.json for
identity/tracking, *.source-pipeline.yaml for all update behavior).
Existing *.update-hooks.yaml files are migrated to *.source-pipeline.yaml definitions. During
migration, the hook system remains as a legacy fallback: if a package has a *.update-hooks.yaml
but no pipeline YAML, the hooks run as before. Once all 12 hook files are migrated, the hook system
is removed.
Integration with Konflux
Initially, the pipeline tool runs pre-build (during the update automation in GitLab CI). The
tarballs it produces are uploaded to the Hummingbird lookaside, and the existing Tekton build
pipeline consumes them via dist-git-client as it does today.
Future: the pipeline tool could run as a Tekton task within the Konflux build pipeline itself. This would move source generation into the SLSA-attested build environment, strengthening the provenance chain. The container image is already compatible — it just needs a Tekton Task definition.
Hummingbird-specific configuration
Pipeline YAML files live at metadata/<package>.source-pipeline.yaml.
GPG keys live at metadata/gpg-keys/<upstream-project>.gpg.
The container image is published as quay.io/hummingbird-ci/source-pipeline:latest.
Local development
Developers can run the tool directly:
podman run --rm \
-v ./rpms/curl:/package:ro \
-v ./metadata/curl.source-pipeline.yaml:/pipeline.yaml:ro \
-v ./metadata/gpg-keys:/gpg-keys:ro \
-v /tmp/output:/output \
quay.io/hummingbird-ci/source-pipeline:latest \
--version 8.21.0
# Outputs in /tmp/output/:
# curl-8.21.0.tar.xz
# curl-8.21.0.tar.xz.asc
# sources
# report.json
To preview without producing artifacts:
podman run --rm \
-v ./rpms/curl:/package:ro \
-v ./metadata/curl.source-pipeline.yaml:/pipeline.yaml:ro \
-v ./metadata/gpg-keys:/gpg-keys:ro \
quay.io/hummingbird-ci/source-pipeline:latest \
--version 8.21.0 \
--dry-run
# Runs Fetch → Transform → Verify → Policy but skips Emit.
# Exit code and report.json reflect what would happen.
# No /output mount needed.
Migration Path
Phase 1: Independent Go packages
Migrate the 22 independent packages that already fetch from upstream. Convert their existing
create-vendor-tarball.sh scripts and download_sources hooks into pipeline YAML definitions.
Validates the tool against known-good packages.
Scope: 7 packages with existing scripts + 15 without.
Of the 7 scripted packages, 4 (caddy, nats-server×2, oauth2-proxy) use an identical clone + go mod vendor + tar pattern that maps directly to vendor: {ecosystem: go}. etcd requires
multi-submodule vendor support. The 2 otel-collector packages need a run: block for OCB code
generation, but their vendor step is generic.
Phase 2: Trivial clean packages
Enable the default pipeline behavior (fetch from spec URLs) for the ~351 clean packages where Fedora’s tarball is identical to upstream. This is the highest-impact, lowest-effort phase.
Prerequisites:
- Audit to classify packages as trivial vs. transformed (HUM-4620)
dist_git.py updateintegration complete (HUM-4621)
Per-package steps:
- Run the pipeline tool, compare output against existing Fedora-sourced tarball
- If identical: add
forked_from: hummingbirdtopackage-overrides.yaml - If different: flag for Phase 3
Phase 3: Transformed packages
Write pipeline YAML definitions for packages where Fedora modifies the tarball. Based on the script audit, most transforms decompose into built-in primitives:
- Strip + repack (nodejs×5, cyrus-sasl, perl-libnet, java in-tree libs):
strip-tarball - Git snapshots (libX11 family×5, oniguruma, php-jsqueeze):
fetch: {git: ...} - Composer vendor (composer):
vendor: {ecosystem: composer}
Only ~7 packages (nss-fips, openssl-fips-provider, ca-certificates, gcc, gdb, python3.14, erlang27)
require run: blocks for genuinely custom logic.
Scope: determined by Phase 2 audit.
Phase 4: Policy enforcement
Add policy sections to pipeline YAMLs for packages with known vendor dependency constraints.
Initially driven by CVE fixes that need to survive upstream updates.
Phase 5: Patch verification
Enable patches.verify and patches.classify for packages. Start with warn-only
(fail-on-unverified: false), gather data on classification accuracy, then selectively enable
fail-on-unverified for high-risk packages.
Operational Health
Escape hatch ratio
The primary health metric for the source pipeline is the escape hatch ratio: the
percentage of pipeline YAML definitions that contain run: blocks (in transform or post
sections) relative to the total number of pipeline YAML definitions.
A run: block means the package needs custom shell commands that the built-in primitives
cannot express. Every concern that accumulates over time – ecosystem coverage gaps, upstream
behavioral drift, lock file format churn, new language ecosystems – ultimately manifests the
same way: a package that cannot be expressed declaratively gets a run: block. This makes the
ratio a single number that aggregates all pressures on the declarative model.
Interpretation:
| Ratio | Signal |
|---|---|
| < 15% | Healthy. The primitive set covers real-world needs. |
| 15-25% | Watch. Look for repeated patterns in run: blocks that should become primitives. |
| > 25% | Action needed. The YAML is becoming a workflow engine. Either expand the primitive set or reconsider the abstraction. |
When the ratio climbs, examine the run: blocks for repeated patterns. If three or more
packages use run: to do the same thing (e.g. a new vendoring ecosystem, a common tarball
repack operation), that pattern should become a built-in primitive. The ratio climbing is not
inherently bad – it is bad only if the response is to keep adding one-off scripts instead of
investing in the primitive set.
CI enforcement
The ci/validate_pipeline_health.py script scans all metadata/*.source-pipeline.yaml files,
calculates the escape hatch ratio, and reports the result. It runs as part of make check.
- During migration (Phases 1-3): warn mode. The script prints the ratio and flags if it
exceeds 25%, but does not fail CI. This avoids blocking legitimate migration work where
some packages temporarily need
run:blocks before new primitives are added. - After Phase 3: enable
--failmode. Once the primitive set is established and migration is complete, exceeding 25% is a hard CI failure that forces a conversation before arun:block is added.
4.2 - Agent-Friendly Documentation
The documentation site provides several features that make it easier for AI agents and LLMs to discover, index, and consume its content.
Discovery and access methods
llms.txt
The site publishes an llms.txt file at the root, following the llmstxt.org specification. This file is a Markdown-formatted index of all documentation pages, grouped by section, with links to each page’s markdown URL. It serves as the primary entry point for agents discovering the site.
llms-full.txt
The site also publishes llms-full.txt – a full content dump of all documentation pages in a single file. This is intended for LLMs with large context windows that want to ingest the entire site in a single request without crawling individual pages.
Pages whose rendered content exceeds a size threshold (currently 50,000 characters) are replaced with a placeholder that includes the page title, description, and a link to the individual page’s markdown URL for on-demand fetching. This keeps the file at a manageable size while ensuring no content is completely inaccessible.
Per-page markdown output
Every page on the site is available in clean markdown format by appending
/index.md to its URL. For example:
- HTML:
https://hummingbird-project.io/docs/contributing/quickstart/ - Markdown:
https://hummingbird-project.io/docs/contributing/quickstart/index.md
The markdown output resolves all shortcodes (such as file includes from other
repositories) while preserving the content as markdown rather than converting
to HTML. Each markdown page includes links to both llms.txt and
llms-full.txt so agents can discover the site-wide indexes from any page.
Content negotiation
A CloudFront distribution sits between DNS and the site’s page host. A
viewer-request CloudFront Function inspects the Accept header on every
request: when the header includes text/markdown, the function rewrites the
URI to serve the pre-built .md equivalent. Only HTML-like patterns (trailing
slash, .html, bare paths) are rewritten; all other URIs pass through
unchanged.
This means agents can use standard HTTP content negotiation on any page URL
without needing to know the /index.md convention:
GET /docs/contributing/quickstart/ HTTP/1.1
Accept: text/markdown
Stable link aliases (/l/)
Pages referenced from external sources have stable short URLs under the /l/
prefix (e.g., /l/adding-images). These return HTTP 301 to the canonical page
URL, so pages can move without breaking incoming links. The
Writing Documentation guide describes how to add aliases.
The CloudFront viewer-request function recognizes /l/ paths and normalizes
them before they reach the origin. Content negotiation (Accept header) is
skipped for /l/ paths — instead, the URL form determines the response:
/l/foo and /l/foo/ redirect to the HTML page, while /l/foo.md and
/l/foo/index.md redirect to the markdown file. If no matching alias exists,
the origin returns a 404.
Specifications and tools
The site aims to follow the emerging standards for agent-friendly documentation:
- llmstxt.org – the llms.txt specification
- Agent-Friendly Documentation Spec – broader specification for documentation sites that serve AI agents
- AFDocs – scoring tool that evaluates a site’s compliance with the Agent-Friendly Documentation Spec
Planned features
HTML llms-directive (HUM-6571)
A Docsy upgrade to v0.17.0 will embed a pointer to llms.txt inside the HTML
<body> of every page, so agents landing on any HTML page can discover the
LLMS index without prior knowledge of the site structure.
Documentation quality evaluation (HUM-6575)
A promptfoo-based evaluation harness will measure how accurately LLMs can answer questions about Hummingbird using the published documentation. This provides a feedback loop for identifying documentation pages that need improvement for agent consumption.
4.3 - Containers repository
Background documentation sourced from the containers repository.
4.3.1 - Image Pipeline
Explanation of the complete container image pipeline, from RPM dependency updates through source templates, generation, build, testing, release, and advisory publication.
Overview
The container image pipeline consists of six main stages:
- Source Templates - Jinja2 templates in
images/*/define container images - Generation - Templates are rendered into Containerfiles and Konflux resources
- Build - Konflux builds multi-architecture container images
- Testing - Testing Farm runs integration tests via Konflux
- Enterprise Contract Validation - Conforma validates policy compliance before release
- Release - Images are published to registries and advisories are created
RPM packages are built and published separately via the RPM Pipeline. This document covers what happens after RPMs are available in the Hummingbird package repository.
RPM Dependency Updates
Container images pin exact RPM versions via lockfiles. When new RPMs are published to the Hummingbird Pulp repository, lockfiles must be regenerated for images to pick up the updates.
Lockfile structure
Each image variant has two RPM-related files:
| File | Purpose |
|---|---|
generated/images/<image>/<distro>/<variant>/rpms/rpms.in.yaml |
Declares required packages |
generated/images/<image>/<distro>/<variant>/rpms/rpms.lock.yaml |
Pins exact versions, URLs, and checksums |
The lockfile contains every RPM (direct and transitive dependencies) with its exact version, architecture, URL pointing to the Pulp repository, and checksum. During build, the locked RPMs are downloaded in hermetic mode — no network access to external repositories.
Automatic lockfile updates
MintMaker / Renovate — A custom Renovate instance (forked
rpm-lockfile manager from redhat-exd-rebuilds/renovate) runs as a
Kubernetes CronJob. It scans each rpms.in.yaml file individually,
resolves dependencies against Pulp repositories, and opens auto-merge
MRs — one per image group and distro (e.g., “Refresh RPM lockfiles for
go-1-25/hummingbird”).
The underlying tool is rpm-lockfile-prototype, which wraps the DNF
dependency solver. For a given rpms.in.yaml, it resolves all direct
and transitive dependencies against the yum repositories defined in
yum-repos/*.repo and outputs a complete rpms.lock.yaml with pinned
versions, download URLs, and checksums. Locally, generate_rpms_lock.py
adds hash-based optimization: it hashes the input file and skips
regeneration when the hash matches the existing lockfile, avoiding
expensive solver runs when nothing has changed.
Manual lockfile updates
To refresh the lockfile for a single image variant:
make generated/images/<image>/<distro>/<variant>/rpms/rpms.lock.yaml FORCE_REFRESH=true
For example:
make generated/images/caddy/hummingbird/default/rpms/rpms.lock.yaml FORCE_REFRESH=true
To regenerate all lockfiles and open MRs for the changes:
# Regenerate all rpms.lock.yaml files
make all-host FORCE_REFRESH=true
# Open MRs for any changed lockfiles
ci/create_lockfile_update_mrs.sh
From lockfile MR to container build
When a lockfile MR merges to main, PipelinesAsCode triggers a container build (Stage 3). Each image variant has CEL expressions in its PipelineRun definition that match on file paths, so only images whose lockfiles changed get rebuilt.
Stage 1: Source Templates
Each container image is defined by templates in images/<image-name>/:
properties.yml- Image configuration (packages, variants, tags, etc.)Containerfile.j2- Jinja2 template for the container buildREADME.md.j2- Documentation templatetests-container.yml- Integration test definitions
Templates use reusable macros from macros/*.yml.j2:
setup_newroot()- Configures DNF and filesysteminstall_newroot()- Installs packagescleanup_newroot()- Cleans up filesfinal_stage()- Creates scratch-based final image
Shared configuration is defined in images/variables.yml. See Global Variables
Reference for details.
Stage 2: Generation
Templates are rendered into concrete artifacts that drive the pipeline:
- Containerfiles - Build instructions for each image variant
- README documentation - Image documentation for Quay.io
- Konflux resources - CI/CD pipeline definitions
Containerfile Generation
Containerfiles are generated from templates for each image variant using Make’s incremental build system:
make
This combines:
- Reusable macros from
macros/ - Service-specific templates from
images/*/Containerfile.j2 - Configuration from
properties.yml(see Image Configuration Reference) - Variables from
images/variables.yml(see Global Variables Reference) - RPM versions from
rpms.lock.yamlfiles - Git submodule information from
.gitmodules
Output: generated/images/<image-name>/<distro>/<variant>/Containerfile, along with VERSION and TAGS files
The build system uses timestamp-based dependency tracking, so only changed files are regenerated.
README Generation
After Containerfiles are generated, README documentation is generated from README.md.j2 templates
using macros from macros/readme.yml.j2. Each image template produces one README per target
defined in images/variables.yml (Hummingbird, Rawhide, Red Hat).
Output: generated/images/<image-name>/hummingbird/README.md, generated/images/<image-name>/rawhide/README.md, generated/images/<image-name>/README.md
Konflux Resource Generation
Konflux CI/CD resources are generated from templates in konflux-templates/:
make
This generates:
- Components - Define what to build (one per image variant)
- ImageRepositories - Define where to push images
- ReleasePlanAdmissions - Define how to release images
Output: konflux-templates/rendered.yml
These resources must be deployed to Konflux before builds can run. See the Konflux Resource Deployment guide for how and when resources are deployed.
Stage 3: Build
Konflux builds container images automatically when changes are pushed to GitLab.
Build Triggers
- Merge Requests: Builds all changed images and triggers tests
- Main Branch: Builds all changed images (tests do not run on main)
Build Process
For each image variant:
- Konflux Component watches the GitLab repository
- On changes, Konflux triggers a build PipelineRun
- The build uses the generated Containerfile from
generated/images/<name>/<distro>/<variant>/Containerfile - Images are built for multiple architectures (x86_64 and aarch64)
- Built images are pushed to the development registry
Build Output
Development images are pushed to the Red Hat User Workloads registry:
quay.io/redhat-user-workloads/hummingbird-tenant/<group>--<variant>--main
Merge request builds are tagged with:
quay.io/redhat-user-workloads/hummingbird-tenant/<group>--<variant>--main:on-mr-<MR_ID>-<COMMIT_SHA>
Examples:
quay.io/redhat-user-workloads/hummingbird-tenant/curl--default--mainquay.io/redhat-user-workloads/hummingbird-tenant/nginx--builder--main:on-mr-123-abc1234
SBOM Generation
Each per-architecture build also produces an SPDX 2.3 Software Bill of Materials (SBOM), attached as an OCI artifact to the image. The SBOM is assembled from two independent scans, merged into a single document:
flowchart LR
syft["Syft\n(buildah-remote-oci-ta)"] --> mobster["Mobster\n(buildah-remote-oci-ta)"]
hermeto["Hermeto\n(prefetch-dependencies-oci-ta)"] --> mobster
mobster --> sbom["Per-arch SPDX SBOM"]
sbom --> index["Mobster\n(build-image-index)"]
index --> oci["Index SBOM\n(.sbom OCI artifact)"]
- Syft runs as the
sbom-syft-generatestep inside the buildah-remote-oci-ta Tekton task. It scans the RPM database of the built per-arch image and finds installed binary RPMs plus non-RPM packages (Go modules, pip packages, etc.). One SBOM is produced per architecture. - Hermeto runs inside the prefetch-dependencies-oci-ta Tekton task. It records all build-time dependencies from lockfiles. A single Hermeto SBOM covers all architectures and source RPMs.
- Mobster runs as the
prepare-sbomsstep inside buildah-remote-oci-ta, merging the Syft and Hermeto SBOMs into one SPDX document per architecture. A second Mobster invocation in the build-image-index task generates an index-level SBOM and attaches it as an OCI artifact.
See Security Labels and Metadata for the SBOM entry structure and how to access SBOMs from the registry.
Stage 4: Testing
Images are validated through two types of integration tests:
- Container tests (
tests-container.yml) - Run with Podman and Docker via Testing Farm on RHEL-9-Nightly systems - K8s tests (
tests-k8s.yml) - Run in Konflux ephemeral Kubernetes namespaces
Test results appear as external jobs in GitLab CI pipelines, providing pass/fail status and links to the Konflux PipelineRun.
Test Triggering
Tests are only triggered on merge requests, not on main branch builds.
For an image group at images/<group-name>/, tests are triggered for all changes below that
directory, excluding documentation-only changes.
Any changes below .tekton/ or ci/ will trigger tests for the caddy image to ensure
infrastructure changes work before merging.
Test Execution
For each image group, pipelines are created per variant. If there is more than one variant in an image group, additional group pipelines are created. See Konflux group snapshot documentation for details.
Tests are selected based on the variants field in test files. Additionally, global tests from
ci/{variant}-tests/ are included. Group pipelines run tests that specify variants: [group],
allowing validation across multiple variants.
Per-component Kubernetes runs execute two categories of tests:
- Own-variant tests: tests defined for the component’s pair (the snapshot image is available
via
IMAGE_URL). - Cross-group reverse dependencies: tests in other image groups that reference the changed
component via
TEST_IMAGES[group/variant]. The changed image comes from the snapshot; their own images are unaffected by the change.
Reverse dependency tests that reference a same-group sibling image — a variant of the same
group other than the component under test — are excluded from per-component Kubernetes runs and run
only in the group pipelines. Own-variant test selection is unchanged. A per-component run provides
only the image under test; a test needing a sibling would fall back to a released image — silently
testing the changed component against an older sibling — or fail to pull when no release exists.
Group snapshot pipelines (the pr group GitLab status) cover all same-group combination tests at
the exact commit, with digest-pinned siblings provided by Konflux.
Reverse Dependency Testing
When a base image like core-runtime changes, dependent images (rust, xcaddy, etc.) need to be
tested to ensure compatibility. Reverse dependency testing is enabled by default. Images can opt-out
by setting reverse_dependency_tests: false in their properties.yml. This is recommended for
images like “curl” which are widely used and only have a small API which their own tests cover well
enough.
For container tests, dependent images are rebuilt locally in the testing environment to prevent version skew and ensure dependent images are in sync with the repository status in the merge request under test.
For Kubernetes tests, reverse dependency testing selects cross-group consumers of the changed component. Same-group combination tests (those referencing sibling variants of the same group) are excluded from per-component runs; they execute in the group pipelines where all sibling images are available at the exact commit.
Integration Test Scenarios
Integration tests are triggered via IntegrationTestScenario resources defined in this
repository (konflux-templates/scenarios/<type>.yml.j2 templates, instantiated by
konflux-templates/scenarios.yml) and deployed through the *-releases sync on merge to main.
The trigger manifests (integration_test_scenarios section) are generated
from the same scenarios.yml matrix so they cannot drift from the deployed CRs:
| Application | Purpose |
|---|---|
containers-hummingbird |
Red Hat supported Hummingbird images |
containers-community-hummingbird |
Community images (support_level: community) |
containers-ci-hummingbird |
Experimental/infrastructure images (support_level: experimental) |
containers-rawhide |
All Rawhide-based images |
Container Testing
Container tests run via Testing Farm on RHEL-9-Nightly systems for both x86_64
and aarch64 architectures.
Container Testing Flow
- Developer opens MR modifying
images/nginx/ - GitLab CI pipeline and Konflux pipelines start in parallel
- Konflux builds
nginximage variants IntegrationTestScenariotriggers Testing Farm job- tmt discovers enabled child plans in
ci/run_tests_container.fmf - Testing Farm provisions one RHEL-9-Nightly machine per enabled plan (per architecture)
- tmt sets up the testing environment on each machine
- tmt runs tests via
ci/run_tests_container.sh
Container Test ITS Configuration
The container tests use the upstream Testing Farm pipeline for Konflux CI:
kind: IntegrationTestScenario
spec:
contexts:
- { name: pull_request }
params:
- { name: COMPOSE, value: RHEL-9-Nightly }
- { name: ARCH, value: x86_64|aarch64 } # one for each
- { name: CONTEXT, value: fips=enabled } # containers-hummingbird only
- { name: IMAGE_TAG, value: v3.22 }
resolverRef:
resolver: bundles
params:
- {
name: bundle,
value: quay.io/testing-farm/tmt-via-testing-farm:$(params.IMAGE_TAG),
}
- { name: name, value: tmt-via-testing-farm }
- { name: kind, value: pipeline }
FMF Test Plan
The root folder of the containers repository is marked with a .fmf directory to enable Testing
Farm support. The test plan in ci/run_tests_container.fmf uses FMF virtual child
plans to define two execution variants:
/standard— inherits all parent configuration, always enabled/fips— inherits all parent configuration plus a FIPS prepare step; disabled by default, enabled viaadjustwhen thefips=enabledcontext is set
The parent node defines shared configuration (prepare, discover, execute) inherited by both
children. Because it has children, FMF treats it as a branch node and does not select it as a
runnable plan. The containers-hummingbird ITS passes CONTEXT=fips=enabled, so both /standard
and /fips run in parallel. Other applications (rawhide, ci-hummingbird, community-hummingbird)
pass no CONTEXT, so only /standard runs.
Both child plans run the following steps:
- Install
podmanfor container testing - Install Docker and start the Docker daemon on the host for Docker integration tests
- Fix git submodules until TFT-3991 is resolved
- For regular pipelines:
- Verify the image built by Konflux is reproducible using
ci/test_rebuild.sh - If needed, build reverse dependency images locally via Buildah
- Run Podman and Docker tests via
ci/run_tests_container.sh --component-name
- Verify the image built by Konflux is reproducible using
- For group pipelines:
- Run Podman and Docker group tests via
ci/run_tests_container.sh --group-component-name
- Run Podman and Docker group tests via
The /fips child additionally enables FIPS mode on the provisioned RHEL machine before all other
prepare steps (via order: 0).
Container Test Environment
Testing Farm provides these environment variables to the test plan:
| Variable | Description |
|---|---|
IMAGE_NAME |
Single component (e.g., curl--default--main) |
IMAGE_NAMES |
Multiple components for group pipelines |
IMAGE_URL |
Image URL from Konflux |
IMAGE_URL_... |
Image URLs from Konflux for group pipelines |
SNAPSHOT_b64 |
Snapshot metadata (base64 encoded) |
K8s Testing
K8s tests run in Konflux ephemeral namespaces provisioned via EaaS (Environment as a Service).
K8s Testing Flow
- Developer opens MR modifying an image in the containers repository (e.g.,
nginx) - GitLab CI pipeline and Konflux pipelines start in parallel
- Konflux builds image variants
IntegrationTestScenariotriggers K8s test pipeline- Pipeline checks for
tests-k8s.yml; skips with SUCCESS if not found - Pipeline provisions ephemeral namespace via Konflux EaaS (tied to PipelineRun lifecycle)
- Pipeline fetches source via Trusted Artifacts
- Tests run via
ci/run_tests_k8s.shwith kubeconfig for ephemeral namespace - Pipeline fails if any test reports non-SUCCESS, ensuring GitLab sees correct status
K8s Test ITS Configuration
The K8s tests use the k8s-test-pipeline:
kind: IntegrationTestScenario
spec:
contexts:
- { name: pull_request }
resolverRef:
resolver: bundles
params:
- { name: bundle, value: quay.io/hummingbird-ci/k8s-test-pipeline:latest }
- { name: name, value: k8s-test }
- { name: kind, value: pipeline }
Stage 5: Enterprise Contract Validation
Before images can be released, they must pass Enterprise Contract (also known as Conforma) policy validation. This ensures images meet security, compliance, and build quality standards. These checks can also be run locally against Konflux-built images.
Policy Validation
Enterprise Contract validates that:
- Images are built using trusted, verified Tekton tasks
- Builds are hermetic (network-isolated with pre-fetched dependencies)
- Required security tests have passed
- Images have proper metadata and labels
- Build artifacts meet supply chain security requirements
Policy Configuration
Policies are defined as EnterpriseContractPolicy resources in
konflux-templates/macros/policy.yml.j2:
containers-hummingbird/containers-rawhide- Strict policies for production imagescontainers-community-hummingbird- Policy for community-supported images in the containers repository (same base exclusions as production policies)
These policies use the @redhat rule collection from the ec-release-policy.
Policy Exclusions
The following checks are excluded from the default @redhat policy set. When modifying exclusions,
update the policy macro and this documentation.
Test Package
The test package verifies that each build was subjected to a set of tests and that those tests all passed.
Snyk SAST checks (test.required_tests_passed:sast-snyk-check, test.no_skipped_tests:sast-snyk-check,
test.required_tests_passed:sast-snyk-check-oci-ta, test.no_skipped_tests:sast-snyk-check-oci-ta)
are excluded because Hummingbird images are currently not supported by Snyk.
Red Hat certification preflight checks (test.no_failed_tests:ecosystem-cert-preflight-checks,
test.no_erred_tests:ecosystem-cert-preflight-checks) are excluded because Hummingbird images are
not yet published to the Red Hat certified container registry.
Informative test failures (test.no_failed_informative_tests) are excluded because these produce
warnings for advisory purposes only and are explicitly non-blocking.
Deprecated image warnings (test.no_test_warnings:deprecated-image-check) are excluded because
final images are built FROM scratch, and the builder image is updated via Renovate like all other
images.
Trusted Task Package
The trusted_task package verifies that all Tekton Tasks involved in building the image are trusted by comparing Task references with a pre-defined list of trusted Tasks.
The trusted_task.current check warns when newer versions of tasks are available. This is excluded
because we use stable pinned task versions and control upgrade timing via Renovate rather than
requiring the latest version at all times.
RPM Repos Package
The rpm_repos package confirms that all RPM packages listed in SBOMs specify a known and permitted repository ID.
The rpm_repos.ids_known check is excluded because images use the internal
hummingbird repository and Fedora repositories, which are not
in the upstream known_rpm_repositories.yml list (that file only contains Red Hat
official repositories).
RPM Signature Package
The rpm_signature package verifies that all RPMs in the image are signed by an allowed signing key.
The upstream default only allows the Red Hat release key 2
(199e2f91fd431d51). Rawhide, community, and experimental images contain
Fedora-signed RPMs, so the containers-rawhide,
containers-community-hummingbird, and containers-ci-hummingbird policies
add Fedora signing keys via the additional_rpm_signature_keys macro
parameter. The containers-hummingbird policy uses the default (Red Hat key
only) because Hummingbird images install RPMs exclusively from the
Hummingbird repository.
The Fedora key list in konflux-resources.yml.j2 should be kept in sync with
the keys published at https://fedoraproject.org/security.
Labels Package
The labels package checks if the image has the expected labels set, including required and optional labels for Red Hat container certification.
Both labels.required_labels and labels.optional_labels are excluded because images currently
only include basic labels (maintainer, license_terms, name, cpe) and version labels, not
the full set of Red Hat certification labels (vendor, version, release, summary,
description, url, etc.) required for Red Hat Ecosystem Catalog publishing.
Buildah Build Task Package
The buildah_build_task package verifies buildah build task parameters.
The buildah_build_task.privileged_nested_param check verifies that PRIVILEGED_NESTED is not set
to true. This is excluded because images use the dnf-installroot helper from
the builder image to build all containers, including the builder image itself. This script requires
privileged operations (unshare, mount -t tmpfs, mount --bind for /proc and /dev/*) to set
up the install root environment (see commit 3668af16).
Schedule Package
The schedule package verifies that releases conform to a given schedule, including weekday restrictions.
The schedule.weekday_restriction check is excluded to allow releases any day including weekends.
The @redhat policy restricts weekend releases, but this project needs the ability to ship urgent
CVE fixes immediately regardless of the day of week.
CVE Package
The cve package checks for blocking and non-blocking CVEs in container images.
The cve.cve_blockers check is excluded because blocking on known CVEs would prevent releasing
images that fix other CVEs. A VEX feed could suppress false positives for RPM-level CVEs, but CVEs
can also originate from other artifact types where VEX does not apply (see MR !2227).
Hermetic Task Package (CI-Only)
The hermetic_task package verifies that tasks were invoked with the proper parameters to perform a hermetic (network-isolated) execution.
All Konflux applications in the containers repository enforce the hermetic task check. All
applications allow quay.io/hummingbird-ci/ as a base image source because images build
FROM quay.io/hummingbird-ci/hummingbird-builder via setup_newroot.yml.j2.
Stage 6: Release
After images pass testing in merge requests and are merged to main, they are released to public registries.
Registry Organization
Images are published to different registries based on distro and purpose:
| Registry | Purpose |
|---|---|
registry.access.redhat.com/hi/ |
Red Hat supported Hummingbird images (production) |
quay.io/hummingbird/ |
Red Hat supported Hummingbird images (mirror) |
quay.io/hummingbird-community/ |
Community-supported Hummingbird images |
quay.io/hummingbird-rawhide/ |
All Rawhide-based images |
quay.io/hummingbird-ci/ |
CI and tools images |
Production Registry: Red Hat supported Hummingbird images are published to
registry.access.redhat.com/hi/ via the rh-advisories release pipeline and
mirrored to quay.io/hummingbird/.
Community Registry (hummingbird-community): Publishes community-supported images
(those with support_level: community in properties.yml). Examples include minio,
minio-client, and os-minimal.
CI/tools Registry (hummingbird-ci): Publishes tooling and infrastructure images. This
includes tools-repo images (built outside the containers repository) and experimental images
from the containers repository (support_level: experimental), such as hummingbird-builder.
Release Process
- Merge request passes all tests
- Merge request is merged to main branch
- Konflux builds images from main
- ReleasePlanAdmission resources trigger the release pipeline
- Images are signed via Cosign for supply chain attestation
- Images are copied from the Konflux registry to target registries
- Tags are applied based on
properties.ymlconfiguration - For production releases: images are registered in the Red Hat container catalog via Pyxis
Release Mechanism
Releases are configured via ReleasePlanAdmission (RPA) resources in the containers repo and ReleasePlan resources in the infrastructure repo:
- ReleasePlanAdmission - Defines per-registry release configuration (target registry, tags, visibility settings)
- ReleasePlan - Triggers the release pipeline for a specific application and registry
Each distro/registry combination has its own RPA.
Production release pipeline
The production release uses the rh-advisories pipeline from the release-service-catalog.
The push-snapshot task pushes images from the development registry to:
- Production:
registry.access.redhat.com/hi/<image>:<tags>
Configuration is defined in releng/hummingbird-containers-prod.yaml.
Image signing
Images are signed using Cosign as part of the release pipeline. Signing provides supply chain attestation, allowing consumers to verify image provenance.
Pyxis catalog registration
The release pipeline registers images in the Red Hat container catalog via the Pyxis API:
- Product: Red Hat Hardened Images (Product ID 1071)
- Metadata: version, categories, layer information, SBOM references
- Configuration:
releng/pyxis-hummingbird.yaml
Pyxis registration makes images discoverable in the Red Hat Ecosystem Catalog and links them to security advisories.
Release Output
Released images are published to registries based on distro and support level:
registry.access.redhat.com/hi/<image-repository>:<image-tag>
quay.io/hummingbird/<image-repository>:<image-tag>
quay.io/hummingbird-community/<image-repository>:<image-tag>
quay.io/hummingbird-rawhide/<image-repository>:<image-tag>
quay.io/hummingbird-ci/<image-repository>:<image-tag>
Examples:
registry.access.redhat.com/hi/curl:8- Production Red Hat supported imagequay.io/hummingbird/nodejs:20- Red Hat supported Hummingbird image (mirror)quay.io/hummingbird-community/minio:latest- Community-supported imagequay.io/hummingbird-ci/hummingbird-builder:latest- Hummingbird Builder imagequay.io/hummingbird-rawhide/curl:latest- Rawhide image
Release Tags
Tags are extracted from Containerfile labels as defined in properties.yml:
latest- Latest version of the image<major>- Major version (e.g.,20for Node.js 20.x)<major>.<minor>- Major.minor version (e.g.,20.11)<full-version>- Complete version with release (e.g.,20.11.1-1.fc42)<YYYYMMDD>- UTC date when RPM lockfile content last changed, available viarpm_lockfile.content_datefor any image that includes it in its tags. Only advances when the resolved RPM set changes.<timestamp>- Build timestamp (production releases only)
Non-default variants receive a -<variant> suffix (e.g., latest-builder).
Advisory Creation
When the production release pipeline runs, advisories are created:
- The Release Service generates an Advisory YAML from ReleasePlan and ReleasePlanAdmission metadata
- The advisory is pushed to the advisories repo on CEE GitLab
- GitLab CI validates the advisory against the schema and enforces field-level permissions
- On merge to main, the advisory is published via UMB and Kafka
Advisory types:
| Type | Meaning |
|---|---|
| RHSA | Security Advisory (contains CVE fixes) |
| RHBA | Bug Fix Advisory |
| RHEA | Enhancement Advisory |
After publication, the Pyxis API links each image record to its advisory via
image_advisory_id.
VEX Feed Update
After advisory publication, Red Hat SDEngine generates public vulnerability data:
- CSAF VEX documents (per-CVE) published at
https://security.access.redhat.com/data/csaf/v2/vex-feed/ - CSAF Advisory documents (per-advisory) published at
https://security.access.redhat.com/data/csaf/v2/advisories/
The hummingbird-vex-feed repository maintains CPE mappings that associate
Hummingbird package repositories with cpe:/a:redhat:hummingbird:1.
Image Documentation
When a commit to main changes a README.md file, the content is automatically
pushed to quay.io as the image description via the update_quay_description
job.
References
- RPM Pipeline - How RPM spec changes flow through the build pipeline to the package repository
- Development Workflow - Practical guide for contributing
- Adding Images - Step-by-step guide for adding new images
- Testing Guide - How to run and write tests locally
- Image Configuration Reference - Complete
properties.ymlreference
4.3.2 - Global Variables Reference
Complete reference for global configuration settings that apply to all container images.
Overview
The images/variables.yml file contains global configuration that applies to
all container images in the repository. These settings define project-wide
defaults, security parameters, and variant-specific behavior.
Location: images/variables.yml (in the containers repository root)
Scope: All images inherit these settings unless overridden in
image-specific properties.yml
Configuration Reference
cpe
- Type: String (CPE 2.2 formatted string)
- Default:
"cpe:/a:redhat:hummingbird:1" - Description: Common Platform Enumeration (CPE) identifier for all Hummingbird containers.
- Usage: Automatically added as a label to all generated container images
and passed to
inject-source-infoduring the build process. - Format: CPE 2.2 URI format:
cpe:/part:vendor:product:version
default_user
- Type: String (numeric UID)
- Default:
"65532" - Description: The default unprivileged user ID that all containers run as.
- Usage: Referenced by the
{{ set_user() }}macro whenuser: defaultis specified (or whenuser:is omitted, asdefaultis the default value). - Value:
65532is chosen as a high, non-conflicting UID that:- Avoids conflicts with system users (typically < 1000)
- Avoids conflicts with regular users (typically 1000-60000)
default_distros
- Type: Array of strings
- Default:
["rawhide", "hummingbird"] - Description: Defines the distribution variants (distros) for which images are built. Each distro represents a different base package source.
- Directory Structure: Images are organized as
images/<name>/<distro>/<variant>/ - Common Values:
rawhide- Uses Fedora Rawhide repositories onlyhummingbird- Uses both Fedora Rawhide and Hummingbird repositories
- Usage: Scripts iterate over distros and variants to find all image variants to build. Each distro/variant combination produces a separate container image.
- See Also: default_variant_repos for repository configuration per distro
default_variants
- Type: Array of strings
- Default:
["default", "builder"] - Description: Defines the default set of variants to generate for each
image within each distro when not explicitly specified in the image’s
properties.yml. - Directory Structure: Variants are subdirectories within each distro:
images/<name>/<distro>/<variant>/ - Common Values:
default- The minimal runtime variantbuilder- Extended variant with build tools and package managers
- Extending: Images can use
additional_variantsinproperties.ymlto add extra variants while keeping the defaults (preferred approach) - Override: Images can specify
variantsinproperties.ymlto replace these defaults entirely (use when you need to exclude default variants) - See Also: Image Configuration Reference - variants
default_rpm_packages
-
Type: Object with
allor variant names as keys, arrays of package names as values -
Default:
default_rpm_packages: all: - system-release builder: - bash - dnf5 - shadow-utils -
Description: Adds packages to the packages from each image’s
properties.yml. Both lockfile generation and Containerfile generation use these defaults. -
Variants:
allapplies to every image and variant.builderalso applies to variants with the-buildersuffix and thehummingbird-builderimage. -
Usage: Use
allfor shared image requirements. Thesystem-releasepackage preserves distribution identity and repository configuration aftersetupremoved its dependency onsystem-release. Usebuilderfor package management tools.
default_variant_repos
-
Type: Object with distro names as keys, arrays of repository filenames as values
-
Default:
default_variant_repos: rawhide: - fedora-44.repo hummingbird: - fedora-44.repo - hummingbird.repo -
Description: Defines which yum repositories each distro uses by default for package installation and lockfile generation. The Rawhide distro uses the latest branched Fedora development repo, while Hummingbird uses a stable Fedora release plus the Hummingbird package repository.
-
Repository Files: References files in the
yum-repos/directory -
Usage: This allows different distros to use different package sources. The
hummingbirddistro includes additional repositories that provide Hummingbird-specific packages. -
Override: Image-specific
additional_reposinproperties.ymlare appended to the distro-specific repos
oscap
-
Type: Object
-
Default:
oscap: enabled: true profiles: cis: true stig: variants: - "*fips*" exclude_rules: - id: xccdf_org.ssgproject.content_rule_root_path_no_dot reason: "False positive: environmentvariable58 probe requires /proc which is unavailable in offline chroot scanning" -
Description: Global defaults for OpenSCAP compliance scanning. Scanning is enabled by default (
enabled: true). Images can opt out by settingenabled: falsein theirproperties.yml. -
Fields:
enabled- Whether compliance scanning is active (overridden per image)profiles- Which compliance profiles run per variant. Each profile can betrue(all variants),false(disabled), or{variants: [...]}with glob patterns. Default: CIS for all variants, STIG for FIPS variants.exclude_rules- Global rule exclusions applied to all oscap-enabled images. Image-specific exclusions inproperties.ymlare concatenated with these (not replaced).
-
Merge Behavior: When an image overrides
oscapfields:enabled(scalar) is replaced by the image valueprofiles(dict) is recursively merged, so an image can override individual profiles without affecting othersexclude_rules(list) is concatenated, so image rules are added after global rules
-
See Also: Image Configuration Reference - Compliance Scanning
readme_targets
-
Type: Object with target names as keys, configuration objects as values
-
Default: See the actual file
-
Description: Defines target-specific configuration for generating multiple README files from a single template. Each target represents a different build/distribution channel (e.g., Hummingbird project vs Red Hat product).
-
Target Configuration Fields:
filename- Output filename for this target’s READMEregistry- Container registry URL for image referencesproduct_name- Full product name for documentationproduct_name_short- Short product name for headings and titlesdoc_base_url- Base URL for documentation links
-
Usage: Templates access these values via
{{ readme_targets[target].registry }}and similar expressions. The build system generates one README file per target from the sameREADME.md.j2template. -
See Also: Image Pipeline - README Generation
Next Steps
- Image Configuration Reference - Per-image configuration
- Image Variants - Understanding variant types
- Adding Images - How to add a new container image
4.3.3 - Konflux Resource Deployment
Konflux resources for container images are split between two repositories:
- containers: Per-image resources generated from
properties.yml - infrastructure: Application-level resources and deployment to the Konflux cluster
This separation allows independent iteration on each concern.
Deployment Matrix
| Resource Type | Source | Containers MR | Containers Push | Infra MR | Infra Push |
|---|---|---|---|---|---|
| Component | containers | manual | manual | - | - |
| ImageRepository | containers | manual | manual | - | - |
| ReleasePlanAdmission | containers | - | automatic | manual | automatic |
| EnterpriseContractPolicy | containers | - | automatic | manual | automatic |
| IntegrationTestScenario | containers | - | automatic | manual | automatic |
| Application | infrastructure | - | - | manual | automatic |
| ReleasePlan | infrastructure | - | - | manual | automatic |
| ServiceAccount, Secret | infrastructure | - | - | manual | automatic |
Legend:
- Containers MR: Downstream pipeline triggered from containers MR (deploys from MR commit)
- Containers Push: Downstream pipeline triggered from containers push to main
- Infra MR: Infrastructure MR pipeline (manual trigger)
- Infra Push: Infrastructure push to main or web pipeline
Resource Locations and Rationale
Containers Repo
Resources are defined in konflux-templates/ and rendered to konflux-templates/rendered.yml.
Component and ImageRepository:
- Ownership: Components are managed by the containers repo. The infrastructure pipeline uses
ONLY_DOWNSTREAMso only explicit downstream triggers from containers deploy changes, giving the containers repo full control over the component lifecycle. - Dynamic generation: Generated from
images/*/properties.ymlviaci/internal/generate_konflux_resources.sh, depending on per-image configuration (variants, tags, repository names). - Timing: Must be deployed early during MR review so Konflux can build and test new images.
ReleasePlanAdmission:
- Dynamic generation: Includes per-image tag mappings extracted from
properties.yml. - Main branch only: Prevents race conditions—deploying from a feature branch risks undoing other images’ onboarding. If branch A adds image-foo and branch B (created earlier) adds image-bar, deploying B’s RPA would remove image-foo from the release config.
EnterpriseContractPolicy:
- Documentation co-location: Policy exclusions are documented in Image Pipeline. Keeping policy and documentation together eases maintenance.
IntegrationTestScenario:
- Ownership co-location: Scenarios live alongside the components,
tests-k8s.ymlinventory, and trigger manifests that describe them. Config is split between per-type CR templates (konflux-templates/scenarios/<type>.yml.j2, owning all params and resolver refs) and a pure application/trigger matrix (konflux-templates/scenarios.yml). Both the CR renderer (konflux-templates/rendered.yml) and the trigger-manifest generator read the same config, so deployed scenarios and manifest expectations cannot drift apart. See the trigger manifests reference for the manifest schema. - Deployment: Deployed through the
*-releasessync (fetchesrendered.ymlfrom the main branch on every merge to main), not the*-componentssync. The components sync deploys from MR-head SHAs, so concurrent or stale MR pipelines would reset the live ITS CR to an older version; ITS is static config that is not supposed to change in MRs. The releases sync avoids this race and also means Renovate bundle-digest and IMAGE_TAG bumps auto-deploy on merge without any manual step. - Renovate: Bundle digest and IMAGE_TAG pins are managed by regex managers in this repo’s
renovate.json.
Infrastructure Repo
Application and ReleasePlan are defined in kubernetes/containers-*/ directories (one per
Konflux application):
- Independent iteration: Changes are decoupled from containers repo activity—they can be modified, test-deployed via manual trigger in an infrastructure MR, verified, and merged without touching the containers repo.
- Testable before merge: If defined in the containers repo, these would only deploy after merging to main (like RPAs), making iteration difficult.
- Static configuration: These resources don’t depend on per-image data from
properties.yml.
ServiceAccount and Secret for releases are defined in
kubernetes/release-quay-hummingbird*/ directories (one per Quay organization):
- Security: Secret specifications (names, structure, credential references) should not be exposed in the containers repo.
- Independent iteration: Like other infrastructure resources, these can be modified and test-deployed without touching the containers repo.
The service accounts are referenced by ReleasePlanAdmissions to authorize pushing images to
each quay.io/hummingbird* organization.
See Also
- Image Pipeline – Build, test, and release stages
4.3.4 - Security Labels and Metadata
Hummingbird container images include labels, embedded metadata, and SBOMs that enable security scanners to perform container-first vulnerability reporting.
Overview
Security scanners need a way to determine which vulnerabilities are applicable to a container image. Three mechanisms provide this information:
- Container Labels - OCI image labels (
name,cpe) set in the Containerfile - Embedded Metadata - A
labels.jsonfile written to the container filesystem during build - Software Bill of Materials (SBOM) - An SPDX document listing all packages, attached as an OCI artifact alongside the image
Labels and embedded metadata identify the product; the SBOM identifies the packages within it. Labels and embedded metadata provide the same core information but serve different access paths: labels are accessible via container inspection tools, while embedded metadata is accessible to scanners with only filesystem access.
Applicability
The cpe label is only added to released Hummingbird distro images (no
CPE identifier for non-product images). All other labels are added to all
images.
The labels.json embedded metadata file contains all Containerfile LABEL
values plus auto-computed fields.
SBOMs are attached to all released images (both Rawhide and Hummingbird).
Container Labels
name
- Type: String
- Format:
<org>/<image>or<org>/<image>-<variant> - Examples:
hummingbird/nodejs-24,hummingbird/nodejs-24-builder,hummingbird/caddy - Description: Canonical name for the container image. This is the name that appears in VEX (Vulnerability Exploitability eXchange) statements and is used by scanners to map vulnerabilities to specific images.
The canonical name is derived from the image directory name (which includes
the version, e.g. nodejs-24) and the variant. For default variants, the
name is hummingbird/<image> (e.g. hummingbird/nodejs-24). For non-default
variants, the variant is appended with a hyphen (e.g.
hummingbird/nodejs-24-builder). Note that the canonical name may differ
from the registry repository path. For example, nodejs-24 and nodejs-20
are both pushed to the hummingbird/nodejs registry repository with
different tags, but their canonical names are hummingbird/nodejs-24 and
hummingbird/nodejs-20 respectively.
See container-image-labels.md for the full
name label format across all image types.
cpe
- Type: String (CPE 2.2 formatted)
- Format:
cpe:/a:redhat:hummingbird:1 - Description: Common Platform Enumeration (CPE) identifier for the container. CPE is a standardized naming scheme for software products that enables correlation with vulnerability databases.
The CPE value is defined globally in images/variables.yml and applied to all
Hummingbird images. Containers with the same CPE are considered the same
software product for vulnerability reporting purposes.
org.opencontainers.image.created
- Type: String (RFC3339 timestamp)
- Format:
2025-01-15T12:00:00Z - Description: Creation timestamp of the container image. Used by scanners to determine if an image predates or postdates a vulnerability fix.
If SOURCE_DATE_EPOCH is set during the build, that timestamp is used instead
of the actual build time, supporting reproducible builds.
Embedded Metadata (labels.json)
The labels.json file is written to /usr/share/buildinfo/labels.json inside
the container filesystem by the inject-source-info script during build. It
contains all Containerfile LABEL values plus auto-computed fields like
architecture and org.opencontainers.image.created.
Schema
The file follows the embedded_metadata.v1 schema published by Red Hat Product Security.
Fields
The embedded_metadata.v1 schema defines the minimum fields required by security scanners:
| Field | Type | Description |
|---|---|---|
name |
string | Canonical container name (e.g., hummingbird/nodejs-24) |
cpe |
string | CPE identifier (e.g., cpe:/a:redhat:hummingbird:1) |
architecture |
string | Target architecture (e.g., amd64, arm64) |
org.opencontainers.image.created |
string | RFC3339 creation timestamp |
The file also includes all other Containerfile LABEL values (e.g.,
description, summary, vendor, version). See
container-image-labels.md for the complete
label reference.
Example
{
"name": "hummingbird/caddy",
"cpe": "cpe:/a:redhat:hummingbird:1",
"architecture": "amd64",
"org.opencontainers.image.created": "2025-01-15T12:00:00Z"
}
Software Bill of Materials (SBOM)
Each released image has a per-architecture SPDX 2.3 SBOM attached as an OCI artifact. See SBOM Generation for how the production SBOMs are built from Syft, Hermeto, and Mobster.
Accessing SBOMs
SBOMs are stored as OCI artifacts alongside each per-architecture image. To download one, resolve the per-arch digest and use cosign:
IMAGE="registry.access.redhat.com/hi/caddy"
TAG="latest"
ARCH="amd64"
# Get per-architecture digests from the image index
skopeo inspect --raw "docker://${IMAGE}:${TAG}" \
| jq '.manifests[] | {digest, platform}'
# Download the SBOM for the chosen architecture
ARCH_DIGEST="$(skopeo inspect --raw "docker://${IMAGE}:${TAG}" \
| jq -r --arg arch "${ARCH}" \
'.manifests[] | select(.platform.architecture == $arch) | .digest')"
cosign download sbom "${IMAGE}@${ARCH_DIGEST}" > sbom.json
Entry Sources
The merged SBOM contains entries from two tools:
| Source | Content | Scope |
|---|---|---|
| Syft | Installed binary RPMs + non-RPM packages | Image architecture only |
| Hermeto | Build-time dependencies from lockfiles | All architectures + source RPMs |
A package installed in the final image may have entries from both Syft and Hermeto (with different metadata), since it was both a build dependency and is present at runtime.
Distinguishing Entry Sources
After the Mobster merge, Syft and Hermeto entries carry mutually exclusive markers:
| Marker | Syft | Hermeto |
|---|---|---|
sourceInfo field present |
✓ | |
licenseDeclared set |
✓ | |
| CPE references | ✓ | |
upstream= in PURL |
✓ | |
| Annotation containing “hermeto” | ✓ | |
repository_id= in PURL |
✓ |
As a general rule, any entry with an annotation containing the word “hermeto” (case-insensitive) originates from Hermeto (build-time provenance). All other entries originate from Syft (runtime image scan).
PURL Format
Syft and Hermeto use different PURL qualifier sets for the same package. Examples from caddy:latest (Hummingbird, amd64):
Syft (runtime):
pkg:rpm/hummingbird/caddy@2.10.2-1.hum1?arch=x86_64&distro=hummingbird-20251124&upstream=caddy-2.10.2-1.hum1.src.rpm
Hermeto (build, binary):
pkg:rpm/caddy@2.10.2-1.hum1?arch=x86_64&checksum=sha256:ca02a0...&repository_id=public-hummingbird-x86_64-rpms
Hermeto (build, source):
pkg:rpm/caddy@2.10.2-1.hum1?arch=src&checksum=sha256:42912d...&repository_id=public-hummingbird-source-rpms
Key differences:
- Distro namespace: Syft includes the distro in the PURL path
(
pkg:rpm/hummingbird/...). Hermeto omits it (pkg:rpm/...). - EVR: Syft populates
versionInfowith the full epoch:version-release. Hermeto setsversionInfoto the bare upstream version; the full EVR is only in the PURL@version. - Architecture: Syft entries match the image architecture. Hermeto entries
carry an
arch=PURL qualifier that may be the image arch, a different arch (cross-arch),noarch, orsrc(source RPMs). - Source RPM: Syft entries carry
upstream=<srpm>in the PURL. Hermeto entries have separatearch=srcentries instead.
Hermeto Annotation Format
Hermeto entries carry annotations with JSON-encoded metadata:
{
"annotationDate": "2026-03-02T11:58:12Z",
"annotationType": "OTHER",
"annotator": "Tool: hermeto:jsonencoded",
"comment": "{\"name\": \"hermeto:found_by\", \"value\": \"hermeto\"}"
}
Entry Breakdown Example
Typical counts for caddy:latest (Hummingbird, amd64); exact counts vary as image dependencies change:
| Source | Typical count | Content |
|---|---|---|
| Syft | ~53 | Binary RPMs (x86_64 + noarch) |
| Syft | 1 | Go module (stdlib) |
| Syft | 2 | OCI image metadata |
| Hermeto | ~41 | Binary RPMs (x86_64) |
| Hermeto | ~41 | Binary RPMs (aarch64, cross-arch) |
| Hermeto | ~11 | Binary RPMs (noarch) |
| Hermeto | ~36 | Source RPMs (arch=src) |
How Scanners Use This Metadata
- CPE Matching: Scanners use the
cpevalue to look up applicable VEX statements for the product - Name Matching: The
nameidentifies which specific container the VEX statements apply to - Version Comparison: The creation timestamp enables comparison between the scanned image and fixed versions reported in VEX statements
- Package Enumeration: Scanners use the SBOM to enumerate all packages in the image and correlate them with vulnerability databases via PURLs and CPEs
Related Files
| File | Purpose |
|---|---|
documentation/background/container-image-labels.md |
Complete reference for all image labels |
documentation/background/image-pipeline.md |
SBOM generation pipeline (Stage 3) |
images/variables.yml |
Defines the global cpe value |
images/hummingbird-builder/inject-source-info.sh |
Script that creates labels.json |
macros/inject_source_info_labels.yml.j2 |
Macro that adds LABEL to Containerfile |
macros/install_newroot.yml.j2 |
Macro that invokes inject-source-info |
See Also
- Container Image Labels – complete reference for all image labels across all categories
- Image Pipeline – SBOM Generation – how the production SBOMs are built from Syft, Hermeto, and Mobster
References
- Embedded Metadata Schema
- Container-First Vulnerability Reporting (Konflux feature specification)
- VEX (Vulnerability Exploitability eXchange)
- Syft – SBOM generation tool
- Hermeto – build-time dependency provenance tool
- Mobster – SBOM merging tool
- SPDX Specification
4.3.5 - Container Image Labels
Hummingbird container images carry labels from multiple standards and namespaces. This page is the single reference for every label.
Standards: C = Conforma (rule dataset), O = OCI Image Spec, H = Hummingbird project, S = Security schema
Labels
| Label | Aliases | Value | C | O | H | S |
|---|---|---|---|---|---|---|
architecture |
Host architecture (e.g., x86_64) |
✓ | ||||
com.redhat.component |
hummingbird |
✓ | ||||
com.redhat.license_terms |
UBI EULA ¹ | |||||
cpe |
CPE identifier (Hummingbird only) ² | ✓ | ✓ | |||
distribution-scope |
public |
✓ | ||||
io.hummingbird-project.containerfile |
Containerfile path relative to repo root ⁹ | ✓ | ||||
io.hummingbird-project.deprecated |
true for deprecated image streams |
✓ | ||||
io.hummingbird-project.major-minor-version |
Major.minor from tags (e.g., 2.10) ³ |
✓ | ||||
io.hummingbird-project.major-version |
Major from tags (e.g., 2) ³ |
✓ | ||||
io.hummingbird-project.repository |
Publishing name (e.g., caddy) ⁴ |
✓ | ||||
io.hummingbird-project.stream |
Version stream (e.g., 2) ⁴ |
✓ | ||||
io.hummingbird-project.variant |
Variant name (e.g., fpm-builder) ⁸ |
✓ | ||||
io.hummingbird-project.variant.base |
Base specialization (e.g., fpm) ⁸ |
✓ | ||||
io.hummingbird-project.variant.builder |
true when builder (absent otherwise) ⁸ |
✓ | ||||
io.hummingbird-project.variant.description |
Base variant description (no modifiers) ⁸ | ✓ | ||||
io.hummingbird-project.variant.fips |
true when FIPS (absent otherwise) ⁸ |
✓ | ||||
io.k8s.description |
Long description ⁵ | ✓ | ||||
maintainer |
Project Hummingbird / Red Hat |
✓ | ||||
name |
Canonical publishing repository ² | ✓ | ✓ | |||
org.opencontainers.image.created |
build-date |
RFC3339 build timestamp | ✓ | ✓ | ✓ | |
org.opencontainers.image.description |
description |
Long description ⁵ | ✓ | ✓ | ||
org.opencontainers.image.revision |
vcs-ref |
Full git commit SHA | ✓ | ✓ | ||
org.opencontainers.image.source |
GitLab repository URL | ✓ | ||||
org.opencontainers.image.title |
Image name (e.g., caddy) |
✓ | ||||
org.opencontainers.image.url |
url |
Upstream project URL ⁵ | ✓ | ✓ | ||
org.opencontainers.image.vendor |
vendor |
Red Hat, Inc. ⁶ |
✓ | ✓ | ||
org.opencontainers.image.version |
version |
Full version from tags (e.g., 2.10.2) ³ |
✓ | ✓ | ||
release |
SOURCE_DATE_EPOCH (commit timestamp) ⁷ |
✓ | ||||
summary |
Short one-liner ⁵ | ✓ | ||||
vcs-type |
git |
✓ |
Notes
¹ Licenses
The com.redhat.license_terms label is not required by Conforma, OCI, or
the security schema. It is a Red Hat convention present on all Red Hat
container images (UBI, language runtimes, etc.), pointing to the UBI EULA.
The org.opencontainers.image.licenses label is intentionally not set.
Per-package license data is derived from
SBOMs
stored as OCI artifacts alongside each image. A manually-set SPDX expression
would be incomplete compared to the SBOM-derived data available in the image
catalog.
² Name and identity
The name label identifies the canonical publishing repository. It uses the
repository value from properties.yml, prefixed by the registry organization:
- Red Hat supported Hummingbird images:
hi/<repository> - Community images:
hummingbird-community/<repository> - Experimental images:
hummingbird-ci/<repository> - Rawhide images:
hummingbird-rawhide/<repository>
All variants published to the same repository have the same name label. For
example, the default and builder variants of Caddy both use hi/caddy.
The cpe label is only set for Hummingbird distro images (not Rawhide).
See Security Labels and Metadata for
details on the scanning workflow and labels.json.
³ Version labels
Three version granularities are derived from the image’s tags:
version (full, e.g., 2.10.2), major-minor-version (e.g., 2.10),
and major-version (e.g., 2). The full version is the OCI/Conforma
org.opencontainers.image.version; the two coarser granularities are
Hummingbird project labels used for tag aliasing. There is no
io.hummingbird-project.version label — it would duplicate the OCI label.
⁴ Repository and stream
The io.hummingbird-project.repository and io.hummingbird-project.stream
fields together form the release identity pair. Both are defined in
properties.yml. See
image-configuration-reference
for field definitions.
The io.hummingbird-project.deprecated label is present with the value true on
the final release of a deprecated stream. It is omitted for non-deprecated streams. The
catalog uses the label to preserve the current deprecation state without changing
older image digests.
⁵ Description, summary, and URL
The description, summary, and url fields are defined in
properties.yml. Description and summary target different display
contexts:
- summary: One-liner (~40-80 chars) for table/list views
- description: Short paragraph (~100-250 chars, 1-2 sentences) for
card views and
podman inspect
Style rules:
- Do not start
summaryordescriptionwith the image name - Use
>-YAML scalar for multi-line readability inproperties.yml - Avoid embedded double quotes and backslashes (no escaping in templates)
⁶ Vendor
All images use Red Hat, Inc. as vendor (the distributing entity).
The OCI org.opencontainers.image.vendor uses Red Hat (without
“, Inc.”) per OCI convention. Both values are set on all distros.
⁷ Release
The release label uses the commit timestamp as a Unix epoch (matching
Red Hat convention). In Konflux this is set via the generate-labels
pipeline task.
⁸ Variant labels
Each variant name is decomposed into a base specialization and
cross-cutting modifiers (builder, fips). The naming convention is
<base>[-fips][-builder], where modifier order does not matter. A bare
modifier like builder has base default.
Examples: fpm-builder → base=fpm, builder=yes; fips-builder →
base=default, builder=yes, fips=yes.
The .description label stores only the base description (e.g.,
“PHP FastCGI process manager”). Modifier display is handled by
consumers using the .builder and .fips boolean labels. Base
descriptions come from variant_descriptions in images/variables.yml
(for default) and image-specific properties.yml (for bases like
fpm, runtime).
⁹ Containerfile path
The io.hummingbird-project.containerfile label contains the path to the
Containerfile source relative to the repository root (e.g.,
generated/images/caddy/hummingbird/default/Containerfile). A full URL to the
source can be constructed from org.opencontainers.image.source +
org.opencontainers.image.revision + this path.
Embedded Metadata (labels.json)
All labels are written to /usr/share/buildinfo/labels.json inside the
container filesystem, providing filesystem-level access for security
scanners. See Security Labels and Metadata
for schema details.
Related Files
| File | Purpose |
|---|---|
images/<image>/properties.yml |
Per-image metadata fields |
ci/internal/generate_jinja2.py |
Builds inject_labels dict from properties |
macros/image_metadata_labels.yml.j2 |
Emits LABEL instructions from inject_labels |
macros/inject_source_info_labels.yml.j2 |
Emits name and cpe LABEL instructions |
macros/install_newroot.yml.j2 |
Invokes inject-source-info.sh with labels |
images/hummingbird-builder/inject-source-info.sh |
Writes labels.json to container filesystem |
ci/build_images.sh |
Adds build-time labels |
ci/check_release_fields.py |
Validates label field values in properties.yml |
4.3.6 - CI Scripts
This section documents the CI scripts used in the Project Hummingbird container build and test pipeline.
These scripts can be run locally for development and testing, and are also used by the automated CI/CD pipeline.
4.3.6.1 - build_images.sh
Purpose
Build container images using buildah with support for multiple architectures and container engines.
Usage
Usage: ci/build_images.sh [OPTIONS] [GROUP_NAMES...] [-- BUILDAH_ARGS...]
OPTIONS:
--verbose, -v Enable verbose output during build
--arch ARCHITECTURE Specify target architecture (e.g., amd64, arm64, s390x)
--engine ENGINE Specify runtime engine for testing/export (podman or docker)
--setup Set up Docker-in-Docker environment before exporting
--component-name NAME
Parse component name (format: group--distro--variant)
Example: curl--rawhide--default → curl/rawhide/default
--local-rpms-dir DIR Directory containing custom RPMs to use during build.
These have a higher priority and thus override the
standard repositories.
--build-deps Build all dependencies of specified groups, but NOT the
specified groups themselves. Collects forward, reverse,
and transitive dependencies automatically.
--affected-tests With --build-deps, select dependencies used by affected tests
--pull Pull images from registry instead of building locally.
Fast alternative for local development. Uses published
images from quay.io.
--dryrun Show what would be built (or pulled with --pull) without
actually building/pulling.
--help, -h Show this help message
Examples
# Build single image group (all distro/variants)
ci/build_images.sh nginx
ci/build_images.sh nodejs-20
# Build specific distro/variant only
ci/build_images.sh nginx/rawhide/builder
ci/build_images.sh nodejs-20/hummingbird/default
# Build multiple image groups at once
ci/build_images.sh nginx curl git
ci/build_images.sh nginx/rawhide/default curl/hummingbird/builder
# Build with options
ci/build_images.sh --verbose curl
ci/build_images.sh --arch arm64 nginx
ci/build_images.sh --engine docker nginx
ci/build_images.sh --engine podman --verbose nginx
# Build multiple image groups with options
ci/build_images.sh --verbose nginx curl git
ci/build_images.sh --arch arm64 nginx postgresql
# Build all dependencies of dotnet-runtime-10-0 (forward + reverse + transitive)
ci/build_images.sh --build-deps dotnet-runtime-10-0
# Pull dependencies from registry instead of building (faster for local dev)
ci/build_images.sh --build-deps --pull dotnet-runtime-10-0
# Pull a specific image from registry
ci/build_images.sh --pull nginx/rawhide/default
# Build from CI component name format (used in CI environments)
ci/build_images.sh --component-name curl--rawhide--default
# Build with custom RPMs (for testing modified packages)
ci/build_images.sh --local-rpms-dir ../rpms/builds/hostname/RPMS git/rawhide/builder
Note: Some images require git submodules initialized (use
git init --recurse-submodules when cloning or git submodule update --init if
already cloned). When building for foreign architectures, make sure qemu-user-static
is available.
Dependency Building
The --build-deps flag builds all dependencies needed for testing an image, but not
the image itself. This is used in CI to ensure all required images are available
before running tests.
Pulling vs Building Dependencies
For local test development, use --pull with --build-deps to pull pre-built
images from the registry instead of building them locally. This is much faster
and uses the same dependency collection logic:
# Slow: Build all dependencies locally
ci/build_images.sh --build-deps nginx
# Fast: Pull all dependencies from registry
ci/build_images.sh --build-deps --pull nginx
The --pull flag works without --build-deps too:
# Pull a specific image instead of building it
ci/build_images.sh --pull caddy/rawhide/default
How Dependency Collection Works
When you run ci/build_images.sh --build-deps <image>, it performs a 2-level expansion:
-
Level 1: Collects forward and reverse dependencies of the specified image
- Forward dependencies: Images that the specified image’s tests depend on
(detected by
TEST_IMAGES[...]references in test files) - Reverse dependencies: Images that depend on the specified image
(filtered by
reverse_dependency_tests: trueinproperties.yml)
- Forward dependencies: Images that the specified image’s tests depend on
(detected by
-
Level 2: Collects forward dependencies of the reverse dependencies
-
Stops: No further expansion (avoids infinite graph traversal)
All dependencies are automatically deduplicated to ensure each image is built exactly once.
Relationship to Testing
The reverse_dependency_tests property in properties.yml affects both building and testing:
- Build phase (
--build-deps): Filters which reverse dependencies to build - Test phase (
ci/run_tests_container.shandci/run_tests_k8s.shwith--include-reverse-deps): Filters which reverse dependencies to test
Set reverse_dependency_tests: false for images like curl that are used pervasively
but don’t need reverse dependency workflows.
Dependency Building Examples
# Build dependencies for dotnet-runtime-10-0
# Builds: dotnet-sdk-10-0 (forward dependency)
ci/build_images.sh --build-deps dotnet-runtime-10-0
# Pull dependencies for dotnet-runtime-10-0 (faster alternative)
# Pulls: dotnet-sdk-10-0 from quay.io/hummingbird-rawhide
ci/build_images.sh --build-deps --pull dotnet-runtime-10-0
# Build dependencies for core-runtime
# Builds: xcaddy, go, rust (reverse deps with reverse_dependency_tests: true)
# Plus their forward dependencies
ci/build_images.sh --build-deps core-runtime
# Use --dryrun to see what would be built/pulled without actually doing it
ci/build_images.sh --dryrun --build-deps core-runtime
ci/build_images.sh --dryrun --build-deps --pull core-runtime
Building with custom RPMs
The --local-rpms-dir option enables testing container images with
custom-built RPM packages, for iterating on package changes before committing
to the RPM repository.
Workflow
-
Build custom RPMs in the
rpmsrepository (see its documentation for the complete workflow):cd ../rpms # modify a package ci/build_rpms.sh packagename # Built RPMs will be in builds/packagename/RPMS/ -
Build container image using the custom RPMs:
cd ../containers ci/build_images.sh --local-rpms-dir ../rpms/builds/packagename/RPMS imagename/builder -
Verify the custom package was installed:
podman run --rm --entrypoint '' registry.access.redhat.com/hi/imagename:latest-builder rpm -qa
Builds for affected container tests
Use --build-deps --affected-tests to select dependencies needed by affected
container checks across both engines. Testing Farm adds --pull to fetch
published dependencies. Supplied Konflux image URL overrides take precedence.
Omit --pull to validate unpublished dependency changes from the checkout.
ci/build_images.sh --build-deps --affected-tests --pull core-runtime/hummingbird/default
--affected-tests requires --build-deps. Without it, dependency building retains
the full reverse suite for callers that need broader coverage.
4.3.6.2 - run_tests_container.sh
Purpose
Run tests for image groups using containerized test environment with support for both Docker and Podman engines.
Usage
Run ci/run_tests_container.sh --help for full usage information.
Usage: ci/run_tests_container.sh [OPTIONS] [GROUP_NAMES...]
OPTIONS:
--verbose, -v Enable verbose output during testing
--engine ENGINE Specify container engine (podman or docker)
--setup Set up Docker-in-Docker environment before running tests
--pause, -p Pause failed tests before cleanup to allow debugging
Prints a message and waits for Enter before cleaning up containers
--hermetic Use hermetic builds (--pull=never, only use prefetched/built images)
Default for CI. Without this, missing images are pulled on demand.
--component-name NAME
Parse component name (format: group--distro--variant)
Example: curl--rawhide--default → curl/rawhide/default
--group-component-name NAME
Parse component name and add /group variant
Example: curl--rawhide--default → curl/rawhide/group
--affected-tests Run affected tests and checks that reference dependencies
--dryrun Print selected tests without invoking container engines
--include-reverse-deps
Also test groups that depend on specified groups
--help, -h Show this help message
Note: If no distro/variant is specified, all combinations will be tested. To test
only a specific variant, use <group_name>/<distro>/<variant>. To test only a
specific test, use <group_name>/<distro>/<variant>/<test>.
Engine Selection: Use --engine to specify the container engine (podman
or docker). When using Docker, add --setup for automatic Docker-in-Docker
environment setup.
Hermetic vs Non-Hermetic Testing:
- Local development (default): Without
--hermetic, the test runner allows pulling missing images from the registry. This is convenient for testing individual images without building all dependencies first. - CI environment: Use
--hermeticto enforce that tests only use prefetched or locally-built images (via--pull=never). This ensures reproducible builds and prevents accidentally using images from the registry that differ from what Konflux built.
Building and Testing
For local development, you’ll typically want to build the images first, then test them:
# Build the images
ci/build_images.sh <group_name>[/distro/variant]
ci/build_images.sh <group_name1> <group_name2> # Build multiple image groups
# Test the images
ci/run_tests_container.sh <group_name>[/distro/variant]
ci/run_tests_container.sh <group_name1> <group_name2> # Test multiple image groups
The build script uses the same syntax as the test script, making it easy to build and test the same image/variant combination.
When working on base images (like core-runtime) that other images depend on,
build the base image and its dependencies before running reverse-dependency tests:
# Build core-runtime, then its forward and reverse dependencies
ci/build_images.sh core-runtime
ci/build_images.sh --build-deps core-runtime
# Then test them all
ci/run_tests_container.sh --include-reverse-deps core-runtime
The -p/--pause option pauses the script on failed tests before cleaning up
to allow interactive and efficient debugging.
Testing with Specific Image Builds
To reproduce CI failures, you can run the test with mapping the image under test to the Konflux build. Set IMAGE_URL_<GROUP>__<DISTRO>__<VARIANT> environment variables (uppercase, hyphens/dots → underscores, __ separates parts):
# Test with a specific CI build
IMAGE_URL_TOMCAT_10__HUMMINGBIRD__BUILDER='quay.io/redhat-user-workloads/.../tomcat-10--hummingbird--builder@sha256:...' \
ci/run_tests_container.sh tomcat-10/hummingbird/builder
Automatic Retries for Transient Infrastructure Failures
The test runner automatically retries tests that fail with certain transient infrastructure errors. This helps avoid false test failures caused by temporary issues with external services like container registries or network problems.
Retry Behavior:
- Failed tests are checked against a list of retriable error patterns
- If a match is found, the test is automatically retried
- If the test still fails after all attempts, it’s reported as a normal failure
Examples
# Test single group (all distro/variants)
ci/run_tests_container.sh curl
ci/run_tests_container.sh nginx
# Test single group (specific distro/variant)
ci/run_tests_container.sh curl/rawhide/default
ci/run_tests_container.sh nodejs-20/hummingbird/builder
# Test multiple groups (all distro/variants)
ci/run_tests_container.sh curl nginx mariadb
# Test multiple groups (mixed variants)
ci/run_tests_container.sh curl/rawhide/default nginx mariadb/hummingbird/builder
# Test with different engines
ci/run_tests_container.sh --engine docker --setup git
ci/run_tests_container.sh --engine podman --verbose nginx
# Test with verbose output for debugging
ci/run_tests_container.sh --verbose nginx mariadb dotnet-runtime-8-0
# Test specific tests
ci/run_tests_container.sh curl/rawhide/default/version
ci/run_tests_container.sh nginx/rawhide/default/port
ci/run_tests_container.sh curl/rawhide/default/version nginx/hummingbird/default/port
# Also test groups with reverse dependencies
ci/run_tests_container.sh --include-reverse-deps core-runtime
# Test from CI component name format (used in CI environments)
ci/run_tests_container.sh --component-name curl--rawhide--default
# Test across variants from CI component name format
ci/run_tests_container.sh --group-component-name curl--rawhide--default
Reverse dependency testing is enabled by default for all images. When enabled, changes to an image will automatically:
- Find images that depend on it by scanning for
TEST_IMAGES[group/distro/variant]references - Run tests for both the changed image and all its dependents
This catches breaking changes in base images early. Images can opt-out by setting
reverse_dependency_tests: false in properties.yml (e.g., curl, which is
widely used but typically doesn’t need reverse dependency testing).
Affected dependency tests
Testing Farm uses --affected-tests for single-component jobs. Dependency pulls
use ci/build_images.sh --build-deps --affected-tests --pull with the same selector.
The requested component runs its complete suite. Dependent variants run only
checks, including global checks, that reference the changed component variant
or a selected dependency. Unrelated global checks remain covered by the
requested component.
Use --dryrun to list runnable checks without invoking container engines.
Use --include-reverse-deps without --affected-tests to run full dependent suites.
Run each suite with an isolated container engine. Cleanup preserves resources present at startup and removes resources created during the suite, including resources created concurrently by other processes.
4.3.6.3 - run_tests_k8s.sh
Purpose
Run K8s tests for image groups using kubectl. Tests execute in a real Kubernetes environment, validating that images work correctly in Kubernetes.
Usage
Run ci/run_tests_k8s.sh --help for full usage information.
Usage: ci/run_tests_k8s.sh [OPTIONS] [GROUP_NAMES...]
OPTIONS:
--context CONTEXT K8s context to use (REQUIRED unless --kubeconfig set)
--kubeconfig FILE Path to kubeconfig file (REQUIRED unless --context set)
--push-image IMAGE Push local image to OpenShift internal registry before testing
Sets TEST_IMAGE to the internal registry reference
--verbose, -v Show full output for passing tests
--pause, -p Pause on test failure before cleanup to allow debugging
--component-name NAME
Parse component name (format: group--distro--variant)
Example: curl--rawhide--default → curl/rawhide/default
--group-component-name NAME
Parse component name and add /group variant
Example: curl--rawhide--default → curl/rawhide/group
--include-reverse-deps
Also run tests that use the specified image variants
--output FILE Write JSON test results to FILE (for CI integration)
When set, exits 0 after writing results regardless of test outcome
--help, -h Show this help message
Safety: The script requires explicit --context or --kubeconfig to prevent
accidental operations on production clusters.
Note: If no distro/variant is specified, all combinations will be tested. To test
only a specific variant, use <group_name>/<distro>/<variant>. To test only a
specific test, use <group_name>/<distro>/<variant>/<test>.
Local Development Workflow
For local development, build images and push them to the internal registry:
# Build the image locally
ci/build_images.sh nginx/hummingbird/default
podman tag registry.access.redhat.com/hi/nginx:latest my-nginx:dev
# Push to internal registry and test
ci/run_tests_k8s.sh --context mpp-preprod --push-image my-nginx:dev nginx/hummingbird/default
The --push-image flag:
- Compares local image digest with remote
- Skips push if image already exists with same digest
- Uses port-forward to registry-proxy for pushing
- Sets
TEST_IMAGEto the internal registry reference
Prerequisites:
oc loginto the target clusteroc project <namespace>to set the target namespace- Port-forward access to
registry-proxyinhummingbird--internal
Testing with Published Images
Test published images without building locally:
# Test using published images (resolved via IMAGE_URL/IMAGE_NAME)
IMAGE_URL=registry.access.redhat.com/hi/nginx:latest \
IMAGE_NAME=nginx--hummingbird--default \
ci/run_tests_k8s.sh --context mpp-preprod nginx/hummingbird/default
Reverse dependency selection
--include-reverse-deps uses ci/internal/image_test_plan.py, shared with the
container runner, to select dependent tests individually. It reads Kubernetes
commands from tests-k8s.yml and follows shell scripts and sourced fixtures for
TEST_IMAGES[group/variant] references. Container-only references do not select
Kubernetes tests.
ci/run_tests_k8s.sh --context kind-kind --include-reverse-deps nginx/hummingbird/default
This selects all tests for the requested nginx variant and only dependent tests
that use nginx/default in the same distro. For example, Python custom CA tests
run, but standalone Python version checks do not. Explicit test selectors remain
restricted to the requested test. reverse_dependency_tests: false disables
selection of consumers of that image.
--include-reverse-deps runs the planner in single-component mode: reverse
dependency tests that need a same-group sibling variant are excluded. Such a
test references TEST_IMAGES[<same-group>/variant] for a variant other than the
one under test (or is itself defined for a same-group variant), but a per-component
run provides only the image under test. The sibling would fall back to a released
image — silently testing the changed component against an older sibling — or fail to
pull when no release exists. Those combinations run in the group pipelines (pr group), which provide every variant of the group at the exact commit. Cross-group
reverse dependencies are still selected, and tests of the requested variant itself
are unaffected.
Kubernetes variants lists match exact variant names. Tests without a list apply
to all image variants except group. Tests with variants: [group] run as group
tests. The planner does not apply container engine or host FIPS filters to
Kubernetes definitions, or infer the cluster architecture from the local host.
Environment Variables
Tests have access to these environment variables:
| Variable | Description |
|---|---|
TEST_IMAGE |
Container image under test (with digest) |
TEST_IMAGES |
Associative array with group/variant image URLs (uses TEST_DISTRO context) |
TEST_IMAGES_PATH |
Path to file containing serialized TEST_IMAGES array |
TEST_GROUP |
The image group being tested |
TEST_DISTRO |
The distro being tested (e.g., rawhide) |
TEST_VARIANT |
The variant being tested (e.g., default) |
TEST_VERBOSE |
Show test command output (true or false) |
TEST_RUN_ID |
Unique ID for this test run (for resource naming) |
TEST_RUN_LABEL |
Label selector for cleanup (hum-k8s-test=<id>) |
Cluster Access
The kubectl command is pre-configured with the context/kubeconfig from CLI args,
so tests can use it directly without additional configuration.
Helper Function
| Function | Description |
|---|---|
test_fail |
Fail the test with a custom error message |
Examples
# Test single group with explicit context (all distro/variants)
ci/run_tests_k8s.sh --context mpp-preprod nginx
# Test specific distro/variant
ci/run_tests_k8s.sh --context mpp-preprod nginx/hummingbird/default
# Test specific test
ci/run_tests_k8s.sh --context mpp-preprod nginx/hummingbird/default/readiness-probe
# Test with verbose output for debugging
ci/run_tests_k8s.sh --verbose --context mpp-preprod nginx
# Test with kubeconfig file (CI environments)
ci/run_tests_k8s.sh --kubeconfig /workspace/kubeconfig nginx/hummingbird/default
# Build locally and push to internal registry
ci/build_images.sh nginx/hummingbird/default
podman tag registry.access.redhat.com/hi/nginx:latest my-nginx:dev
ci/run_tests_k8s.sh --context mpp-preprod --push-image my-nginx:dev nginx/hummingbird/default
# Test from CI component name format
ci/run_tests_k8s.sh --kubeconfig /workspace/kubeconfig --component-name nginx--hummingbird--default
# Write JSON results for CI integration
ci/run_tests_k8s.sh --output /tmp/results.json --kubeconfig /workspace/kubeconfig nginx/hummingbird/default
Resource Cleanup
Tests should label resources with TEST_RUN_LABEL for automatic cleanup:
kubectl create configmap my-config --from-literal=key=value
kubectl label configmap my-config "${TEST_RUN_LABEL}"
The test runner automatically cleans up all resources with the test run label after each test and at script exit.
CI Integration
In CI environments (Konflux), the script receives:
--kubeconfigpointing to the ephemeral namespace kubeconfig--component-nameor--group-component-namefor component identification--outputfor structured JSON results
The --output flag writes results in Konflux-compatible format and exits 0,
allowing the pipeline to read results without relying on exit codes.
4.3.6.4 - retrigger_failed_checks.py
Purpose
Retrigger failed Konflux CI checks by posting retest commands and waiting for the checks to start running.
Usage
Run ci/retrigger_failed_checks.py --help for full usage information.
Usage: ci/retrigger_failed_checks.py <MR_URL> [OPTIONS]
OPTIONS:
-h, --help show this help message and exit
--dry-run just print what comments would be posted
--token-path, -t PATH path to file containing GitLab API token
Authentication (in order of precedence):
1. --token-path: Path to file containing GitLab API token (highest priority)
2. GITLAB_TOKEN_PATH: Environment variable with path to token file
3. GITLAB_TOKEN: Environment variable with GitLab API token (fallback)
Behavior
The script will:
- Extract failed pipeline runs from the commit statuses on the MR
- Generate
/retest {pipeline-name}commands for each failed run - Post the retest commands as MR comments (unless –dry-run)
- Wait for the pipeline runs to start (unless –dry-run)
Examples
# Using --token-path option (highest priority)
ci/retrigger_failed_checks.py --token-path /path/to/token https://gitlab.com/group/project/-/merge_requests/1234
# Using GITLAB_TOKEN_PATH environment variable
GITLAB_TOKEN_PATH=/path/to/token ci/retrigger_failed_checks.py https://gitlab.com/group/project/-/merge_requests/1234
# Using GITLAB_TOKEN environment variable (legacy)
GITLAB_TOKEN=your-token-here ci/retrigger_failed_checks.py https://gitlab.com/group/project/-/merge_requests/1234
# Dry-run mode (show what would be done, don't post or wait)
ci/retrigger_failed_checks.py --token-path /path/to/token --dry-run https://gitlab.com/group/project/-/merge_requests/1234
4.3.6.5 - gitlab_sync.py
Purpose
Sync a file from a public source URL to a target GitLab repository via merge
request. The script always waits for the MR to be merged and for the
post-merge pipeline to complete before returning. It is invoked by
infrastructure CI jobs to keep external repos (pyxis-repo-configs,
konflux-release-data) in sync with generated files from the source repos.
Usage
Run gitlab_sync.py --help for full usage information.
usage: gitlab_sync.py [-h] --source-url SOURCE_URL
--target-project TARGET_PROJECT
--target-file TARGET_FILE --sync-branch SYNC_BRANCH
--mr-title MR_TITLE --gitlab-url GITLAB_URL
[--merge-timeout MERGE_TIMEOUT]
[--post-merge-timeout POST_MERGE_TIMEOUT]
[--mr-description MR_DESCRIPTION] [--squash-on-merge]
[--dry-run]
Authentication:
GITLAB_TOKEN Environment variable with GitLab API token (required)
Monitoring:
SENTRY_DSN Sentry DSN for alerting on failures (optional)
How It Works
- Fetch source from the public
--source-url(with HTTP retries) - Determine comparison ref: use the sync branch if it exists, otherwise the project’s default branch
- Compare source content against the comparison ref (stripped whitespace)
- Early exit if content matches and the comparison ref is the default branch (nothing to deploy)
- Commit the source to the sync branch using the GitLab Commits API with
force: true(creates a single-commit branch on top of the default branch). Also auto-squashes if the branch has accumulated multiple commits. - Create or update MR targeting the default branch
- Self-approve the MR (best-effort; continues if already approved)
- Post takeover comment with
CI_JOB_IDfor preemption tracking - Wait for MR merge (polling every 30s, up to
--merge-timeout) - Wait for post-merge pipeline to succeed (up to
--post-merge-timeout)
Exit Codes
| Code | Meaning |
|---|---|
| 0 | Success (MR merged) or no changes needed |
| 1 | Failure (timeout, pipeline failure, unrecoverable error) |
| 42 | Preempted by a newer CI job managing the same MR |
Preemption
When multiple CI jobs target the same sync branch, the script uses MR
comments to coordinate. Each job posts a comment containing its CI_JOB_ID
(monotonically increasing within a GitLab instance). Before each merge
attempt, the script checks for comments with a higher job ID. If found, it
yields by exiting with code 42.
The caller should handle exit 42 to stop processing (a newer job will handle
all remaining syncs). When running outside CI (CI_JOB_ID not set),
preemption is disabled.
Error Recovery
- Pipeline failure: Retries the MR pipeline with exponential backoff (60s, 300s, 900s). Sends a Sentry alert if all retries are exhausted.
- Merge conflict: Rebases the MR and re-approves (rebase resets approvals in GitLab). Up to 3 attempts before failing with a Sentry alert.
- Post-merge pipeline failure: Retries once, then returns exit code 1. This blocks downstream sync steps (e.g., Pyxis must succeed before RPA).
- Draft MR: Keeps polling without attempting to merge (allows manual intervention).
Examples
# Dry run: show diff without modifying anything
GITLAB_TOKEN=$TOKEN gitlab_sync.py \
--source-url "https://gitlab.com/redhat/hummingbird/containers/-/raw/main/releng/pyxis-hummingbird.yaml" \
--target-project "releng/pyxis-repo-configs" \
--target-file "products/hummingbird/hummingbird.yaml" \
--sync-branch "hummingbird/sync-containers-pyxis" \
--mr-title "chore: Update hummingbird Pyxis config" \
--gitlab-url "https://gitlab.cee.redhat.com" \
--dry-run
# Sync Pyxis config (requires squash merge)
gitlab_sync.py \
--gitlab-url "https://gitlab.cee.redhat.com" \
--source-url "https://gitlab.com/redhat/hummingbird/containers/-/raw/main/releng/pyxis-hummingbird.yaml" \
--target-project "releng/pyxis-repo-configs" \
--target-file "products/hummingbird/hummingbird.yaml" \
--sync-branch "hummingbird/sync-containers-pyxis" \
--mr-title "chore: Update hummingbird Pyxis config" \
--squash-on-merge
# Sync RPM RPA to konflux-release-data
gitlab_sync.py \
--gitlab-url "https://gitlab.cee.redhat.com" \
--source-url "https://gitlab.com/redhat/hummingbird/rpms/-/raw/main/releng/hummingbird-rpms-staging.yaml" \
--target-project "releng/konflux-release-data" \
--target-file "config/kflux-prd-rh03.nnv1.p1/product/ReleasePlanAdmission/hummingbird/hummingbird-rpms-staging.yaml" \
--sync-branch "hummingbird/sync-rpms-rpa" \
--mr-title "Update hummingbird RPM ReleasePlanAdmission"
Preemption wrapper for sequential syncs in a CI job:
gitlab_sync.py --gitlab-url ... --source-url ... --squash-on-merge
rc=$?
if [ "$rc" -eq 42 ]; then
echo "Preempted by newer job, stopping"
exit 0
elif [ "$rc" -ne 0 ]; then
exit "$rc"
fi
# Proceed to next sync only on success
gitlab_sync.py --gitlab-url ... --source-url ...
Environment Variables
| Variable | Required | Description |
|---|---|---|
GITLAB_TOKEN |
Yes | API token with Developer+ access on the target project |
CI_JOB_ID |
No | Set automatically in CI; enables preemption detection |
SENTRY_DSN |
No | Sentry DSN for failure alerting |
Development
The script and gitlab-ci tool image are maintained in the
Hummingbird tools repository, not this repository.
Make implementation and test changes there.
4.4 - K8s Test Pipeline
Background documentation sourced from the K8s test pipeline repository, covering pipeline design, test format, and EaaS debugging for Kubernetes integration tests.
4.4.1 - Pipeline Design
Task/Step Overview
| Task | Type | Steps | Results | Retries | Timeout |
|---|---|---|---|---|---|
check-for-tests |
Inline taskSpec |
write-snapshot, resolve-component, fetch-source, check-for-tests, write-results | HAS_TESTS, TEST_OUTPUT |
2 | 10m |
provision-namespace |
External bundle (task-eaas-provision-space) |
(managed by task) | secretRef |
2 | 10m |
run-tests |
Inline taskSpec |
write-snapshot, resolve-component, fetch-source, run-tests, write-results | TEST_OUTPUT |
2 | 24h |
fail-pipeline-on-test-failure |
Inline taskSpec (finally) |
check-results | — | 0 | 10m |
Both check-for-tests and run-tests independently fetch source — each task
gets its own emptyDir volume, so no state carries across tasks.
Design Guidelines
1. Filesystem over results
Pass data between steps via files on a shared emptyDir, not Tekton results.
Tekton results have a 4KB termination message limit
(tektoncd/pipeline#4060)
and large $(params.*) substitutions hit ARG_MAX.
Each task mounts an emptyDir at /workdir. Steps communicate via files:
snapshot.json, component, image_names, src/, kubeconfig, result,
error. Zero $(params.*) references remain inside script: blocks — all
param values are passed via env vars or written to /workdir.
2. Distinguish retryable from non-retryable errors
Use exit 1 for transient failures (network, registry) that benefit from
retry. Use exit 0 + a sentinel file for permanent failures (bad data) to
prevent wasted retries.
Non-retryable errors write Konflux-format JSON to /workdir/error and
exit 0. Subsequent steps check for this sentinel and skip. The
write-results step copies it to TEST_OUTPUT. Tasks default to retries: 2,
but fail-pipeline-on-test-failure has retries: 0 — test failures should not
be retried.
3. Fail fast
Every shell script enables set -euo pipefail so unexpected command failures
surface immediately rather than cascading silently.
4. Separate data from status
Report test results as structured data for the CI system to consume, and separately translate results into pipeline pass/fail for source control integration.
Konflux reads TEST_OUTPUT results as data but does not fail the pipeline
based on them. The fail-pipeline-on-test-failure finally-task reads both
TEST_OUTPUT values and exit 1 on non-SUCCESS. This is needed because
GitLab PAC integration requires the pipeline itself to fail for MR status
reporting.
5. Gate expensive work behind cheap checks
Run a fast, cheap check before provisioning resources. Skip the expensive path when there is nothing to do.
provision-namespace and run-tests are gated on
check-for-tests.results.HAS_TESTS == "true" via Tekton when expressions.
If no tests-k8s.yml exists, the pipeline completes with just
check-for-tests, skipping EaaS provisioning. This is the common case for
components without K8s tests.
6. Make failures reproducible
Preserve all diagnostic output (do not swallow stderr). Log exact commands so developers can copy-paste to reproduce locally.
Stderr from cosign/jq/oras is not swallowed. The full run_tests_k8s.sh
command line is logged.
7. Set task timeouts deliberately
Tekton applies a global default TaskRun timeout (typically 1h from
default-timeout-minutes in config-defaults) to every task that has no
explicit timeout, regardless of pipeline-level timeouts. This means omitting
a task timeout does NOT let the pipeline timeout control the bound — the task
gets killed at 1h.
Every task in this pipeline has an explicit timeout. Bounded operations
(check-for-tests, provision-namespace, fail-pipeline-on-test-failure)
use 10m. The variable-duration run-tests task uses 24h so the
pipeline-level timeout (controlled per-application via ITS annotations like
test.appstudio.openshift.io/pipeline_timeout) is the effective bound,
following the Testing Farm pipeline pattern.
8. Define shared logic once
When multiple tasks need identical steps, use YAML anchors to define them once and reuse via aliases. Tekton pipelines have no function/import mechanism, so anchors are the only DRY tool available within a single pipeline file. Without this, step logic drifts between tasks when one copy is updated but not the other.
write-snapshot, resolve-component, and fetch-source are defined as
anchors (&write-snapshot-step, etc.) in check-for-tests and reused via
*write-snapshot-step in run-tests. Both tasks independently execute the
same steps to fetch and process source, staying in sync.
Trusted Artifacts Source Fetch
Source code is fetched via the Konflux Trusted Artifacts chain:
cosign download attestation— retrieves build attestations (handles multiple JSONL attestations)- Extract
SOURCE_ARTIFACTfrom.predicate.buildConfig.tasks[].results[] oras blob fetch— downloads the source tarballtar -xzf— extracts to/workdir/src
The oci: prefix is stripped from the artifact URI. --no-same-owner avoids
permission issues during extraction.
4.4.2 - Test Format
Test Discovery
The pipeline delegates test discovery to each repository’s own test runner.
It calls ci/run_tests_k8s.sh --has-tests with the component names from the
snapshot and reads true/false from stdout to determine whether any
component has K8s tests defined. This keeps test file location knowledge
in the repository that owns the tests, not in the pipeline.
The actual tests-k8s.yml format is opaque to the pipeline — it is passed
to the repository’s test runner script, which defines the structure and
semantics. See the respective repository documentation for the YAML format.
Environment Variables
The test runner receives these variables from the pipeline:
| Variable | Description |
|---|---|
TEST_IMAGE |
Container image under test (with digest) |
TEST_GROUP |
Image group name (e.g., git, postgresql) |
TEST_VARIANT |
Variant being tested (e.g., default, builder) |
KUBECONFIG |
Path to kubeconfig for the ephemeral namespace |
TEST_IMAGES |
Bash associative array of all images in the snapshot |
TEST_IMAGES_PATH |
Path to source TEST_IMAGES in external scripts |
TEST_RUN_ID |
Unique ID for this test run (for resource naming) |
TEST_RUN_LABEL |
Label selector for cleanup (e.g., hum-k8s-test=<id>) |
test_fail |
Shell function — call test_fail "message" to fail a test |
Pipeline Parameters
| Parameter | Description | Default |
|---|---|---|
SNAPSHOT |
Konflux snapshot JSON or snapshot name (with snapshot-param-as-name annotation) |
— |
EXTRA_SINGLE_COMPONENT_ARGS |
Additional arguments appended to the test runner for single-component (PR) snapshots | --include-reverse-deps |
Group Snapshot Handling
For group snapshots (post-merge), the pipeline exports:
IMAGE_NAMES— space-separated list of all component namesIMAGE_URL_<COMPONENT>— image URL for each component (name uppercased, dashes replaced with underscores)
The test runner script is called once with multiple --group-component-name
arguments. Test discovery checks all components for tests-k8s.yml. Source is
fetched from the first component’s build attestation (shared source tree).
4.4.3 - EaaS and Debugging
Environment as a Service (EaaS)
EaaS is a Konflux-managed service that provisions ephemeral OpenShift namespaces on member clusters for integration testing.
Provisioning Flow
- The
provision-namespacetask creates aneaas.konflux-ci.dev/v1alpha1 Namespaceclaim, owned by the PipelineRun. - The EaaS controller allocates a namespace on a member cluster (currently
kflux-prd-es01) and writes a scoped kubeconfig into a Kubernetes Secret. - The
run-testsstep reads the kubeconfig from the Secret to get cluster access. - When the PipelineRun is deleted, the owner reference triggers garbage collection of the claim, namespace, and Secret.
The member cluster may change — it is dynamically assigned by the EaaS controller.
Debugging Test Failures
Using Kubearchive for Historical PLRs
Kubearchive stores historical PipelineRun, TaskRun, and Pod data. The
containers repo includes
a helper script at ci/internal/k8s_helper.py.
Example: find PLRs for a specific PR:
cd /path/to/rprm-containers
python3 -c "
from ci.internal import k8s_helper
h = k8s_helper.K8sHelper('https://konflux-ui.apps.kflux-prd-rh03.nnv1.p1.openshiftapps.com/ns/hummingbird-tenant/')
endpoint = h._tekton_endpoint('pipelineruns')
items = h._fetch_from_both_sources(endpoint, 'pac.test.appstudio.openshift.io/pull-request=421,test.appstudio.openshift.io/scenario=tools-k8s-test')
for plr in items:
print(plr['metadata']['name'], plr['status']['conditions'][0]['reason'])
"
Useful label selectors:
test.appstudio.openshift.io/scenario=<scenario-name>— filter by ITS namepac.test.appstudio.openshift.io/pull-request=<number>— filter by MRappstudio.openshift.io/component=<component>— filter by component
Accessing the EaaS Namespace During a Live PLR
While a PipelineRun is running, you can access the ephemeral namespace:
-
Find the provision secret name from the
provision-namespacetask results (visible in the PLR status or Konflux UI). -
Extract the kubeconfig:
kubectl get secret <secret-name> -n hummingbird-tenant \ -o jsonpath='{.data.kubeconfig}' | base64 -d > /tmp/eaas-kubeconfig -
Use it to inspect the namespace:
export KUBECONFIG=/tmp/eaas-kubeconfig kubectl get pods kubectl get events --sort-by='.lastTimestamp' kubectl logs <pod-name>
Finding Which Cluster EaaS Uses
To determine the current EaaS member cluster, extract the server URL from a kubeconfig provisioned by EaaS:
kubectl get secret <secret-name> -n hummingbird-tenant \
-o jsonpath='{.data.kubeconfig}' | base64 -d | grep server
As of this writing, the EaaS member cluster is
kflux-prd-es01.1ion.p1.openshiftapps.com. This may change as the EaaS
controller dynamically assigns clusters.
4.5 - Tools repository
Background documentation sourced from the tools repository, covering infrastructure components like event forwarders, the message bus, and monitoring tools.
4.5.1 - Message Bus Architecture
The Hummingbird message bus is an event-driven architecture built on AWS SNS. Events from multiple sources flow through a central topic, enabling subscribers to filter and process only the events they need.
Architecture
flowchart TD
subgraph Publishers
CVE[CVE Analysis]
GL[GitLab Webhooks]
JIRA[Jira Automation]
K8S[Kubernetes Clusters]
end
subgraph MessageBus [Message Bus]
SNS[(SNS Topic)]
end
subgraph Archiver
ARCH[SNS S3 Archiver]
S3[(S3 Bucket)]
end
subgraph StatusDB [Status Database]
SQS1[SQS Queue]
WORKER[hummingbird-status]
PG[(PostgreSQL)]
end
subgraph ConsumerQueue [Consumer Queue]
SQS2[SQS Queue]
end
subgraph Catalog [Container Catalog]
SYNC[SyncFunction]
DDB[(DynamoDB)]
end
subgraph Consumers
CONSUMER[Event Consumer]
end
CVE -->|CVE-agent sweep request| SNS
GL -->|gitlab-event-forwarder| SNS
JIRA -->|jira-event-forwarder| SNS
K8S -->|kubernetes-event-forwarder| SNS
SNS --> ARCH
ARCH --> S3
SNS --> SQS1
SQS1 --> WORKER
WORKER --> PG
SNS -->|"FilterPolicy\nkind=Release"| SYNC
SYNC -->|fetch manifests| Registry[(OCI Registry)]
SYNC -->|read/write| DDB
SNS --> SQS2
SQS2 -->|new events| CONSUMER
S3 -.->|historic events| CONSUMER
PG -.->|aggregated status| CONSUMER
Components
| Component | Role | Description | Staging |
|---|---|---|---|
| hummingbird-events-topic | Infrastructure | Central SNS topic for all events | Yes |
| hummingbird-cve-analysis | Publisher | Publishes requests for CVE-agent sweeps | No |
| hummingbird-cve-agent | Subscriber | Consumes sweep requests and investigates eligible Jira tickets | No |
| gitlab-event-forwarder | Publisher | Receives GitLab webhooks, publishes to SNS | Yes |
| jira-event-forwarder | Publisher | Receives Jira Automation webhooks, publishes to SNS | Yes |
| kubernetes-event-forwarder | Publisher | Watches K8s resources, publishes changes to SNS | Yes |
| sns-s3-archiver | Subscriber | Archives all events to S3 for querying/replay | Yes |
| hummingbird-status | Subscriber | Ingests events to PostgreSQL for structured queries | Yes |
| pipeline-timing | Subscriber | Stores pipeline timing in its own schema and queue | Yes |
| hummingbird-agent | Subscriber | Processes events to drive automated workflows | Yes |
| workqueue-service | Subscriber | Queues and dispatches work items from events | Yes |
| container-catalog | Subscriber | Incrementally syncs image metadata to DynamoDB on Release events | No |
For subscriber rows, the Staging column describes running consumers. Pipeline Timing queues and subscriptions can be provisioned in both environments before the workers run; their presence does not mark the service as deployed.
Message Format
All messages include standard attributes for filtering:
| Attribute | Description | Examples |
|---|---|---|
source |
Event origin | cve-analysis, gitlab, jira, kubernetes |
event_type |
Type of event | cve_agent_sweep_requested, push, merge_request, pipeline, … |
Additional attributes vary by source - see individual publisher docs for details.
GitLab Events
Published by gitlab-event-forwarder:
| Attribute | Description | Example |
|---|---|---|
project_path |
Full project path | redhat/hummingbird/containers |
group_path |
Full group path | redhat/hummingbird |
When configured, Pipeline Hooks use event_type: pipeline; Job Hooks use
event_type: build.
CVE-agent Sweep Requests
Published by hummingbird-cve-analysis:
| Attribute | Value | Description |
|---|---|---|
source |
cve-analysis |
Identifies the publisher |
event_type |
cve_agent_sweep_requested |
Requests a CVE-agent sweep (wake-up request) |
The SNS Message is plain JSON, with no content_encoding attribute.
The subscription uses SNS envelope delivery, so the SQS body contains the SNS
notification envelope and its Message field contains the JSON above. Publish
errors are logged but do not fail the completed analysis cycle; a later enabled
cycle can publish another request.
Jira Events
Published by jira-event-forwarder:
| Attribute | Description | Example |
|---|---|---|
project_key |
Jira project key | HUM |
issue_type |
Issue type name | Story, Bug |
The event_type attribute is set from the X-Jira-Event-Type request header,
which each Jira Automation rule hardcodes for its trigger type
(e.g., issue_created, issue_updated).
Kubernetes Events
Published by kubernetes-event-forwarder:
| Attribute | Description | Example |
|---|---|---|
cluster |
Logical kubeconfig cluster entry name | konflux-rh03 |
namespace |
Object namespace | production |
kind |
Resource kind | PipelineRun, TaskRun, Snapshot, Release |
api_version |
API version | tekton.dev/v1, appstudio.redhat.com/v1alpha1 |
object_name |
Resource name | nginx-7d8c4c9d6f |
Published kinds depend on the deployed forwarder configuration. TaskRun source
activation is separate rollout work. The forwarder obtains cluster from the
selected kubeconfig context’s context.cluster value, not the API server URL.
ExternalSyncQueue
SQS queue fed by ExternalReleaseFunction (triggered by EventBridge schedule
every 6 hours) and consumed by ExternalSyncFunction (one message per
invocation, BatchSize: 1, ReportBatchItemFailures) for registry tag
resolution and metadata sync.
Message body:
{ "registry": "docker.io/library/nginx" }
Same-account SQS with no resource policy — the sender’s Lambda execution role
carries the sqs:SendMessage policy. Failed messages redrive to
ExternalSyncDLQ after 3 receives.
ExternalIndexQueue
SQS queue fed by ExternalSyncFunction and consumed by
ExternalIndexFunction (one message per invocation, BatchSize: 1,
ReportBatchItemFailures) for Syft SBOM generation. One message per digest,
emitted after each TAGS write for every non-superseded digest of the registry
(registries with no changes at all emit nothing).
Message body:
{
"registry": "docker.io/library/python",
"canonicals": ["3.15"],
"digest": "sha256:…",
"architectures": ["amd64", "arm64"]
}
Same-account SQS with no resource policy — the sender’s Lambda execution role
carries the sqs:SendMessage policy. Failed messages redrive to
ExternalIndexDLQ after 3 receives; DLQ entries are not retried automatically
(see the container-catalog documentation for recovery).
Subscription Filtering
SNS filter policies enable subscribers to receive only relevant events:
{
"source": ["gitlab"],
"event_type": ["push", "merge_request"]
}
{
"source": ["jira"],
"event_type": ["issue_created", "issue_updated"]
}
{
"source": ["kubernetes"],
"kind": ["Deployment"],
"event_type": ["MODIFIED"]
}
Kubernetes Release events (used by container-catalog sync Lambda):
{
"kind": ["Release"]
}
See hummingbird-events-topic for subscription setup instructions.
Event Flow Example
- Developer pushes to GitLab repository
- GitLab sends webhook to gitlab-event-forwarder Lambda
- Lambda validates token, extracts metadata, publishes to SNS
- SNS delivers to all matching subscribers:
- sns-s3-archiver stores event in S3
- Event consumers process new events in real-time
Archive keys
The S3 archiver writes one deterministic key for each SNS message. Duplicate
delivery rewrites the same object. GitLab pipeline and job identities include
the project and their numeric IDs; all Kubernetes identities include logical
cluster, namespace, and name. URL-valued cluster attributes from older
forwarders retain their name-only archive identities.
See SNS S3 Archiver for the exact key format and the machine-readable archive-key contract used by the archiver and GitLab forwarder tests.
Consuming Events
Consumers can receive events from multiple sources:
- New events: Subscribe to SNS topic for real-time processing
- Historic events: Download from S3 archive for replay or catch-up
- Structured queries: Query PostgreSQL via hummingbird-status for pipeline status, component state, and cross-referenced data
The S3 archive enables consumers to bootstrap state, then switch to live SNS
events. Hummingbird Status provides a queryable view of its current pipeline
status data. Pipeline Timing consumes its own SQS queue and stores selected
reporting fields in the pipeline_timing schema. Follow a domain row’s
execution identity and parentage to query creation/start/completion times and
overall status. After historical creation-timestamp recovery, the legacy event
catalogue and pointer columns are removed. Timing ingestion maintains only reporting
state and ordering metadata; the independent S3 archiver continues unchanged.
Archived notifications can be searched using publication time and resource identity,
subject to archive retention, without a permanent database row for every message.
Replaying Events
The sns-s3-archiver stores complete SNS records with decoded payloads, enabling easy replay:
# Download archived events
aws s3 sync s3://bucket-name/2025/12/13/ ./local-events/
# Browse events
zcat ./local-events/2025/12/13/12/34/*.json.gz | jq
import gzip
import json
from pathlib import Path
# Replay archived events to handler
for path in sorted(Path("./local-events").rglob("*.json.gz")):
with gzip.open(path, "rt") as f:
sns_record = json.load(f)
event = {"Records": [{"Sns": sns_record}]}
my_handler.lambda_handler(event, None)
See sns-s3-archiver documentation for details on storage format and
the _decode_message pattern for handlers.
Staging Environment
Production and staging use independent SNS topics. Each environment has its own publishers, archiver, and consumer queues so that changes can be tested in staging before production deployment.
flowchart TD
GL["GitLab Projects\n(dual webhook delivery)"]
JIRA["Jira Automation\n(dual webhook delivery)"]
CVE["CVE Analysis\n(CronJob)"]
K8S["Konflux Cluster\n(K8s resources)"]
subgraph production ["Production"]
CVE_P["hummingbird-cve-analysis\n(CronJob)"]
CVE_Q_P["CVE-agent SQS queue"]
CVE_A_P["hummingbird-cve-agent"]
GEF_P["gitlab-event-forwarder\n(Lambda)"]
JEF_P["jira-event-forwarder\n(Lambda)"]
KEF_P["kubernetes-event-forwarder\n(K8s pod)"]
SNS_P["SNS: arr-hummingbird-prod-events"]
prodConsumers["consumers\n(status, agent, workqueue,\narchiver, etc.)"]
CVE_P --> SNS_P
SNS_P -->|sweep request filter| CVE_Q_P
CVE_Q_P --> CVE_A_P
GEF_P --> SNS_P
JEF_P --> SNS_P
KEF_P --> SNS_P
SNS_P --> prodConsumers
end
subgraph staging ["Staging"]
CVE_S["hummingbird-cve-analysis\n(CronJob)"]
CVE_Q_S["CVE-agent SQS queue"]
CVE_A_S["hummingbird-cve-agent"]
GEF_S["gitlab-event-forwarder\n(Lambda)"]
JEF_S["jira-event-forwarder\n(Lambda)"]
KEF_S["kubernetes-event-forwarder\n(K8s pod)"]
SNS_S["SNS: arr-hummingbird-staging-events"]
stagConsumers["consumers\n(status, agent, workqueue,\narchiver)"]
CVE_S --> SNS_S
SNS_S -->|sweep request filter| CVE_Q_S
CVE_Q_S --> CVE_A_S
GEF_S --> SNS_S
JEF_S --> SNS_S
KEF_S --> SNS_S
SNS_S --> stagConsumers
end
GL --> GEF_P
GL --> GEF_S
CVE --> CVE_P
CVE --> CVE_S
JIRA --> JEF_P
JIRA --> JEF_S
K8S --> KEF_P
K8S --> KEF_S
How it works
- Duplicate webhook delivery: GitLab sends every event to both
gitlab-event-forwarder.hummingbird-project.io(prod) andgitlab-event-forwarder.staging.hummingbird-project.io(staging). Each forwarder publishes to its own SNS topic. - Duplicate webhook delivery: Jira Automation sends every event to both
jira-event-forwarder.hummingbird-project.io(prod) andjira-event-forwarder.staging.hummingbird-project.io(staging). Each forwarder publishes to its own SNS topic. - Dual Kubernetes watches: Both prod and staging kubernetes-event-forwarder pods watch the same Konflux cluster resources and publish to their respective topics.
- CVE-agent sweep requests: Each analysis CronJob publishes to its environment’s topic only when its sweep switch is enabled. Attribute filters route the request to that environment’s dedicated agent queue.
- Consumer isolation: Staging consumers (hummingbird-status,
pipeline-timing, hummingbird-agent, workqueue-service) have separate SQS queues
subscribed to the staging topic. Pipeline Timing shares Hummingbird Status’s
staging
eventsdatabase and owns only thepipeline_timingschema. - Staging archiver: A separate sns-s3-archiver subscribes to the staging topic and archives to its own S3 bucket with a shorter retention period than production.
4.5.2 - Alloy CloudWatch
An AWS CloudFormation/SAM stack that provisions a read-only IAM user for
Grafana Alloy’s prometheus.exporter.cloudwatch component. Grants access
to CloudWatch Metrics APIs so selected AWS service metrics (Lambda, SQS,
SNS, DynamoDB) can be ingested into Mimir for alerting.
Features
- Read-Only IAM User: Dedicated user for Alloy CloudWatch metrics exporter
- CloudWatch Metrics:
GetMetricData,GetMetricStatistics,ListMetrics - Resource Discovery:
tag:GetResourcesfor filtering byapp-code=RPRM-001 - Account Discovery:
sts:GetCallerIdentityandec2:DescribeRegionsfor YACE auto-discovery
Architecture
Single SAM template that creates:
- IAM User - Read-only service account (
${ResourcePrefix}-user) - IAM Policy - CloudWatch Metrics read, tag and region discovery, STS caller identity
The user’s access keys are created manually after deployment and stored in Vault. The companion infrastructure repo injects the credentials into the Alloy Hub pod on MPP via a Kubernetes Secret.
flowchart LR
subgraph aws [AWS]
iam["IAM User\n(read-only)"]
cwMetrics["CloudWatch Metrics"]
end
subgraph mpp [MPP Hub]
alloy["Alloy Hub"]
mimir["Mimir"]
end
alloy -->|"GetMetricData"| cwMetrics
alloy -->|"remote_write"| mimir
alloy -.->|"credentials from Vault"| iam
IAM Permissions
All permissions are read-only with Resource: "*" (CloudWatch does not
support resource-level restrictions for most read operations).
| Category | Actions |
|---|---|
| Metrics | GetMetricData, GetMetricStatistics, ListMetrics |
| Discovery | ec2:DescribeRegions, tag:GetResources, sts:GetCallerIdentity |
Prerequisites
- AWS CLI configured with appropriate credentials (IAM permissions for CloudFormation)
- Podman or Docker (for containerized SAM build/deploy)
Deployment
Build and deploy using containerized AWS SAM CLI:
cd alloy-cloudwatch
sam deploy --guided # First deployment
sam deploy # Subsequent deployments
After deployment, create an access key for the IAM user and store it in
Vault using cki_tools.credentials.manager.
Related
- Alloy Hub deployment:
kubernetes/alloy-hub/in the infrastructure repo - Monitoring documentation:
documentation/monitoring.mdin the infrastructure repo
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.
4.5.3 - Deployment Bot
An AWS CloudFormation/SAM stack that provisions IAM resources for automated AWS deployments from GitLab CI. Creates a service account with permissions to deploy SAM applications including Lambda functions, API Gateway, SNS topics, and related resources.
Features
- IAM Service Account: Dedicated user for CI/CD pipelines
- Scoped Permissions: Least-privilege access for SAM deployments
- Resource Naming: All managed resources use consistent prefix
Architecture
Single SAM template that creates:
- IAM User - Service account for deployments (
${ResourcePrefix}-deployment-bot) - IAM Policy - Permissions for SAM deployment operations
Prerequisites
- AWS CLI configured with admin credentials
- Podman or Docker (for containerized SAM build/deploy)
Deployment
Deploy using containerized AWS SAM CLI:
cd deployment-bot
sam build
sam deploy --guided # First deployment (interactive)
sam deploy # Subsequent deployments
After deployment, create access keys for the IAM user and store them securely in HashiCorp Vault.
SAM Parameters
| Parameter | Description | Default |
|---|---|---|
ResourcePrefix |
Prefix for resources | myapp-prod |
AdditionalResourcePrefix |
Additional prefix this bot can manage (e.g. staging) | "" |
Resource naming: IAM resources follow {ResourcePrefix}-deployment-bot
pattern (e.g., myapp-prod-deployment-bot). Use AdditionalResourcePrefix
to allow the same bot to manage resources in another environment.
Permissions
The deployment bot has permissions to manage:
| Service | Scope |
|---|---|
| CloudFormation | Stacks matching ${ResourcePrefix}-*, SAM bootstrap stack |
| S3 | SAM CLI managed buckets |
| Lambda | Functions matching ${ResourcePrefix}-* |
| API Gateway | All REST APIs and domains |
| SNS | Topics matching ${ResourcePrefix}-* |
| IAM | Users/policies/roles matching ${ResourcePrefix}-* |
| CloudWatch | Log groups for managed Lambda functions |
| Route53 | Record changes (for custom domains) |
| ACM | Certificate management |
Usage
Bootstrap (One-Time)
The deployment bot must be deployed first using admin credentials:
# With admin AWS credentials
cd deployment-bot
sam build
sam deploy --guided
Create access keys for the new IAM user and store in HashiCorp Vault.
Automated Deployments
Once bootstrapped, the deployment bot credentials are used by aws_deploy.sh
in the infrastructure repository for all subsequent SAM deployments:
export PROJECT_NAME=gitlab-event-forwarder
./aws_deploy.sh
GitLab CI runs this automatically via the aws matrix job.
Development
This project contains only a SAM template with no application code.
Security
- Access keys should be stored in HashiCorp Vault
- Permissions are scoped to resources with the configured prefix
- Route53 and ACM have broader permissions due to API limitations
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.
4.5.4 - Grafana CloudWatch
An AWS CloudFormation/SAM stack that provisions a read-only IAM user for Grafana’s CloudWatch datasource plugin. Grants access to CloudWatch Logs and Metrics APIs so Lambda execution logs and AWS service metrics are browsable directly in Grafana.
Features
- Read-Only IAM User: Dedicated user for Grafana CloudWatch datasource
- CloudWatch Logs: Browse Lambda and AWS service log groups via Logs Insights
- CloudWatch Metrics: Browse metrics across all AWS namespaces (Lambda, SQS, SNS, DynamoDB, etc.)
- Resource Discovery:
tag:GetResourcesandec2:DescribeRegionsfor Grafana plugin autodiscovery
Architecture
Single SAM template that creates:
- IAM User - Read-only service account (
${ResourcePrefix}-user) - IAM Policy - CloudWatch Logs read, Metrics read, tag and region discovery
The user’s access keys are created manually after deployment and stored in Vault. The companion infrastructure repo injects the credentials into the Grafana pod on MPP via a Kubernetes Secret.
flowchart LR
subgraph aws [AWS]
iam["IAM User\n(read-only)"]
cwLogs["CloudWatch Logs"]
cwMetrics["CloudWatch Metrics"]
end
subgraph mpp [MPP Hub]
grafana["Grafana\n(CloudWatch datasource)"]
end
grafana -->|"Logs Insights API"| cwLogs
grafana -->|"Metrics API"| cwMetrics
grafana -.->|"credentials from Vault"| iam
IAM Permissions
All permissions are read-only with Resource: "*" (CloudWatch does not
support resource-level restrictions for most read operations).
| Category | Actions |
|---|---|
| Logs | DescribeLogGroups, DescribeLogStreams, GetLogEvents, FilterLogEvents, StartQuery, StopQuery, GetQueryResults, DescribeQueries |
| Metrics | GetMetricData, GetMetricStatistics, ListMetrics, DescribeAlarms, DescribeAlarmsForMetric |
| Discovery | ec2:DescribeRegions, tag:GetResources |
Prerequisites
- AWS CLI configured with appropriate credentials (IAM permissions for CloudFormation)
- Podman or Docker (for containerized SAM build/deploy)
Deployment
Build and deploy using containerized AWS SAM CLI:
cd grafana-cloudwatch
sam deploy --guided # First deployment
sam deploy # Subsequent deployments
After deployment, create an access key for the IAM user and store it in
Vault using cki_tools.credentials.manager.
Related
- Grafana deployment:
kubernetes/grafana/in the infrastructure repo - CloudWatch datasource:
grafana_data/datasource/in the infrastructure repo - Monitoring documentation:
documentation/monitoring.mdin the infrastructure repo
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.
4.5.5 - Hummingbird AI
hummingbird-ai is a synchronous Python library used in-process by Hummingbird
agents to call Gemini and Anthropic models through normalized interfaces.
Features
- Shared Gemini API/Vertex AI and Anthropic Vertex AI adapters.
- Model construction, regional selection, token pricing, and usage accounting.
- Retryable model-call handling and Prometheus request/token/cost metrics.
- HTTP sessions and Google Application Default Credentials authentication.
The library is not a service and has no deployment of its own. Model adapter instances are scoped to individual conversations. Callers provide bounded concurrency for independent work.
Model interface
hummingbird_ai.models defines the ModelAdapter protocol and normalized
ToolDef, ToolCall, Usage, ModelResponse, and ModelError types. Adapters
accept provider-neutral tool definitions and return normalized responses.
Their make_user_content() and make_tool_responses() methods create
provider-specific conversation content; callers keep contents opaque.
Adapters also implement to_canonical() and from_canonical() for persisted
conversation history. The canonical message format lets an agent resume a
session with a different provider without storing either provider’s native
wire format.
Provider adapters
Gemini
GeminiModel supports the direct Gemini API with an API key and Vertex AI with
Google credentials. It translates the normalized tool and response types to
Gemini REST JSON. Gemini’s implicit prompt cache is reflected in
Usage.cache_read_tokens from cachedContentTokenCount.
Anthropic on Vertex AI
AnthropicVertexModel uses Vertex AI rawPredict. It handles Anthropic’s
assistant/user turn alternation and associates tool results with the
conversation-local tool-use IDs. For prompt caching it adds sliding
cache_control breakpoints to shallow copies of messages; the original
conversation history is not mutated. Transient messages can use ephemeral
so warning text is excluded from the cache write prefix.
When Anthropic web search is enabled, the adapter adds the server-side search
tool. Search blocks remain opaque to the agent tool loop and survive canonical
session serialization. The adapter also follows pause_turn continuations and
merges content and usage from those provider-side calls into the returned
response.
Provider wire formats
The agent uses normalized types; adapters translate them to provider payloads:
| Normalized data | Gemini | Anthropic on Vertex AI |
|---|---|---|
| System instruction | systemInstruction |
top-level system string |
| Tool definitions | functionDeclarations with JSON Schema |
tools with input_schema |
| Tool calls | functionCall parts |
tool_use content blocks |
| Tool results | functionResponse parts |
tool_result content blocks in a user turn |
Gemini API-key requests use the direct Gemini endpoint; Vertex requests use
Google authentication and a configured region. Anthropic requests use Vertex
AI rawPredict. The adapters keep provider-specific message normalization
inside generate(); the agent appends the returned raw_content without
interpreting it.
Canonical sessions
Adapters convert their native conversation content to and from the shared canonical message format at persistence boundaries. Anthropic thinking and server-tool blocks are preserved as opaque pass-through fields and restored in their original order. This allows sessions saved by one provider to resume on another while keeping the agent loop independent of provider payload shapes. Extended thinking blocks in assistant responses are part of the cached prefix. They do not break cache hits when the following user message contains only tool_result blocks.
Prompt caching
Gemini Vertex AI prompt caching is implicit. The adapter reads
cachedContentTokenCount from usage metadata and reports it as
cache_read_tokens; it adds no cache annotations to requests.
Anthropic Vertex caching uses explicit cache_control: {"type": "ephemeral"}
annotations. The adapter uses two of the four available breakpoints on
messages: B2 writes the latest prefix and B1 reads the previous prefix. The
prefix hash includes prior tools and system instructions, so extra breakpoints
on those components are redundant. Breakpoints are added to shallow copies so
the agent-owned history is unchanged. Ephemeral turns keep warning text out of
the cache: B1 still reads the last stable prefix, while B2 and the tracked write
position advance only to the last stable message before the transient tail. If
there is no stable message to annotate, no breakpoint is moved.
Vertex AI Anthropic prompt caches have a five-minute TTL refreshed on a hit. Cache writes use 1.25 times the base input-token rate and reads use 0.10 times that rate. Minimum cacheable prefixes are 1,024 tokens for Sonnet/Opus and 4,096 for Haiku. A newly constructed adapter starts without a prior breakpoint position, so the first resumed call writes a fresh prefix before later calls can read it.
T1: U1·B2 prev=0
T2: U1·B1 M1 U2·B2 prev=2
T3(E): U1 M1 U2·B1 M2·B2 U3+E prev=3
T4(E): U1 M1 U2 M2·B1 U3 M3·B2 U4+E prev=5
T5: U1 M1 U2 M2 U3 M3·B1 U4 M4 U5·B2 prev=8
Calls, retries, and accounting
hummingbird_ai.calls.generate_with_retry() owns model-call retry and usage
accounting. A caller selects either an elapsed-time retry budget or a maximum
attempt count. It can also pass response validation into the same attempt loop,
so validation failures do not multiply a caller’s transport retry limit.
Provider-specific HTTP status handling remains in each adapter.
Each returned ModelResponse contributes its input, output, cache-read, and
cache-creation tokens and estimated cost exactly once, even when caller-side
validation rejects that response. The library emits shared Vertex request,
token, and cost counters while callers retain workflow-specific error handling
and summaries.
Pricing is kept in models.MODEL_PRICING as per-million-token rates for input,
output, cache reads, and cache creation. models.estimate_cost() returns an
estimated USD cost or None for an unpriced model.
HTTP and authentication
hummingbird_ai.http.new_session() creates requests sessions with configurable
transport retries. Model adapters disable transport-level retries for 429 so
that rate-limit failures use the model-call retry policy. VertexAuth uses
Google Application Default Credentials and resolves the project from an
explicit argument, GOOGLE_CLOUD_PROJECT, then ADC.
requests.Timeout is caught before requests.ConnectionError in adapters
because ConnectTimeout is a subclass of both. The urllib3 retry adapter
does not retry POST requests (non-idempotent), so all transport errors
from model calls propagate to the generate_with_retry retry layer.
Prerequisites
- Python 3.11 or later.
- Google credentials and a project for Vertex AI models; Gemini API key mode is also supported.
Installation
The library is installed as an in-repository dependency of both agents:
make hummingbird-agent/setup
make hummingbird-cve-agent/setup
Usage
from hummingbird_ai import models
model = models.build_model(
"gemini-2.5-flash",
google_api_key="",
google_cloud_project="my-project",
model_regions={"gemini-2.5-": "us-central1"},
)
Shared calls and usage accounting are provided by hummingbird_ai.calls.
Agent-specific prompts, tools, response parsing, and decisions remain in the
consuming agent.
Configuration
| Setting | Use |
|---|---|
GOOGLE_API_KEY |
Passed by the caller for direct Gemini API access |
GOOGLE_CLOUD_PROJECT |
Vertex AI project fallback when credentials do not specify a project |
Model regions and Vertex labels are passed by the consuming agent when it constructs an adapter.
Development
See the main README for development workflows.
make check # lint
make test # run tests
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.
4.5.6 - Hummingbird Events Topic
An SNS topic for all Hummingbird project events. This topic serves as a central hub for distributing events from multiple sources (GitLab webhooks, Kubernetes, etc.) to multiple subscribers, enabling event-driven architectures and integrations.
Features
- Central Event Hub: Dedicated SNS topic per environment (production and staging) for all project events from multiple sources
- Multiple Subscribers: Supports Lambda, SQS, HTTP endpoints, email, SMS, and more
- Event Filtering: Subscribers can filter events using SNS subscription filter policies
Prerequisites
- AWS CLI configured with appropriate credentials (IAM permissions for SNS, CloudFormation)
- Podman or Docker (for containerized SAM build/deploy)
Deployment
Build and deploy using containerized AWS SAM CLI:
cd hummingbird-events-topic
make build # Build SAM application
make deploy # First deployment (interactive/guided)
make redeploy # Subsequent deployments (non-interactive)
Deployment outputs:
TopicArn- SNS topic ARN (for event publishers)TopicName- SNS topic name
Parameters
| Parameter | Description | Default |
|---|---|---|
TopicName |
SNS topic name | myapp-prod-events |
Resource naming: The SNS topic uses the provided TopicName parameter.
Usage
Publishing Events
Event publishers need the topic ARN to publish messages:
# Get topic ARN from CloudFormation stack
aws cloudformation describe-stacks \
--stack-name <stack-name> \
--query 'Stacks[0].Outputs[?OutputKey==`TopicArn`].OutputValue' \
--output text
Event publishers:
- gitlab-event-forwarder - Publishes GitLab webhook events
- kubernetes-event-forwarder - Publishes Kubernetes resource changes
- Custom event sources
Subscribing to Events
Subscribe services to receive events:
Via AWS Console:
- Open SNS console → Topics
- Select the topic
- Create subscription (choose protocol: Lambda, SQS, HTTP, Email, etc.)
- Add subscription filter policy (optional)
Via AWS CLI:
aws sns subscribe \
--topic-arn <topic-arn> \
--protocol lambda \
--notification-endpoint <lambda-arn>
Add filter policy:
aws sns set-subscription-attributes \
--subscription-arn <subscription-arn> \
--attribute-name FilterPolicy \
--attribute-value '{"source": ["gitlab"], "event_type": ["push"]}'
Subscription Filter Examples
GitLab push events from specific project:
{
"source": ["gitlab"],
"event_type": ["push"],
"project_path": ["redhat/hummingbird/containers"]
}
All merge request events:
{
"source": ["gitlab"],
"event_type": ["merge_request"]
}
All events from GitLab:
{
"source": ["gitlab"]
}
Event metadata: See publisher documentation for available metadata:
- gitlab-event-forwarder - GitLab webhook event metadata
- kubernetes-event-forwarder - Kubernetes resource event metadata
Pipeline Timing queue and recovery
Pipeline Timing uses a separate standard SQS queue and dead-letter queue in each
environment. The source queue subscribes to its
environment’s SNS topic through pipeline-timing/template.yaml; the DLQ receives
messages from the source queue after the receive limit. The Status queue and
worker remain separate and unchanged. Provision queues and subscriptions in both
environments before deploying the workers. Messages received before worker
deployment may expire after four days; this pre-launch loss is accepted. Without
a consumer, messages are not received and do not automatically move to the DLQ.
After launch, monitor queue age during worker pauses to avoid losing events.
Validate the staging worker before deploying the production worker or enabling
new production event sources.
| Setting | Value |
|---|---|
| Queue | arr-hummingbird-<env>-pipeline-timing-sqs |
| Dead-letter queue | arr-hummingbird-<env>-pipeline-timing-sqs-dlq |
| Message retention | 4 days on the source queue; 14 days on the DLQ |
| Redrive limit | 3 receives |
| Visibility timeout | 300 seconds |
| SNS raw message delivery | false, preserving the SNS envelope, message ID, and timestamp |
| Subscription filtering | None, the worker receives each topic event |
When SQS automatically moves a standard-queue message to its DLQ, it preserves the original enqueue timestamp. The DLQ’s 14-day retention is measured from that timestamp, not added after the source queue’s four days. An operator redrive from the DLQ back to the source queue resets the enqueue timestamp.
The worker follows Status’s 20-second long polling and receives up to 10 messages per batch. Timing commits accepted messages individually, then acknowledges eligible receipts in one batch-delete request per receive. Partial or uncertain delete failures leave committed messages available for idempotent redelivery. The 300-second visibility timeout controls SQS redelivery; it does not interrupt message processing.
The queue names include sqs-dlq so the existing CloudWatch rules
classify the DLQ separately. Grafana Alloy is Hummingbird’s monitoring
collector. SAM stack tags let its CloudWatch exporter find the source queue
and DLQ and forward their queue-depth and oldest-message-age metrics to Mimir.
Alloy does not read or consume SQS messages.
The dedicated worker IAM user can call sqs:ReceiveMessage and
sqs:DeleteMessage on the timing queue only. It has no access to the Status
queue or the DLQ. The queue policy grants SNS delivery only from the matching
environment topic and denies non-HTTPS requests. The DLQ redrive policy accepts
only its matching source queue. The template uses the Status stack’s UserName
and UserArn output names. Operator redrive uses separate AWS credentials.
Fix malformed events before redriving them. An operator can move messages from
the DLQ back to the timing queue and check the move task. Set ENV_PREFIX to
staging or prod, then read the queue ARNs from the stack outputs:
STACK_NAME="arr-hummingbird-${ENV_PREFIX}-pipeline-timing"
TIMING_QUEUE_ARN=$(aws cloudformation describe-stacks \
--stack-name "$STACK_NAME" \
--query "Stacks[0].Outputs[?OutputKey=='QueueArn'].OutputValue" \
--output text)
TIMING_DLQ_ARN=$(aws cloudformation describe-stacks \
--stack-name "$STACK_NAME" \
--query "Stacks[0].Outputs[?OutputKey=='DeadLetterQueueArn'].OutputValue" \
--output text)
aws sqs start-message-move-task \
--source-arn "$TIMING_DLQ_ARN" \
--destination-arn "$TIMING_QUEUE_ARN"
aws sqs list-message-move-tasks --source-arn "$TIMING_DLQ_ARN"
Verify that the message returns to the timing queue and that the worker processes it. During the initial infrastructure test, if the worker is not deployed yet, remove only the identifiable test message after confirming the move. Do not purge the queue. SNS message IDs make worker redelivery idempotent.
Development
This is a pure infrastructure project (no application code). See the main README for SAM build/deploy commands.
Security & Limitations
Security:
- SNS topic follows least privilege principle
- Access controlled via IAM policies
- Supports server-side encryption (optional)
- Publishers require
sns:Publishpermission - Subscribers require appropriate protocol permissions
Limitations:
- Message size: Up to 256 KB
- Maximum subscriptions: 12,500,000 per topic
- Message retention: Not supported (use SQS for durable queuing)
- Delivery retries: Protocol-dependent
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.
4.5.7 - Vertex AI Cost Metrics
This document defines the shared cost-observability contract for all Hummingbird services that call Vertex AI. It covers two obligations:
- Prometheus metrics — estimated cost counters scraped into Mimir.
- Vertex request labels — billing metadata attached to every API call.
All services that make Vertex AI calls MUST implement both.
hummingbird-ai emits shared model request, token, and cost metrics for its
consumers. The dashboard continues to expose independent database-backed
gauges with the same names and labels.
1. Prometheus metrics (estimated cost)
Every Vertex AI caller MUST emit the following counters on its scraped
/metrics endpoint. Using identical names and label sets across services
enables a single Grafana dashboard without per-app query logic.
Counters
| Name | Unit | Labels | Description |
|---|---|---|---|
hummingbird_vertex_cost_dollars_total |
USD | model, workflow |
Estimated cost based on MODEL_PRICING x tokens |
hummingbird_vertex_tokens_total |
tokens | model, workflow, direction |
Token counts per API call |
hummingbird_vertex_requests_total |
requests | model, workflow, status |
API call attempts |
Labels
Applications MUST set these labels. Additional labels are permitted if cardinality remains bounded.
| Label | Values |
|---|---|
model |
Model name as sent to Vertex (e.g. gemini-2.5-flash, claude-sonnet-4-6) |
workflow |
Workflow or operation name (e.g. code-review, renovate-babysit) |
direction |
input, output, cache_read, cache_creation |
status |
success, error, retry, parse_error |
Request statuses are mutually exclusive per model attempt: success means the
response was accepted, error is a terminal model-call failure, retry marks
an attempt that will be retried, and parse_error marks a terminal
response-validation failure. Token and cost metrics still include every model
response, including responses rejected by validation.
Applications MUST NOT set cluster, namespace, app, service, or
pod — Alloy adds these automatically at scrape time via relabel rules.
Cardinality
model: bounded by deployed model set (currently 2-4)workflow: bounded by workflow config entries (~10 for agent, 1 for dashboard)direction: fixed set of 4status: fixed set of 4
Total series per app: model x workflow x max(direction, status) = 4 x 10 x 4 = 160. Well within Mimir per-metric limits.
2. Vertex request labels (billed cost)
Every Vertex API call MUST include billing labels so that GCP Cloud Billing rows can be attributed to a specific application and workflow once BigQuery export is enabled.
Required labels
| Key | Value | Example |
|---|---|---|
app |
Application name (matches Kubernetes app) | agent, dashboard |
workflow |
Workflow or operation name within the app | code-review, analyze-failures |
Callers MUST set both keys on every generateContent,
streamGenerateContent, or rawPredict request.
How to attach
| Endpoint | Mechanism |
|---|---|
generateContent (Gemini) |
Top-level "labels" key in JSON body |
rawPredict / streamRawPredict |
X-Vertex-AI-Labels HTTP header |
For generateContent, add the labels object to the request body:
{
"contents": [...],
"labels": {"app": "agent", "workflow": "code-review"}
}
For rawPredict, base64-encode the labels JSON and pass as a header
(shown decoded for clarity):
import base64
import json
labels = {"app": "dashboard", "workflow": "analyze-failures"}
headers["X-Vertex-AI-Labels"] = base64.b64encode(
json.dumps(labels).encode(),
).decode()
Constraints (GCP requirements)
- Keys and values: lowercase letters, numbers, underscores, dashes only
- Max 63 characters each
- Keys MUST start with a letter
- Labels are only forwarded for PayGo consumption (Provisioned Throughput silently ignores them)
Reference
Example PromQL
Total estimated spend yesterday (all apps):
sum(increase(hummingbird_vertex_cost_dollars_total[1d]))
Per-app daily cost:
sum by (app)(increase(hummingbird_vertex_cost_dollars_total[1d]))
Per-workflow breakdown for the agent:
sum by (workflow, model)(
increase(hummingbird_vertex_cost_dollars_total{app="hummingbird-agent"}[1d])
)
Token consumption by direction:
sum by (app, direction)(increase(hummingbird_vertex_tokens_total[1d]))
Implementing apps
| Application | Port | Workflow values |
|---|---|---|
| hummingbird-agent | 9090 | from wf_cfg.name per workflow |
| hummingbird-dashboard | 8080 | analyze-failures |
4.5.8 - Hummingbird Sandbox
Shared Python library for isolated command execution used by Hummingbird agents.
Features
- Podman, Kubernetes, and KubeVirt sandbox backends.
- Command execution, streamed input/output, file transfer, and sandbox cleanup.
- Kubernetes and KubeVirt pools for reusing pre-warmed sandboxes.
- Dedicated execution images for Hummingbird Agent and CVE preparation.
Prerequisites
- Python 3.11 or later.
- Podman for local containers, or
kubectl, Kubernetes credentials, and namespace-scoped permissions for Kubernetes-backed containers. - An image appropriate for the selected backend; Kubernetes deployments must also provide the required network policy and resource controls.
Installation
Install the development package from the tools repository:
make hummingbird-sandbox/setup
Usage
Create a backend through the shared factory, then always clean it up:
from hummingbird_sandbox import config, sandbox
backend = sandbox.create_sandbox(config.SandboxConfig(image="example:latest"))
backend.start()
try:
result = backend.exec("printf 'hello\\n'")
if result.exit_code:
raise RuntimeError(result.stderr)
print(result.stdout)
finally:
backend.cleanup()
create_sandbox accepts sandbox_type="podman", "k8s", or "kubevirt";
Kubernetes-backed modes also accept a context and namespace. Pool backends are
created separately by callers and require pre-warmed pods or VMIs.
Execution Images
| Image | Source path | Purpose |
|---|---|---|
Hummingbird Agent sandbox (hummingbird-agent-sandbox) |
hummingbird-sandbox/images/agent/ |
Command tools for Podman, direct Kubernetes, and Kubernetes pod pools. |
CVE preparation sandbox (hummingbird-cve-sandbox) |
hummingbird-sandbox/images/cve/ |
RPM preparation tools for the CVE agent; optional and not yet its default image. |
The agent’s config example and pod-pool Deployment still use the digest-pinned
gitlab-ci image. Switch them only after the agent sandbox image is published
and verified. Direct Podman/K8s uses settings.sandbox.image; k8spool uses
the image in its pre-warmed Deployment.
Configuration
SandboxConfig holds the backend image, namespace, active deadline, pool linger
limits, pod metadata/resources, and KubeVirt image and SSH settings. Callers are
responsible for loading configuration and keeping their credentials outside the
sandbox.
Hummingbird Agent supplies these settings in settings.sandbox. This example
is moved from the agent documentation; the library itself accepts a
SandboxConfig object and does not load YAML or environment variables:
sandbox: # sandbox pod configuration
image: quay.io/.../gitlab-ci:latest # container image (Podman or direct K8s)
namespace: default # sandbox namespace (or CLI --namespace)
active_deadline_seconds: 1800 # pod hard timeout / reap interval
linger_seconds: 300 # keep pod alive after success (pool mode, 0=disable)
max_lingering_pods: 2 # max idle lingering pods before eviction (pool mode)
vm_image: quay.io/.../vm-disk:latest # containerDisk image (kubevirt mode only)
vm_memory: 1Gi # VM guest memory (kubevirt mode only)
vm_ssh_private_key: /path/to/key # SSH private key for pool VMs (kubevirtpool mode)
metadata: # pod/VMI metadata (k8s/kubevirt mode)
labels:
app.kubernetes.io/name: hummingbird-agent-sandbox
resources: # K8s resource requests/limits (k8s mode)
requests:
cpu: "100m"
memory: "256Mi"
ephemeral-storage: "256Mi"
limits:
cpu: "1"
memory: "1Gi"
ephemeral-storage: "2Gi"
Security and Design Constraints
hummingbird-sandbox supplies execution backends; it does not create the
caller’s namespace RBAC or NetworkPolicy, load configuration, or make
deployment decisions. Callers own configuration and credentials, and must
provide the infrastructure controls described below. The following security
and backend documentation is moved from Hummingbird Agent without changing its
behavior claims.
The agent is designed to run in a shared OpenShift cluster without cluster-admin access, processing potentially untrusted merge requests. These constraints shaped the architecture:
Sandbox isolation. All arbitrary commands executed by the LLM run inside an ephemeral container, never on the host:
- Podman (local):
--network=none,--user 65532, no host mounts. Complete network isolation. - Kubernetes (production): Pods comply with OpenShift’s
restricted-v2Security Context Constraint:runAsNonRoot,seccompProfile: RuntimeDefault,allowPrivilegeEscalation: false,capabilities.drop: ["ALL"],automountServiceAccountToken: false(no K8s API access from sandbox),activeDeadlineSeconds(configurable, default 1800). Security context fields are hardcoded in the pod manifest for portability to vanilla Kubernetes with Pod Security Admission (restricted level). Resource requests/limits, metadata, andactiveDeadlineSecondsare configurable viasettings.sandboxin the config file. Network access is denied by aNetworkPolicyon the sandbox namespace that blocks all egress from all pods (podSelector: {}).
No cluster-admin required. The agent operates with namespace-scoped permissions only. The orchestrator’s ServiceAccount needs only:
pods: create, get, list, delete, patch– sandbox pod lifecycle and pool claimspods/exec: create– command execution viakubectl execvirtualmachineinstances.kubevirt.io: create, get, list, delete, patch– KubeVirt VMI lifecycle (kubevirt/kubevirtpool modes only)secrets: create, get, delete– SSH key Secrets for VMI provisioning (kubevirt/kubevirtpool modes only)
These permissions are granted via a Role in the sandbox namespace, not the orchestrator’s own namespace. No CRDs, no custom runtimes, no cluster-scoped resources. Konflux data is fetched via bearer token from kubeconfig, not from inside the cluster.
Namespace separation. Sandbox pods are created in a dedicated namespace, separate from the orchestrator. This limits blast radius: even if a sandbox pod is compromised, it has no visibility into the orchestrator’s Secrets, Pods, or ServiceAccount tokens. The sandbox namespace is locked down with standard K8s resources:
- RBAC: Role + RoleBinding scoped to the namespace, granting only the permissions above to the orchestrator’s ServiceAccount
- NetworkPolicy: uses
podSelector: {}to select all pods in the dedicated sandbox namespace, denying all egress (egress: []). The sandbox cannot reach the internet, the K8s API, or other pods. - activeDeadlineSeconds: sandbox pods self-terminate after the configured timeout (default 1800s / 30 minutes) even if the orchestrator crashes or is killed, preventing orphaned pods
Credential separation. Data source credentials (GitLab tokens, kubeconfig)
live in the orchestrator process only, injected via K8s Secrets. The sandbox
container has no credentials, no SA token
(automountServiceAccountToken: false), and no network access. Data flows
into the sandbox via stdin piping through write_file.
Command execution via kubectl exec. The K8s sandbox uses a hybrid approach:
the Kubernetes Python client manages pod lifecycle (create, wait, delete),
while kubectl exec handles command execution. This avoids the complexity and
reliability issues of the websocket-based exec API.
KubeVirt VM isolation. KubeVirt sandbox VMs run as root inside the guest
OS, but the VM itself is contained by the KubeVirt hypervisor (QEMU/KVM). The
VM has no access to the Kubernetes API, no ServiceAccount token, and no
credentials. SSH keys are ephemeral (generated per sandbox start for direct
mode, or shared per pool for pool mode) and cleaned up with the VMI.
Sandbox Backends
| Podman (local) | K8s (direct) | K8sPool (production) | KubeVirt (direct) | KubeVirtPool | |
|---|---|---|---|---|---|
| Start | podman run -d --network=none |
create_namespaced_pod |
Claim standby pod from Deployment | Create VMI + SSH Secret | Claim standby VMI from VMIRS |
| Exec | podman exec |
kubectl exec |
kubectl exec |
ssh |
ssh |
| Auth | Local Podman socket | In-cluster SA or kubeconfig | In-cluster SA or kubeconfig | In-cluster SA or kubeconfig | In-cluster SA or kubeconfig |
| Network | None (--network=none) |
None (deny-all NetworkPolicy) |
None (deny-all NetworkPolicy) |
Cluster pod network (SSH) | Cluster pod network (SSH) |
| User | 65532 (fixed) |
Namespace UID range (SCC) | Namespace UID range (SCC) | root (inside VM) |
root (inside VM) |
| Cleanup | podman rm -f |
delete_namespaced_pod |
delete_namespaced_pod |
Delete VMI + Secret + temp keys | Delete VMI |
All five implement the Sandbox protocol: start(), exec(),
write_file(), read_file(), cleanup(), linger().
The pod pool backend (k8spool) eliminates pod startup latency by claiming
pre-warmed pods from a Kubernetes Deployment. Claimed pods are detached
from the ReplicaSet and the Deployment automatically creates replacements.
After a successful workflow, pool pods linger for linger_seconds (default
300, configurable, 0 to disable) so that user replies can reuse the same
pod without re-creating it or restoring from S3. The reaper runs once per
workflow execution and deletes expired lingering pods. It also evicts excess
lingering pods beyond max_lingering_pods (default 2), starting with those
closest to their deadline. See the design doc (section 8.8) for details.
The KubeVirt backends (kubevirt, kubevirtpool) provide full VM isolation
using KubeVirt VirtualMachineInstances. The VM boots from a containerDisk
image and SSH keys are injected via a Kubernetes Secret volume (the VM
image’s inject-ssh-keys.service reads from /dev/disk/by-id/virtio-ssh-pubkeys).
Command execution uses SSH instead of kubectl exec. The kubevirtpool
backend claims pre-warmed VMIs from a VirtualMachineInstanceReplicaSet,
with the same claim/reap/linger semantics as the pod pool. The sandbox
backend can be set per-workflow via the sandbox: field in the workflow
config, with resolution order: workflow config > CLI --sandbox > default.
The Python API does not pass caller credentials into sandbox processes. Callers must continue to keep credentials on the trusted side of the API and provide the namespace network and resource controls required by their backend. For workflow and CLI integration details, see Hummingbird Agent.
Development
Build the Hummingbird Agent execution image from the tools repository root:
podman build -f hummingbird-sandbox/images/agent/Containerfile \
-t localhost/hb-agent-sandbox:latest hummingbird-sandbox/
The CVE agent’s optional RPM-preparation image can be built locally with:
podman build -f hummingbird-sandbox/images/cve/Containerfile \
-t localhost/hb-cve-sandbox:latest hummingbird-sandbox/
Run the sandbox library tests and lint from the tools repository root:
cd hummingbird-sandbox
python3 -m unittest discover -s tests -t .
ruff check .
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.
4.5.9 - SNS S3 Archiver
A Lambda function that archives all messages from the Hummingbird Events SNS
topic to S3. Events are stored as clean JSON with decoded payloads, enabling
easy browsing with zcat | jq and replay to handlers.
Features
- S3 Storage: Events stored with time-hierarchical key structure for efficient time-range queries
- Clean JSON: Decodes gzip+base64 Message payload for human-readable storage
- Replay Support: Preserves all MessageAttributes with updated
content_encodingfor replay fidelity - Easy Access: Download via
aws s3 sync, browse withzcat | jq
Prerequisites
- AWS CLI configured with appropriate credentials (IAM permissions for Lambda, S3, SNS, CloudFormation)
- Podman or Docker (for containerized SAM build/deploy)
- Python 3.11 or later (for development)
Deployment
Build and deploy using containerized AWS SAM CLI:
cd sns-s3-archiver
make build # Build SAM application
make deploy # First deployment (interactive/guided)
make redeploy # Subsequent deployments (non-interactive)
Deployment outputs:
BucketName- S3 bucket name for archived eventsBucketArn- S3 bucket ARNFunctionArn- Lambda function ARN
Parameters
| Parameter | Description | Default |
|---|---|---|
ResourcePrefix |
Prefix for all resource names | myapp-prod |
SnsTopicArn |
ARN of the SNS topic to subscribe to | (required) |
SentryDsn |
Optional Sentry DSN for error tracking | "" |
RetentionDays |
Days to retain events (0=indefinite) | 0 |
Resource naming: Bucket and Lambda names are derived from ResourcePrefix:
- S3 bucket:
{ResourcePrefix}-s3-events - Lambda function:
{ResourcePrefix}-lambda-s3-archiver
Prerequisites: Requires an existing SNS topic. Deploy
hummingbird-events-topic first to create the topic, then use its
ARN for the SnsTopicArn parameter.
S3 Key Structure
Events are stored with a time-hierarchical key structure optimized for time-range queries:
sns/YYYY/MM/DD/HH/MM/TIMESTAMP#MSGID::source::kind::name.json.gz
Examples:
sns/2025/12/13/12/34/2025-12-13T12:34:56.789Z#abc12345::kubernetes::Snapshot::konflux-rh03--tenant--my-snapshot.json.gz
sns/2025/12/13/12/34/2025-12-13T12:34:56.789Z#def67890::gitlab::push::abc123def456.json.gz
sns/2025/12/13/12/34/2025-12-13T12:34:56.789Z#0123abcd::gitlab::pipeline::org-project-42.json.gz
sns/2025/12/13/12/34/2025-12-13T12:34:56.789Z#4567efab::gitlab::build::org-project-99.json.gz
sns/2025/12/13/12/34/2025-12-13T12:34:56.789Z#89abcdef::kubernetes::TaskRun::konflux-rh03--tenant--task-1.json.gz
The sns/ prefix separates the new format from legacy data, enabling incremental
migration. The :: delimiter separates metadata fields (avoiding conflicts with K8s
names that contain --). The resource name in the key enables efficient “latest state
per resource” queries without downloading all files.
The minute-level subdirectory (/MM/) provides write distribution for high-throughput scenarios.
Resource identities
The timestamp and first eight characters of the SNS MessageId make each key
deterministic for one SNS delivery. The archiver never uses an SQS message ID.
Repeated archiver delivery of the same SNS record writes the same S3 object key.
| Source | Kind | Resource identity |
|---|---|---|
| GitLab | push |
First 12 characters of checkout_sha, falling back to after |
| GitLab | pipeline |
Project path with / replaced by -, followed by object_attributes.id |
| GitLab | build |
Project path with / replaced by -, followed by build_id |
| GitLab | Other events | Project path with / replaced by - |
| Kubernetes | All kinds | Logical cluster name, namespace, and name joined by -- |
The logical cluster name comes from the Kubernetes event forwarder’s selected
kubeconfig context, not from the API server URL. URL-valued cluster
attributes from an older forwarder retain the name-only identity used by
existing archives.
When a Pipeline or Job Hook omits its numeric ID, the archiver retains the project-only GitLab identity.
sns-s3-archiver/tests/fixtures/archive-key-contract.json is the versioned,
machine-readable contract for these identities. Archiver and GitLab forwarder
tests load it directly. Hummingbird Status tests will load the same repository
file as part of HUM-7529. It is test data, not a runtime dependency.
Storage Format
Each S3 object contains a gzip-compressed JSON record:
{
"MessageId": "abc12345-1234-5678-9abc-def012345678",
"Timestamp": "2025-12-13T12:34:56.789Z",
"Message": {
"apiVersion": "appstudio.redhat.com/v1alpha1",
"kind": "Snapshot",
"metadata": { ... },
"spec": { ... }
},
"MessageAttributes": {
"source": { "Type": "String", "Value": "kubernetes" },
"kind": { "Type": "String", "Value": "Snapshot" },
"content_encoding": { "Type": "String", "Value": "json" }
}
}
Key points:
Messageis a decoded JSON object (not the original base64+gzip string from SNS)content_encodingis set to"json"to indicate the decoded format
Usage
Download Events Locally
# Sync entire bucket
aws s3 sync s3://bucket-name/ ./local-events/
# Sync specific time range
aws s3 sync s3://bucket-name/2025/12/13/ ./local-events/2025/12/13/
# Sync with filtering
aws s3 sync s3://bucket-name/ ./local-events/ --exclude "*" --include "*kubernetes*Snapshot*"
Browse Events
# List events for a specific hour
aws s3 ls s3://bucket-name/2025/12/13/12/ --recursive
# View a single event
aws s3 cp s3://bucket-name/2025/12/13/12/34/event.json.gz - | zcat | jq
# Local browsing
zcat ./local-events/2025/12/13/12/34/event.json.gz | jq
Replay Events
Consumer handlers should support both live SNS format and archived format:
def _decode_message(sns_record: dict) -> dict:
"""Decode Message - handles live SNS and archived formats."""
message = sns_record.get("Message", "")
encoding = sns_record.get("MessageAttributes", {}).get("content_encoding", {}).get("Value")
# Archived format: Message is already decoded
if encoding == "json" or isinstance(message, dict):
return message if isinstance(message, dict) else json.loads(message)
# Live SNS format: Message is gzip+base64
if encoding == "gzip+base64":
return json.loads(gzip.decompress(base64.b64decode(message)))
# Fallback: plain JSON string
return json.loads(message)
Compression Flow
Forwarder → SNS → Archiver → S3
1. Forwarder: gzip(payload) → base64 → SNS Message
2. SNS: Passes through with content_encoding="gzip+base64"
3. Archiver: Decodes Message, sets content_encoding="json", gzip(record) → S3
4. S3: Stores clean JSON record, gzip-compressed at object level
Development
See the main README for development workflows.
make setup # Install dependencies
make check # Lint code (ruff)
make fmt # Format code
make test # Run unit tests
make coverage # Run tests with coverage
S3 Lifecycle Policies
The bucket has two lifecycle rules:
- AbortIncompleteUploads: Automatically cleans up incomplete multipart uploads after 1 day (prevents storage cost from failed uploads)
- ExpireOldEvents: When
RetentionDays > 0, automatically deletes objects older than the specified retention period. Disabled whenRetentionDays = 0(indefinite retention, the default).
Security & Limitations
Security:
- S3 bucket blocks all public access by default
- S3 objects encrypted at rest (AES256)
- Lambda follows least privilege principle (PutObject only)
- CloudWatch logs capture all archive operations (7-day retention)
- Sentry integration for error tracking
Limitations:
- Lambda timeout: 30 seconds
- Lambda memory: 256 MB
- S3 object key length: max 1024 bytes
- Single-region deployment (follows SNS topic region)
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.
4.5.10 - Hummingbird Tools
Database backup and restore tools for the Hummingbird status database. Creates compressed PostgreSQL dumps, uploads to S3 with rotation, and supports restore from the latest backup.
The image also contains Tekton Tasks for MR artifact promotion: they find the tested image and create a push-style Component Snapshot after merge.
Features
- Streaming Backup: Streams pg_dump through gzip to temp file, then S3 multipart upload - handles arbitrarily large databases
- Streaming Restore: Streams S3 through gzip and psql with bounded memory
- Automatic Rotation: Keeps configurable number of backups per group (daily/weekly/monthly)
- Separate IAM Users: Write user for production backups, read-only user for staging restore
- MR Artifact Lookup: Derive the MR head from a merge commit and verify its tested Snapshot against the linked build PipelineRun
- Promotion Component Snapshot: Create a single-Component Snapshot using the tested revision and digest, allowing Integration Service to release it
Prerequisites
- AWS CLI configured with appropriate credentials
- PostgreSQL client tools (pg_dump, psql)
- Python 3.11 or later
- Access to the hummingbird-status PostgreSQL database
The MR artifact lookup Task needs a Git checkout Trusted Artifact at the merge commit, a digest-pinned Hummingbird tools image, and a ServiceAccount that can list and get Snapshots and PipelineRuns in the Konflux namespace. KubeArchive must be reachable by HTTPS; it authorizes reads using the Task’s projected ServiceAccount token and the same namespace permissions.
The promotion Component Snapshot Task also needs permission to list IntegrationTestScenarios and create Snapshots in that namespace. Creating a Component Snapshot for an Application with automatic ReleasePlans can start a release.
Both Tasks unpack the source Trusted Artifact into /var/workdir/source and run their Python
step from /var/workdir. Tekton creates a step’s working directory before any step in the Task runs;
using the extraction path as the working directory makes it root-owned and prevents unpacking
under the restricted SCC.
MR Artifact Lookup
The Task manifest is hummingbird-tools/tekton/find-tested-mr-artifact.yaml.
Pass the merge commit SHA, target repository URL, application, Component, and
namespace. Pass the clone Task’s SOURCE_ARTIFACT result as sourceArtifact. The
clone must include full Git history. Supply the
Hummingbird tools image by digest through toolImage. Set archiveUrl to the
cluster’s KubeArchive API URL, for example
https://kubearchive.apps.cluster.example.com.
The Task reads the merge commit’s second Git parent, then selects the newest
passed MR Snapshot from the live API and KubeArchive. A passed /test or
/retest build is eligible when its Snapshot and build carry the same PaC
test or retest event. Multiple passed Snapshots for one commit are normal — a
/test run creates another — and while RPM builds are not reproducible their
image digests may differ (HUM-9785); the newest eligible Snapshot is selected.
It reads a linked build PipelineRun from KubeArchive when the live object has
been removed, then checks the source URL and revision, target branch, and image
digest. The Task fails when KubeArchive cannot be queried. Its results are
SNAPSHOT, BUILD_PIPELINERUN, IMAGE, SOURCE_SHA, and SOURCE_URL. It
does not create a Snapshot or Release.
Promotion Component Snapshot
The Task manifest is hummingbird-tools/tekton/create-promotion-component.yaml.
Pass the same merge commit, target repository, Application, Component, and
namespace as the lookup Task. Pass the same sourceArtifact and the lookup
Task’s SNAPSHOT, BUILD_PIPELINERUN, IMAGE, SOURCE_SHA, and SOURCE_URL
results as sourceSnapshot,
buildPipelineRun, image, sourceSha, and sourceUrl. Supply the tools image
by digest through toolImage. Pass the same archiveUrl used by the lookup
Task and the current PipelineRun name as promotionPipelineRun.
The Task rechecks Git parent B, the MR Snapshot, its linked build PipelineRun,
and the selected digest. It reads removed MR evidence from KubeArchive while
checking current IntegrationTestScenarios and creating the Component Snapshot
through the live Kubernetes API. It fails if an IntegrationTestScenario for the
Application could run on the push-style component Snapshot. It then creates a Snapshot
named deterministically from the full Component name and merge SHA, containing
only the selected Component entry. The Snapshot has Application and Component
labels, type=component, and a push event label. Its annotations link the merge
commit, MR Snapshot, build PipelineRun, and promotion PipelineRun. The
test.appstudio.openshift.io/pipelinerunstarttime annotation is the promotion
PipelineRun’s status.startTime in epoch milliseconds. The COMPONENT_SNAPSHOT
result contains its name.
Before creation, the Task checks the target branch with Git. For RPM builds,
it compares the build’s monorepo-subdir with the current branch when other
merges have advanced it. An unrelated package merge does not block promotion;
a newer change to the same package does. Builds without a monorepo-subdir
require the merge commit to remain the branch head. A retry reuses a live
Component Snapshot only when its Application, Component, source revision, image
digest, provenance, and original promotion start time match; otherwise it reports
the conflicting fields. It never reuses an archived Snapshot. Git branch access
is required for this check.
The Task creates no Release or build attestation. Integration Service handles the automatic Release. See MR artifact promotion for the complete flow and Snapshot ordering. If a retry finds a matching Snapshot, the Task returns that name without creating another one.
Promotion failure and recovery
If a stale run stops, let promotion for the current target branch proceed. If the Component Snapshot conflicts, inspect it and its source MR evidence; do not delete or replace it to force another Release. A failed automatic Release needs an explicit operator-owned retry through the release workflow. If the tested MR Snapshot or build evidence is unavailable, recover through an explicit new build and required tests before release. Check Git access for branch lookup failures, ServiceAccount permissions for forbidden writes, and tenant quota for quota failures. A normal Task retry does not repair a failed Release.
Deployment
The SAM template creates:
- S3 bucket for database backups
- IAM user with write permissions (for production backup CronJobs)
- IAM user with read-only permissions (for staging restore CronJob)
Build and deploy using containerized AWS SAM CLI:
cd hummingbird-tools
make build # Build SAM application
make deploy # First deployment (interactive/guided)
make redeploy # Subsequent deployments (non-interactive)
Note: Only deployed in production - staging restore reads from the production backup bucket.
Parameters
| Parameter | Description | Default |
|---|---|---|
ResourcePrefix |
Prefix for all resource names | myapp-prod |
Resource naming:
- S3 bucket:
{ResourcePrefix} - Write IAM user:
{ResourcePrefix}-write-user - Read IAM user:
{ResourcePrefix}-read-user
Usage
Backup
Run a backup with rotation:
python3 -m hummingbird_tools.backup <group> <rotation>
Arguments:
group: Backup group name (e.g.,daily,weekly,monthly)rotation: Number of backups to keep for this group
Environment variables:
| Variable | Description |
|---|---|
S3_BUCKET |
S3 bucket name |
S3_PREFIX |
Optional prefix for S3 keys |
POSTGRESQL_HOST |
Database hostname |
POSTGRESQL_USER |
Database username |
POSTGRESQL_PASSWORD |
Database password |
POSTGRESQL_DATABASE |
Database name |
Example:
export S3_BUCKET=myapp-prod-db-backups
export S3_PREFIX=mydb/
export POSTGRESQL_HOST=myapp-postgres
export POSTGRESQL_USER=postgres
export POSTGRESQL_PASSWORD=secret
export POSTGRESQL_DATABASE=mydb
python3 -m hummingbird_tools.backup daily 7
This creates a backup like mydb/2026-01-16-02-30.daily.sql.gz and removes
any daily backups beyond 7.
Restore
Restore from the latest backup:
python3 -m hummingbird_tools.restore
Environment variables are the same as for backup.
The restore process:
- Lists all backups matching
*.{daily,weekly,monthly}.sql.gz - Selects the latest by filename sort (most recent timestamp)
- Opens the backup stream and starts a database transaction
- Removes the
rpm_trackerschema and existingpublictables - Streams the decompressed SQL through psql with
ON_ERROR_STOP=1 - Commits only after the complete gzip stream validates and all SQL succeeds
SQL errors and damaged backups fail the restore and roll back both cleanup and import. This replaces tracker data rather than merging it. An older backup without tracker tables removes the target tracker schema rather than retaining stale data. Other schemas are not removed automatically. If the dump conflicts with an existing schema outside this cleanup scope, the restore fails and rolls back instead of merging it.
Pause all database writers and wait for active Jobs to finish before restoring.
SCALE_DEPLOYMENT pauses only its configured Deployment, not tracker CronJobs or other
services. Ensure database roles referenced by the dump already exist. After restoring,
run service initializers to apply migrations and refresh environment specific permissions
before resuming workloads.
CronJob Schedule
Deployed via Kubernetes CronJobs in kubernetes/hummingbird-status/:
| CronJob | Schedule | Environment | Command |
|---|---|---|---|
| daily | 30 2 * * * |
production | python3 -m hummingbird_tools.backup daily 7 |
| weekly | 30 6 * * 0 |
production | python3 -m hummingbird_tools.backup weekly 4 |
| monthly | 30 10 1 * * |
production | python3 -m hummingbird_tools.backup monthly 12 |
| restore | (suspended) | staging | python3 -m hummingbird_tools.restore |
To manually trigger a staging restore:
kubectl create job --from=cronjob/hummingbird-status-restore manual-restore-$(date +%s)
S3 Key Structure
Backups are stored with a flat key structure per database:
{database}/{timestamp}.{group}.sql.gz
Examples:
mydb/2026-01-16-02-30.daily.sql.gz
mydb/2026-01-12-06-30.weekly.sql.gz
mydb/2026-01-01-10-30.monthly.sql.gz
Multiple databases can share the same bucket using different prefixes.
Development
See the main README for development workflows.
make hummingbird-tools/setup # Install dependencies
make check # Lint code (ruff)
make fmt # Format code
make test # Run unit tests
make coverage # Run tests with coverage
Restore integration tests require PostgreSQL client tools and an empty disposable database
whose name starts with test_restore_. They refuse an existing populated database and use
fake S3 input, without contacting AWS or Kubernetes:
cd hummingbird-tools
export PGHOST=localhost PGPORT=5432 PGUSER=postgres
export RESTORE_TEST_DATABASE=test_restore_local
python3 -m unittest tests.test_restore_db
Create the empty test database first. Supply PGPASSWORD through your local test environment
if required. Never point these tests at a deployed database.
Security
- S3 bucket blocks all public access
- S3 objects encrypted at rest (AES256)
- Separate IAM users with least privilege:
- Write user:
s3:PutObject,s3:DeleteObject,s3:ListBucket - Read user:
s3:GetObject,s3:ListBucket
- Write user:
- Database passwords passed via environment variables (Kubernetes secrets)
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.
4.5.11 - ProdSec RPM Catalog
Builds an RPM catalog from each Hummingbird Pulp domain and publishes it to
that domain’s rpm-catalog file repository. The public catalog reads RPM
metadata through DNF. Private catalogs read repository versions through the
authenticated Pulp API so content guards on downloads do not block catalog
generation. A monitoring companion prints the last public publication time.
Features
- DNF-Based Public Query: Reads package metadata from all 5 public RPM repos (source, x86_64, aarch64, ppc64le, s390x)
- Authenticated Private Query: Reads the same repos from a pinned Pulp repository version, requesting only the fields used in the catalog
- Pulp REST API Upload: Creates file content in the selected repository; autopublish updates its distribution
- Artifact Deduplication: Skips upload when content SHA-256 matches an existing artifact
- Async Task Polling: Handles Pulp’s asynchronous tasks with configurable timeout (10 minutes)
- Sentry Error Reporting: Errors are reported to Sentry when
SENTRY_DSNis set - Monitoring Check: Companion module prints the publication timestamp for automated freshness monitoring
Prerequisites
- Python 3.11 or later
dnfsystem library for the public catalog (available on Fedora/RHEL)- Network access to the Pulp API and public repo metadata
- Pulp API access with client certificate or username/password auth
- An autopublishing file repository and distribution named
rpm-catalogin each target domain - Writer access to view and modify the file repository in each target domain
Usage
Build and Publish Catalog
python3 -m hummingbird_tools.prodsec_catalog rpm-catalog
For a private domain, use its restricted-reader config for package listing and its writer config for catalog publication:
PULP_READER_CONFIG=./reader-cli.toml PULP_CONFIG=./writer-cli.toml \
python3 -m hummingbird_tools.prodsec_catalog rpm-catalog \
--domain private-hummingbird-example --source pulp-api
The private CronJob mounts a ConfigMap with one domain per line and processes each domain with the same reader and writer credentials:
PULP_READER_CONFIG=./reader-cli.toml PULP_CONFIG=./writer-cli.toml \
python3 -m hummingbird_tools.prodsec_catalog rpm-catalog \
--domains-file ./domains.txt --source pulp-api
Blank lines and lines beginning with # are ignored. Only private-*
domains are accepted. If one domain fails, the updater continues with the
others and exits unsuccessfully after processing the full list.
The updater refuses to publish an empty catalog. A missing RPM repository or an incomplete Pulp API page also fails the run instead of publishing partial data.
Check Last Publication Time
python3 -m hummingbird_tools.prodsec_catalog_check
Prints the timestamp of the most recent catalog publication in local timezone. Uses the same authentication as the catalog builder.
Configuration
Authentication
Client certificate auth (when configured):
| Variable | Description |
|---|---|
HUMMINGBIRD_PULP_BOT_CERTIFICATE |
Path to client certificate PEM file |
HUMMINGBIRD_PULP_BOT_KEY |
Path to private key PEM file (optional) |
HUMMINGBIRD_PULP_BOT_PASSWORD |
Optional passphrase for the key |
Basic auth (alternative for local use):
| Variable | Description |
|---|---|
PULP_USERNAME |
Pulp API username |
PULP_PASSWORD |
Pulp API password |
Credentials can also be configured in ~/.config/pulp/cli.toml (same format
as the pulp CLI). Set PULP_CONFIG to override the config file path (useful
in containers where the home directory may vary).
Pulp API
| Variable | Description |
|---|---|
PULP_BASE_URL |
Pulp API base URL (or read from cli.toml) |
PULP_CONFIG |
Path to pulp CLI config file (default ~/.config/pulp/cli.toml) |
PULP_READER_CONFIG |
Optional separate read-only CLI config for API enumeration |
--domain selects one Pulp domain (default public-hummingbird), while
--domains-file reads private domains from a mounted ConfigMap. --source
selects dnf (public default) or pulp-api (private domains). PULP_BASE_URL,
when set, must already include the selected domain’s /api/v3 path.
When PULP_READER_CONFIG is set, its credentials are used only for package
listing; PULP_CONFIG remains the writer used to upload the catalog. Without
a separate reader config, the writer performs both operations.
Error Reporting
| Variable | Description |
|---|---|
SENTRY_DSN |
Optional Sentry DSN for error tracking |
CronJob Schedule
Deployed via Kubernetes CronJobs in the infrastructure repository’s
kubernetes/hummingbird-tools/ templates:
| CronJob | Schedule | Package source |
|---|---|---|
hummingbird-tools-prodsec-catalog |
@daily |
Public DNF metadata |
hummingbird-tools-private-catalog |
@daily |
Pulp API for configured private domains |
The private CronJob has one container and mounts a ConfigMap containing the
private domain list. To onboard a domain, provision its RPM repositories and
an autopublishing rpm-catalog file repository and distribution, grant the
shared restricted-reader and writer accounts access to that domain, then add
its name to the ConfigMap in the infrastructure repository. Regenerate and
commit the infrastructure manifests. The CronJob mounts generic reader and
writer Kubernetes Secrets; one domain’s failure does not stop updates for
the remaining domains, but the Job still fails overall.
Catalog Format
The catalog is a tab-separated file with one RPM per line:
{name}-{version}-{release}.{arch}.rpm\t{repo}\t{build_timestamp}
Example:
nginx-1.28.0-1.hum1.x86_64.rpm x86_64 2026-01-15 14:30:00
kernel-6.12.5-1.hum1.src.rpm source 2026-01-10 08:00:00
For example, a catalog can be served at
https://packages.example.com/api/pulp-content/private-hummingbird-example/rpm-catalog/hummingbird-rpm-catalog.txt.
The distribution’s content guard still applies when downloading it.
Development
See the main README for development workflows.
make hummingbird-tools/setup # Install dependencies
make check # Lint code (ruff)
make test # Run unit tests
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.
4.5.12 - Chainguard Comparison
Collects CVE posture data comparing Hummingbird and Chainguard container images and surfaces the results in the Hummingbird dashboard under the Competitive tab.
Features
- Fetches Hummingbird CVE data from the catalog API — the same source used by
images.redhat.com, updated hourly by the scan pipeline - Scans Chainguard images with Grype (the only available source for Chainguard CVE data)
- Stores results, including the individual CVE IDs found by each side, in PostgreSQL for trend tracking and historical comparison
- Dashboard shows per-image CVE counts, severity breakdowns (C/H/M/L), delta (HB minus CG), 30-day trend charts, aggregate summary stats, and a per-image CVE ID diff drawer
- Runs every 6 hours as a Kubernetes CronJob
Prerequisites
- Access to the Hummingbird cluster (PostgreSQL, image pull credentials)
- The
chainguard_comparisonpackage from competitive-tools installed in the container image
Deployment
The tool is deployed as a Kubernetes CronJob. The container image is built by
Konflux from chainguard-comparison/Containerfile.
Environment variables
| Variable | Required | Description |
|---|---|---|
DATABASE_URL |
Yes | PostgreSQL connection string, e.g. postgresql://user:pass@host/db |
GRYPE_TIMEOUT |
No | Per-image Grype scan timeout in seconds (default: 300) |
SENTRY_DSN |
No | Sentry DSN for error reporting |
How it works
- Discovery — lists Hummingbird images from the catalog API and Chainguard images via the OCI distribution API
- Matching — pairs images by name using the alias map in
competitive-tools(e.g.nodejs↔node,postgresql↔postgres). Each pair also captures the catalog’s own real image name and stream for the chosen Hummingbird tag (e.g. real namevalkey, stream9.0) alongside the “common” display name/parsed version used for comparison (e.g.redis,9) — the two can disagree, either because the display name is a cross-vendor alias or because the parsed version doesn’t match the catalog’s own stream label. Both are stored (see schema below) so the dashboard can match HUM tickets against the catalog’s real identity instead of the display name (see hummingbird-dashboard.md) - HB collection — calls
GET /v1/images/{name}/vulnerabilities/{tag}on the catalog API; no image pull or local scan required. The catalog API’s own scan is Grype-based too, without--by-cve, so itsvulnerabilities[].idcan also be a raw GHSA/Go vulnerability DB ID - CG collection — pulls each Chainguard image and runs
grypelocally with--by-cve; Chainguard has no publicly accessible pre-scanned CVE feed.--by-cvereports a CVE ID instead of the native GHSA/Go vulnerability DB ID wherever grype’s own local vulnerability DB has that cross-reference, with no added network call - Alias resolution — every non-CVE ID collected from either side
(
cg_comparison.__main__._resolve_non_cve_ids) is looked up in one batched call togrype db search vulnagainst grype’s already-downloaded local database, which checks every data provider’s record for that ID, not just the one that produced a live scan match – this finds cross-references--by-cve’s own live match misses. An ID with no known CVE anywhere in the database is stored as-is - Storage — writes both sides to the
chainguard_scanstable in PostgreSQL
Dashboard
The /competitive page in hummingbird-dashboard shows:
- Summary cards: total images compared, count worse/better/same
- Sortable table with per-image CVE counts and delta
- Inline severity breakdown: Critical / High / Medium / Low
- Per-image 30-day trend chart (click trend on any row)
- Filter box for quick image lookup
- A per-image CVE diff drawer (click the Delta value on any row) listing
every CVE ID that differs between the two scans:
+— a CVE ID reported only by Hummingbird’s scan-— a CVE ID reported only by Chainguard’s scan- This is a scan result difference, not a confirmed determination of whether either side is actually affected
- CVE IDs both sides report are shared and excluded from the diff
- Each CVE ID links to its Red Hat CVE page; when a HUM Jira ticket tracks
it, the ticket is shown and linked alongside it (
HUM-5678/CVE-2026-1234) - If a scan predates the
cve_idscolumn, the drawer shows a notice instead of a misleading empty diff
Database schema
CREATE TABLE chainguard_scans (
id SERIAL PRIMARY KEY,
collected_at TIMESTAMPTZ NOT NULL,
source TEXT NOT NULL, -- 'hummingbird' or 'chainguard'
image_name TEXT NOT NULL,
tag TEXT NOT NULL DEFAULT 'latest',
image_ref TEXT,
total_cves INTEGER NOT NULL DEFAULT 0,
critical INTEGER NOT NULL DEFAULT 0,
high INTEGER NOT NULL DEFAULT 0,
medium INTEGER NOT NULL DEFAULT 0,
low INTEGER NOT NULL DEFAULT 0,
negligible INTEGER NOT NULL DEFAULT 0,
unknown INTEGER NOT NULL DEFAULT 0,
cve_ids JSONB NOT NULL DEFAULT '[]'::jsonb, -- distinct CVE IDs found in this scan
catalog_image_name TEXT, -- real Hummingbird image name, e.g. "valkey" (image_name may be a display alias, e.g. "redis")
catalog_stream TEXT -- real catalog stream, e.g. "9.0" (tag may be a parsed, less precise version)
);
The table is created by the Alembic migration
b4e7f8a1c3d5_add_chainguard_scans.py; cve_ids was added later by
9c4f7e21ab36_add_cve_ids_to_chainguard_scans.py, and
catalog_image_name / catalog_stream by
d3f6a8b1c2e4_add_catalog_identity_to_chainguard_scans.py. Scans collected
before the cve_ids migration shipped have an empty cve_ids array even
though total_cves is nonzero — the dashboard detects this case and shows
a “check back after the next collection run” notice instead of a false
empty diff. Scans collected before catalog_image_name / catalog_stream
existed have both as NULL; the dashboard’s HUM ticket matching falls back
to image_name / tag (the display name/parsed version) in that case,
matching its pre-migration behavior exactly.
Development
cd chainguard-comparison
pip install -e ".[dev]"
python -m pytest tests/
To run the collector locally against a development database:
export DATABASE_URL=postgresql://postgres:dev@localhost:5432/events
python -m cg_comparison
License
GPL-3.0-or-later
4.5.13 - CVE Analysis
Automated analysis of HUM Jira Security (CVE) tickets. For each ticket, the tool fetches vulnerability data from MITRE and NVD, compares it against the versions shipped in the Hummingbird package repository, searches for upstream and Fedora fixes, and computes whether the ticket should be closed, moved to In Progress, or flagged for manual investigation.
CVE Lifecycle & State Vocabulary
A CVE tracker moves through a shared lifecycle that both this tool and the
CVE agent reason
about. The deterministic
analysis decides a target resolution for each ticket; the residual it
cannot resolve (cve-needs-attention) is handed to the agent; closed tickets
are later reconciled against the VEX feed.
stateDiagram-v2
direction TB
[*] --> New
New --> InProgress: affected / needs investigation
New --> DoneErrata: fixed by analysis
New --> NotABug: component not present
New --> WontDo: package EOL
InProgress --> Agent: upstream-fix-available / cve-needs-attention
Agent: CVE Agent
Agent --> Review: fix MR opened
Agent --> DoneErrata: already_fixed
Agent --> NotABug: not_affected
Agent --> InProgress: no_upstream_fix / needs_human / error
Review --> DoneErrata: MR merged
Review --> Review: MR conflicts / needs human
state "Closed (JiraStatus)" as Closed {
DoneErrata: Done-Errata
NotABug: Not a Bug
WontDo: Won't Do
}
DoneErrata --> AwaitingVEX: enqueue awaiting-vex
NotABug --> AwaitingVEX: enqueue awaiting-vex
WontDo --> [*]: no VEX reconciliation
AwaitingVEX --> [*]: VEX matches resolution
AwaitingVEX --> AwaitingVEX: pending / mismatch
classDef jiraStatus fill:#e5e7e9,stroke:#27ae60,color:#1f2937
classDef closedStatus fill:#e5e7e9,stroke:#1f5f8b,color:#fff
classDef resolution fill:#d5f5e3,stroke:#27ae60,color:#1f2937
classDef automation fill:#e5e7e9,stroke:#7f8c8d,color:#1f2937
class New,InProgress,Review jiraStatus
class Closed closedStatus
class DoneErrata,NotABug,WontDo resolution
class Agent,AwaitingVEX automation
Notes on the diagram:
- Agent is the CVE agent. See that tool’s doc for the full internals and automated sweep deployment.
- Blue states are Jira statuses, green states are resolutions, and gray states represent supporting automation.
- Won’t Do (package EOL) closes without VEX reconciliation;
Done-ErrataandNot a Bugenqueueawaiting-vexand leaveAwaitingVEXonly once the VEX feed matches the resolution.
These state names live in exactly one place,
hummingbird_cve_analysis/lib/states.py, imported by both tools:
JiraStatus—New,In Progress,Review,Closed.Resolution—Done-Errata,Not a Bug,Won't Do(set when closing).Disposition— the second field of acomputed_resolutionline: the three resolutions above plus the open-ticket pseudo-dispositionsaffectedandneeds investigation.Verdict— the CVE agent’s per-ticket outcome (needs_change,not_affected,already_fixed,no_upstream_fix,needs_human,error), withVERDICT_OUTCOMEmapping each verdict to the Jira status/resolution it implies.
How It Works
1. Ticket Selection
The tool queries Jira for HUM project tickets with component Security.
Tickets can be filtered by creation period (--show-since "2 weeks"),
limited to specific keys (HUM-796 HUM-518), or scanned in bulk. Closed
tickets are skipped by default unless --include-closed is specified or
specific ticket keys are given on the command line.
Embargoed tickets (security level “Embargoed Security Issue” or embargo status field set to true) are always skipped.
Trackers filed against a Hummingbird image rather than an RPM (a
pscomponent:hi/<name> label, or CVE-YYYY-NNNN hi/<name>: in the
Summary) are filed in error and are always skipped. Like other automation,
this respects the human override: trackers assigned to someone other than
the bot only get the skip-assignee notice. Otherwise each run logs a
warning with the alert text and ticket link, and with --resolve open
trackers get a tracker comment, the cve-needs-attention label, and a
one-time Slack alert via --slack-webhook-url (":alert-siren: HUM CVE
TRACKER FILED IN ERROR – TRACKER FILED AGAINST IMAGE AND NOT AN RPM",
with the ticket link and an @-mention of prarit). A separate comment marker
records successful Slack delivery. The notice, the delivery marker, and
the label are checked independently, so a failed webhook call or Jira
write is retried on the next run.
Tickets with no valid CVE-YYYY-NNNN in the CVE ID field or Summary
(for example GHSA/OSV-only trackers) are postponed: analysis is skipped,
an INFO message is logged, and with --resolve a one-time tracker comment
is posted and the cve-needs-attention label is applied (open tickets
only). Use --skip-no-cve to omit those tickets without posting the
postpone notice or adding the label.
Jira mutations driven by ticket selection itself (the image tracker alert,
skip-assignee notice, and no-CVE postponement above) run serially, before the per-ticket pool
starts. Per-ticket analysis (sections 2-7) is read-only. With --resolve,
each ticket’s decide step (analyze plus the Jira/advisory mutations in
section 9) also runs on that same pool. Both use --max-workers threads
(default: 8; also CVE_ANALYSIS_MAX_WORKERS). The advisory batch MR
apply (section 8, push/create) stays serial after the pool drains.
Results are collected as each ticket finishes but reported — both on
stdout and in the --handoff-file payload — back in the same order Jira
returned the tickets, regardless of which ticket finished first.
2. CVE Data Collection
For each ticket the tool:
-
Extracts the CVE ID from the Jira CVE ID field (
customfield_10667) when populated, otherwise from the Summary via a regex match for CVE-YYYY-NNNN patterns. If neither source has a CVE ID, analysis is postponed (see above). -
Loads the CVE record from a local clone of the cvelistV5 repository and parses the CVE 5.0
affectedblock, following the CVE 5.0 Product and Version Encodings specification. This includes:- Parsing
lessThanandlessThanOrEqualversion ranges - Handling
lessThan: "*"(no upper bound) andlessThan: "4.*"(end-of-series, all versions in the 4.x series) wildcards per the spec - Processing
changeslists that subdivide ranges into affected and unaffected segments - Respecting
defaultStatusat the product level - Filtering out
versionType: "git"entries that use commit hashes instead of numeric versions (these are preserved as informational data but not used for version comparison) - Detecting CNA data errors where git commit hashes are used in
version fields without
versionType: "git", including hashes wrapped in operator syntax (e.g.,< 6374ae0bcdfe...) - Parsing inline comparison operators (
< 12.3.0,>= 4.0,<= 2.5) that are not part of the CVE 5.0 schema but are widely used by CNAs in practice - Parsing compound range bands (
>= 2.0, < 2.2.26and> 1.32.3, < 1.34.6) used by some CNAs to express a closed-open range in a single version string
- Parsing
-
Looks up version ranges from the NVD data feeds and merges them with the MITRE data. NVD data is obtained from the JSON 2.0 data feeds at
nvd.nist.gov/feeds/json/cve/2.0/(per-year.json.gztarballs updated daily, plus a CVE-Modified overlay updated every 2 hours). Feeds are cached locally (--nvd-cache-dir) and only re-downloaded when the NVD.metafile SHA256 indicates newer data is available. When MITRE has no vendor-specific data (all vendors are “n/a”), NVD ranges replace the MITRE data entirely. NVD references tagged “Patch” or matching commit URL patterns are also extracted and used for upstream fix detection (see below). -
Looks up the Hummingbird package version by scraping the Pulp repository index at
packages.redhat.comfor the latest RPM matching the package name extracted from the ticket summary.
Pulp directory listings are parsed once and shared by version and SRPM lookups.
The parsed filenames and upload times are cached by directory URL until
clear_listing_cache() clears the cache between analysis cycles, so repository
updates become visible after the cache is cleared.
3. Resolution Computation
The tool compares the shipped Hummingbird version against the affected version ranges to compute a recommended resolution:
- Closed / Done-Errata: The repo version is explicitly unaffected (an
exact version, an unaffected range, or product
defaultStatus: unaffected), or NVD-only data establishes that it is past the fix/outside affected ranges. An exact unaffected version does not cover later versions. For CVE 5 data, when no entry matches, omitteddefaultStatusmeans unknown, not safe. Overlapping affected ranges from multiple streams prevent a version-based close unless the version is explicitly unaffected. - In Progress / affected: The repo version falls within an affected range.
- In Progress / affected (no version data): Neither MITRE nor NVD has
affected version information. The ticket is moved to In Progress for
manual review but does not receive the
cve-needs-attentionlabel. - In Progress / needs investigation: Version data was available but
could not be compared (e.g., CVE only provides git commit hashes), or
multiple distinct products with different versioning schemes are listed
and no
cve_productoverride is configured. These cases receive thecve-needs-attentionlabel.
4. Product Mismatch Detection
The tool detects when a CVE was filed against the wrong Hummingbird
package. It compares the CVE vendor/product names against the Hummingbird
package name and upstream repo URL from the package map. For example, a CVE
for isaacs/node-tar (an npm package) filed against the tar RPM (GNU
tar) is flagged as a mismatch. This detection works even without a repo URL
by comparing normalized product names.
Mismatch triage uses a two-phase evidence gate:
- Identity mismatch — CVE product/vendor does not match the package
(
cve_productoverride or heuristic name/repo matching) - SBOM (+ binary) evidence — only then decide misfiled vs vendored vs needs investigation
Phase 2 fetches the latest package SBOM from the Hummingbird Pulp
repository and records the artifact used (NVR/URL) plus match evidence.
SBOM JSON is parsed incrementally; an invalid document is treated as unavailable.
Go module matching accepts the module root and major-version paths
(github.com/vendor/product or .../v5). Deeper import paths such as
.../api or .../daemon only count as a hit when a CVE path hint
(from malformed/version-path fields) aligns with that subpackage, so a
client/API module does not satisfy a daemon-only CVE.
If the CVE product is found as a vendored dependency, the tool runs
binary confirmation in this order:
- SPDX SBOM evidence —
sourceInfopointing at shipped paths (e.g. Go buildinfo under/usr/bin/...) orCONTAINSfrom binary RPM roots (*.x86_64,*.aarch64, …) markspresent_in_binarywithout Syft. Evidence only under.src/go.modis treated as source-only for the next step. - RPM
Provides: bundled(...)— downloaded non-debug binary RPMs are queried withrpm -qp --provides. A matching Fedora bundled Provide (npmbundled(npm(name)), Gobundled(golang(IMPORT_PATH)), Node corebundled(nodejs-undici), etc.) markspresent_in_binarywithevidence_method=rpm_bundled_provides. Ecosystem wrappers such asnpm(...)are unwrapped so a CVE/SBOM search fornanoidmatchesbundled(npm(nanoid)). This catches dependencies embedded into binaries (including minified frontend JS) that Syft cannot inventory as on-disk packages. - Syft on binary RPMs — Syft scans the same RPMs, even after a
Provides or SPDX hit, to find additional shipped copies. When OSIDB reports
subpackage names that match RPMs for the NVR, only those RPMs are
scanned for absence checks; vendored-version checks scan the full NVR.
If OSIDB data is missing or the names do not match this
package’s RPMs (for example a Redis CVE filed against boost), the
full NVR set is used. Pulp download URLs use the arch from the RPM
filename (
aarch64files underaarch64/,x86_64underx86_64/; noarch stays on thex86_64listing).
Those scans are reused. The process keeps provides tuples and Syft artifact
lists in memory, and when RPM_SCAN_CACHE_S3_BUCKET is set it also stores
that JSON in S3, keyed by RPM URL. An object is written only after both
rpm --provides and Syft ran, so a cycle that could not run Syft is not
later reused as a confirmed absence. --rpm-scan-cache-max-bytes (default
1.5 GiB) is a soft LRU budget on the bucket. Each analysis cycle clears the
in-memory cache and leaves the S3 objects in place. The client is configured
with RPM_SCAN_CACHE_S3_REGION, RPM_SCAN_CACHE_S3_ACCESS_KEY_ID, and
RPM_SCAN_CACHE_S3_SECRET_ACCESS_KEY from the RPM scan cache ObjectBucketClaim.
Syft absence only becomes absent_in_binary_confirmed when SBOM
evidence is source-only (or no SBOM document was available) and Syft
actually ran. A miss with ambiguous SBOM evidence stays unknown so
embedded Go/Rust modules are not closed as source-only when SPDX
already ties them to a binary. A Provides-only scan (Syft unavailable)
never confirms absence. Per-RPM download or scan failures are skipped so
sibling RPMs can still confirm presence; a failed scan never proves
absence or that all vendored copies are fixed. A Provides query failure
falls through to Syft for that same RPM but leaves the version evidence
incomplete for auto-closure.
present_in_binary(mismatch_vendored): legitimate vendored ticket; compare the confirmed binary/Provide version (falling back to the SBOM vendored dep version) against the CVE ranges (parentlatest_repo_version/ Hummingbird repo version stay the package), and applycve-needs-attentionso operators can review the binary hit. Matching versions across binary RPMs, Syft artifacts, SPDX binary packages, and bundled Provides are all collected. A binary scan must complete before the versions can justify auto-closure; a confirmed presence with an incomplete scan remains open for investigation. Resolution is worst-of / all-must-be-fixed: the ticket stays affected if any copy is still affected;Done-Errataonly when every copy is not affected. The tool does not invent multi-major fix mappings from a single CVE fixed line — Fixed in Build remains the override when CVE version data only describes one release line. Placeholder dep versions such as cargostd@0.0.0are not comparable: fall back to the parent package version when available; otherwise setnon_comparable_vendored_version, recommendNew / needs investigation, and applycve-needs-attention— neverDone-Erratafrom “0.0.0 is not in any affected range”absent_in_binary_confirmed(mismatch_source_only): source-only; eligible to close as Not a Bug / Component not Present. Must never set Fixed in Build or take the advisory / Done-Errata path.unknown(mismatch_binary_unknown): leave for investigation; do not auto-close or call the ticket misfiled
Stable reason codes are emitted in analysis output (Reason codes:)
and on each CVE entry (reason_code / mismatch_gate) for automation:
| Reason code | Meaning |
|---|---|
mismatch_sbom_miss |
Identity mismatch + SBOM miss → misfiled candidate |
mismatch_sbom_unavailable |
Identity mismatch but SBOM could not be checked |
mismatch_binary_unknown |
SBOM hit but binary presence unknown |
mismatch_source_only |
SBOM hit confirmed absent from binaries |
mismatch_vendored |
SBOM hit confirmed present in binaries → label for review |
non_comparable_vendored_version |
Vendored version is a placeholder (e.g. 0.0.0) with no parent fallback → needs attention |
A misfiled recommendation requires identity mismatch + SBOM miss
(no vendored hit). Ambiguous checks and confirmed binary RPM matches keep
cve-needs-attention with the reason code above. When present in binaries,
normal resolution flow still proceeds (In Progress / Close) in addition to
the attention label. The upstream fix search targets the parent package’s
repo. Each analysis cycle re-fetches the SBOM and re-runs binary
confirmation.
5. Upstream Fix Detection
Fix search uses a precedence model so tickets are not dual-labeled from both Fedora and upstream when a definitive path exists:
- CVE/NVD fix links (preferred): NVD references tagged “Patch” or matching commit URL patterns, plus CVE/NVD PR/MR references that match the package’s upstream repo. When these yield a fix label, Bodhi and forge search are skipped.
- Otherwise, repo-aware search from
metadata/*.jsonupstream_repo:- Fedora-based (
src.fedoraproject.org,pagure.io, or missing repo URL): Bodhi + DistGit only (see §6) - Non-Fedora forge (GitHub/GitLab/cgit): forge search only
- Fedora-based (
Forge Search (non-Fedora fallback):
- GitHub: Searches PRs via the GitHub API (
/search/issues) for PRs mentioning the CVE ID. Fetches commit counts from the pulls API. - GitLab: Searches merge requests via the GitLab API on supported
instances (gitlab.com, gitlab.gnome.org, gitlab.freedesktop.org, etc.).
The
--gitlab-tokenis only sent to gitlab.com to avoid 401 errors on other instances. - cgit: Scrapes commit log pages on cgit hosts (Savannah, Sourceware,
kernel.org, busybox.net, etc.) for commits mentioning the CVE ID.
Handles URL rewrites for Savannah hosts (e.g.,
git.savannah.gnu.org/git/tocgit.git.savannah.gnu.org/cgit/).
Fix status is classified as:
upstream-fix-available: At least one merged/closed PR, committed fix, or NVD patch referenceupstream-fix-in-progress: At least one open PR with no merged fixes
6. Fedora Fix Detection
Used only for Fedora-based packages (or when no upstream repo is configured)
and only when CVE/NVD fix links did not already produce a label. Hits are
classified with the same upstream-fix-* labels as forge/NVD evidence
(Bodhi update links are still shown in analysis output):
- Bodhi: Queries the Fedora Bodhi API for updates matching the package
name that reference the CVE ID (in the
cveslist ornotesfield). Supports multi-page results. Stable →upstream-fix-available; testing/pending →upstream-fix-in-progress. - DistGit Spec Scan: When Bodhi has no matches, fetches the Fedora
DistGit spec file and scans for CVE references in patch filenames,
changelog entries, and comments to detect backported fixes
(
upstream-fix-available).
7. Fixed Build Detection
The tool automatically detects whether the current Hummingbird build fixed a CVE by checking if all CVE IDs are mentioned anywhere in the package directory (patch names, changelogs, comments in the .spec file and so on). If found, it returns the current source RPM name. Assuming that we run the analysis frequently enough, this is precise enough, otherwise it errs on the side of caution (i.e. it possibly marks a higher version as fixed even if an earlier one already carried the fix).
The detected build is shown in the output as Detected Fix: package-version-release.hum1.src.rpm.
When --resolve is specified and a fixed build is detected, the tool
automatically populates the “Fixed in Build” Jira field with the detected SRPM
name, but only if:
- The field is not already set (manual values take precedence)
- The issue is not in Closed status (respects human closure decisions)
- The assessment is not
mismatch_source_only/source_only_confirmed(Component not Present closes as Not a Bug and clears any Fixed in Build) - The computed resolution is already Closed / Done-Errata or every ticket
CVE ID appears on an uncommented
PatchN:line (HUM-6182). A spec comment or leftover unapplied file that merely mentions the CVE is not enough. Product-mismatch and EOL tickets never take this PatchN override.
Version-range analysis can still say “affected” after a backport that does not
bump Version (for example an unbounded range such as 1.1.1+ on popt 1.19).
In that case an applied CVE-YYYY-NNNN.patch listed as PatchN: is allowed to
set Fixed in Build. The SRPM must already be in Pulp (the existing
fix_committed_not_built gate is unchanged). The ticket is not closed here;
advisory review remains the backstop.
The RPMs repository can be provided via --rpms-repo, pointing to a local
clone. This is preferable with multiple runs to avoid the git clone operation.
When not given, the repository will be (shallow) cloned into a temporary
directory.
Similarly, the cvelistV5 repository (used for CVE record lookups) can be
provided via --cve-repo. When not given, a shallow clone is performed into a
temporary directory. For production use, pre-clone and update the repo
externally (e.g. via a CronJob) to avoid the clone overhead on each run. The
tool does not attempt to update the provided repo itself.
8. Advisory Integration (with --resolve)
When a CVE is resolved as “Closed / Done-Errata”, the tool integrates with the CEE GitLab advisories repo to document the fix:
- Clone: The bot fork of the advisories repo (default upstream:
releng/advisories) is cloned blobless (--filter=blob:none), synced to upstreammain, and force-pushed to the fork. A blobless clone keeps the commit history that pushes to a stale fork need while downloading only the checked-out files; a--depth 1clone would be rejected by the fork. The token is embedded in the git URL and redacted from logs. - Modify: For each resolved ticket, the advisory YAML is updated:
cves.fixedentries are added, the type is changed from RHBA to RHSA, and CVE references are appended. - Batch MR: Per-ticket advisory YAML edits run inside the decide
worker pool (serialized by an advisory write lock). Changes accumulate
as individual commits on a single branch (
cve-analysis/batch). One merge request is created after the pool drains, covering all tickets. - Review state: Tickets are transitioned to “Review” (not directly closed) and the batch MR URL is posted to each affected ticket. The MR is approved and queued onto the GitLab merge train; the run does not wait for merge or the post-merge pipeline (HUM-7562).
- Auto-close: On subsequent runs, tickets in “Review” are checked:
if the advisory MR has been merged, the ticket is closed as
“Done-Errata”. The MR URL is taken from the ticket comment, or, if that
comment is missing, from an advisories search by ticket key. If the MR
has unresolvable rebase conflicts, a comment with manual resolution
steps is posted and the
advisory-mr-failedlabel is added. - Slack notifications: When
--slack-webhook-url(or theSLACK_WEBHOOK_URLenv var) is set, the tool sends a Slack message when batch MR creation fails. Failure messages @-mention prarit and jstibran. It also alerts, with an @-mention of prarit, when a tracker is filed against an image instead of an RPM (see above).
The advisory flow is skipped when --advisories-project points to a
non-production URL, allowing safe testing without modifying Jira.
The clone lives in a temporary directory and is removed at the end of the
run. It is intentionally left on disk when the batch MR push fails or
--keep-advisory-repo is passed, so an engineer can inspect it; a leftover
clone from a prior run is automatically removed once it is a few hours old
to prevent repeated failures from exhausting disk/inode space.
8.1 Recovery for failed advisory MRs
Troubleshoot an advisory MR
An advisory MR updates release metadata for a published fixed build.
| What changes | Advisory field | Purpose |
|---|---|---|
| CVE associations | spec.content.artifacts[].cves.fixed |
Records which CVEs the applicable artifacts fix. |
| Tracker links | spec.issues.fixed |
Links the HUM trackers covered by the advisory. |
| Security references | spec.references |
Adds links to CVE details and security information. |
| Advisory type | spec.type |
Changes RHBA to RHSA when the update includes security fixes. |
| Advisory text, when needed | spec.synopsis, spec.topic, spec.description |
Describes the security fixes and packages covered. |
If a retry is not enough, the fix depends on the failure: resolve merge conflicts against the current advisory, or check that the CVE, tracker, Fixed in Build and artifact entries refer to the right package/build. Correct advisory text and references where needed, preserving unrelated fixes, artifact provenance, advisory IDs and ship dates. Permission failures need an authorized author/workflow rather than another retry.
- Open the Advisory MR linked in the Jira tracker comments or the failure notification. Check its pipeline, failed job logs and merge status.
- For a transient failure, retry the failed job or rerun the pipeline in GitLab. This is often enough; no advisory edits are needed.
- If retrying does not help, or the MR has conflicts, permission failures or
needs an advisory correction, ask ProdSec Team for help. Include
the MR URL, affected HUM tracker(s), failed job/error and what you retried.
For manual changes, see the advisories repository’s
contribution guide
and current
field permissions.
Note: changes to
spec.typerequire an allowlisted author and must be committed through theHummingbird CVE Analysis <hummingbird-jira-bot-admins@redhat.com>bot. - Confirm that the MR merges and its default-branch publication pipeline succeeds. Leave Done-Errata closure to advisory automation. If the tracker or dashboard remains out of sync, use the recovery steps below.
Recover lifecycle events after advisory delivery
In some failure cases (for example, advisory MR conflicts followed by manual ticket closure), Jira status can be corrected to Done-Errata while dashboard lifecycle metadata remains incomplete. This leaves R-Time rows PENDING (elapsed-to-now) until image, VEX, and close all exist.
Use the recovery script to backfill delivery events:
rpm_fix_published_to_pulp(hb_rpm_fix) frompackages.redhat.comsource RPM timestampsimage_rebuilt_on_quay(hb_image_fix) from catalog/quay image historyhum_ticket_closedfrom Jiraresolutiondate
Only tickets that are Closed / Done-Errata are used for event derivation and
dashboard import. Tickets that remain in Review (or any non-closed state) are
reported and skipped for event backfill.
For tickets still in Review, the script also attempts advisory reconciliation
before deriving events:
- Read advisory MR URL from Jira comments (
Advisory MR: https://...) - If no MR URL is present in comments, search merged advisories MRs for the HUM ticket key in MR description/title
- When a merged MR is found and
--applyis used, transition the ticket toClosed / Done-Errataand then derivehum_ticket_closed
Script location:
hummingbird-cve-analysis/scripts/cleanup_advisory_mr_failure_events.py
Run in dry-run mode first (default):
cd /path/to/tools
export JIRA_TOKEN=your_jira_token
export CEE_GITLAB_TOKEN=your_cee_gitlab_token
export CVE_REPORT_TOKEN=your_dashboard_token
python3 hummingbird-cve-analysis/scripts/cleanup_advisory_mr_failure_events.py \
--jira-user your-jira-user@example.com \
--cee-gitlab-token your_cee_gitlab_token \
--output-json /tmp/cleanup-advisory-derived.json
Apply the backfill to dashboard event storage:
python3 hummingbird-cve-analysis/scripts/cleanup_advisory_mr_failure_events.py \
--jira-user your-jira-user@example.com \
--cee-gitlab-token your_cee_gitlab_token \
--output-json /tmp/cleanup-advisory-derived.json \
--apply
Target specific tickets (instead of default JQL):
python3 hummingbird-cve-analysis/scripts/cleanup_advisory_mr_failure_events.py \
--jira-user your-jira-user@example.com \
--cee-gitlab-token your_cee_gitlab_token \
--apply \
HUM-4884 HUM-4871 HUM-4844
Verification example:
# Replace with the actual dashboard URL (same as --dashboard-url default)
DASHBOARD_URL=https://hummingbird-dashboard-hummingbird--internal.apps.int.spoke.prod.us-east-1.aws.paas.redhat.com
curl -s "${DASHBOARD_URL}/api/cve/r-time?days=30" | jq -r '
.entries[]
| select(.key=="HUM-4884")
| [.key, (.fix_delivered_at // "-"), (.stages.hb_rpm_fix // "-"), (.stages.hb_image_fix // "-")]
| @tsv
'
If hb_rpm_fix and hb_image_fix remain empty in the derived JSON output,
the issue is source-data availability (no resolvable RPM/image publish signal),
not dashboard ingestion.
If a ticket is in Review and no advisory MR can be found in Jira comments or
merged advisories MRs, it is skipped and reported for manual follow-up.
By default, the script targets advisory-mr-failed tickets (for both
Closed / Done-Errata and Review states); override with --jql when needed.
8.2 Dashboard collection ticket selection
Dashboard lifecycle collection (HUM-5603) uses a separate ticket-selection
path from cve_analysis mutations. Helpers live in
hummingbird_cve_analysis/dashboard_selection.py.
Watermark (start of previous run): read the newest entry from the
dashboard Run Log (GET /api/cve/run-log?limit=1). Use run_started_at when
available; older entries estimate the start as recorded_at - duration_seconds
(recorded_at is ingest time). Subtract a small buffer (default 2 minutes) for
clock skew. The overlap avoids missing tickets updated while the previous
collector run was in progress and is safe because /api/cve-report event
upserts are idempotent.
JQL selection:
- All still-open HUM Security tickets (rescan for updates)
- Closed tickets with Jira
updated >=watermark - Bootstrap (empty Run Log): open tickets only
- Explicit ticket keys:
key in (...)(any status)
Use --start-time (ISO-8601, e.g. 2026-08-01T00:00:00Z) to override the
Run Log watermark and recollect Closed tickets with updated >= that
start, so dashboard data can be overwritten/backfilled.
Inspect watermark and JQL without modifying Jira or the dashboard.
From the tools repo root, either install the package
(make hummingbird-cve-analysis/setup) or set PYTHONPATH:
cd /path/to/tools
export PYTHONPATH=hummingbird-cve-analysis
export CVE_REPORT_TOKEN=your_dashboard_token
python3 -m hummingbird_cve_analysis.dashboard_selection --prod
# Optional: also search Jira and list matching keys
export JIRA_TOKEN=your_jira_token
python3 -m hummingbird_cve_analysis.dashboard_selection \
--prod \
--jira-user user@example.com \
--fetch-issues -o json-pretty
| Option | Environment Variable | Description |
|---|---|---|
--prod / --preprod |
Required. Hardcoded production or preprod dashboard URL | |
--cve-report-token |
CVE_REPORT_TOKEN |
Bearer token for /api/cve/run-log |
--start-time |
Explicit UTC collection start (ISO-8601); skips Run Log fetch | |
--watermark |
Alias for --start-time |
|
--watermark-buffer-minutes |
Minutes subtracted from previous-run start estimate (default: 2) | |
--fetch-issues |
Also search Jira with the constructed JQL | |
--jira-token / --jira-user |
JIRA_TOKEN |
Jira credentials (only with --fetch-issues) |
--jira-url |
JIRA_URL |
Jira base URL (only with --fetch-issues) |
--max-results |
Max issues to fetch with --fetch-issues (default: 2000) |
|
--output, -o |
human, json, or json-pretty |
8.3 Analysis → collector handoff file
cve_analysis writes a small JSON handoff file for
collect_cve_dashboard (HUM-5799 / HUM-5800). This carries Run Log counters,
captured log_output, and per-ticket fields that are expensive to
recompute (upstream_fix, fedora_fix) plus human_text for Run Log
details. Delivery timestamps (hb_rpm_fix / hb_image_fix) and
hum_ticket_closed are gathered by the collector, not this file. Analysis
stdout is human text only; it does not emit /api/cve-report JSON.
PYTHONPATH=hummingbird-cve-analysis python -u -m hummingbird_cve_analysis.cve_analysis \
--jira-user "$BOT_USER" --jira-token "$BOT_JIRA_TOKEN" \
--handoff-file /tmp/cve_analysis_handoff.json \
...
Handoff shape (schema_version: 1):
- Top level:
run_started_at,duration_seconds, mutation counters (labels_changed,advisories_created/advisories_failed,tickets_closed,comments_posted,attachments_uploaded,fatal_errors),log_output tickets[]:key,upstream_fix,fedora_fix,human_textvex_updates[](HUM-5843):key,vex_status,vex_match_state,vex_resolved,labelsfrom the awaiting-vex reconcile pass
8.4 collect_cve_dashboard
collect_cve_dashboard (HUM-5798 / HUM-5800) owns dashboard JSON emission.
cve_analysis no longer prints /api/cve-report JSON on stdout; it prints
human analysis text and optionally writes --handoff-file for the collector.
- Resolve Run Log watermark + open-ticket rescan (section 8.2). Selection
always includes Closed tickets labeled
awaiting-vex(HUM-5843),awaiting-cve-published, orawaiting-image-rebuild(HUM-9975), then anyvex_updates[]keys missing from that JQL - For each ticket, gather slim lifecycle fields from Jira / cvelistV5 /
Fixed-in-Build → Pulp (
cve_published,hum_ticket_created,hum_ticket_closed,hb_rpm_fix) and catalog image history (hb_image_fix), pluscve_ids– every CVE ID the ticket tracks, from its dedicated CVE ID Jira field (falling back to Summary text), not just the first found in labels; a ticket can legitimately track more than one CVE, and consumers keyed by CVE ID (e.g. the dashboard’s Competitive page HUM ticket matching) index it under every entry here, not just one (see hummingbird-dashboard.md), plus OSIDBosidb_flaw_created/osidb_affect_createdfrom the public OSIDB API (omitted when OSIDB has no data),cve_embargo_lifted– the OSIDB flawunembargo_dt, included only when it postdates the flaw’screated_dt(i.e. the CVE was genuinely embargoed; omitted otherwise) – andosidb_flaw_public– the sameunembargo_dt, but included unconditionally whenever OSIDB reports one, regardless of whether the flaw was ever embargoed –computed_resolutionfrom Jira status/resolution (R-Time gates onDone-Erratain that string),catalog_image_sourcefrom the catalog source map (R-Time delivery is image publish when true, RPM publish when false), andcatalog_images– every{image}:{stream}the ticket’s package actually appears in, per the same catalog source map’s SBOM data (used by the dashboard’s Competitive page to match a ticket to a specific image precisely instead of guessing from the package name; see hummingbird-dashboard.md). Timestamp helpers live inlib/lifecycle.pyand are shared withanalyze_issue - Merge
--handoff-filecounters,log_output, per-ticketupstream_fix/fedora_fix/human_text, andvex_updates - Collect agent costs: every ticket carrying a
cve-agent-*label is found with one paginated key-only Jira search, independent of this run’s watermark selection – an unchanged Closed ticket would otherwise never be retried after a failed download. Unrecorded keys (checked in batches against/api/cve/agent-cost-keys) have their<TICKET>-cve-agent-analysis.jsonattachment downloaded for itstoken_costblock, and the result travels as the report’sagent_costslist. A ticket without an attachment yet is retried on the next run; an attachment predating cost reporting is recorded once with a nullestimated_cost_usdso it is not downloaded again. - Dry-run prints
/api/cve-reportJSON;--applyPOSTs it
The collector builds the catalog source map once per run for
hb_image_fix, catalog_image_source, and catalog_images. When a delivery
timestamp is omitted, the dashboard keeps any existing
rpm_fix_published_to_pulp / image_rebuilt_on_quay value rather than
clearing it. An empty or failed catalog map omits catalog_image_source and
catalog_images so the dashboard keeps requiring an image / falls back to
its own package-name heuristics.
Catalog fetches (an image’s tag listing, or a specific stream’s SBOM) are
not retried; a failure is logged and that image/stream is silently skipped
when building the source map, so catalog_images’s absence of an
image/stream is not always confirmed ground truth – it can also mean that
image/stream’s data simply failed to collect this run.
build_catalog_source_map() returns the set of (image, stream) pairs it
couldn’t verify alongside the map (stream is None when the image’s own
tag listing failed, so every one of its streams is unknown). Every ticket
in the run carries this set as catalog_unverified_images (rendered as
{image}:{stream}, or {image}:* for a whole unverified image) so a
downstream consumer – the dashboard’s _match_cve_tickets – can tell
“confirmed absent” from “unknown” before excluding a ticket candidate
based on catalog_images alone (see hummingbird-dashboard.md).
PYTHONPATH=hummingbird-cve-analysis python -u -m hummingbird_cve_analysis.collect_cve_dashboard \
--preprod \
--cve-report-token "$CVE_REPORT_TOKEN" \
--jira-user "$BOT_USER" --jira-token "$BOT_JIRA_TOKEN" \
--cve-repo /tmp/cvelistV5 \
--handoff-file /tmp/cve_analysis_handoff.json
# POST to preprod dashboard
PYTHONPATH=hummingbird-cve-analysis python -u -m hummingbird_cve_analysis.collect_cve_dashboard \
--preprod \
--cve-report-token "$CVE_REPORT_TOKEN" \
--jira-user "$BOT_USER" --jira-token "$BOT_JIRA_TOKEN" \
--cve-repo /tmp/cvelistV5 \
--handoff-file /tmp/cve_analysis_handoff.json \
--apply
| Option | Environment Variable | Description |
|---|---|---|
--prod / --preprod |
Required. Hardcoded production or preprod dashboard URL | |
--cve-report-token |
CVE_REPORT_TOKEN |
Bearer token for Run Log + /api/cve-report |
--handoff-file |
CVE_ANALYSIS_HANDOFF_FILE |
Analysis handoff JSON from cve_analysis --handoff-file |
--apply |
POST report to /api/cve-report (default: dry-run JSON on stdout) |
|
--start-time |
Explicit UTC collection start (ISO-8601); skips Run Log fetch | |
--watermark |
Alias for --start-time |
|
--cve-repo |
Local cvelistV5 repo for cve_published |
|
--jira-token / --jira-user |
JIRA_TOKEN |
Jira credentials |
--output, -o |
human, json, or json-pretty |
8.5 verify_image_publish
verify_image_publish (HUM-721) catches CVE tickets that claim a fix via
image rebuild (hb_image_fix in the whiteboard cve_cycle) but whose image
was never actually published to the registry — hb_image_fix is set from
catalog release-history metadata (oldest_created), which can lag or be
wrong; this re-verifies with a live check.
- Search Jira for Closed HUM Security tickets
updatedwithin the last--window-minutes(default 60) - For each, read
hb_rpm_fix/hb_image_fixfrom the whiteboard; skip tickets wherehb_image_fixfalls outside the window - Re-derive the source package name from the summary/labels (no SBOM /
network calls) and re-run the same catalog-history lookup
(
find_image_publish_release) that originally producedhb_image_fix, recovering the specific image/stream/variant/digest - HEAD the image manifest by digest directly against the registry
(
image_manifest_exists) — this is the parthb_image_fixitself never did - Post a Slack alert for each unverified claim; log a summary either way
Read-only: never modifies Jira, the dashboard, or the CVE analysis pipeline.
PYTHONPATH=hummingbird-cve-analysis python3 -m hummingbird_cve_analysis.verify_image_publish \
--jira-user "$BOT_USER" --jira-token "$BOT_JIRA_TOKEN" \
--window-minutes 60
| Option | Environment Variable | Description |
|---|---|---|
--window-minutes |
Look back this many minutes for hb_image_fix (default: 60) |
|
--slack-webhook-url |
SLACK_WEBHOOK_URL |
Webhook for unverified-claim alerts |
--dry-run |
Check and log, but do not post to Slack | |
--catalog-api-base |
Catalog API base URL | |
--jira-token / --jira-user |
JIRA_TOKEN |
Jira credentials |
--jira-url |
JIRA_URL |
Jira base URL |
9. Jira Actions (with --resolve)
When --resolve is specified, the tool modifies Jira tickets. Those
per-ticket mutations run concurrently with analysis on the
--max-workers pool (HUM-11537); only the advisory batch MR push/create
(section 8) runs once after the pool finishes. Shared advisory-repo
writes are serialized by an internal lock; Jira API GET/PUT retries
cover transient conflicts.
The deployed analysis CronJob also uses --publish-cve-agent-sweep with
--resolve. After a completed cycle it publishes a small SNS event
(source=cve-analysis, event_type=cve_agent_sweep_requested) to wake the
CVE-agent SQS worker. The event contains only the run timestamp; the agent
searches Jira for eligible tickets itself. SNS publishing failures are logged
without undoing or failing completed Jira analysis; the next scheduled run
publishes another request. See the message bus contract
for the exact message body and attributes.
- Fixed in Build: When a fixed build is detected and the field is not
already set, the tool populates it with the detected SRPM name (only for
non-closed issues). Not-affected
Closed / Done-Erratacloses also set Fixed in Build fromdetected_fixed_buildorlatest_srpmwhen empty, so R-Time image timestamps can be resolved. - Closed / Done-Errata: Tickets where the shipped version is not affected are transitioned to Review with an advisory MR (see above). After the MR merges, they are closed as Done-Errata on the next run.
- Move to In Progress: Tickets where the shipped version is affected are transitioned to In Progress with a comment including affected version ranges, upstream fix status, and Fedora update links.
- Needs Investigation: Tickets with incomplete version data get a comment explaining why automatic resolution was not possible.
- Product Mismatch: A comment is posted explaining the mismatch (with
SBOM artifact evidence) and the
cve-needs-attentionlabel is applied. No transition is performed. Auto-detected Fixed-build / delivery timestamps (hb_rpm_fix,hb_image_fix) are cleared so R-Time does not treat misfiled tickets as done. Human Fixed in Build overrides are kept. When--resolveruns against the production advisories project, the analyzed package SBOM JSON is also attached to the ticket as{nvr}.sbom.json(skipped if that filename is already present). The same attachment is applied for other SBOM-backed review paths (vendored binary hits, source-only closes, binary-unknown). - Package Not Present: If the package is missing from the rpms repo,
the tool still runs an SBOM-first check against the CVE product(s). An
SBOM hit triggers Syft binary confirmation:
absent_in_binary_confirmedcloses as Not a Bug / Component not Present; presence or unknown leaves the ticket for investigation. An SBOM miss/unavailable closes as Not a Bug with VEX Component not Present. The close timestamp is persisted ashum_ticket_closedfor dashboard ingestion. - Package EOL (
fix_status: 0): Tickets for packages marked End-Of-Life in rpms metadata are closed as Won’t Do with no VEX Justification. This takes precedence over Done-Errata and Fixed-in-Build / advisory paths. Automation labels (upstream-fix-*, legacyfedora-fix-*,cve-needs-attention,cve-next-release,advisory-mr-failed) are removed;fedora-bz-filedis kept as an audit record. The close timestamp is persisted ashum_ticket_closedfor dashboard ingestion. - Label Management: Labels are applied and updated:
upstream-fix-available/upstream-fix-in-progress(including Bodhi/DistGit evidence for Fedora-based packages)cve-needs-attention(applied when human review is needed; removed when resolved)- Legacy
fedora-fix-*labels are stripped on subsequent runs - Labels are upgraded (in-progress to available) and stale labels are cleaned up on ticket closure.
Every Jira action (label add/remove, status transition, product mismatch) includes a single comment with a bold action summary heading followed by the full analysis output in a preformatted code block, giving the reader the same detail they would see on the CLI.
10. Assignee-Based Automation Control
When --resolve is active, the tool checks each ticket’s assignee. If
the assignee is not the bot account (--jira-user), the tool skips all
automated actions (comments, labels, transitions) for that ticket. A
one-time comment is posted explaining that automation is skipped. This
allows humans to take ownership of a ticket by assigning it to
themselves, preventing the bot from interfering with manual work.
Reassigning back to the bot account re-enables automation.
For testing purposes, use --skip-assignee-check to process all tickets
regardless of assignee.
11. Human Closure Protection
When the tool encounters a Closed ticket whose analysis recommends a non-Closed resolution (e.g., In Progress), it checks the Jira changelog to determine who performed the last status transition. If a human (not the bot) closed the ticket, the tool skips the transition to respect the human’s decision. This prevents the tool from reopening tickets where a fix was backported without a version bump or where a human determined the CVE does not apply.
12. Stale Closure Detection (with --include-closed)
When --include-closed is used, the tool also analyzes Closed tickets. If
the current analysis recommends a non-Closed resolution (e.g., the package
version changed and is now affected), a warning is emitted flagging the
ticket for review.
13. Sprint and Epic Assignment
When --resolve is active, the tool automatically adds CVE tickets to the
current Hummingbird sprint and links them to a per-sprint CVE tracking epic
when human interaction is detected. This ensures sprint metrics capture
human work on CVE tickets.
The tool queries the Jira Agile API for the active sprint on the
Hummingbird board (--board-id, default 1489) and filters by name prefix
(--sprint-prefix, default “Hum S”) to identify the correct sprint among
multiple teams sharing the HUM project.
Sprint and epic assignment is triggered in two cases:
- Self-assigned tickets: When a ticket is assigned to someone other than the bot, the ticket is added to the current sprint and linked to the sprint’s CVE epic.
- Human-set Fixed in Build: When the Fixed in Build field was set by a human (not the bot), as determined by the issue changelog, the ticket is added to the current sprint.
If no matching epic exists for the current sprint, one is created automatically with the summary “CVE tickets for sprint <name>”. The epic key is cached for the duration of the run. A Jira comment is posted on the ticket recording the sprint and epic assignment.
Tickets that already have a sprint assigned are skipped. Tickets closed manually without going through either automated path should be added to the sprint by hand (see the manual process guide).
14. Continuous Operation
When --time-between-runs N is set (N > 0), the tool runs in a loop,
re-executing the full analysis every N minutes. The default is 0 (single
run). Shutdown is graceful: SIGINT/SIGTERM finishes the current cycle
before exiting.
15. Feature Flag
When --feature-flag-name is set (default: cve_analysis_enabled), the
tool checks a GitLab feature flag on the project specified
by --feature-flag-project-id (default: 73447720, i.e. redhat/hummingbird/rpms) at
the start of each cycle. If the flag is inactive, the cycle is skipped
and the tool sleeps for 60 seconds before checking again.
The check uses the --gitlab-token / GITLAB_TOKEN credential, which
must have Developer role or higher on the target project. The check is
fail-open: if GitLab is unreachable or the token lacks permissions, the
tool assumes the flag is enabled and proceeds.
Set --feature-flag-name "" to disable the check entirely.
16. Timing Instrumentation
A @timed(label) decorator
(hummingbird_cve_analysis.lib.timing) wraps the heavier per-ticket
network calls – NVD (fetch_nvd_ranges, get_nvd_references), Pulp
(fetch_hummingbird_versions, fetch_hummingbird_latest_srpm_with_time,
fetch_hummingbird_latest_srpm_as_of, fetch_srpm_upload_time), forge
search (search_forge), Fedora (fetch_bodhi_updates), and SBOM/Syft
(fetch_latest_sbom, confirm_component_in_binary_rpms) – timing each
call with time.perf_counter().
Two levels of output:
- Per ticket:
_analyze_one_issue()(read-only) logs<KEY> analysis timing: ...; with--resolve,_process_one_ticket()logs<KEY> process timing: ...covering analyze plus decide. Both include the per-label breakdown for that ticket and are logged viafinally, so they still appear when the ticket raised. - Per cycle: after the ticket loop, one
Cycle sub-step timing: ...line aggregates every label’s total seconds and call count across the whole cycle (all tickets, all workers).
The per-ticket breakdown is isolated per ticket even under the
--max-workers thread pool (HUM-10139 / HUM-11537): it’s tracked with a
contextvars.ContextVar, which each worker thread gets its own copy of,
rather than a single shared dict two concurrent tickets could stomp on.
The cycle-level totals are a lock-guarded dict shared across all
workers, the same pattern as resolve.increment_run_stat()
(HUM-10136).
Before/after: validating the HUM-10139 parallelization
Three rounds of paired runs (sequential baseline at the commit just before HUM-10139, vs. the parallelized version, same frozen ticket list and shared NVD/cvelistV5/rpms-repo caches for both runs in a pair, run back-to-back) measured the actual wall-clock effect of parallelizing Phase 1:
| Round | Pairs | Tickets/run | Mean speedup | Aggregate speedup | Worst single run |
|---|---|---|---|---|---|
| 1: most-recent 150 tickets | 10 | 150 | 1.72x | 1.47x | 0.43x |
| 2: random sample of 200 tickets | 10 | 200 | 2.77x | 2.74x | 1.84x |
| 3: all currently-open tickets | 10 | 309 | 1.43x | 1.41x | 0.95x |
Across all three rounds (30 pairs, ~5,400 ticket-analyses total), resolution output matched between the sequential and parallel run of each pair with zero correctness regressions; every mismatch traced to a live external cause (a Hummingbird build publishing to Pulp between the two runs of a pair, or the real production bot mutating a ticket’s Jira fields concurrently), not to the code.
Run-to-run variance (both within and across rounds) is dominated by a small number of tickets needing large, multi-architecture RPM downloads for SBOM/Syft binary confirmation – real network/CDN time that varies independent of sequential vs. parallel execution, and that a bounded thread pool can’t finish faster than its slowest in-flight ticket. Round 2’s random sample, less weighted toward any one such cluster, shows both the highest and the most consistent speedup of the three.
See Also
- CVE Analysis Reference — Package map, vendored dependency detection, prerequisites, usage, configuration, managed labels, output, logs, library layout, development, license
4.5.13.1 - CVE Analysis Reference
Reference and operational documentation for the CVE Analysis tool. This page covers package map, vendored dependency detection, prerequisites, usage, configuration, managed labels, output format, logs, library layout, development, and license. For conceptual documentation (CVE lifecycle, how the analysis pipeline works), see CVE Analysis.
Package Map
The package map is loaded from the rpms repo’s per-package metadata files
(metadata/<package>.json). Each file contains an upstream_repo field
pointing to the canonical upstream git repository, plus optional fields:
upstream_branch– upstream branch (for versioned packages sharing a repo)cve_product– CVE vendor/product overrideversion_transform– version transform rulefix_status– package fix policy (0= End-Of-Life / will not fix CVEs; close as Won’t Do)
If fix_status is 0, analysis always resolves to Closed / Won’t Do
(including when the shipped version is outside the CVE affected range or a
product mismatch would otherwise need investigation). No VEX Justification
is set. If fix_status is missing or set to 1, behavior is unchanged from
the default analysis flow.
When an upstream fix is available, Jira backport guidance keeps the
NEXT STEPS: Backport upstream patches to Hummingbird marker and reminds
maintainers to keep independent packages’ modification_status as
independent and omit modification_reason, mark Fedora-imported packages
modified with a modification_reason, and leave metadata release
unchanged. This guidance does not affect the CVE assessment or fix decision.
The cve_product field is an optional human-curated value that specifies
which CVE vendor/product entry maps to this package. It supports these
formats:
Vendor / Product(exact match): matches a specific vendor and product pair. Example:"F5 / NGINX Open Source"for thenginxpackage, which excludes NGINX Plus entries that use incompatible R-versioning.Vendor(vendor prefix match): matches any product whose vendor starts with the given string. Example:"Go "forgolangpackages, where the vendor varies (Go standard library,Go toolchain, etc.) and the product varies by module (net/url,os,crypto/x509).- Multiple selectors: provide more than one acceptable value either
as a JSON list in metadata (preferred) or as a semicolon-delimited
string. Example:
["vda-linux / busybox_mirror", "BusyBox / BusyBox"]or"vda-linux / busybox_mirror; BusyBox / BusyBox". The tool treats this as “match any selector”.
When cve_product is set, the tool uses it for exact product matching in
both resolution computation (filtering to the correct product in
multi-product CVEs) and mismatch detection. When empty, the tool falls
back to heuristic name matching.
The rpms repo is cloned automatically at startup (or provided via
--rpms-repo). A CSV override can be passed via --package-map for
backward compatibility.
Vendored Dependency Detection
Vendored dependency detection uses live SBOM lookups from the
Hummingbird Pulp repository. When a CVE product mismatch is detected,
the tool fetches the SPDX SBOM for the ticket’s package from
packages.redhat.com/.../metadata/sboms/{package}-main/ (same public
index used by the rpms CVE skill), selects the newest dated
sha256-….sbom entry from that listing, and searches that document for
the CVE product(s) only (via purl references), stopping on the first
product hit. Binary evidence then collects all matching copies of that
product. The analysis output always records the SBOM artifact used and
whether the lookup hit, missed, or was unavailable.
The same SBOM-first check also gates Component not Present closures
for packages missing from the rpms repo. An SBOM hit alone no longer
blocks that closure: binary confirmation must reach
absent_in_binary_confirmed (no bundled Provide match and Syft miss on
source-only SPDX evidence) before Not a Bug / Component not Present is
recommended. A binary hit or unknown result keeps the ticket open for
investigation.
Go subpackages are matched by progressively stripping path components
(e.g., github.com/jackc/pgx/v5/pgproto3 matches
github.com/jackc/pgx/v5). When a match is found and confirmed in
binaries, the confirmed binary/Provide version is preferred for
resolution over the SBOM lockfile version. The SBOM and binary
confirmation are re-run each analysis cycle.
Requires the rpm and/or syft CLI on PATH (both are available in
the analysis container image via rpm-build and the Syft install).
Without either tool, binary confirmation returns unknown. With only
rpm, a bundled Provide can still confirm present_in_binary, but
absence is never auto-confirmed without Syft.
The generate_vendored_map_sbom.py script in package_maps/ can still
be used for auditing vendored dependencies across all packages, but is
no longer required at runtime.
Prerequisites
- Python 3.11 or later
- Jira API token (Bearer or Basic auth)
- Network access to
redhat.atlassian.net,github.com,nvd.nist.gov,bodhi.fedoraproject.org,src.fedoraproject.org, andpackages.redhat.com(note:github.comis only needed when--cve-repois not provided and the tool must clone cvelistV5 itself) rpmandsyftCLIs onPATH(for binary RPM confirmation after SBOM hits: bundled Provides viarpm, component inventory via Syft)- GitHub API token (optional, for upstream PR search)
- GitLab API token (optional, for gitlab.com MR search)
- CEE GitLab API token (required with
--resolve, for advisory repo)
Usage
# Basic usage with Jira token from environment
export JIRA_TOKEN=your_token
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com
# Analyze specific tickets
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com HUM-796 HUM-518
# Show only tickets from the last 2 weeks
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com --show-since "2 weeks"
# Analyze with upstream fix detection (report findings without modifying Jira)
export GITHUB_TOKEN=your_github_token
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com
# Resolve tickets and apply labels
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com --resolve
# Include closed tickets and check for stale closures
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com --include-closed
# Include closed tickets but exclude specific ones
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com \
--include-closed --exclude "HUM-555,HUM-552"
# Skip issues without CVE links; write collector handoff
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com \
--skip-no-cve --handoff-file /tmp/cve_analysis_handoff.json
# Continuous mode: resolve tickets every 30 minutes
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com \
--resolve --time-between-runs 30
Dashboard /api/cve-report JSON is produced by collect_cve_dashboard
(section 8.4),
not by cve_analysis stdout (HUM-5800).
Inspecting lifecycle timing with dump_lifecycle
scripts/dump_lifecycle reads lifecycle milestones and package data from the
dashboard HTTP API and prints them for one or more HUM tickets. CVE_REPORT_TOKEN is needed.
# Text output for one ticket against prod
scripts/dump_lifecycle --prod HUM-1234
# JSON output for multiple tickets against preprod
scripts/dump_lifecycle --preprod --json HUM-1234 HUM-5678
# Custom dashboard URL
scripts/dump_lifecycle --dashboard-url https://... HUM-1234
Output includes all milestone timestamps (cve_published, osidb_flaw_created,
osidb_affect_created, cve_embargo_lifted, hum_ticket_created, …), computed
duration legs, and
rpm_first_published (package) sourced from
package_lifecycle. The --json flag emits the output as a JSON array instead
of human-readable text.
Backfilling OSIDB timestamps with backfill_osidb_timestamps.py
scripts/backfill_osidb_timestamps.py is a one-time helper that fills
osidb_flaw_created and osidb_affect_created for tickets already stored
in cve_ticket_events. It reads existing rows from /api/cve-export,
looks up timestamps from the public OSIDB API, and POSTs new rows to
/api/cve-import with reset=false (write-once; existing timestamps are
kept). Tickets that already have both OSIDB events are skipped unless
--force is set. CVE_REPORT_TOKEN is needed.
# Backfill prod
scripts/backfill_osidb_timestamps.py --prod
# Dry run against preprod
scripts/backfill_osidb_timestamps.py --preprod --dry-run
# Limit to specific tickets
scripts/backfill_osidb_timestamps.py --prod HUM-1234 HUM-5678
Ongoing collection is handled by collect_cve_dashboard. After a prod
backfill, copy_prod_to_preprod_db copies the new event types with the
rest of cve_ticket_events (they are on the dashboard import allowlist).
Syncing prod CVE data to preprod with copy_prod_to_preprod_db
scripts/copy_prod_to_preprod_db copies cve_analysis_log,
cve_ticket_events, package_lifecycle, and cve_ticket_blame from prod to
preprod via a direct pg_dump/pg_restore over two kubectl port-forward
tunnels, instead of paginating them through /api/cve-export ->
/api/cve-import. cve_analysis_log rows can carry up to 512 KiB of
log_output, so the HTTP path needs hundreds of small paginated requests and
aborts entirely on the first 5xx; the direct-DB path does the whole sync in
one dump/restore pair, typically well under a minute end to end.
The script manages its own port-forwards — killing anything already bound to
the local ports first — and retries pg_dump/pg_restore if a tunnel goes
stale mid-transfer (a kubectl port-forward can keep its local listener up
while silently dropping all traffic). It also checks kubectl auth whoami on
both contexts before doing anything, printing the OpenShift console login
URL if a token has expired.
Prerequisites: kubectl, pg_dump, pg_restore, and pg_isready on PATH,
and an active oc login to both the prod and preprod OpenShift clusters.
# Check auth, start tunnels, and preview the planned pg_dump/pg_restore
# commands; no data is transferred
scripts/copy_prod_to_preprod_db --dry-run
# Run the sync (destructive on preprod; --yes is required)
scripts/copy_prod_to_preprod_db --yes
# Sync a subset of tables only
scripts/copy_prod_to_preprod_db --yes --table cve_analysis_log
A stalled tunnel is detected by watching for no progress (growing dump file
size on export, pg_isready on restore) for --stall-timeout seconds
(default 300s — preprod’s Postgres pod runs under a 500m CPU limit and has
been observed taking 150s+ to respond during a checkpoint, so the default
gives 2x margin above that; a shorter timeout causes false-positive “dead
tunnel” restarts). On a stall, the
affected tunnel is restarted and the step retried up to --max-attempts
times (default 3); pg_restore --clean --if-exists makes every retry
idempotent. A timing summary (per phase and total) prints at the end.
Collecting rpm_first_published with collect_rpm_first_published
scripts/collect_rpm_first_published sweeps the Hummingbird Pulp repo for all
packages, finds the earliest SRPM upload timestamp for each, and writes the
results to the package_lifecycle table via POST /api/cve-export - no direct
DB access.
# Sweep all packages from the rpms repo and write to prod
scripts/collect_rpm_first_published --prod --rpms-repo /path/to/rpms
# Only process specific packages
scripts/collect_rpm_first_published --prod --rpms-repo /path/to/rpms --package curl wget
# Dry run — log what would be written without posting
scripts/collect_rpm_first_published --prod --rpms-repo /path/to/rpms --dry-run
Re-runs are safe: the dashboard upsert only updates occurred_at if the
incoming timestamp is earlier than the stored one.
Analyzing CVE Response Delays with analyze_rpm_first_published
scripts/analyze_rpm_first_published inspects the relation between CVE publication,
package onboarding (rpm_first_published), and HUM ticket creation:
# Analyze all tickets on prod
scripts/analyze_rpm_first_published --prod --token $CVE_REPORT_TOKEN
# Filter to new packages onboarded after CVE publication
scripts/analyze_rpm_first_published --prod --filter new-pkg
# Filter to pre-existing packages with at least 30 days actionable delay
scripts/analyze_rpm_first_published --prod --filter pre-existing --min-days 30 --sort actionable
CLI options:
--filter {all,new-pkg,pre-existing}: filter by onboarding timing relative to CVE.--min-days N: minimum delay threshold in days (default: 0).--sort {cve_to_hum,actionable,cve_to_rpm}: sort column (default:actionable).--limit N: max rows to display (default: 50).
Configuration
| Option | Environment Variable | Description |
|---|---|---|
--jira-token |
JIRA_TOKEN |
Jira API or Bearer token |
--jira-url |
JIRA_URL |
Jira base URL (default: https://redhat.atlassian.net) |
--jira-user |
Username for Basic auth | |
--output, -o |
Stdout format: human only (json / json-pretty removed in HUM-5800) |
|
--show-since |
Filter by creation period (e.g. 2 weeks, 3 hours) |
|
--resolve |
Transition Jira tickets and apply labels | |
--skip-assignee-check |
Skip assignee validation (for testing) | |
--include-closed |
Include Closed tickets in analysis; warn on stale closures | |
--exclude |
Comma-separated ticket keys to skip (e.g. HUM-555,HUM-552) |
|
--skip-no-cve |
Omit issues with no CVE ID (CVE ID field or Summary); no postpone comment/label | |
--max-results |
Max number of issues to fetch (default: 2000) | |
--max-workers |
CVE_ANALYSIS_MAX_WORKERS |
Worker threads for per-ticket analysis and (with --resolve) decide mutations (default: 8) |
--github-token |
GITHUB_TOKEN |
GitHub API token for upstream PR search |
--gitlab-token |
GITLAB_TOKEN |
GitLab API token for gitlab.com MR search |
--nvd-cache-dir |
NVD_CACHE_DIR |
Directory for caching NVD data feed files; avoids re-downloading unchanged feeds |
--rpm-scan-cache-s3-bucket |
RPM_SCAN_CACHE_S3_BUCKET |
S3 bucket for durable RPM provides and Syft artifact JSON; disabled if unset |
--rpm-scan-cache-s3-region |
RPM_SCAN_CACHE_S3_REGION |
Region for the RPM scan cache bucket; required when the bucket is set |
--rpm-scan-cache-s3-access-key-id |
RPM_SCAN_CACHE_S3_ACCESS_KEY_ID |
Access key for the RPM scan cache bucket; required when the bucket is set |
--rpm-scan-cache-s3-secret-access-key |
RPM_SCAN_CACHE_S3_SECRET_ACCESS_KEY |
Secret key for the RPM scan cache bucket; required when the bucket is set |
--rpm-scan-cache-max-bytes |
RPM_SCAN_CACHE_MAX_BYTES |
Soft LRU size budget in bytes for the RPM scan S3 cache (default 1.5 GiB) |
--cee-gitlab-token |
CEE_GITLAB_TOKEN |
CEE GitLab token for advisory repo operations |
--advisories-project |
ADVISORIES_REPO |
Advisories repo URL (default: releng/advisories) |
--advisories-fork |
ADVISORIES_FORK |
Bot’s fork URL for advisory MR creation |
--slack-webhook-url |
SLACK_WEBHOOK_URL |
Slack webhook URL for failure and image-tracker alerts; empty or unset disables Slack |
--no-merge-request |
Skip advisory MR creation during --resolve |
|
--keep-advisory-repo |
Do not delete the cloned advisory repo after the run (useful for debugging) | |
--test-advisory |
Create advisory MR then immediately close it (for testing) | |
--package-map |
Path to CSV override; by default uses rpms repo metadata | |
--rpms-repo |
Path to RPMs git repo for fixed build detection | |
--cve-repo |
Path to local cvelistV5 git repo for CVE record lookups | |
--time-between-runs |
Re-run every N minutes; 0 = single run (default) | |
--feature-flag-project-id |
GitLab project ID for feature flag lookup (default: 73447720, rpms project) |
|
--feature-flag-name |
Feature flag name to check each cycle (default: cve_analysis_enabled; empty = disabled) |
|
--board-id |
Jira Agile board ID for sprint lookup (default: 1489) |
|
--sprint-prefix |
Sprint name prefix to identify Hummingbird sprints (default: Hum S) |
|
--handoff-file |
CVE_ANALYSIS_HANDOFF_FILE |
Write analysis→collector handoff JSON (counters, log, fix times, human_text) |
SENTRY_DSN |
Optional Sentry DSN for error tracking |
Managed Labels
The tool manages the following labels on Jira tickets. These are automatically applied, upgraded, and cleaned up:
| Label | Meaning |
|---|---|
upstream-fix-available |
A fix exists (forge PR/commit, NVD/CVE link, or Fedora Bodhi/DistGit) |
upstream-fix-in-progress |
A fix is in progress (open forge PR, or testing/pending Bodhi update) |
fedora-bz-filed |
A Fedora Bugzilla has been filed for this CVE |
cve-needs-attention |
Analysis needs human attention (see below) |
advisory-mr-failed |
Advisory MR has unresolvable rebase conflicts |
cve-next-release |
Fix will arrive via the next upstream release (set manually; see below) |
awaiting-vex |
Closed ticket awaiting VEX agreement with Jira resolution (HUM-5843 work queue) |
awaiting-cve-published |
Ticket (open or closed) awaiting a cve_published date from cvelistV5 |
awaiting-image-rebuild |
Closed Done-Errata ticket awaiting its image rebuild on Quay (HUM-9975) |
Legacy fedora-fix-available / fedora-fix-in-progress labels are no
longer applied; the bot removes them on subsequent runs (Bodhi/DistGit
evidence now uses upstream-fix-*).
Labels are upgraded automatically (e.g., upstream-fix-in-progress is
replaced by upstream-fix-available when a fix is merged). Stale labels
are removed when tickets are closed, except fedora-bz-filed which is
preserved as an audit record and awaiting-vex which is managed by the
VEX reconcile pass. The cve-needs-attention label is removed
automatically when the warning condition no longer applies.
awaiting-vex (HUM-5843)
When a ticket is closed as Done-Errata or Not a Bug in production,
analysis adds awaiting-vex. Each --resolve cycle then:
- Queries Closed tickets labeled
awaiting-vex, plus Closed Done-Errata / Not a Bug tickets resolved in the last 7 days (so human/agent closes that skipped the close-time enqueue still enter the queue) - Fetches Hummingbird status from the Red Hat CSAF VEX feed for every CVE ID in the ticket summary, scoped to that ticket’s package (not a CVE-wide worst-case across other Hummingbird products)
- Compares to the Jira resolution for the
awaiting-vexwork queue (Done-Errata→fixed,Not a Bug→known_not_affected,package_not_listedwhen CSAF omits the package, orfixed– CSAF’s known_not_affected/fixed split tracks dependency-version presence, not code-path reachability, so a Not a Bug verdict is compatible with either, e.g. a downstream package swept into an unrelated dependency’s rebuild-and-fix); match requires all CVE IDs to agree - On match: records
vex_status+vex_resolved(scan time) in the collector handoff and removes the label if present - On pending/mismatch: adds
awaiting-vexif missing, otherwise keeps it, and still emits currentvex_status. Catch-up closes with no CVE ID in the summary are not labeled: MATCH is impossible, and adding the label would re-select the ticket forever.
The Closed tab computes MATCH from stored vex_status plus Jira resolution;
it does not use stored vex_match_state as source of truth. Dashboard
reads merged event metadata (later rows overlay earlier ones) so a ticket
closed after cve_published does not keep leftover New / In Progress
analysis text as Resolution (HUM-6091).
The collector also fetches any vex_updates[] keys that the watermark JQL
missed, so a same-cycle catch-up match still lands on the Closed tab.
Won’t Do closures do not enqueue awaiting-vex. Open-ticket VEX
mismatches remain a vex-checker reconcile concern.
awaiting-image-rebuild (HUM-9975)
A ticket closes Done-Errata the moment the RPM-level fix ships – resolution
is a version comparison with no dependency on whether the Hummingbird image
has rebuilt with that fix yet, and image rebuilds routinely lag RPM publish
by hours. collect_cve_dashboard’s watermark JQL only re-examines a Closed
ticket when it was updated within the current watermark window or it carries
an awaiting-* workflow label, so without a dedicated label a ticket that
closes before its image finishes rebuilding falls out of the watermark
window on the very next run and image_rebuilt_on_quay is never captured.
_sync_awaiting_image_rebuild_labels (collect_cve_dashboard.py) adds
awaiting-image-rebuild to a Closed Done-Errata ticket once it has an
hb_rpm_fix but no hb_image_fix yet, so the JQL keeps re-selecting it
every run; the label is removed once hb_image_fix resolves or the ticket
is no longer Done-Errata. This mirrors the awaiting-vex /
awaiting-cve-published pattern.
cve-next-release
The cve-next-release label is used for CVEs where no immediate action is
possible and the fix will arrive via the next upstream release. It is set
manually. Common scenarios include:
- Vendored dependencies: A fix exists in a vendored package (e.g.
wsinsidedotnet) but cannot be consumed until the parent package updates its vendored copy. - No backport path: The fix cannot be backported to the current release and must wait for a future upstream version.
- Upstream fix pending: A fix is expected upstream but has not landed yet, and no interim mitigation is available.
Package metadata fix_status: 0 (EOL) does not use this label; those
tickets are closed as Won’t Do instead.
When this label is present on a ticket:
- The automation skips applying
upstream-fix-availableandupstream-fix-in-progresslabels, since they would be misleading, and removes them if already present - The SBOM version check continues each analysis cycle, so when a new upstream release containing the fix is consumed, the ticket is closed normally
- Fedora labels and
cve-needs-attentionare still managed normally - The label is not removed automatically; it must be removed manually when no longer applicable
cve-next-release tickets are not closed just because Hummingbird shipped a
new build. They close when analysis concludes the CVE is no longer affected;
for vendored dependencies, that means the SBOM shows the fixed upstream
version, not merely a new parent NVR.
cve-needs-attention conditions
The cve-needs-attention label is applied when any of these conditions
are detected:
- Tracker filed against an image: the tracker component is a
Hummingbird image (
hi/<name>) instead of an RPM; a Slack alert is also sent - Product mismatch: CVE vendor/product does not match the Hummingbird package (e.g. node-tar CVE filed against GNU tar) after an SBOM-first check finds no vendored dependency hit
- Package missing from rpms but present in SBOM/binaries: package directory is absent from the rpms repo, yet the CVE product appears in the package SBOM and binary confirmation is present or unknown
- Multiple products: CVE lists multiple distinct products with different
versioning schemes and no
cve_productoverride is configured (e.g. NGINX Open Source + NGINX Plus) - Git-only version data: CVE version data uses commit hashes instead of
numeric versions (e.g. libsodium with
lessThan: ad3004ec...) - CNA data error: A git commit hash is used where a version number is
expected without setting
versionType: "git", including hashes embedded in operator syntax (e.g.,"version": "< 6374ae0bcdfe..."). The warning includes a link to the CVE 5.0 source control versions spec - Malformed version field: The version field contains syntax that could
not be parsed (e.g., compound operator+hash ranges like
>= hash1, < hash2) - CVE record not found: the CVE record file was not found in the cvelistV5 repo
- No repo version: the Hummingbird package was not found in the Pulp repository (package name mismatch or missing SRPM)
- Stale closure: ticket is Closed in Jira but current analysis recommends a different resolution
Human override workflow
When a human wants to take over a ticket from the bot:
-
Assign the ticket to yourself – the bot skips all automation on tickets not assigned to the bot account. Reassign to the bot to re-enable automation.
-
Set “Fixed in Build” – if you know the fix is in a specific SRPM, set the “Fixed in Build” field to the SRPM name (e.g.
libarchive-3.8.7-1.hum1.src.rpm). The bot will use this to create the advisory MR and close the ticket, bypassing its own analysis.
Migration note: The cve-analysis-okay label is deprecated and no
longer suppresses automation. Existing tickets with this label will be
re-processed by the bot on its next run. To keep the bot from touching a
specific ticket, reassign it to yourself before the next run. The
cve-analysis-okay label can then be removed manually.
Output
Human-Readable
Project: HUM Component: Security
HUM-796 CVE-2026-2673 openssl: buffer overflow [hummingbird-1]
Open since: 7 days (Mar 24 2026)
Labels resolution: upstream-fix-available
OpenSSL / OpenSSL:
Affected versions: 3.5.0 < 3.5.6
Fixed in: 3.5.6
Hummingbird repo (latest): 3.5.5 / openssl-3.5.5-1.hum1.src.rpm
Repo: https://github.com/openssl/openssl
CVE-2026-2673: upstream-fix-available (3 PRs)
[CLOSED, 3 commits] Fix group tuple handling in DEFAULT expansion (3.5)
https://github.com/openssl/openssl/pull/30110
Fedora update: FEDORA-2026-abc123 openssl-3.5.6-1.fc44 (stable, security)
https://bodhi.fedoraproject.org/updates/FEDORA-2026-abc123
Jira current: In Progress / (none)
Jira resolution: In Progress / affected (repo 3.5.5 is in affected range 3.5.0 < 3.5.6)
JSON
Each issue includes: key, summary, labels, open_since, cve_ids,
cves (with version and resolution data), jira_current (status/resolution),
computed_resolution, and upstream (with PR/MR search results, Bodhi
updates, and Fedora version data).
Viewing Logs
The CVE analysis tool runs as a pod in the hummingbird--internal
namespace on mpp-prod. See the OpenShift MP+ internal docs
page for general log access instructions.
For CVE analysis specifically:
- Live: Open the mpp-prod pods list in the OpenShift
console, filter for
cve-analysis, and open the Logs tab. - Grafana/Loki: Open the Hummingbird Grafana logs dashboard, select
cluster
mpp-prodand namespacehummingbird--internal, then filter forhummingbird-cve-analysispods.
Library layout
Reusable library code lives under hummingbird_cve_analysis/lib/. The CLI
entry point remains cve_analysis.py at the package root.
| Module | Responsibility |
|---|---|
lib/github_client.py |
GitHub PR and release API reads |
lib/gitlab_client.py |
gitlab.com merge request, release, and feature-flag reads |
lib/fedora.py |
Fedora Bodhi, DistGit spec parsing, and Bugzilla helpers |
lib/upstream.py |
Upstream forge search, cgit commits, and fix-status analysis |
lib/net.py |
Shared network error tuple used by library HTTP callers |
lib/nvd.py |
NVD JSON 2.0 feed cache, download, and reference parsing |
lib/cvelist.py |
Local cvelistV5 repository access and CVE 5.0 affected parsing |
lib/jira_client.py |
Jira REST read/write primitives and ADF comment helpers |
lib/versions.py |
Pure version comparison and range helpers |
lib/analysis.py |
CVE decision pipeline, product matching, and lifecycle analysis |
lib/formatting.py |
Human and Jira comment/output formatting |
lib/resolve.py |
Jira/advisory mutation helpers (labels, close, attach SBOM) |
lib/ticket.py |
Shared one-ticket analyze/resolve API |
lib/advisory_handler.py |
CEE advisories repo clone/edit/MR helpers |
lib/osidb_client.py |
OSIDB subpackage PURL lookups and flaw/affect created timestamps |
lib/pulp.py |
Hummingbird Pulp repo RPM/SRPM listings and repodata lookups |
lib/catalog.py |
Container catalog API for image publish times |
lib/rpms_repo.py |
Local rpms git repo, package map loading, fixed-build detection |
lib/rpm_scan_cache.py |
S3 LRU cache of RPM provides and Syft artifact JSON |
lib/sbom.py |
SBOM fetch, vendored dependency lookup, binary RPM confirmation |
lib/slack.py |
Slack webhook helper |
lib/version_transforms.py |
Named version-transform helpers |
cve_analysis.py is the CLI entry point only (argument parsing, JQL building,
signal handling, and the main orchestration loop). Library code lives under
lib/; import and patch those modules directly. New cron jobs, webhooks, and
other integrations should prefer ticket.process_ticket for analyzing (and
optionally resolving) a single Jira issue:
from hummingbird_cve_analysis.lib import ticket
result = ticket.process_ticket(
issue,
pkg_map,
catalog_source_map,
github_token,
gitlab_token,
base_url=base_url,
token=token,
resolve=False,
)
Development
See the main README for development workflows.
make hummingbird-cve-analysis/setup # Install dependencies
make check # Lint code (ruff)
make test # Run unit tests
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.
4.5.14 - Dashboard MR Linker
An AWS Lambda function that posts internal notes on newly opened GitLab merge requests linking to the Hummingbird dashboard status page. This provides an easy way for developers to check Konflux build status directly from their MRs.
Features
- Internal Notes: Posts an internal note with dashboard link when MRs are opened (visible to project members only)
- Configurable Projects: Only monitors specified projects via space-delimited list
- SNS Integration: Subscribes to GitLab events via SNS with filter policy
Architecture
The Lambda subscribes to the existing SNS topic (from gitlab-event-forwarder)
with a filter for merge_request events:
GitLab Webhook → gitlab-event-forwarder → SNS Topic → dashboard-mr-linker → GitLab API
When a new MR is opened on a configured repository, the Lambda posts an internal note:
:robot: **Hummingbird Status**
View Konflux build status for this MR: [org/group/repo!123](https://dashboard.example.com/mr/org/group/repo/123)
Prerequisites
- AWS CLI configured with appropriate credentials (IAM permissions for Lambda, SNS, CloudFormation, CloudWatch Logs)
- Podman or Docker (for containerized SAM build/deploy)
- Python 3.11 or later (for development)
- GitLab API token with
apiscope (Reporter role or higher on monitored projects)
Deployment
Build and deploy using containerized AWS SAM CLI:
cd dashboard-mr-linker
make build # Build Lambda package
make deploy # First deployment (interactive/guided)
make redeploy # Subsequent deployments (non-interactive)
Parameters
| Parameter | Description | Default |
|---|---|---|
ResourcePrefix |
Prefix for resources | myapp-prod |
SnsTopicArn |
SNS topic ARN | (required) |
GitLabToken |
GitLab API token (api scope, Reporter+) |
(required) |
GitLabUrl |
GitLab instance URL | https://gitlab.com |
DashboardBaseUrl |
Base URL of the dashboard | (required) |
GitLabProjects |
Space-delimited list of GitLab project paths | (required) |
SentryDsn |
Optional Sentry DSN | `` |
Resource naming: Lambda follows {ResourcePrefix}-lambda-linker pattern.
Prerequisites: Requires an existing SNS topic. Deploy
hummingbird-events-topic first to create the topic, then use its
ARN for the SnsTopicArn parameter.
Usage
Configure the GITLAB_PROJECTS parameter with space-delimited project paths:
org/group/containers org/group/rpms org/other/project
Only MRs opened on these projects will receive dashboard link notes.
Development
See the main README for development workflows.
make setup # Install dependencies
make check # Lint code (ruff)
make fmt # Format code
make test # Run unit tests
make coverage # Run tests with coverage
Configuration
Lambda function receives configuration via environment variables (automatically set by CloudFormation):
| Variable | Description |
|---|---|
GITLAB_TOKEN |
GitLab API token (api scope, Reporter+) |
GITLAB_URL |
GitLab instance URL |
DASHBOARD_BASE_URL |
Base URL of the dashboard |
GITLAB_PROJECTS |
Space-delimited list of GitLab project paths |
SENTRY_DSN |
Optional Sentry DSN |
AWS_REGION |
AWS region (auto-set) |
Security & Limitations
Security:
- GitLab API token should have minimal required scopes (
api) - Token stored as CloudFormation parameter with
NoEcho: true - SNS subscription uses filter policy to only receive relevant events
- Notes are posted as internal (visible to project members only)
- CloudWatch logs capture all note posts (7-day retention)
- Sentry integration for error tracking
Limitations:
- Lambda timeout: 30 seconds
- Lambda memory: 256 MB
- Only processes
action: openevents (not updates or other actions) - Skips MRs with the
managed-by::workqueuelabel (created by workqueue-service). The workqueue-service manages its own dashboard integration for these MRs, so the Lambda backs off to avoid duplicate notes
Integration
The workqueue-service creates the dashboard note itself for MRs it
manages. The managed-by::workqueue scoped label, applied by the
create_mr executor, causes this Lambda to skip those MRs. The
workqueue-service’s ensure_dashboard_note executor then posts the
note and stores the dashboard_link dimension with the note ID.
For non-managed MRs (those not created by the workqueue-service), the Lambda remains the sole creator of the dashboard note.
Both systems use the same marker in the note body:
:robot: **Hummingbird Status**. The Lambda’s COMMENT_MARKER and
the workqueue-service’s DASHBOARD_LINK_MARKER constant must stay in
sync — changing one without the other will break identification and
may cause issues with future reconciliation or status rendering.
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.
4.5.15 - GitLab Event Forwarder
An AWS Lambda function that receives GitLab webhook events and forwards them to an SNS topic with structured metadata for filtering. Validates webhook signatures and extracts minimal metadata (source, event type, project/group paths) as SNS message attributes, enabling downstream subscribers to filter events precisely.
The original GitLab JSON payload is forwarded unchanged as the SNS message body.
Features
- Webhook Signature Validation: Verifies authenticity using HMAC-SHA256
signing tokens (Standard Webhooks), with legacy
X-Gitlab-Tokenfallback - Structured Metadata: Extracts event type, project/group paths as SNS message attributes
- SNS Subscription Filtering: Enables precise event routing to subscribers
- Custom Domain: Optional custom domain with automatic TLS certificate management via ACM and Route53
Architecture
Single Lambda function handles the webhook processing:
- Handler - API Gateway endpoint (
/webhook) that validates GitLab webhook signatures, extracts metadata, and publishes to SNS
Prerequisites
- AWS CLI configured with appropriate credentials (IAM permissions for Lambda, API Gateway, SNS, CloudFormation, CloudWatch Logs, and optionally Route53/ACM for custom domain)
- Podman or Docker (for containerized SAM build/deploy)
- Python 3.11 or later (for development)
Deployment
Build and deploy using containerized AWS SAM CLI:
cd gitlab-event-forwarder
make build # Build Lambda package
make deploy # First deployment (interactive/guided)
make redeploy # Subsequent deployments (non-interactive)
Deployment outputs: ApiEndpoint - API Gateway endpoint URL (<api-endpoint>)
Custom Domain
Optional custom domain with automatic TLS certificate management (ACM +
Route53). Requires a Route53 hosted zone. Deploy with CustomDomainName and
HostedZoneId parameters - CloudFormation handles certificate creation, DNS
validation, and configuration. Certificate validation takes 5-30 minutes; allow
up to 1 hour for DNS propagation.
Parameters
| Parameter | Description | Default |
|---|---|---|
ResourcePrefix |
Prefix for resources | myapp-prod |
SnsTopicArn |
SNS topic ARN | (required) |
GitLabWebhookTokens |
GitLab webhook tokens (JSON) | (required) |
GitLabSigningTokens |
Signing tokens (JSON array) | [] |
SentryDsn |
Optional Sentry DSN | `` |
CustomDomainName |
Custom domain name | `` |
HostedZoneId |
Route53 hosted zone | `` |
Resource naming: Lambda and API resources follow
{ResourcePrefix}-{type}-{name} pattern (e.g., myapp-prod-lambda-handler,
myapp-prod-api).
Prerequisites: Requires an existing SNS topic. Deploy
hummingbird-events-topic first to create the topic, then use its
ARN for the SnsTopicArn parameter.
Usage
Configure GitLab projects or groups to send webhooks to <api-endpoint>:
webhooks:
<api-endpoint>:
token: <secret-token>
signing_token: <whsec-signing-token>
push_events: true
merge_requests_events: true
pipeline_events: true
job_events: true
signing_token (recommended) uses HMAC-SHA256 verification with replay
protection (±5 min tolerance) per the Standard Webhooks spec. token uses
a static X-Gitlab-Token header. When signing tokens are configured on
the forwarder, they take priority and webhook-signature is required on
all requests.
SNS Subscription Filter Examples:
Push events from a specific project:
{
"source": ["gitlab"],
"event_type": ["push"],
"project_path": ["redhat/hummingbird/containers"]
}
All merge request events:
{
"source": ["gitlab"],
"event_type": ["merge_request"]
}
Member events from a group:
{
"source": ["gitlab"],
"event_type": ["member"],
"group_path": ["redhat/hummingbird"]
}
Development
See the main README for development workflows.
make setup # Install dependencies
make check # Lint code (ruff)
make fmt # Format code
make test # Run unit tests
make coverage # Run tests with coverage
Configuration
Lambda function receives configuration via environment variables (automatically set by CloudFormation):
| Variable | Description |
|---|---|
SNS_TOPIC_ARN |
SNS topic ARN |
GITLAB_WEBHOOK_TOKENS |
GitLab webhook tokens (JSON array) |
GITLAB_SIGNING_TOKENS |
Signing tokens (JSON array) |
SENTRY_DSN |
Optional Sentry DSN |
AWS_REGION |
AWS region (auto-set) |
Event Metadata
The Lambda function extracts minimal metadata from GitLab webhook events and adds them as SNS message attributes:
| Attribute | Description | Example |
|---|---|---|
source |
Always "gitlab" |
gitlab |
event_type |
From object_kind |
push, merge_request, pipeline, build |
project_path |
Full project path | redhat/hummingbird/containers |
group_path |
Full group path | redhat/hummingbird |
Security & Limitations
Security:
- HMAC-SHA256 signing token verification via
webhook-signatureheader (Standard Webhooks spec) with ±5 min replay protection - When signing tokens are configured, static
X-Gitlab-Tokenis not accepted - Legacy
X-Gitlab-Tokenvalidation when no signing tokens are configured - Supports multiple active tokens for zero-downtime rotation
- Invalid/missing tokens return HTTP 401
- SNS topic follows least privilege principle
- CloudWatch logs capture all webhook deliveries (7-day retention)
- Sentry integration for error tracking
Limitations:
- Lambda timeout: 30 seconds
- Lambda memory: 256 MB
- Webhook payload size: Up to 6 MB (API Gateway limit)
- SNS limits the compressed and base64-encoded message plus attributes to 256 KiB
Pipeline Hooks use object_kind: pipeline; Job Hooks use object_kind: build.
The forwarder compresses the original JSON for SNS transport. Consumers that
decode gzip+base64 receive the unchanged webhook payload.
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.
4.5.16 - Hummingbird MR Human Tracker
A CLI tool that detects human intervention on automation-opened GitLab merge requests (Renovate, Dependabot, etc.) and links them to a per-sprint Jira epic for tracking.
Features
- Bot MR Detection: Identifies MRs authored by bot accounts across multiple GitLab projects using configurable username patterns
- Human Activity Detection: Finds MRs where humans have commented or pushed commits (via GitLab system notes)
- Per-Sprint Jira Epic: Automatically finds or creates an epic per sprint to track automation fix work (e.g. “Automation MR fixes for sprint Hum S16”)
- Remote Link Tracking: Adds affected MR URLs as remote links on the Jira epic, with deduplication to avoid duplicates on repeated runs
- Merged/Closed MR Scanning:
--sinceflag catches MRs that were merged or closed before the next scheduled run - Dry Run: Preview what would be linked without making any changes
Prerequisites
- Python 3.11 or later
- GitLab API token with read access to target projects (including MR notes)
- Jira API token with permission to create epics and remote links in the HUM project
Installation
pip install -e hummingbird-mr-human-tracker
Usage
mr-human-tracker --jira-user user@redhat.com --dry-run -v
Common invocations
# Preview what would be linked (no changes made)
mr-human-tracker --jira-user user@redhat.com --dry-run -v
# Run for real, including MRs merged in the last 2 days
mr-human-tracker --jira-user user@redhat.com --since 2d
# Scan only specific projects
mr-human-tracker --jira-user user@redhat.com --projects redhat/hummingbird/tools
# Verbose output for debugging
mr-human-tracker --jira-user user@redhat.com --since 1w -v
Output
On completion, the tool prints a summary of linked MRs:
INFO hummingbird_mr_human_tracker.tracker: Active sprint: Hum S16 [June 11 - June 25] (id=68851)
INFO hummingbird_mr_human_tracker.tracker: Using epic HUM-2602
INFO hummingbird_mr_human_tracker.tracker: Linked 2 MR(s) to HUM-2602:
INFO hummingbird_mr_human_tracker.tracker: https://gitlab.com/redhat/hummingbird/rpms/-/merge_requests/2395
INFO hummingbird_mr_human_tracker.tracker: https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/8476
Configuration
All configuration is via CLI arguments and environment variables.
CLI arguments
| Argument | Description | Default |
|---|---|---|
--gitlab-url |
GitLab instance URL | GITLAB_URL env |
--gitlab-token |
GitLab private token | GITLAB_TOKEN env |
--jira-url |
Jira base URL | JIRA_URL env |
--jira-token |
Jira API token | JIRA_TOKEN env |
--jira-user |
Jira email for Basic auth (omit for Bearer) | None |
--projects |
GitLab project paths to scan | tools, rpms, containers |
--board-id |
Jira board ID for sprint lookup | 1489 |
--sprint-prefix |
Sprint name prefix to match | Hum S |
--since |
Also scan merged/closed MRs updated within window | None (open MRs only) |
--dry-run |
Report without making changes | False |
-v, --verbose |
Enable debug logging | False |
Environment variables
| Variable | Description |
|---|---|
GITLAB_URL |
GitLab instance URL |
GITLAB_TOKEN |
GitLab private token |
JIRA_URL |
Jira base URL |
JIRA_TOKEN |
Jira API token |
Duration format for --since
The --since flag accepts a number followed by a unit:
| Unit | Meaning |
|---|---|
h |
Hours |
d |
Days |
w |
Weeks |
Examples: 12h, 2d, 1w
How it works
- Fetch active sprint from the Jira Agile API (board 1489, prefix “Hum S”)
- Find or create a Jira epic named “Automation MR fixes for sprint {sprint_name}” in the HUM project, assigned to the active sprint
- Scan GitLab projects for open MRs authored by bot accounts; if
--sinceis set, also scan recently merged/closed MRs - Detect human activity on each bot MR by checking:
- Non-bot, non-system comments
- System notes indicating a human pushed commits (“added N commit”)
- Reopen events by non-bot users
- Add remote links on the Jira epic for each human-touched MR (skipping URLs already linked)
Bot detection
A username is considered a bot if it matches any of:
- Contains
bot_orbot-(e.g.renovate_bot) - Ends with
_bot(e.g.some_bot) - Ends with
[bot](e.g.renovate[bot],dependabot[bot]) - Starts with
project_<digits>_botorgroup_<digits>_bot(GitLab service accounts)
Development
make setup # Install dependencies
make check # Lint code (ruff)
make fmt # Format code
make test # Run unit tests
make coverage # Run tests with coverage
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.
4.5.17 - MR Auto-Approver
An AWS Lambda function that auto-approves GitLab merge requests based on configurable per-project rules. Replaces CI/CD-based approval jobs and moves approval tokens out of GitLab CI/CD variables.
Features
- Rules-Based Config: YAML config with per-project ordered rules matching on user IDs, username patterns, branch patterns, and Konflux pipeline status
- First-Match-Wins: Rules evaluated in order; first match determines action (approve or deny)
- Konflux Integration: Optional check that Konflux pipelines have posted commit statuses before approving
- Fork Rejection: Unconditionally rejects MRs from forked projects
- User Verification: Matches against the authenticated webhook event user, not forgeable Git commit metadata
- Stateless & Idempotent: No queues or stored state; re-approving is a no-op
Architecture
The Lambda subscribes to the existing SNS topic (from gitlab-event-forwarder)
with a filter for merge_request and pipeline events:
GitLab Webhook → gitlab-event-forwarder → SNS Topic → mr-auto-approver → GitLab API
Merge request events (open, update): evaluate rules using the
authenticated pusher from the webhook payload. If matched and no Konflux check
needed, approve immediately. If Konflux check needed, check statuses now and
approve only if present and not failed.
Pipeline events (success, failed): look up the MR from the pipeline
event via the GitLab API, evaluate rules, and approve if Konflux statuses are
ready. This handles the case where Konflux posts commit statuses after the
initial MR event.
Prerequisites
- AWS CLI configured with appropriate credentials (IAM permissions for Lambda, SNS, CloudFormation, CloudWatch Logs)
- Podman or Docker (for containerized SAM build/deploy)
- Python 3.11 or later (for development)
- GitLab API token(s) with
apiscope and permission to approve MRs on target projects - Target project webhooks must send
merge_requests_eventsandpipeline_eventsto the gitlab-event-forwarder endpoint
Deployment
Build and deploy using containerized AWS SAM CLI:
cd mr-auto-approver
make build # Build Lambda package
make deploy # First deployment (interactive/guided)
make redeploy # Subsequent deployments (non-interactive)
Parameters
| Parameter | Description | Default |
|---|---|---|
ResourcePrefix |
Prefix for resources | myapp-prod |
SnsTopicArn |
SNS topic ARN | (required) |
GitLabUrl |
GitLab instance URL | https://gitlab.com |
ConfigPath |
Path to YAML config (bundled with Lambda) | config.yaml |
ApprovalTokens |
JSON map of env var names to tokens | {} |
SentryDsn |
Optional Sentry DSN | `` |
Resource naming: Lambda follows {ResourcePrefix}-lambda-approver pattern.
Prerequisites: Requires an existing SNS topic. Deploy
hummingbird-events-topic first to create the topic, then use its
ARN for the SnsTopicArn parameter.
Configuration
The Lambda uses a YAML config file bundled at deploy time. The config defines per-project rules:
settings:
gitlab_url: https://gitlab.com
projects:
org/group/repo:
token_env: APPROVAL_GITLAB_TOKEN_REPO
rules:
- branch_regexes: ["renovate/skip/.*"]
action: deny
- user_ids: [12345678]
branch_regexes: ["renovate/.*"]
- user_ids: [87654321]
check_konflux: true
Rule fields
| Field | Type | Default | Description |
|---|---|---|---|
user_ids |
list[int] | [] |
GitLab user IDs (immutable, preferred) |
user_regexes |
list[str] | [] |
Fullmatch regexes for username |
branch_regexes |
list[str] | [] |
Fullmatch regexes for source branch |
check_konflux |
bool | false |
Require Konflux statuses before approval |
action |
str | approve |
approve or deny |
Match logic: AND across field types, OR within a field. Empty fields match
anything. First matching rule wins. user_ids and user_regexes are both user
identity constraints – if either is set, the user must match at least one entry
from across both lists. Prefer user_ids over user_regexes to prevent
username confusion attacks.
Adding a new project
- Create a project-scoped GitLab token (Developer,
apiscope) in Vault - Add the token to
vars.shand thejqcommand that buildsAPPROVAL_TOKENS - Add a project entry to
config.yamlin the infrastructure repo withtoken_envand rules - Deploy the Lambda
- Ensure the project’s GitLab webhook sends events to the gitlab-event-forwarder endpoint
Environment variables
| Variable | Description |
|---|---|
CONFIG_PATH |
Path to YAML config file |
GITLAB_URL |
GitLab instance URL |
APPROVAL_TOKENS |
JSON object mapping env var names to GitLab tokens |
SENTRY_DSN |
Optional Sentry DSN |
Security
- Fork MRs are unconditionally rejected before any rule evaluation
(
source_project_id != target_project_id) - User matching supports immutable
user.id(preferred) in addition touser.usernamefrom the webhook payload, preventing username confusion attacks. Never uses Git commit author/committer metadata - For pipeline events, the Lambda looks up
head_pipeline.user(bothidandusername) via the GitLab API to verify the last pusher - Approval tokens stored as CloudFormation parameters with
NoEcho: true - SNS subscription filter policy limits events to
merge_requestandpipeline - CloudWatch logs capture all approval decisions (7-day retention)
Development
See the main README for development workflows.
make setup # Install dependencies
make check # Lint code (ruff)
make fmt # Format code
make test # Run unit tests
make coverage # Run tests with coverage
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.
4.5.18 - Jira Event Forwarder
An AWS Lambda function that receives Jira Automation webhook events and forwards them to an SNS topic with structured metadata for filtering. Validates Bearer token authentication and restricts access to Atlassian egress IP ranges via an API Gateway resource policy.
The original Jira JSON payload is forwarded unchanged as the SNS message body.
Features
- Bearer Token Authentication: Validates
Authorization: Bearer <token>headers with constant-time comparison, supporting multiple tokens for zero-downtime rotation - Atlassian IP Restriction: API Gateway resource policy restricts access to Atlassian egress IP ranges (IPv4 and IPv6), fetched dynamically at deploy time
- Structured Metadata: Extracts event type, project key, and issue type as SNS message attributes
- SNS Subscription Filtering: Enables precise event routing to subscribers
- Custom Domain: Optional custom domain with automatic TLS certificate management via ACM and Route53
Architecture
Single Lambda function handles the webhook processing:
- Handler - API Gateway endpoint (
/webhook) that validates the Bearer token, extracts metadata from the Jira payload and request headers, and publishes to SNS
Prerequisites
- AWS CLI configured with appropriate credentials (IAM permissions for Lambda, API Gateway, SNS, CloudFormation, CloudWatch Logs, and optionally Route53/ACM for custom domain)
- Podman or Docker (for containerized SAM build/deploy)
- Python 3.11 or later (for development)
Deployment
Build and deploy using containerized AWS SAM CLI:
cd jira-event-forwarder
make build # Build Lambda package
make deploy # First deployment (interactive/guided)
make redeploy # Subsequent deployments (non-interactive)
Deployment outputs: ApiEndpoint - API Gateway endpoint URL
(<api-endpoint>)
Custom Domain
Optional custom domain with automatic TLS certificate management (ACM +
Route53). Requires a Route53 hosted zone. Deploy with CustomDomainName and
HostedZoneId parameters - CloudFormation handles certificate creation, DNS
validation, and configuration. Certificate validation takes 5-30 minutes; allow
up to 1 hour for DNS propagation.
Parameters
| Parameter | Description | Default |
|---|---|---|
ResourcePrefix |
Prefix for resources | myapp-prod |
SnsTopicArn |
SNS topic ARN | (required) |
JiraWebhookTokens |
Jira webhook tokens (JSON array) | [] |
SentryDsn |
Optional Sentry DSN | `` |
AtlassianIpv4Cidrs |
Atlassian egress IPv4 CIDRs (comma-list) | `` |
AtlassianIpv6Cidrs |
Atlassian egress IPv6 CIDRs (comma-list) | `` |
CustomDomainName |
Custom domain name | `` |
HostedZoneId |
Route53 hosted zone | `` |
Resource naming: Lambda and API resources follow
{ResourcePrefix}-{type}-{name} pattern (e.g., myapp-prod-lambda-handler,
myapp-prod-api).
Prerequisites: Requires an existing SNS topic. Deploy
hummingbird-events-topic first to create the topic, then use its
ARN for the SnsTopicArn parameter.
Usage
Jira Automation allows only one trigger per rule, and the “Multiple issue
events” trigger does not expose which event fired. Since each event needs a
distinct X-Jira-Event-Type header, configure 6 separate automation rules
— one per event type:
| Trigger | X-Jira-Event-Type value |
|---|---|
| Created | issue_created |
| Updated | issue_updated |
| Deleted | issue_deleted |
| Assigned | issue_assigned |
| Transitioned | issue_transitioned |
| Moved | issue_moved |
Each rule’s “Send web request” action:
- URL:
<api-endpoint> - Method: POST
- Headers:
Authorization: Bearer <token>— enter the token value and click “Hidden” to mask it in the Jira Automation UIX-Jira-Event-Type: <value from table above>
- Body: “Issue data (Jira format)”
SNS Subscription Filter Examples:
All Jira issue events:
{
"source": ["jira"]
}
Specific Jira event types:
{
"source": ["jira"],
"event_type": ["issue_created", "issue_updated"]
}
Events from a specific project:
{
"source": ["jira"],
"project_key": ["HUM"]
}
Development
See the main README for development workflows.
make setup # Install dependencies
make check # Lint code (ruff)
make fmt # Format code
make test # Run unit tests
make coverage # Run tests with coverage
Configuration
Lambda function receives configuration via environment variables (automatically set by CloudFormation):
| Variable | Description |
|---|---|
SNS_TOPIC_ARN |
SNS topic ARN |
JIRA_WEBHOOK_TOKENS |
Jira webhook tokens (JSON array) |
SENTRY_DSN |
Optional Sentry DSN |
AWS_REGION |
AWS region (auto-set) |
Event Metadata
The Lambda function extracts metadata from Jira webhook events and request headers, adding them as SNS message attributes:
| Attribute | Description | Example |
|---|---|---|
source |
Always "jira" |
jira |
event_type |
From X-Jira-Event-Type header |
issue_created |
project_key |
Jira project key | HUM |
issue_type |
Issue type name | Story, Bug |
content_encoding |
Compression format | gzip+base64 |
The event_type attribute is set from the X-Jira-Event-Type request header.
Each Jira Automation rule hardcodes the appropriate value for its trigger type
in this header (see Usage). If the header is absent, the event_type
attribute is omitted from the SNS message.
Security & Limitations
Security:
- Bearer token validation via
Authorizationheader with constant-time comparison (hmac.compare_digest) - API Gateway resource policy restricting source IPs to Atlassian egress
ranges (IPv4 and IPv6 CIDRs both via
aws:SourceIp), fetched dynamically at deploy time fromip-ranges.atlassian.com - Supports multiple active tokens for zero-downtime rotation
- Missing or invalid tokens return HTTP 401; no configured tokens returns HTTP 500
- SNS topic follows least privilege principle
- CloudWatch logs capture all webhook deliveries (7-day retention)
- Sentry integration for error tracking
Limitations:
- Lambda timeout: 30 seconds
- Lambda memory: 256 MB
- Webhook payload size: Up to 6 MB (API Gateway limit)
- SNS limits the compressed and base64-encoded message plus attributes to 256 KiB
The forwarder compresses the original JSON for SNS transport. Consumers that
decode gzip+base64 receive the unchanged webhook payload.
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.
4.5.19 - Container Catalog
Per-distro serverless catalog API for Hummingbird container images.
Features
- Image Directory - Browse all container images with metadata
- Tag Browser - View tags, digests, architectures per image
- Specifications - Per-architecture OCI config details (env, cmd, user, labels)
- SBOM - Per-architecture package lists from syft scans with a Hermeto build-only merge
- Vulnerabilities - CVE scanning results via Grype
- CVE Metrics - Catalog-wide vulnerability aggregate
- Provenance - Source traceability from SLSA attestations
- Release History - Timeline of past builds with drill-down
- Deprecation status - Stream and release status from OCI labels
- OpenAPI Spec - Machine-readable API documentation at /v1/openapi.json
- Swagger UI - Interactive API explorer at /v1/docs/
- External Images - CVE comparison with external image sources
- External Comparison UI - Side-by-side CVE and SBOM comparison between Hummingbird and external counterparts
Architecture
Rust serverless stack deployed as two isolated per-distro stacks:
- API Lambda - DynamoDB pass-through (~10ms response)
- Sync Lambda - Incremental DynamoDB sync from SNS Release events (~22 registry calls per release), then enqueues one index event per release
- Index Lambda (
index-lambda) - Queue-driven Syft runs per architecture, writes the SBOM items to DynamoDB, uploads native Syft JSON to S3 for scanner consumption (same function shape as the external index — the future unified function) - Scan Lambda (
scan-lambda) - SQS-triggered CVE scanning via Grype from native Syft JSON in S3, per-digest processing withfirst_seentracking and partial batch failure reporting - Enqueue Lambda (
enqueue-lambda) - Hourly EventBridge-triggered fan-out that checks Grype DB for updates and enqueues non-superseded digests to SQS - Metrics Lambda (
metrics-lambda) - DynamoDB Stream-triggered vulnerability aggregate - External Release Lambda (
external-release-lambda) - Scheduled trigger feeding the external sync queue - External Sync Lambda (
external-sync-lambda) - Registry tag resolution and DynamoDB metadata writes for external images - External Index Lambda (
external-index-lambda) - Syft SBOM generation for external images, uploads to S3 for Grype scanning - DynamoDB - Pre-computed JSON items (single-table PK/SK design, Streams: KEYS_ONLY)
- S3 ScanDataBucket - Native Syft JSON storage for scanner data (gzipped,
keyed by
grype/{image}/{digest_hex}.json.gzfor Hummingbird andgrype-external/{registry}/{digest_hex}.json.gzfor external images; SPDX SBOMs are stored atsbom/{image}/{digest_hex}.json.gzfor Hummingbird andsbom-external/{registry}/{digest_hex}.json.gzfor external images — thesbom*prefixes match no scan notification filter) - SQS IndexQueue - Decouples the index (syft) path from sync: the Sync Lambda enqueues one tag-release event per release after its DynamoDB writes; failed index records are retried individually via SQS batch-item-failure semantics, redriving to the IndexDLQ after 3 attempts
- SQS ScanQueue - Central queue for scan tasks (fed by both
grype/andgrype-external/S3 event notifications and Enqueue Lambda) - CloudFront - CDN with per-endpoint cache TTLs
- catalog sync - Full DynamoDB population from GitLab + Quay.io OCI v2 registry
- catalog scan - CLI CVE scanning via Grype; reads native Syft JSON from
S3 (
--bucketrequired); historic releases read the same S3 objects, and digests whose object is missing are skipped with a warning - catalog index - CLI Syft JSON generation for backfilling S3
- Catalog SPA - Lit 3 web app served from S3 via CloudFront
Scanner Architecture
CVE vulnerability data must match direct grype <image> scans exactly for all
images. Synthetic Syft JSON (built from stored SBOM packages) cannot reliably
reproduce the native output because Grype relies on metadata fields, artifact
relationships, and deduplication logic that are lost in the SPDX-to-API
roundtrip. Additionally, raw SPDX from the build system contains package bloat
(e.g. Go sub-modules, empty-version entries) that native Syft filters out
when scanning a compiled binary directly.
To guarantee exact-match results, the scanner chain stores native Syft JSON
in S3 rather than DynamoDB: a single image’s Syft JSON is typically 1-10 MB
(gzipped to 100 KB - 1 MB), which exceeds DynamoDB’s 400 KB item limit. S3
has no per-object size constraint, avoids provisioned throughput costs for
large blobs, and supports event notifications for triggering downstream
scanners. The Index Lambda runs syft <image> on each new
release and uploads the full output to
s3://{bucket}/grype/{image}/{hex}.json.gz. The S3 upload fires an event
notification to SQS, which triggers the Scan Lambda to read the native JSON,
run Grype, and write per-canonical vulnerability data to DynamoDB.
Each syft run extracts image layers into its own temp directory, passed to
the syft child via TMPDIR and removed when the run finishes; concurrent
runs never share temp data. Dirs left behind by killed runs are swept before
each new run (age-gated to never touch a possibly-active run).
When running inside a Lambda, the syft child process inherits GOMEMLIMIT
derived as 90% of AWS_LAMBDA_FUNCTION_MEMORY_SIZE (injected by Lambda, in
MB; e.g. 3072 MB → 2764 MiB). This makes the cgroup limit and the GC soft
limit share a single source of truth — a memory bump auto-rescales both with
no code change. Outside Lambda (CLI catalog index backfill, local runs) the
variable is absent and Go’s defaults apply. Without GOMEMLIMIT, Go’s default
GOGC=100 lets syft’s heap grow to 2× its live size (~2.8 GB for large
multi-layer images → ~5.7 GB peak), exceeding the Lambda cgroup and causing
OOM-kills. If a future image’s live heap grows beyond the limit, the scan
degrades to slower GC thrash rather than an instant kill.
The scanner chain is decoupled from the catalog data chain by a queue: the SNS Release event triggers the Sync Lambda (catalog data to DynamoDB), which then enqueues one index event per release; the Index Lambda consumes those events (Syft JSON to S3). The event carries everything the index needs, so the slow syft path never blocks the sync invocation and a failed record is retried individually by the queue.
Grype runs with --by-cve: some ecosystems (e.g. Go modules) are otherwise
reported using their native GHSA or Go vulnerability DB ID rather than a
CVE. --by-cve reports the CVE ID instead wherever grype’s own local
vulnerability database has that cross-reference, with no added network
call. An ID with no known CVE anywhere in that database is returned as-is.
Vulnerability components include the source RPM filename from the syft scan when available. Consumers can group affected binary RPMs by this value.
CVE Data Flow
graph TD
subgraph Hummingbird
Attestation[(Registry .sbom attestations)] -->|"fetch + merge"| Index
Konflux([Konflux Release])
Konflux -->|SNS| Sync[SyncFunction]
Sync -->|"index events via SQS"| Index[IndexFunction]
Sync -->|update| Tags[(Tags)]
Index -->|upload| S3Syft[(S3 grype/)]
Index -->|write| SbomItems[(SBOM items)]
end
subgraph External Images
ExtSchedule([Schedule 6h]) -->|invoke| ExtRelease[ExternalReleaseFunction]
ExtRelease -->|"per registry via SQS"| ExtSync[ExternalSyncFunction]
ExtSync -->|"per-tag release events via SQS"| ExtIndex[ExternalIndexFunction]
ExtIndex -->|upload| S3Ext[(S3 grype-external/)]
ExtIndex -->|write| ExtSbomItems[(External SBOM items)]
ExtSync -->|update| ExtTags[(External Tags)]
end
Tags -->|read| Scan[ScanFunction]
ExtTags -->|read| Scan
S3Syft -->|"S3 event via SQS"| Scan
S3Ext -->|"S3 event via SQS"| Scan
Hourly([Schedule 1h]) -->|invoke| Enqueue[EnqueueFunction]
Enqueue --> Gate[check Grype DB]
Gate -->|"fan out via SQS"| Scan
Scan -->|update| Vulns[(Vulnerabilities)]
Vulns -->|"stream via ESM"| Metrics[MetricsFunction]
Tags -->|"stream via ESM"| Metrics
Metrics -->|"write aggregate"| CatalogVulns[(Catalog Vulnerabilities)]
classDef lambda fill:#d4e6f1,stroke:#2980b9
classDef dynamo fill:#fdebd0,stroke:#e67e22
classDef trigger fill:#d5f5e3,stroke:#27ae60
classDef gate fill:#e5e7e9,stroke:#7f8c8d
class Sync,Scan,Enqueue,Metrics,Index,ExtRelease,ExtSync,ExtIndex lambda
class Vulns,Tags,ExtTags,CatalogVulns,S3Syft,S3Ext,S3Spdx,S3ExtSpdx,SbomItems,ExtSbomItems,Attestation dynamo
class Konflux,Hourly,ExtSchedule trigger
class Gate gate
The sbom/ and sbom-external/ records are the recovery inputs for the
already-indexed path: each index function persists its primary-arch SPDX
before the grype* blob, which is deliberately the last write — it is the
dedup key and the scan trigger, so it never arms before its recovery input
exists. Both planes share the same ordering guarantee: the sync function’s
DynamoDB writes complete before its index events are enqueued — the queue
is the ordering barrier. The attestation edge is internal-only: the
Hummingbird index fetches the per-arch Hermeto attestations from the
registry and merges build-only entries into each arch’s SBOM; external
images have no attestation input.
Prerequisites
- Rust 1.75+ (for building backend)
- Node.js 22+ (for building frontend)
- AWS credentials (for DynamoDB access and deployment)
- SAM CLI (for deployment)
API Endpoints
Both namespaces share the same response types and URL structure. The
Hummingbird prefix is /v1/images/{name}, the external prefix is
/v1/external-images/{name} where {name} is a percent-encoded registry
reference (e.g. docker.io%2Flibrary%2Fnginx). DynamoDB partition keys are
IMAGE#{name} and EXTERNAL#{registry} respectively.
| Suffix | Response type | HB | Ext |
|---|---|---|---|
| (directory) | DirectoryResponse |
✅ | ✅ |
| (config) | ExternalImagesConfig |
✅ | |
/{…} |
ImageResponse |
✅ | ✅ |
/{…}/tags |
TagsResponse |
✅ | ✅ |
/{…}/details/{canonical} |
DetailsResponse |
✅ | ✅ |
/{…}/sbom/{canonical} |
SbomResponse |
✅ | ✅ |
/{…}/vulnerabilities/{canonical} |
VulnerabilitiesResponse |
✅ | ✅ |
/{…}/history/{stream}/{variant} |
HistoryResponse |
✅ | ✅ |
/{…}/releases/details/{digest} |
DetailsResponse |
✅ | ✅ |
/{…}/releases/sbom/{digest} |
SbomResponse |
✅ | ✅ |
/{…}/releases/vulnerabilities/{digest} |
VulnerabilitiesResponse |
✅ | ✅ |
/v1/vulnerabilities |
CatalogVulnerabilitiesResponse |
✅ |
GET /v1/openapi.json and GET /v1/docs/ serve the OpenAPI 3.1 spec and
Swagger UI for all endpoints.
Response types
DirectoryResponse
Catalog-level image listing with aggregate statistics. Used for both
Hummingbird (CATALOG/DIRECTORY) and external (CATALOG/EXTERNAL_DIRECTORY)
image directories.
| Field | Type | Description |
|---|---|---|
images |
ImageSummary[] |
All images in the catalog |
image_count |
u32 |
Total number of images |
variant_count |
u32 |
Total unique variant names across all images |
stream_variant_count |
u32 |
Total published (stream, variant) pairs |
stream_count |
u32 |
Total version streams across all images |
tag_count |
u32 |
Total tags across all images |
release_count |
u32 |
Total releases across all images |
ImageSummary
Per-image summary used in both directory listings and as the base of
ImageResponse.
| Field | Type | Hummingbird | External |
|---|---|---|---|
name |
String |
Image name (e.g. python) |
Registry reference (e.g. docker.io/library/nginx) |
summary |
String |
From summary OCI label |
Omitted — no OCI labels available from upstream registries |
application_category |
String |
From io.hummingbird-project.application-category label |
Omitted — Hummingbird-specific label |
description |
String |
From org.opencontainers.image.description label or README |
Omitted — no README available from upstream registries |
latest_tag |
String |
Most recent canonical tag | Same |
pull_url |
String |
quay.io/…/{name}:latest |
{registry}:{latest_tag} |
oldest_created |
String? |
Earliest OCI created timestamp |
Same |
architectures |
String[] |
From OCI manifest list | Same |
variants |
VariantDescription[] |
From containers repo | From config |
stream_variant_count |
u32 |
Computed from tags | Same |
streams |
String[] |
From containers repo directory names | From config |
deprecated_streams |
String[] |
Streams with deprecated=true label |
Always [] — external images have no deprecation mechanism |
tag_count |
u32 |
Number of tags | Same |
release_count |
u32 |
Sum of history entries | Same |
ImageResponse
Full image detail — flattens ImageSummary into the top level with
additional fields.
| Field | Type | Hummingbird | External |
|---|---|---|---|
(all ImageSummary fields) |
(see above) | (see above) | |
readme_html |
String |
README rendered as HTML | Omitted — no README available from upstream |
licenses |
String[] |
SPDX identifiers from the attestation SBOM’s runtime packages | Omitted — no license labels on upstream images |
TagsResponse
| Field | Type | Description |
|---|---|---|
tags |
Tag[] |
Tags ordered by oldest_created desc |
variants |
VariantDescription[] |
Available variant descriptions |
Tag
| Field | Type | Hummingbird | External |
|---|---|---|---|
name |
String |
Tag name (e.g. 3.14.2) |
Same |
canonical |
String |
Most specific tag for this release | Same (by semver specificity) |
variant |
String |
From containers repo | From config |
pull_url |
String |
quay.io/…:{tag} |
{registry}:{tag} |
digest |
String |
Manifest list digest | Same |
sizes |
{arch: bytes} |
Compressed layer sizes from OCI manifest | Same |
architectures |
String[] |
From OCI manifest list | Same |
oldest_created |
String? |
Earliest OCI created timestamp |
Same |
stream |
String |
From containers repo directory name | From config |
deprecated |
bool |
From deprecated=true OCI label |
Always false — no deprecation mechanism for external images |
superseded |
bool |
Computed by mark_superseded |
Same |
DetailsResponse / ArchDetails
Per-architecture OCI container configuration keyed by architecture name
(e.g. {"amd64": {...}, "arm64": {...}}).
| Field | Type | Hummingbird | External |
|---|---|---|---|
user |
String |
From OCI config | Same |
env |
{key: value} |
From OCI config | Same |
entrypoint |
String[] |
From OCI config | Same |
cmd |
String[] |
From OCI config | Same |
exposed_ports |
String[] |
From OCI config | Same |
volumes |
String[] |
From OCI config | Same |
working_dir |
String |
From OCI config | Same |
stop_signal |
String |
From OCI config | Same |
labels |
{key: value} |
From OCI config | Same |
created |
String? |
From OCI config root | Same |
source_repo |
String |
From SLSA attestation | Omitted — no signed attestations on upstream images |
source_commit |
String |
From SLSA attestation | Omitted — no signed attestations on upstream images |
containerfile |
String |
From SLSA attestation | Omitted — no signed attestations on upstream images |
slsa_level |
String |
From SLSA attestation | Omitted — no signed attestations on upstream images |
build_system |
String |
From SLSA attestation | Omitted — no signed attestations on upstream images |
SbomResponse / ArchSbom
Per-architecture package list keyed by architecture name.
| Field | Type | Hummingbird | Rawhide | External |
|---|---|---|---|---|
runtime_count |
u32 |
From the syft scan | From the syft scan (Hermeto merge disabled — see below) | Same as Hummingbird |
build_count |
u32 |
From the Hermeto merge | Always 0 |
Always 0 — Hermeto is a Konflux-specific build tool |
source_count |
u32 |
Always 0 — arch=src entries are skipped |
Always 0 |
Always 0 — Hermeto is a Konflux-specific build tool |
packages |
SbomPackage[] |
All known architectures — one ArchSbom per arch from the manifest list |
Same | All known architectures — one ArchSbom per arch from the manifest list |
SbomPackage
| Field | Type | Notes |
|---|---|---|
name |
String |
Package name (e.g. python3-libs, github.com/caddyserver/caddy/v2). |
evr |
String |
Version. RPM: version-release (e.g. 1.33-5.1.hum1); other types: upstream version (e.g. v2.9.1). |
type |
String |
Package ecosystem, carried verbatim from the Syft artifact type: rpm, go-module, python, npm, gem, dotnet, java-archive. Items written before the syft-native switch may carry purl-derived names (golang, pypi, nuget, maven). |
license |
String |
SPDX license expression, NOASSERTION when unknown. |
sourcerpm |
String |
Full source RPM filename (e.g. python3-3.14.2-1.fc42.src.rpm). Only set for RPM runtime packages. |
category |
String |
runtime (Syft) or build (Hermeto binary with no syft twin). source appears only on legacy items — arch=src entries are skipped. |
purl |
String |
Full purl string (e.g. pkg:golang/golang.org/x/crypto@0.35.0). |
cpes |
String[] |
CPE 2.3 identifiers. |
Schema revision 1.1.0: the arch map of SbomResponse serializes its keys
sorted (byte-stable writes of identical data; semantically identical JSON for
consumers). As SBOM content regeneration from the native syft scans rolls out,
the type vocabulary switches to the syft-native names above — until an
image’s items are regenerated they keep the purl-derived names, so the
type value itself distinguishes regenerated from legacy items (go-module
and python only appear on regenerated items; legacy items carry golang and
pypi).
SBOM content sources: the Index Lambda builds items from the native syft
scan of each architecture — all runtime packages, with the type vocabulary
carried verbatim from syft — and merges the per-arch Hermeto attestation on
top: attestation entries with a syft runtime twin (same name and
version-release; a leading RPM epoch prefix on the syft side, e.g.
1:3.5.5-1.hum1, is excluded from the comparison) are dropped, and no-twin
entries with a matching arch (or noarch) become build rows carrying
their attestation purl verbatim. arch=src entries are skipped — source
rows are not emitted, the SRPM identity is derivable from any binary row’s
sourcerpm. A version skew keeps both entries — visible,
never a silent drop. The merge is gated by the distro switch
(classify_by_hermeto): Hummingbird merges; Rawhide serves syft-only items.
The attestation SPDX itself remains load-bearing only for derived fields in
sync (OVERVIEW licenses, the variant-label fallback) and the
conditionally-seeded RELEASE_SBOM for digests that were never scanned
while current (an atomic write-if-absent write) —
SBOM item content no longer depends on the mobster-merged attestation.
Historical items: items written before the cutover keep their
attestation-derived content and purl-derived type vocabulary until the SBOM
backfill regenerates them from the stored scan blobs (see the schema
revision note above for the interim marker). Digests with no scan blob at
all keep their attestation-derived RELEASE_SBOM seed indefinitely.
HistoryResponse / HistorySummary
Release timeline for a specific (stream, variant) combination.
| Field | Type | Hummingbird | External |
|---|---|---|---|
digest |
String |
Manifest list digest | Same |
oldest_created |
String |
Earliest OCI created |
Same |
architectures |
String[] |
From OCI manifest list | Same |
source_commit |
String |
From SLSA attestation | Omitted — no signed attestations on upstream images |
source_repo |
String |
From SLSA attestation | Omitted — no signed attestations on upstream images |
VulnerabilitiesResponse
Vulnerability scan results are identical in structure. The only behavioral
difference: ignore_list (CPE-based false-positive suppression) is populated
for Hummingbird images but empty for external images, since suppression rules
are maintained per Hummingbird image.
Timestamp Fields
The oldest_created field on ImageSummary, Tag, and HistorySummary is
the earliest OCI created timestamp across all architectures in the release.
All architectures were built at or after this date, making it useful for
conservative staleness detection.
The specifications endpoint returns per-architecture data keyed by architecture
name. Each architecture’s created field is the direct OCI config root
timestamp for that specific architecture.
Deprecation status
The containers repository can mark an image stream with the
io.hummingbird-project.deprecated=true OCI label. The catalog exposes this
status in two places:
Tag.deprecatedidentifies a release carrying the label.ImageSummary.deprecated_streamslists streams whose current release is deprecated.
Both fields are additive API fields. Older DynamoDB items without these fields are read as active releases with no deprecated streams.
Usage
All tools are built as a single catalog binary with subcommands (api,
sync, sync-lambda, scan, scan-lambda, index, index-lambda,
enqueue-lambda, metrics, metrics-lambda). The binary is built in the
Rust container and CLI subcommands are run in the gitlab-ci container (which
provides grype, syft, and other tools). Only make and podman are required.
Sync
# Dry run (print items to stdout)
make container-catalog/sync ARGS="--distro rawhide --dry-run"
# Populate DynamoDB
make container-catalog/sync ARGS="--distro rawhide --table-name <table>"
Sync Lambda
The sync-lambda subcommand runs as an AWS Lambda function triggered by SNS
Release events from kubernetes-event-forwarder.
It incrementally syncs a single image release to DynamoDB (~22 registry API
calls per release vs ~104 for a full sync).
The Lambda:
- Decodes gzip+base64 SNS messages
- Filters for Succeeded releases targeting the configured Quay.io namespace.
Ignores
DELETEDevents and releases with a non-nulldeletionTimestamp. - Fetches OCI manifest data for the new digest
- Writes per-digest items (DETAILS, RELEASE_DETAILS, and a RELEASE_SBOM
seed for digests that were never scanned while current — an atomic
write-if-absent conditional write, so a sweep or a retried invocation can
never overwrite index-normalized content);
SBOM#{canonical}items are written by the Index Lambda - Merges into aggregate items (TAGS, HISTORY, OVERVIEW, DIRECTORY)
- Fetches README from GitLab for OVERVIEW content (uses
README.mdfor Hummingbird,rawhide/README.mdfor Rawhide) - Enqueues one index event per release on the IndexQueue — the message carries the image, the canonical just written, the digest, and the manifest-list architectures, everything the Index Lambda needs (it never reads TAGS). A redelivered release re-emits (the known-digest early exit is cheap and the index dedups); historical releases do not emit.
The attestation SBOM fetch remains load-bearing for OVERVIEW licenses and the variant-label fallback — moving licenses off the attestation is follow-up polish.
Registry fetch errors (manifest, SBOM, attestation) propagate as hard failures so the Lambda retries automatically (up to 2 retries with backoff) before sending to the DLQ. GitLab README failures are non-fatal – the existing README is preserved if the fetch fails.
| Environment Variable | Description |
|---|---|
TABLE_NAME |
DynamoDB table name |
INDEX_QUEUE_URL |
SQS queue URL for index events |
DISTRO |
rawhide or hummingbird |
SENTRY_DSN |
Optional Sentry DSN for error tracking |
Index
The index subcommand generates native Syft JSON for all non-superseded image
digests and uploads them to S3 for scanner consumption. It reads the image
directory and TAGS from DynamoDB, checks S3 for existing objects (dedup via
HeadObject), runs syft <image> --platform linux/amd64, gzips the output,
and uploads to s3://{bucket}/grype/{image}/{hex}.json.gz.
# Dry run (show what would be uploaded)
make container-catalog/index ARGS="--distro hummingbird --table-name <table> --bucket <bucket> --dry-run"
# Backfill S3 for all images
make container-catalog/index ARGS="--distro hummingbird --table-name <table> --bucket <bucket>"
# Backfill a single image
make container-catalog/index ARGS="--distro hummingbird --table-name <table> --bucket <bucket> --image caddy"
Index Lambda
The index-lambda subcommand runs as an AWS Lambda function triggered by
the IndexQueue (SQS, BatchSize: 1, batch-item-failure reporting). The
Sync Lambda enqueues one TagReleaseEvent per release after its writes; it
is the same event shape and per-event processing structure as the external
index — the function is the future unified index, with the blob-key
prefixes and the Hermeto-merge gating as per-deployment configuration. It
generates the native Syft JSON for scanner consumption and owns the SBOM
items.
The Lambda:
- Parses
TagReleaseEventrecords (image, canonicals, digest, architectures); a record that fails to parse or process is reported as an SQS batch-item failure and retried individually by the queue, redriving to the IndexDLQ after 3 attempts. - Checks S3 for existing objects (HeadObject dedup by digest). On a hit, it
heals any canonical whose
SBOM#item is missing (canonical rename, DLQ replay) by rebuilding it from the persisted SPDX record —RELEASE_SBOM#{hex}is only written when absent, so a recovered primary-arch-only item cannot downgrade a multi-arch item. Blobs written by thecatalog indexCLI carry no persisted SPDX; the recovery skips those with a warning (the SBOM backfill is their designated repair). - Resolves the architecture list and the per-arch Hermeto attestations from the manifest list over the public registry (no IAM). A missing per-arch attestation degrades that arch to syft-only; a manifest-list failure falls back to the event’s recorded architectures.
- Runs
syft <image>@<digest> --platform linux/{arch}for every architecture, bounded by the invocation deadline: the primary arch (amd64 when present, otherwise the first) runs with dual output (syft-json + spdx-json — its spdx is the persisted recovery input) and is a hard failure; the remaining arches run native-only, and their failures degrade the SBOM to primary-arch-only. - Merges Hermeto build-only entries per arch (see SBOM content sources).
- Re-verifies every canonical against the registry after all arch scans, immediately before the writes: canonicals that moved to a newer digest are dropped, a digest no canonical references anymore is discarded entirely (no items, no upload), and a failed verification fails the record after the writes so the retry heals the skipped canonical.
- Writes
SBOM#{canonical}per still-current canonical plusRELEASE_SBOM#{hex}to DynamoDB, then the primary-arch SPDX tos3://{bucket}/sbom/{image}/{hex}.json.gzwithprimary-archobject metadata (the recovery input for step 2), then the gzipped syft JSON tos3://{bucket}/grype/{image}/{hex}.json.gz— the last write and the completion marker that triggers the Scan Lambda via S3→SQS and arms the dedup key.
TAGS is never read — the event carries everything the function needs, and canonical identity is re-resolved against the registry.
Failures are retried by the queue (up to 3 attempts) before the IndexDLQ. S3
gaps can be backfilled with the catalog index CLI (it writes S3 objects
only); SBOM item gaps are healed by the already-indexed recovery on the next
event or DLQ redrive, or by the SBOM backfill.
| Environment Variable | Description |
|---|---|
SCAN_DATA_BUCKET |
S3 bucket for scanner data |
TABLE_NAME |
DynamoDB table name |
SCAN_KEY_PREFIX |
Scan-blob key prefix (default grype/) |
SBOM_KEY_PREFIX |
SPDX-record key prefix (default sbom/) |
DISTRO |
rawhide or hummingbird |
SENTRY_DSN |
Optional Sentry DSN for error tracking |
Scan
The scan subcommand reads image listings and tags from DynamoDB and runs Grype
against each image using native Syft JSON from S3 (--bucket is required).
Results include a first_seen timestamp per CVE, tracked at the
group+variant+stream level and carried across releases for SLI computation.
Tagged scans read native Syft JSON from S3
(grype/{image}/{hex}.json.gz), matching the lambda pipeline. Historic
releases (--scope all) read the same S3 objects; digests whose object is
missing are skipped with a warning.
# Dry run (print items to stdout)
make container-catalog/scan ARGS="--distro hummingbird --table-name <table> --bucket <bucket> --dry-run"
# Scan and write to DynamoDB (purge stale vuln data first, implies --scope=all)
make container-catalog/scan ARGS="--distro hummingbird --table-name <table> --bucket <bucket> --purge"
# Scan only non-superseded (current) tags
make container-catalog/scan ARGS="--distro hummingbird --table-name <table> --bucket <bucket> --scope non-superseded"
# Scan all releases including historic (tagless) releases
make container-catalog/scan ARGS="--distro hummingbird --table-name <table> --bucket <bucket> --scope all"
# Scan a single image
make container-catalog/scan ARGS="--distro hummingbird --table-name <table> --bucket <bucket> --image caddy --dry-run"
Scan Lambda
The scan-lambda subcommand runs as an AWS Lambda function triggered by SQS
messages. Two paths feed the ScanQueue for Hummingbird images, and two
parallel paths for external images:
- Real-time (S3 event notification): When the Index Lambda uploads a new
Syft JSON to S3, an
s3:ObjectCreatedevent (filtered ongrype/prefix) is sent directly to SQS - Hourly (Enqueue Lambda): Fans out all non-superseded digests when the
Grype vulnerability database has been updated, sending
{"bucket": "...", "key": "grype/{image}/{hex}.json.gz"}messages - External real-time: When the External Index Lambda uploads Syft JSON
to
grype-external/{registry}/{hex}.json.gz, the same ScanQueue receives the event - External hourly: The Enqueue Lambda also fans out external re-scans
using
grype-external/{registry}/{hex}.json.gzkeys
The Lambda accepts both S3 event notification JSON and direct
{"bucket", "key"} messages. It detects the S3 key prefix to dispatch
between two code paths:
grype/prefix → Hummingbird path: writesIMAGE#{name}/VULNERABILITIES#{canonical}andIMAGE#{name}/RELEASE_VULNERABILITIES#{hex}grype-external/prefix → external path: parses(registry, hex)from the key, then writesEXTERNAL#{registry}/VULNERABILITIES#{canonical}andEXTERNAL#{registry}/RELEASE_VULNERABILITIES#{hex}using the same Grype scan andfirst_seentracking logic as Hummingbird
For each message the Lambda:
- Downloads and decompresses the native Syft JSON from S3
- Runs Grype on the raw bytes
- Looks up all canonical tags matching the digest from the TAGS item
- Writes
VULNERABILITIES#{canonical}for each matching tag (withfirst_seentracking) andRELEASE_VULNERABILITIES#{hex}once per digest
Processing is per-digest: a single Syft JSON serves all canonicals sharing that digest, avoiding redundant Grype invocations.
The Lambda downloads and hydrates the Grype DB on the first handler invocation
of each cold start (cached in /tmp for warm invocations), processes one
message per batch, and reports partial batch failures so failed records return
to the queue.
Concurrency is capped at 100 concurrent invocations (ScalingConfig on the
SQS event source). Every Grype DB update triggers a full-catalog rescan
fan-out of several hundred scans, and without a cap the event source scales
with that backlog — measured at ~490 concurrent executions, about half the
account’s shared 1000-execution limit, with each cold container downloading
the multi-GB Grype DB. The cap keeps one function at ~10% of the account
budget and amortizes the DB setup across warm invocations. The template
deploys to both the hummingbird and rawhide stacks, so the theoretical
simultaneous maximum is 2 × 100 from the scan Lambdas plus 2 × 5 from the
external index Lambdas — 210 of the account limit.
| Environment Variable | Description |
|---|---|
TABLE_NAME |
DynamoDB table name |
DISTRO |
rawhide or hummingbird |
SENTRY_DSN |
Optional Sentry DSN for error tracking |
Enqueue Lambda
The enqueue-lambda subcommand runs hourly via EventBridge Schedule. On each
invocation it performs a lightweight HTTP GET of the public Grype DB listing
(latest.json, ~200 bytes) and compares the built timestamp against the
stored CATALOG/LAST_FULL_SCAN_DB item in DynamoDB. If the DB hasn’t changed,
it returns early (~23 of 24 hourly invocations short-circuit). When an update
is detected it runs two fan-out loops:
- Hummingbird images: reads the image directory and all TAGS items,
deduplicates by digest, and sends one SQS message per unique digest.
Messages use
{"bucket": "...", "key": "grype/{image}/{hex}.json.gz"}. - External images: reads the external-images config from DynamoDB
(
CATALOG/EXTERNAL_CONFIG), then for each configured registry readsEXTERNAL#{registry}/TAGSand fans out non-superseded digests withgrype-external/{registry}/{hex}.json.gzkeys. If no TAGS item exists (first run, before External Sync Lambda), the registry is skipped. A malformed or missingEXTERNAL_CONFIGdegrades gracefully — a warning is logged and the Hummingbird fan-out already sent is unaffected.
Both loops use the same SendMessageBatch batching and !superseded digest
filter.
| Environment Variable | Description |
|---|---|
TABLE_NAME |
DynamoDB table name |
DISTRO |
rawhide or hummingbird |
SCAN_QUEUE_URL |
SQS queue URL for scan messages |
SCAN_DATA_BUCKET |
S3 bucket for scanner data |
GRYPE_DB_LATEST_URL |
Grype DB listing URL (has sensible default) |
SENTRY_DSN |
Optional Sentry DSN for error tracking |
Metrics
The metrics subcommand performs a one-shot read of all non-superseded
vulnerability data from DynamoDB and writes the catalog-wide vulnerability
aggregate to DynamoDB (PK=CATALOG, SK=VULNERABILITIES).
make container-catalog/metrics ARGS="--table-name <table>"
Metrics Lambda
The metrics-lambda subcommand runs as a DynamoDB Stream-triggered Lambda that
writes a catalog-wide vulnerability aggregate to DynamoDB (served by
GET /v1/vulnerabilities).
How It Works
The Lambda is triggered by DynamoDB Stream events filtered on VULNERABILITIES#
and TAGS changes, with a 60-second batching window and reserved concurrency
of 1 (single instance). It maintains an in-memory active CVE table across warm
invocations:
- Cold start: Reads
CATALOG/DIRECTORY, allTAGS, and all non-supersededVULNERABILITIES#items from DynamoDB to build the full table (~2-3s at 10000 tags) - Warm invocations: Incrementally updates the table from stream event keys (~50-100 GetItem calls per batch)
- After each invocation: Recomputes the aggregate
Stream events for EXTERNAL# PKs (external images) are intentionally
excluded from the aggregate. The incremental_update path filters on
pk.starts_with("IMAGE#") so that external-image VULNERABILITIES# stream
events do not pollute the Hummingbird-only CATALOG/VULNERABILITIES item.
The GET /v1/vulnerabilities endpoint reflects only Hummingbird CVE data.
| Environment Variable | Description |
|---|---|
TABLE_NAME |
DynamoDB table name |
DISTRO |
rawhide or hummingbird |
SENTRY_DSN |
Optional Sentry DSN |
Deployment
make container-catalog/build
make container-catalog/deploy
External Images
The external images feature enables CVE comparison between Hummingbird and
external image sources (e.g. standard upstream registries). External image data
is exposed under the /v1/external-images/ namespace, which is fully additive —
existing /v1/images/ clients are unaffected.
Overview
External images share the same DynamoDB table as Hummingbird images but use
an EXTERNAL#{registry} PK prefix instead of IMAGE#{name}, where {registry}
is the full registry pull reference (e.g. docker.io/library/nginx). The same
response types, SK patterns, and API conventions apply: tags, details, SBOM,
vulnerabilities, history, and release snapshots all work identically. Provenance
fields (source_repo, source_commit, containerfile, slsa_level,
build_system) are empty strings for external images — they are not available
from upstream registries.
The /v1/external-images/config endpoint returns the config that maps
Hummingbird images to their external sources. The join key for a given image
detail page (IMAGE#{repository}) is the repository on each tag entry
(resolved from the source-level default when not set per-tag). Clients filter
tag entries by repository, then match stream and variant values against
the page’s stream/variant selection to construct the
/v1/external-images/{percent-encoded-registry}/… URLs (registry refs
containing / must be percent-encoded in the URL path).
Tag-based discovery and re-verification (resolve_digest — HEAD manifest
by tag, used by both External Sync and External Index) goes directly to
registry-1.docker.io with anonymous tokens from auth.docker.io
(service=registry.docker.io). HEAD manifest requests are free against
Docker Hub’s pull limit (100 manifest GETs per hour per IP, verified
empirically), return docker-content-digest, and always resolve to Docker
Hub’s current view of the tag. ratelimit-remaining from each HEAD
response is logged as a numeric budget gauge.
Digest-pinned architecture fetches (fetch_details — manifest GETs by
digest) and syft image pulls both route through mirror.gcr.io first, with
a direct-docker.io fallback on any failure. Digest-pinned requests are
content-addressed and cannot be stale, so the mirror is a safe first
attempt; the fallback covers mirror ingestion lag on a freshly discovered
digest. In Lambda each syft attempt runs under a per-call budget derived
from the invocation deadline; in CLI mode syft runs until it exits or the
operator interrupts it. Stored data uses the canonical docker.io/...
registry name; other registries are accessed directly.
Configuration Format
The external-images configuration is stored in DynamoDB at
CATALOG/EXTERNAL_CONFIG and served at GET /v1/external-images/config.
The JSON config is maintained outside this repo (e.g. in the infrastructure
repo) and applied to DynamoDB via the catalog config set CLI command.
The config is a flat list of sources. Each source maps registry tags to
(repository, stream, variant) triples — the same concepts Hummingbird uses
for canonical determination and mark_superseded. Stream and repository values
are whatever the corresponding Hummingbird image uses. Variant names use
Hummingbird names where an equivalent exists (e.g. builder for upstream dev
images); upstream-only variants like slim or alpine keep their upstream
names.
Each tag carries its own repository — the Hummingbird repository the tag’s
comparison belongs to. This allows tags from the same upstream registry to
join different Hummingbird repositories (e.g. docker.io/library/caddy tags
2 → repository caddy and 2-builder → repository xcaddy):
{
"sources": [
{
"registry": "docker.io/library/caddy",
"tags": [
{
"tag": "2",
"repository": "caddy",
"stream": "2",
"variant": "default"
},
{
"tag": "2-builder",
"repository": "xcaddy",
"stream": "2",
"variant": "builder"
}
],
"variants": [
{
"name": "default",
"description": "Caddy server",
"base": "default",
"builder": false,
"fips": false
},
{
"name": "builder",
"description": "Build image with xcaddy",
"base": "default",
"builder": true,
"fips": false
}
]
}
]
}
Every tag must carry a repository — the per-tag form is the only shape.
The older source-level repository field (a default inherited by all tags)
was transitional and has been removed: configs are parsed with
deny_unknown_fields, so a source-level repository is now a parse error.
The containers aggregation emits per-tag repository on every tag.
Stream Assignment
External image stream values follow the same rules as Hummingbird images (see Choosing a Stream Value in the containers repo). Key rules for external image config:
- If the Hummingbird image uses numbered streams (e.g.
"1.27","3.14"), external image tags must map to the same stream values. A Docker Hublatesttag maps to the current numbered stream — it is never a separate"latest"stream alongside numbered ones. - Use
"latest"as a stream value only for sources where no meaningful version series exists (rolling releases, date-based versions). - The
assign_canonicalsfunction groups tags by(stream, variant, digest). Within each group, the most semver-specific tag becomes canonical. Whenlatestand1.27both map to stream"1.27"and resolve to the same digest,1.27wins by specificity.
Variant values share the Hummingbird vocabulary where equivalents exist: use
"builder" for upstream dev images that add a shell and package manager
(e.g. Chainguard’s latest-dev), and "default" for plain runtime images.
Upstream-only variants with no Hummingbird equivalent (e.g. "slim",
"alpine" on Docker Hub) keep their upstream names. CVE data for all
tracked variants is served the same way, so clients can present them
however fits.
To add a new image or source, update the config JSON and run
catalog config set --file <path> --table-name <table> to apply the change
to DynamoDB (no Lambda redeploy needed).
External Release Lambda
Triggered by EventBridge on a schedule (every 6 hours). Reads the
external-images config from DynamoDB and sends one SQS message per configured
registry to ExternalSyncQueue via send_batch. Sends unconditionally —
change detection is handled downstream by External Sync Lambda.
The Lambda is lightweight: one DynamoDB GetItem and one or two SQS
SendMessageBatch calls per invocation. No registry calls, no S3 access.
| Environment Variable | Description |
|---|---|
TABLE_NAME |
DynamoDB table name |
DISTRO |
hummingbird (no-op on rawhide) |
EXTERNAL_SYNC_QUEUE_URL |
SQS queue URL for per-registry messages |
SENTRY_DSN |
Optional Sentry DSN for error tracking |
External Sync Lambda
Triggered by ExternalSyncQueue (SQS, BatchSize: 1). Each invocation
processes one registry reference — identical scope to SyncFunction processing
one Konflux release.
- Resolves all configured tags to digests via registry HEAD requests.
Digests are validated (
sha256:[0-9a-f]{64}) before use. Attestation manifests (platform.architecture = "unknown") are filtered out. All configured tags must resolve successfully; partial resolution aborts the write to prevent silently shrinking the TAGS item. - Compares with current
EXTERNAL#{registry}/TAGSin DynamoDB; early-returns if no digests changed (no-op on most invocations). - Groups tags by digest within each
(stream, variant)pair; picks canonical by semver specificity (1.27.2beats1.27). - Writes DETAILS, RELEASE_DETAILS, HISTORY, OVERVIEW, and EXTERNAL_DIRECTORY to DynamoDB.
- Emits one SQS message per non-superseded digest to
ExternalIndexQueue. Emission is unconditional rather than change-detection based: the index Lambda’s S3 existence check makes an event for an already-indexed digest a cheap no-op, and unconditional emission restores the self-healing of the old per-registry topology — canonical renames with unchanged digests get their SBOM records backfilled on the next sync write. When several canonicals resolve to the same digest, they share one event and one scan. Each message carries everything External Index Lambda needs:registry,canonicals,digest,architectures. - Writes TAGS last so it serves as a completion marker — a crashed invocation will be retried in full, and an SQS send failure in step 5 is retried without the early-exit firing (TAGS still holds the old state).
The moment sync observes a changed digest is the release event for an external image — sync stands where the Hummingbird build system stands.
| Environment Variable | Description |
|---|---|
TABLE_NAME |
DynamoDB table name |
DISTRO |
rawhide or hummingbird |
EXTERNAL_INDEX_QUEUE_URL |
SQS queue URL for per-tag index events |
SENTRY_DSN |
Optional Sentry DSN for error tracking |
External Index Lambda
Triggered by ExternalIndexQueue (SQS, BatchSize: 1). Each invocation
processes one digest from a release event emitted by External Sync
Lambda. The event carries everything needed — registry, canonicals,
digest, architectures — so TAGS is not read, eliminating the first-run
race between the two lambdas.
- Validates the event at the queue boundary (registry, digest, canonicals,
architectures). Checks S3 for an existing
grype-external/{registry}/{hex}.json.gzobject; skips if already present (idempotent, mirrors IndexFunction). If the S3 object exists but a canonical’s DynamoDB SBOM item is missing (e.g. a canonical rename after the last syft run), the SBOM is backfilled from the persisted SPDX atsbom-external/{registry}/{hex}.json.gz— one scan serves every canonical of the digest. - Runs Syft with dual output for the primary architecture (
amd64when present, otherwise the first architecture of the event — some upstream images are single-arch non-amd64):syft-jsonfor Grype,spdx-jsonfor the SBOM. Then onespdx-jsonrun per remaining architecture; oneArchSbomper arch is stored in theSbomResponse. - Re-verifies the registry digest has not changed during the Syft run. If the digest changed, the output is discarded to avoid indexing a stale digest. SBOM writes happen only after this verification.
- Writes one
SBOM#{canonical}per canonical of the digest (all from this single scan) plusRELEASE_SBOM#{hex}to DynamoDB. - Uploads the gzipped Syft JSON to S3; the S3 event notification triggers
the Scan Lambda via SQS. SPDX is stored separately at
sbom-external/{registry}/{hex}.json.gz(different prefix to avoid triggering a spurious scan).
Each invocation is bounded to one digest’s architecture matrix (minutes, not the whole registry), so timeout and OOM blast radius are per-digest rather than per-registry. DLQ entries are single-digest releases — replayable and meaningful. DLQ entries are not retried automatically: recover via SQS DLQ redrive, or wait for the registry’s next change (any change re-emits every non-superseded digest).
| Environment Variable | Description |
|---|---|
TABLE_NAME |
DynamoDB table name |
SCAN_DATA_BUCKET |
S3 bucket for scanner data |
DISTRO |
rawhide or hummingbird |
SENTRY_DSN |
Optional Sentry DSN for error tracking |
Data Freshness and Known Asymmetries
- External images are polled every 6 hours. Grype DB re-scans run hourly, same as Hummingbird images.
- The
CATALOG/VULNERABILITIESaggregate (GET /v1/vulnerabilities) reflects only Hummingbird CVE data. There is no external-images catalog-wide CVE aggregate. - Sources that offer only a
latesttag and have no numbered version series will always have"latest"as their canonical.
For per-field availability differences between the two namespaces, see the response type tables in the API Endpoints section.
Docker Hub Limits and mirror.gcr.io Behavior
Reference data for the routing decisions in the External Sync and External Index Lambdas (measured 2026-11-10).
Docker Hub metering
Only manifest GETs count against the pull limit — one unit each (index and platform manifests separately). HEAD manifest requests, blob HEAD/GETs, auth token requests, and tag listing GETs are unmetered.
| Access type | Limit (anonymous) | Notes |
|---|---|---|
| Manifest GETs | 100 / hour / IPv4 (or IPv6 /64) | ratelimit-limit: 100;w=3600 in headers |
| Manifest HEADs | Unmetered | Still returns docker-content-digest |
| Blob GETs / HEADs | Unmetered | Measured; Docker docs do not confirm |
| Auth token fetches | Unmetered | |
| Tag listing GETs | Unmetered | Full listing in one response; paginated via ?n=&last=; mirror.gcr.io serves the same endpoint as its cache inventory |
The registry API endpoint is registry-1.docker.io (the bare docker.io
host redirects to the Docker website). Anonymous tokens are issued by
auth.docker.io with service=registry.docker.io and per-repository scope;
they are valid for approximately 5 minutes. ratelimit-remaining is
present on HEAD responses; resolve_digest logs it as a budget gauge.
Routing policy
| Request | Route | Reason |
|---|---|---|
resolve_digest (HEAD by tag) |
registry-1.docker.io direct |
Tags are mutable; mirror can serve stale. HEADs are unmetered. |
fetch_details (GET by digest) |
mirror.gcr.io → docker.io fallback |
Content-addressed — cannot be stale. Mirror-first; falls back direct on any failure. |
| Syft image pulls | mirror.gcr.io → docker.io fallback |
Mirror-first; retries direct on any failure. |
mirror.gcr.io receives all digest-addressed traffic from this service as the first attempt.
Routing requirements
The routing design is governed by the following requirements:
- R1 — tag resolution is authoritative:
resolve_digestHEADs go directly toregistry-1.docker.iowith anonymousauth.docker.iotokens. Tags are mutable;HEADmanifest requests are unmetered and always return Docker Hub’s current view of the tag. - R2 — digest-addressed fetches go mirror-first with direct fallback:
fetch_details(manifest list, per-arch manifests, config blobs) and the@digest-pinned syft pulls both trymirror.gcr.iofirst. Content-addressed fetches cannot be stale — only missing — so the mirror is safe; the direct fallback covers mirror ingestion lag on a freshly discovered digest (the window between direct tag resolution and the mirror ingesting the new object). The fallback costs ~1 + archs metered manifest GETs, only on a mirror miss; when the mirror serves the request the metered budget stays at zero. - R3 — mirror failures fall back to docker.io: any failure on a mirror
GET retries the same request directly against
docker.io— both for syft pulls (viarun_syft_outputsinscanner.rs) and forfetch_detailsmetadata GETs (viarouted_getinexternal_sync_lambda.rs). Reliability is the reason; error-classification by response body would be brittle. A fallback costs approximately 2–3 metered manifest GETs for a syft pull (blobs are unmetered) or ~1 + archs GETs for afetch_detailsmiss; the fallback granularity for a pull is the whole ref since syft fetches from a single reference. In Lambda each attempt runs under a per-call budget derived from the invocation deadline (deadline − now − 60 sheadroom for post-scan work); a zero remaining budget returns the error before spawning. In CLI mode syft runs until it exits or the operator interrupts it — Ctrl-C causeskill_on_dropto terminate the child andTempDirGuardto remove its temp dir. The stale sweep uses the owner PID as the garbage authority: a syft temp dir is removed when its owner process has exited. - R4 — Accept-set consistency:
resolve_digestHEADs include both index and platform manifest media types (application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json, plus the single-manifest types) so a tag resolves to the index digest rather than a platform-specific one. The same Accept set is used forfetch_detailsGETs via the mirror. - R5 — bounded concurrency against the mirror: the mirror’s anonymous
limiter escalates to persistent per-IP 429s under sustained load, so pull
fan-out must stay bounded. Partially holds via External Index Lambda
MaximumConcurrency: 5. - R6 —
ratelimit-remainingbudget gauge:resolve_digestlogs theratelimit-remainingheader from each Docker HubHEADresponse as a numeric structured field (remaining), including from error responses (e.g. 429) whereremaining=0is most actionable. The header is returned onHEADresponses at no additional cost. The header value may carry a window suffix (100;w=3600) — only the numeric prefix is parsed.
mirror.gcr.io behavior
- Demand-driven cache: objects ingest only when requested; each object (tag mapping, manifest, blob) warms independently.
- Digest-addressed content: served cold (fetched from Docker Hub on miss). Whole-image cold pulls work end-to-end once warmed.
- Blob ingestion is async and flaky: a pull can fail with
BLOB_UNKNOWNfor layers that ingest over subsequent requests (observed taking days). Retries serve as cache warming. - Eviction: images not requested for approximately 5 days are removed. Cached does not mean stable.
- Rate limiting: burst 429s after approximately 10 rapid manifest requests, escalating to a persistent per-IP throttle under sustained load. Bounds pull fan-out concurrency.
- HEAD is unreliable as a cache probe: can 404 for content that is
cached and servable.
GET /v2/{repo}/tags/listlists cached tag mappings only — digest-addressed content can be served while its tag is unlisted.
Configuration
catalog sync
| Argument | Description |
|---|---|
--distro |
rawhide or hummingbird |
--table-name |
DynamoDB table name |
--purge |
Delete all items before writing |
--cache-dir |
Cache directory (auto-detected) |
--image |
Sync only a specific repo |
--legacy-discovery |
Use GitLab-based repo discovery |
catalog index
| Argument | Description |
|---|---|
--distro |
rawhide or hummingbird |
--table-name |
DynamoDB table name (for image/tag discovery) |
--bucket |
S3 bucket for scanner data |
--dry-run |
Print what would be uploaded without uploading |
--image |
Index only a specific image |
--parallel |
Number of concurrent operations (default: 2) |
catalog scan
| Argument | Description |
|---|---|
--distro |
rawhide or hummingbird |
--table-name |
DynamoDB table name (required) |
--bucket |
S3 bucket with native Syft JSON (required) |
--scope |
non-superseded, tags (default), or all |
--dry-run |
Print items without writing |
--purge |
Purge vuln data before writing (implies --scope all) |
--parallel |
Number of concurrent scans (default: 4) |
--image |
Scan only a specific image |
--tag |
Scan only a specific tag (requires --image) |
catalog config
catalog config set --file <path> --table-name <table>
reads the external-images JSON config and writes it to the DynamoDB item
CATALOG/EXTERNAL_CONFIG. Run once on initial setup and after each config
change. No Lambda redeploy is needed — all External* Lambdas read config from
DynamoDB at invocation time.
| Argument | Description |
|---|---|
--table-name |
DynamoDB table name |
--file |
Path to external-images JSON file |
SAM Parameters
| Parameter | Description |
|---|---|
Distro |
rawhide or hummingbird |
CacheEnabled |
Enable CloudFront caching |
CatalogDomainName |
Catalog web UI domain |
ApiDomainName |
API domain |
HostedZoneId |
Route53 hosted zone |
CorsOrigins |
Comma-separated CORS origins (default *) |
SnsTopicArn |
SNS topic ARN for Release events (enables sync/index/external Lambdas) |
The External Release, Sync, and Index Lambdas are all conditioned on
HasSnsTopicArn (a feature flag historically tied to the Release-events topic)
and are only deployed for the hummingbird stack. The external chain itself is
queue-based: the EXTERNAL_SYNC_QUEUE_URL and EXTERNAL_INDEX_QUEUE_URL
environment variables are set automatically from the queue resources defined
in the same SAM template.
Frontend
The catalog web UI is a Lit 3 SPA (Web Components) with Tailwind CSS, built
per-distro with Vite. Source is in container-catalog/frontend/.
The image detail view includes an External tab (gated behind
VITE_VULNERABILITIES_ENABLED) that compares Hummingbird images against their
external counterparts. It shows a summary card with CVE and runtime package
count deltas, side-by-side severity breakdowns, a categorized CVE table
(only-in-upstream / only-in-Hummingbird / shared), and an SBOM package diff.
Only make and podman are required (no local Node.js needed).
Defaults from .envrc.defaults are applied automatically.
# Install dependencies
make container-catalog/frontend/setup
# Development server at http://localhost:5173
make container-catalog/frontend/dev
# Production build
make container-catalog/frontend/build
Host variants (*-host) run without podman (for CI or local Node.js).
Frontend Build Variables
| Variable | Description |
|---|---|
VITE_API_URL |
API base URL for the distro |
VITE_DISTRO |
rawhide or hummingbird |
VITE_DISTRO_LABEL |
Display label for current distro |
VITE_OTHER_CATALOG_URL |
URL of the other distro’s catalog (optional, hides link if unset) |
VITE_OTHER_DISTRO_LABEL |
Display label for other distro (optional) |
VITE_VULNERABILITIES_ENABLED |
Show vulnerabilities and external comparison tabs |
Development
# Backend
cargo test # Run tests
cargo clippy --all-targets # Lint
cargo fmt # Format
# Frontend (host variants, requires local Node.js)
cd container-catalog/frontend
npm run typecheck # Type check
npm run build # Production build
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.
4.5.19.1 - Container Catalog Design Decisions
Design Decision Registry (DDR) for container-catalog — every major design
choice, the alternatives considered, and why the chosen approach won. For
operational usage, see Container Catalog.
DDR-1: Historical RELEASE_SBOM writer survives the SBOM cutover as a write-if-absent seed
Decision: When the index function takes over SBOM-item writes, sync’s
build_release_sbom_item keeps its RELEASE_SBOM#{hex} construction,
written via an atomic write-if-absent conditional write
(put_content_versioned with attribute_not_exists(ver) — every
post-cutover writer stamps ver, so the condition blocks any rewrite of
normalized content); the historical branch of merge_release writes only
digest-keyed items; SBOM#{canonical} becomes index-exclusive.
Context: The SBOM cutover (W1 MR2) moves SBOM-item writes from sync to
index. Index only writes for digests current in TAGS; the SBOM backfill
enumerates scan blobs. Digests that were never scanned while current
therefore lose every writer. Production census (2026-11-11, 10 images,
10,697 history digests, vuln-item presence as the scan/blob proxy): 228
digests (2.1%) have an SBOM item but were never scanned — 225 from 2026-04,
before the scan pipeline wrote blobs, plus ~1/month of new misses (events
raced out of TAGS, dropped events). The registry tagging model (unique
sha256-{hex}/timestamp tags per build; ~a dozen moving plain version tags)
means every rebuild supersedes the previous digest out of TAGS, so this
population accrues slowly but indefinitely. The only code path that can
serve these digests is the catalog sync CLI’s historical sweep — the
production sync Lambda processes only each event’s release and never sweeps.
The sweep is an on-demand repair/repopulation tool (“Full DynamoDB
population”, --purge).
Alternatives: (a) Accept the gap — delete the RELEASE_SBOM construction outright; SBOM items become exclusively the index/backfill domain; the frozen 225 keep their existing items, new misses get history and details but no SBOM, documented as a known gap. (b) Keep an unconditional retained write — preserve today’s behavior for the historical branch unchanged. (c) Extend the backfill to synthesize RELEASE_SBOM from attestations for no-blob digests.
Rationale: (a) rejected because a repair tool with a measured blind spot
is an operational footgun: after an incident or data loss, a catalog sync
run would leave SBOM holes nothing else can fill (no blob → the backfill
enumerates blobs and skips them; no event → index never sees them). (b)
rejected because the sweep population is dominated by ex-current, blob-backed
digests that the SBOM backfill normalizes to scan-derived content; sync’s
writes are unconditional (batch_write; put_content_versioned has no
content comparison), so every CLI run would overwrite blob-derived items
back to attestation-derived content, silently undoing the backfill. (c)
rejected because the backfill is one-off by design — it cannot cover future
misses, only the current cohort. The write-if-absent seed keeps the repair
tool complete for exactly the population only it can reach, while the
conditional write makes it converge with the backfill instead of fighting
it: sync seeds once
(attestation-derived, today’s field spellings — consistent with the accepted
mixed-vocabulary story for no-blob digests), index and the backfill own the
final content for blob-backed digests, and a CLI run never rewrites an item
that already exists. Zero new IAM (one DDB read); ~20 lines retained plus
the guard, confined to a CLI-only code path.
DDR-2: Index invocation converges on the queue-decoupled pattern
Decision: The internal index function drops its SNS Release subscription
and consumes a queue of tag-release events; sync performs its DynamoDB
writes and then enqueues one event per release canonical, using the shared
TagReleaseEvent shape (the promoted ExternalTagReleaseEvent). The
function is shaped as the future unified index — the same per-event
processing structure as the external index, with the event shape and
blob-key prefixes already shared — so the external plane’s unification
(retiring external-index-lambda into this function) remains a small,
well-bounded change.
Context: The internal index was SNS-triggered like sync, which forced it to read TAGS to learn a release’s canonicals — introducing the sync-fast/index-slow race (TAGS-absent → retry → DLQ) and requiring per-image error aggregation inside one invocation. The external plane had already solved invocation decoupling: its sync resolves tags, writes TAGS, then enqueues per-digest events to an index queue whose batch-item-failure semantics provide per-record retry for free. Downstream of sync’s writes the two planes are the same pattern; only the chain’s head differs (the internal producer is the near-realtime bus event, the external one a 6-hour poll).
Alternatives: (a) Keep the SNS trigger and heal the TAGS race with retries — the race class stays, one slow image can block the record, and the two index functions keep divergent shapes. (b) Unify into a single shared deployment rather than two deployments of one function — rejected for blast-radius isolation: a bad deployment would take out both planes at once. (c) Defer the convergence to the external-plane MR — rejected: every week of divergence adds code that both index functions must carry, and the queue convergence was already decided for the internal plane on its own merits.
Rationale: Queue decoupling deletes the TAGS-race class by construction:
the event is emitted after the writes it depends on and carries everything
the index needs (canonicals, digest, architectures), so the index never
reads TAGS and canonical identity is re-resolved against the registry. SQS
batch-item failures replace hand-rolled per-image aggregation with the
platform’s own per-record retry. Sync redelivery is harmless — a duplicate
event hits the index’s dedup skip and flows into the recovery path, which is
the recovery’s designed purpose. The canonical re-verification stays
regardless of ordering: the moved-digest mid-scan race is real in any
invocation shape. Shaping the internal index as the unified function now
(shared event type, external’s processing structure, env-configured
prefixes) keeps MR4 small: point the external queue at the function and
delete external-index-lambda, plus four parameterizations the planes
still differ on — the DDB partition key (IMAGE# vs EXTERNAL#), the
pull-reference source (config.pull_registry vs the raw registry
reference), the arch source (manifest list via fetch_arch_attestations
vs the event’s architectures), and the Hermeto-merge gating
(classify_by_hermeto from DISTRO vs none). The event shape and the
blob-key prefixes are already zero-code.
4.5.20 - Hummingbird Agent
An event-driven LLM agent that investigates CI/CD failures and posts findings as GitLab merge request notes. The agent executes markdown-defined workflows using tool calling, with all data processing running inside an isolated sandbox container.
Model adapters, provider calls, and shared AI usage metrics come from the
in-process hummingbird-ai library.
For the architectural design rationale, module boundaries, security invariants, and design decision registry, see Agent Design. For agent turn flow and tool handling, see Agent Model Loop. Provider adapters and wire formats are documented in Hummingbird AI.
flowchart TD
Pipeline["Pipeline Fails"]
MREvent["MR Created / Updated"]
Slash["/hummingbird command"]
Pipeline -->|"event"| Agent
MREvent -->|"event"| Agent
Slash -->|"command"| Agent
subgraph Agent ["Hummingbird Agent"]
APIs["GitLab + Konflux +<br/>Testing Farm"]
Model["LLM<br/>(Gemini / Claude)"]
subgraph Sandbox ["Isolated Sandbox (no network)"]
Tools["jq / python3 / yq<br/>data processing"]
end
APIs <-->|"data"| Model
Model <-->|"commands"| Tools
end
Note["MR Note<br/>(analysis or review)"]
Agent -->|"posts"| Note
Note -->|"reply to continue"| Agent
Features
- Workflow-driven analysis - Investigation logic lives in
.mdfiles, not in code; easy to iterate without redeployment - Centralized YAML config - A single config file defines operational settings, workflows, enabled data sources with token env var names, project allowlists, and per-project limits
- Sandboxed execution - All untrusted commands (
jq,python3, shell) run in an isolated container, never on the host - Five sandbox backends - Podman for local development (network-isolated), direct K8s pod creation, Deployment-backed pod pool for low-latency production use (restricted-v2 SCC compliant), KubeVirt VM (direct), or VMIRS-backed VM pool for heavier workloads requiring full OS isolation
- Data source abstraction - GitLab, Konflux, and Testing Farm are registered as tool-calling functions the model invokes directly
- Auto-spill for large outputs - Stdout/stderr and data source responses exceeding 4 KB are automatically saved to sandbox files with a compact preview returned to the model, keeping context usage bounded
- Prompt caching - Gemini uses implicit server-side caching automatically; Claude uses explicit sliding-window cache breakpoints that reduce input token costs by ~80% on multi-turn agent runs
- Token budget management - Per-call context ceiling and iteration-based soft/hard limits prevent runaway sessions
- Session persistence - Conversation history, transcript, and sandbox files saved to S3 (production) or local directory (development) for debugging and future session resumption. Sessions are stored in a provider-neutral format, enabling model switching between conversations.
Architecture
flowchart LR
subgraph input [Input]
SQS["SQS Queue"]
CLI["CLI --event"]
end
subgraph agentLoop [Agent Loop]
WF["Workflow .md<br/>(system prompt)"]
LLM["LLM<br/>(Gemini / Claude)"]
TR["Tool Registry"]
end
subgraph tools [Tools]
SE["sandbox_exec"]
FTS["fetch_to_sandbox"]
DS["Data Sources"]
end
subgraph sandbox [Sandbox Container]
JQ["jq / python3 / yq"]
Files["Spilled files"]
end
subgraph external [External APIs]
GL["GitLab API"]
KX["Konflux K8s"]
TF["Testing Farm"]
end
subgraph output [Output]
Note["GitLab MR Note"]
Session["Session State"]
end
SQS --> WF
CLI --> WF
WF --> LLM
LLM -->|"tool calls"| TR
TR --> SE --> sandbox
TR --> FTS --> sandbox
TR --> DS
DS --> GL
DS --> KX
DS --> TF
DS -->|"auto-spill"| sandbox
LLM -->|"final text"| Note
LLM --> Session
An event (CLI --event JSON or SQS message) identifies a GitLab project and
MR IID. The config file maps project paths to workflows and provides
action, max_iterations, enabled data sources, and token env var names.
Both run and serve use this config. The markdown body of the prompt file
becomes the LLM system prompt. The agent
loop iterates until the model produces a final text response or hits the
iteration/context limit. Tool calls are dispatched through the ToolRegistry:
sandbox_exec runs shell commands, fetch_to_sandbox pipes data source
output into the sandbox, and direct data source calls return results inline
(or auto-spill large responses to files).
For a detailed walkthrough of the model loop – what gets sent to the model each iteration, how tool calls flow, the exact wire format, and how user replies integrate for session resumption – see Agent Model Loop.
Event-Driven Triggers
In production, the agent consumes events from an SQS queue subscribed to the central SNS topic. The SNS filter policy delivers three event types:
-
gitlab::pipeline– Fires when a GitLab CI pipeline completes. The agent triggers onstatus=failedpipelines frommerge_request_eventsources where the triggering user has at least Developer access on the project. Only workflows withtrigger: pipelineare executed. Since the pipeline stays open until all Konflux external stages resolve, this naturally waits for all builds and tests to finish before triggering. -
gitlab::merge_request– Fires on MR open and update events. Only workflows withtrigger: merge_requestare executed. The agent triggers in three cases: a new MR is opened (action=open), code is pushed to an existing MR (action=updatewitholdrev), or a draft MR is marked as ready (action=updatewith achanges.drafttransition fromtruetofalse). All other update events (labels, assignees, description changes) are skipped – they carry no new code. Draft MRs are skipped; SHA-based deduplication prevents reviewing the same code revision twice. Access checks andtrigger_rulesfiltering evaluate the push author: for open and push events the webhook user is the pusher; for undraft transitions the agent resolves the actual pusher from MR system notes so that the person undrafting cannot bypass the access check on code pushed by an untrusted user. -
gitlab::note– MR comment events. Two sub-flows:- Slash command (
/hummingbird <workflow-name>): triggers a specific workflow. Prefix matching is supported (e.g./hummingbird analyzematchesanalyze-failures)./hummingbirdor/hummingbird helplists available workflows. The note author must have Developer+ access on the project. Optional runtime overrides can be appended:/hummingbird code-review model=claude-sonnet-4-6 max_iterations=25. - Reply to agent note: when a user replies to an existing agent note
(which contains a session marker), the agent loads the previous session
from S3 (conversation history + sandbox files) and continues the
conversation with the user’s reply as input. The system prompt includes
a
CONTINUATION_PROMPTthat prevents the model from re-running the full workflow. If the session is not found (expired/deleted), the agent falls back to a cold start. Replies may include overrides on a separate line (e.g./hummingbird model=claude-opus-4-6); override lines are stripped from the user message. Overrides persist in the session until explicitly changed.
Notes generated by the agent itself (containing session markers) are skipped to prevent infinite loops. Reply threading respects the
internal_notesconfig: if the project requires internal notes but the original thread was public, the reply is posted as a new top-level internal note instead. - Slash command (
Both pipeline and merge_request triggers apply per-workflow trigger
rules filtering. Each workflow may define an ordered trigger_rules
array; the first matching rule decides whether the event is allowed or
denied (implicit deny on fallthrough). Rules can match on pipeline
status, user_regex/user_regexes, branch_regex/branch_regexes,
and title_regex/title_regexes. All conditions within a rule are ANDed;
multiple patterns within a regex field are ORed. The ! prefix negates a
pattern. If no trigger_rules are specified, merge_request triggers
allow all events and pipeline triggers default to allowing only failed
status (preserving backward compatibility).
The legacy ignore_users and ignore_branches fields are still accepted
and automatically desugared into equivalent trigger_rules (deny +
catch-all allow). They cannot be mixed with explicit trigger_rules.
Rate limiting is per-workflow: each workflow’s thread count is tracked
independently via JSON session markers that embed the workflow name and
commit SHA. The max_runs_per_mr limit applies separately to each
workflow on a given MR.
Events flow through a two-stage SQS pipeline. A slim ingress router
forwards webhooks from the standard queue to an SQS FIFO queue, grouped
by discussion_id for note events (ensuring same-discussion ordering)
and by SQS MessageId for pipeline/MR events (no serialization needed).
Phase 1 handlers validate the event and resolve the session, then post a
continuation message back to the FIFO grouped by session_id. Phase 2
picks up the continuation and runs the workflow. This ensures all work
for a given session is serialized even across multiple discussion threads.
The SQS infrastructure is defined in template.yaml (SAM/CloudFormation):
a standard ingress queue (60s visibility timeout for the router hop) and
a FIFO work queue (30-minute visibility timeout for workflow execution).
Model Configuration
The agent supports multiple LLM providers via a unified adapter interface.
Gemini
- API key (local dev) – Set
GOOGLE_API_KEY. Callsgenerativelanguage.googleapis.comdirectly. No region configuration needed. - Vertex AI (production) – Set
GOOGLE_CLOUD_PROJECTand configuremodel_regionsin the YAML settings. Uses Application Default Credentials (ADC) viagoogle-auth. For OpenShift, mount a service account key JSON file and setGOOGLE_APPLICATION_CREDENTIALS, or use workload identity.
Gemini uses implicit server-side caching – repeated prefixes are automatically cached by the Vertex AI backend with no opt-in required. Cached input tokens are billed at 25% of the base input rate. The agent tracks cached token counts from API responses for cost estimation.
Anthropic Claude
Claude models are accessed via Vertex AI using the same GCP project
(GOOGLE_CLOUD_PROJECT) with the region resolved from model_regions.
Enable the desired model in the GCP Model Garden. Use
model: claude-sonnet-4-20250514 in workflow config.
Prompt caching is enabled automatically for Claude models. The agent places explicit cache breakpoints on messages so that the full conversation prefix is served from cache on every turn after the first. Cache writes cost 1.25x the base input rate; cache reads cost 0.10x (90% discount). For a typical 20-iteration agent run, this reduces input token costs by approximately 80%. Ephemeral messages (iteration warnings, nudges) are excluded from cache writes to avoid polluting the cache with transient content.
Region configuration
GCP regions are configured via settings.model_regions in the config file –
a map of model name prefixes to GCP regions. The agent resolves the region by
longest-prefix match on the effective model name (same algorithm as cost
estimation). Example:
settings:
model_regions:
gemini-2.5-pro: us-east5
gemini-3.1-pro: global
claude: global
At least one of GOOGLE_API_KEY or GOOGLE_CLOUD_PROJECT must be set. If both
are set, the API key takes precedence for Gemini models. Claude models always
require Vertex AI mode (GOOGLE_CLOUD_PROJECT + model_regions);
GOOGLE_API_KEY direct mode is for Gemini only.
Prerequisites
- Python 3.11+
- Podman for local sandboxes, or
kubectland Kubernetes access to a dedicated sandbox namespace for K8s sandboxes. - Authentication:
GOOGLE_API_KEYorGOOGLE_CLOUD_PROJECT(see above) - For Claude models: enable the desired model in GCP Model Garden
- GitLab tokens referenced in the config file (model tool tokens, orchestrator tokens)
- Kubeconfig with access to the Konflux cluster (if using Konflux data sources)
Installation
cd hummingbird-agent
pip install -e .
Usage
Local development (run)
The run command uses the config file for workflow lookup, data source
registration, and token resolution – the same code path as serve. By
default, results are printed to stdout (dry-run). Use --execute to post
the result as a GitLab MR note.
There are two ways to select what to run:
Direct workflow selection (--workflow + --project + --event):
# Run a specific workflow on an MR, Podman sandbox, print to stdout
hummingbird-agent run \
--workflow analyze-failures \
--project org/group/project \
--event '{"iid": 123, "sha": "abc123"}'
# Same but with K8s sandbox
hummingbird-agent run \
--workflow analyze-failures \
--project org/group/project \
--event '{"iid": 123, "sha": "abc123"}' \
--context my-cluster/my-namespace
# Post the result as a GitLab MR note (also saves session to S3 if configured)
hummingbird-agent run \
--workflow analyze-failures \
--project org/group/project \
--event '{"iid": 123, "sha": "abc123"}' \
--execute
# Save session locally for debugging (context.json, transcript.md, sandbox.tar.gz)
hummingbird-agent run \
--workflow analyze-failures \
--project org/group/project \
--event '{"iid": 123, "sha": "abc123"}' \
--save-session /tmp/my-session
# Resume from a saved session with a follow-up question
hummingbird-agent run \
--workflow analyze-failures \
--project org/group/project \
--event '{"iid": 123, "sha": "abc123"}' \
--resume-session /tmp/my-session \
--message "Can you look at the clair-scan timeout more closely?"
# Chain: resume and save the new session for another round
hummingbird-agent run \
--event-file event.json \
--resume-session /tmp/my-session \
--message "What layer hash failed?" \
--save-session /tmp/my-session-2
Event-file replay (--event-file): routes the event through the same
project-index lookup as serve, but skips rate limiting and status filtering:
# Replay a real webhook event, dry-run with Podman
hummingbird-agent run \
--event-file event.json
# Replay with K8s sandbox and post notes
hummingbird-agent run \
--event-file event.json \
--context my-cluster/my-namespace \
--execute
Production (serve)
# Poll SQS queue for events (pool sandbox by default, posts results as MR notes)
CONFIG_PATH=config.yml hummingbird-agent serve
Requires CONFIG_PATH pointing to a config file with settings.sqs_queue_url
and settings.sqs_fifo_queue_url set. The config file defines which workflows
run on which projects, with per-project limits and data source token mappings.
Handles SIGTERM/SIGINT for graceful shutdown. A background router thread
forwards events from the standard queue to the FIFO; the main thread consumes
the FIFO with a semaphore-gated thread pool (controlled by
settings.max_concurrent_agents) so excess messages stay in SQS for other
instances.
Config hot-reload: In serve mode a background thread polls the config file for changes (every 5 seconds by default). When the file changes, the new config is validated and atomically swapped in – subsequent event dispatches use the updated config. If the new config is invalid, the previous config is kept and a warning is logged. No restart required for config changes.
The serve command also accepts --sandbox, --context, and --namespace
for local development with a different sandbox backend (e.g. Podman).
CLI Options
run subcommand:
| Option | Description | Default |
|---|---|---|
--event |
Inline event JSON string (mutually exclusive with --event-file) |
- |
--event-file |
Read event from a JSON file (mutually exclusive with --event) |
- |
--workflow |
Workflow name from config file (requires --project) |
- |
--project |
GitLab project path (requires --workflow) |
- |
--execute |
Post result as GitLab MR note and save session to S3 | - |
--save-session |
Save session artifacts to this directory | - |
--resume-session |
Resume from a saved session directory (requires --message) |
- |
--message |
Follow-up message for session resumption (requires --resume-session) |
- |
--sandbox |
Sandbox backend (podman, k8s, k8spool, kubevirt, or kubevirtpool) |
podman |
--context |
K8s context (implies K8s backend) | - |
--namespace |
K8s namespace | - |
-v, --verbose |
Enable debug logging | - |
serve subcommand:
| Option | Description | Default |
|---|---|---|
--sandbox |
Sandbox backend (podman, k8s, k8spool, kubevirt, or kubevirtpool) |
k8spool |
--context |
K8s context (implies K8s backend) | - |
--namespace |
K8s namespace | - |
-v, --verbose |
Enable debug logging | - |
Configuration
Config file
Both run and serve use a single YAML config file. Set the path via
CONFIG_PATH (default: config.example.yaml). The config has two sections:
settings for operational parameters, and workflows for workflow
definitions. Most settings have defaults; configure the sandbox image and
namespace as needed for the selected backend.
Sandbox options for settings.sandbox are documented by the shared
Hummingbird Sandbox library.
The dedicated agent execution image is not yet published or selected in runtime configuration. See sandbox images for build and rollout details.
settings:
gitlab_url: https://gitlab.com # GitLab instance URL
sandbox: {}
max_concurrent_agents: 4 # max concurrent workflows (serve)
sqs_queue_url: "" # standard SQS ingress queue URL (serve)
sqs_fifo_queue_url: "" # FIFO work queue URL (serve)
s3_session_bucket: "" # S3 bucket for session persistence
model: gemini-3.1-pro-preview # or claude-sonnet-4-20250514
model_regions: # model prefix -> GCP region
gemini-2.5-pro: us-east5
gemini-3.1-pro: global
claude: global
max_iterations: 30 # default iteration limit
max_runs_per_mr: 5 # default per-MR rate limit
internal_notes: true # default note visibility
docs_url: https://gitlab.com/org/group/project/-/blob/main/docs/agent.md
source_url: https://gitlab.com/org/group/project
slack_url: https://slack.example.com/archives/C0123456789
slack_label: "#my-channel"
workflows:
code-review:
trigger: merge_request # auto-trigger on MR events
description: Performs AI-powered code review
sandbox: k8spool # per-workflow sandbox override (optional)
workflow_url: https://gitlab.com/org/group/project/-/blob/main/workflows/code-review.md
trigger_rules: # ordered rule chain, first match wins
- user_regex: "renovate\\[bot\\]" # deny bot MRs (regex fullmatch)
action: deny
- branch_regex: "chore/.*" # deny maintenance branches
action: deny
- action: allow # allow everything else
prompt: workflows/code-review.md
action: post_gitlab_note
model: gemini-3.1-pro-preview # or claude-sonnet-4-20250514
max_iterations: 15
max_inline_size: 200000 # keep full diffs in context
context_limit: 500000 # Gemini 3.1 Pro / Claude Sonnet 4 have large context windows
data_sources:
gitlab:
token_env: GITLAB_TOKEN_RO
projects:
org/group/project: {}
analyze-failures:
trigger: pipeline # auto-trigger on failed pipelines
description: Investigates CI/CD pipeline failures
sandbox: kubevirtpool # use VM sandbox for heavier workloads
auto_resolve_on_push: true # resolve threads when a new SHA is pushed
auto_resolve_on_success: true # resolve threads when pipeline succeeds
workflow_url: https://gitlab.com/org/group/project/-/blob/main/workflows/analyze-failures.md
prompt: workflows/analyze-failures.md # relative to config file dir
action: post_gitlab_note
model: gemini-3.1-pro-preview # or claude-sonnet-4-20250514
max_iterations: 50 # per-workflow iteration override
data_sources: # model tool tokens (read-only)
gitlab:
token_env: GITLAB_TOKEN_RO # env var name, not the token
konflux:
cluster_url: https://example.com:6443/ns/my-tenant
kubeconfig_env: KUBECONFIG
kubearchive_url: https://kubearchive-api-server-product-kubearchive.apps.example.com
testing_farm: {}
projects:
redhat/hummingbird/containers:
tokens: # per-project model token overrides
gitlab: GITLAB_TOKEN_CONTAINERS_RO
action_tokens: # per-workflow write tokens
gitlab: HUMMINGBIRD_AGENT_ACTION_ANALYZE_FAILURES_GITLAB_TOKEN_CONTAINERS
With --workflow/--project, the workflow and project are looked up directly
in the config. With --event-file, the project is extracted from the event
body and matched against the project index to find applicable workflows.
Pool overrides in this example require the corresponding pool to be provisioned
and selected with --sandbox at startup. Only the CLI-selected pool is
initialized, so these two pool overrides cannot both be used in one service.
Without a matching pool, a workflow can fall back to a direct sandbox instead.
Discussion threads
All workflow results are posted as discussion threads: a placeholder note starts the discussion and the full result is posted as a reply. The placeholder is never edited, so email notifications include the actual result text. For slash commands, the result is posted as a reply in the triggering discussion.
Auto-resolve
Workflows can opt in to automatic resolution of their discussion threads:
auto_resolve_on_push(defaultfalse): When a new commit is pushed to the MR (i.e. amerge_requestevent withaction: update), all agent discussion threads for the workflow whose SHA differs from the new HEAD are resolved. This clears stale failure analyses when the developer pushes a fix.auto_resolve_on_success(defaultfalse): When the head pipeline succeeds, all agent discussion threads for the workflow on that MR are resolved (regardless of SHA). This handles pipeline reruns on the same SHA where a transient failure is now green.
Both flags are independent and can be combined. Resolution runs before rate limit checks, so threads are resolved even if the workflow’s per-MR run limit has been reached.
Web search
Workflows can enable Anthropic’s built-in web search server tool by
listing web_search as a data source:
workflows:
renovate-babysit:
model: claude-sonnet-4-20250514
data_sources:
gitlab: {}
web_search: {}
# ...
When web_search is present in data_sources, the web_search_20250305
server tool is included in API requests to Claude. The model decides
autonomously when to search. Search execution happens server-side (no
client-side tool dispatch), and results appear as server_tool_use /
web_search_tool_result content blocks in the response. These blocks
are preserved through session save/resume.
Unlike other data sources, web_search has no configuration options and
does not register any orchestrator-side tools – it is handled entirely
by the model provider.
If the API returns a pause_turn stop reason (server-side search loop
hit its iteration limit), the adapter automatically re-sends the
conversation to continue, up to 5 continuations per generate() call.
Web search is only supported with Claude models. The setting is ignored for Gemini.
Placeholder footer
The first agent-authored note (placeholder) includes a footer with links to documentation, source code, the Slack channel, the workflow prompt, and a continuation prompt. These links are configured via global settings:
| Setting | Description |
|---|---|
settings.docs_url |
Link to agent documentation |
settings.source_url |
Link to agent source repository |
settings.slack_url |
Link to support Slack channel |
settings.slack_label |
Display text for Slack link (default: “Slack”) |
Per-workflow, set workflow_url to link to the workflow’s prompt file.
The footer stays on the placeholder and the result is a separate reply.
Token separation
Tokens are split into three tiers that never mix:
- Model tool tokens (in YAML
data_sources/tokens): read-only tokens passed to the LLM’s tool calls. Declared in the config file as env var names. These are user-defined and resolved at runtime from the referenced env vars. Create as project access tokens with Reporter role andread_apiscope. Reporter is the minimum role required to see internal (confidential) notes in the discussions tool. - Workflow action tokens (in YAML
action_tokens, per-project): write tokens used by the orchestrator for per-workflow GitLab writes (notes, thread resolution). Each workflow gets a dedicated bot user per project, providing clear audit trails for which agent produced each note. Declared in the project config as env var names. Create as project access tokens with Developer role andapiscope. These tokens are never exposed to the LLM and never enter the ToolRegistry. Naming convention:HUMMINGBIRD_AGENT_ACTION_<WORKFLOW>_GITLAB_TOKEN_<PROJECT>. - Orchestrator tokens (
ORCHESTRATOR_*env vars, NOT in YAML): tokens used by the runner for operational reads (member access checks, push author lookup, head pipeline queries) and infrastructure notes (access-denied replies, rate-limit notices). Create as project access tokens with Developer role andread_apiscope (orapifor infrastructure notes). Developer role is required because the discussions tool trust-filters notes by author access level (>= Developer); if the orchestrator bot has only Reporter access, its own notes are redacted. Resolved by convention:ORCHESTRATOR_GITLAB_TOKEN_<MANGLED_PROJECT>(per-project) orORCHESTRATOR_GITLAB_TOKEN(fallback). TheORCHESTRATOR_prefix makes these impossible to confuse with model tokens.
Environment Variables
The agent reads only secrets and authentication from environment variables.
All operational settings come from the config file’s settings section.
| Variable | Required | Default | Description |
|---|---|---|---|
CONFIG_PATH |
no | config.example.yaml |
Path to config YAML |
GOOGLE_API_KEY |
yes* | - | Gemini API key; Gemini direct mode only (not for Claude) |
GOOGLE_CLOUD_PROJECT |
yes* | - | GCP project ID (Vertex AI mode); required for Claude models |
ORCHESTRATOR_GITLAB_TOKEN |
serve | - | Orchestrator GitLab token (global fallback) |
ORCHESTRATOR_GITLAB_TOKEN_<PROJECT> |
no | - | Per-project orchestrator token |
SENTRY_DSN |
no | - | Sentry DSN for error tracking |
*One of GOOGLE_API_KEY or GOOGLE_CLOUD_PROJECT is required. Claude models
require Vertex AI (GOOGLE_CLOUD_PROJECT); GOOGLE_API_KEY is for Gemini
direct mode only.
Model tool tokens (e.g. GITLAB_TOKEN_RO, GITLAB_TOKEN_CONTAINERS_RO) and
data source credentials (e.g. KONFLUX_CLUSTER_URL, KUBECONFIG) are
referenced by name in the config file’s data_sources and tokens sections.
They are not listed in the table above because their names are user-defined.
Sandbox Security
Sandbox isolation, security constraints, permissions, and backend behavior are documented in the shared Hummingbird Sandbox library. The agent continues to own workflow selection, CLI options, session persistence, and the decision to acquire or reuse a sandbox.
Data Sources
Data sources are registered as tool-calling functions. The model invokes them by name; the orchestrator executes them and returns results (or auto-spills large responses to the sandbox).
GitLab
| Tool | Description |
|---|---|
gitlab_get_mr_details |
MR metadata (title, author, state, SHA, labels) |
gitlab_get_mr_unified_diff |
Complete unified diff in patch format |
gitlab_get_mr_diff |
Per-file structured change data |
gitlab_get_mr_commits |
List of commits in a merge request |
gitlab_get_mr_discussions |
Discussion threads with redacted agent transcripts and trust-filtered comments |
gitlab_get_commit_statuses |
CI/CD pipeline statuses for a commit |
gitlab_get_file_at_ref |
Raw file content at a git ref |
gitlab_get_repo_archive |
Repository tar.gz (binary, auto-spilled) |
gitlab_get_job_log |
CI job trace output (ANSI codes stripped) |
Konflux
Fetches Tekton PipelineRuns and TaskRuns from both the live K8s API and Kubearchive (for completed resources), with deduplication by UID.
| Tool | Description |
|---|---|
konflux_list_pipelineruns |
PipelineRuns for a commit SHA; optional status allowlist filter |
konflux_list_taskruns |
TaskRuns by commit SHA or parent PipelineRun name; optional status filter |
konflux_list_pods |
Pods for a commit SHA; optional status allowlist filter |
konflux_get_pod |
Full pod resource (spec, status, conditions, container statuses) |
konflux_get_pod_log |
Pod logs; optional container param, fetches all containers when omitted |
Status filtering: All three list tools accept an optional status
parameter — an allowlist of status.conditions[0].status values
("True" = success, "False" = failure, "Unknown" = in-progress).
Only items matching the allowlist are written to the sandbox file. Response
metadata always includes counts (all items seen, by reason), included,
and excluded — the agent knows the full reason breakdown even when fetching
only failures. Items without conditions always pass.
Per-PLR TaskRun fetching: konflux_list_taskruns accepts a pipelinerun
parameter. When set, it uses tekton.dev/pipelineRun=<name> as a server-side
label selector, returning only the ~15-21 TaskRuns for that PipelineRun (~600 KB)
instead of all TaskRuns for the commit (~530 MB for 340 components).
Recommended pattern for analyze-failures: Fetch failing PLRs and TaskRuns
together in one batch with status=["False"]. This captures every failure
reason (Failed, Cancelled, PipelineRunTimeout, StepFailed, etc.)
without enumerating them. Use response metadata counts for the summary.
Use per-PLR fetch only when deeper investigation is needed.
Response metadata includes konflux_ui base URL for building reviewer-facing
links.
Testing Farm
| Tool | Description |
|---|---|
tf_get_results |
JUnit XML results for a request ID |
tf_get_test_log |
Individual test log by URL (restricted to Testing Farm artifact URLs) |
tf_get_request_status |
Request state, queue/run times |
Response metadata includes artifacts_base URL for building artifact links.
Workflow System
Workflows are .md files whose content becomes the LLM system prompt
verbatim. Workflow metadata (action, model, max_iterations, enabled
data sources, project allowlists) is defined in the config file. The .md
file is pure system prompt text.
Available workflows:
analyze-failures.md- Investigates CI/CD pipeline failures by fetching MR details, identifying failed pipelines via commit statuses, retrieving PipelineRuns/TaskRuns from Konflux, analyzing test results from Testing Farm, and producing a grouped root-cause report with reviewer-facing URLs.code-review.md- Performs AI-powered code review by fetching MR details, the unified diff, and prior discussion threads in parallel, then attempting to load per-project rules fromworkflows/repo-rules/(a mandatory step – rules take precedence over generic standards when present) and producing structured feedback with severity ratings, code examples, and actionable suggestions. On follow-up reviews (after SHA updates), the agent sees its own previous findings, developer responses, and resolved threads – avoiding duplicate findings and respecting developer explanations. Uses elevatedmax_inline_size(200 KB) andcontext_limit(500K tokens) to keep the full diff in context.renovate-babysit.md- Triages a single Renovate-authored MR triggered by a successful pipeline. Classifies the MR as safe or risky based on diff scope, upstream dependency changes (from MR description, web search), and local codebase impact (viagitlab_get_repo_archive+grep). Posts a structured note with verdict, upstream change summary, and suggested actions for risky MRs. Requires a Claude model withweb_searchdata source.
Token Budget Management
Prompt caching, cached token accounting, and provider-specific cost estimates
are implemented in hummingbird-ai.
The agent uses a dual-limit approach instead of a cumulative token budget:
- Iteration limit (
settings.max_iterations, default 30, overridable per-workflow) - Hard cap on tool-calling rounds. A wrap-up prompt is injected at 80% (SOFT_ITERATION_RATIO). - Context limit (
CONTEXT_LIMIT, default 60,000 tokens, overridable per-workflow viacontext_limit) - Per-call input token ceiling. When exceeded, a wrap-up prompt forces the model to finalize.
Large outputs are automatically redirected to sandbox files to keep the LLM
context small. The spill threshold defaults to 4 KB (MAX_INLINE_SIZE) but
can be overridden per-workflow via max_inline_size in the config:
sandbox_exec- stdout/stderr exceeding the threshold saved to/tmp/_out/{N}.txt; model receives a preview (head + tail) with file path- Data sources - text exceeding the threshold saved to
/tmp/_out/{name}_{N}.txtwith preview; binary data saved to.bin fetch_to_sandbox- always writes to the caller-specified path; returns metadata only
All tuning constants are centralized in config.py:
| Constant | Default | Purpose |
|---|---|---|
DEFAULT_MAX_ITERATIONS |
30 | Hard iteration cap |
SOFT_ITERATION_RATIO |
0.8 | Inject wrap-up at this fraction |
CONTEXT_LIMIT |
60,000 | Per-call input token ceiling |
OUTPUT_PREVIEW_BYTES |
4,096 | Preview size for spilled outputs |
OUTPUT_TAIL_BYTES |
512 | Extra tail appended to previews |
MAX_INLINE_SIZE |
4,096 | Max inline size for data source responses |
Evaluation Framework
The agent includes an evaluation framework for scientific measurement of
prompt and model quality. See
Hummingbird Agent Evals for the full guide
covering the shared hummingbird_agent.eval library, existing
evaluations, and how to write new ones.
Development
See the main README for development workflows.
make hummingbird-agent/setup # Install dependencies
make check # Lint code (ruff)
make fmt # Format code
make test # Run unit tests
make coverage # Run tests with coverage
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.
4.5.20.1 - Hummingbird Agent Model Loop
How the Hummingbird agent runs conversation turns, calls tools, and integrates user replies.
For the architectural design rationale behind the loop (why full history replay, why iteration-based budgets, why nudge via system prompt), see Agent Design – Section 6. For operational usage documentation, see Hummingbird Agent. Provider adapters, wire formats, canonical message conversion, and caching are documented in Hummingbird AI.
What gets sent to the model
Every call to the model API sends three things:
1. System instruction
A single text string, rebuilt every iteration. The adapter maps it to the active provider’s request format. It is composed of:
BASE_SYSTEM_PROMPT # agent.py: sandbox rules, tool usage tips
+ tool_notes # per-data-source notes from ToolRegistry
+ workflow_prompt # full content of e.g. workflows/analyze-failures.md
Near the iteration or context limit, a warning suffix is appended to the system prompt for that call. The suffix escalates through three levels:
ITERATION_WARNING/CONTEXT_WARNING– soft “start wrapping up” at 80% of the iteration or context limit. Tools remain available.FINAL_TURN_WARNING– hard stop on the absolute last iteration or after the context limit is exceeded. Combined with tool-calling disabled for that request to mechanically prevent further tool calls.EMPTY_RESPONSE_NUDGE– appended when the model returns an empty response (no text, no tool calls). Tools remain available.
2. contents
A list of messages representing the full conversation history. It grows every iteration. Provider-specific content conversion is handled by the adapter; see Hummingbird AI.
3. tools
Tool definitions, static across all iterations. Adapters translate the shared tool definition into the provider’s schema.
4. Tool availability (conditional)
On the final turn (last iteration or after context limit exceeded), the request disables tools to mechanically prevent function calls. The active adapter applies the provider-specific setting.
This is only sent when FINAL_TURN_WARNING is active. All other iterations
omit this constraint, allowing the model to choose freely between text and tools.
The agent loop, turn by turn
The loop in run_agent_loop runs up to max_iterations times. Each
iteration is one round-trip to the model.
Initialization
system_prompt = BASE_SYSTEM_PROMPT + tool_notes + workflow_prompt
tool_defs = [sandbox_exec, fetch_to_sandbox, fetch_batch_to_sandbox, <data sources...>]
contents = [user: {"project": "org/repo", "iid": 42, "sha": "abc...", "session_id": "uuid"}]
Iteration 1
-> Send: system_instruction + contents (1 item) + tool_defs
<- Response: tool_call(gitlab_get_mr_details, {project: "org/repo", iid: 42})
contents.append(model: response.raw_content) # opaque provider content
execute tool -> result = {"result": "{\"title\": \"Fix auth\",...}"}
contents.append(tool: model.make_tool_responses([(name, result)]))
Iteration 2
-> Send: system_instruction + contents (3 items) + tool_defs
<- Response: tool_call(fetch_batch_to_sandbox, {requests: [...]})
contents.append(model: ...)
execute tool -> result = {"results": [{"saved_to": "/tmp/data/pipelineruns.json", "bytes": 85432}]}
contents.append(tool: ...)
Iteration 3
-> Send: system_instruction + contents (5 items) + tool_defs
<- Response: tool_call(sandbox_exec, {command: "jq '[...]' /tmp/data/pipelineruns.json"})
contents.append(model: ...)
execute tool -> result = {"exit_code": 0, "stdout": "...(preview)...", "stdout_file": "/tmp/_out/0.txt"}
contents.append(tool: ...)
Iteration N (final)
-> Send: system_instruction + contents (2N-1 items) + tool_defs
<- Response: text("## Konflux Failure Analysis\n\n...") // text + no tool calls = DONE
contents.append(model: {text: "## Konflux Failure Analysis..."})
BREAK -- return (text, usage, transcript, contents)
Response handling and termination
Each iteration, the model’s response can contain text, tool calls, both, or neither. The agent handles each case:
| Response | Action |
|---|---|
| Text only | Final response. Save text, break. (Happy path.) |
| Text + tool calls | Save text as last_text fallback, execute the tool calls, continue loop. The model is thinking aloud while also acting. |
| Tool calls only | Execute the tool calls, continue loop. |
| Neither | Empty response – retry up to MAX_EMPTY_RETRIES (2) times. Each retry nudges the model via EMPTY_RESPONSE_NUDGE appended to the system prompt (not injected as a user message, to keep the contents list clean). Tools remain available during nudge retries. |
The text + tool calls case is worth noting: the model’s text is not
returned to the user immediately. It is stored as last_text (a fallback in
case the loop terminates later without a clean final text, e.g. by hitting max
iterations). The provider response is appended to contents unchanged; its
wire format is documented in Hummingbird AI.
The loop terminates on:
- Text only – the model produced a final response.
- Max iterations reached – the final iteration uses
FINAL_TURN_WARNINGand disables tools through the adapter to force text output. Falls back to whateverlast_textwas seen, or"[Agent did not produce a final response]". - Context limit exceeded –
input_tokens >= CONTEXT_LIMITsetscontext_exceeded, making the next iteration final (same as #2). A soft warning (CONTEXT_WARNING) fires earlier at 80% of the limit. - Fatal model error – a provider error after shared retry handling is exhausted. Error classification and retries are described in Hummingbird AI.
- Empty responses exhausted –
MAX_EMPTY_RETRIES(2) nudge retries failed. - Unexpected error – any exception not handled by the retry mechanism (e.g. malformed API response) breaks the loop and returns partial results. Accumulated contents, transcript, and sandbox are preserved and the session is saved normally.
Budget escalation
Both iterations and context size use the same two-tier pattern:
| Soft warning (80%) | Hard stop (100%) | |
|---|---|---|
| Iterations | ITERATION_WARNING at 80% of max_iterations |
FINAL_TURN_WARNING on last iteration |
| Context | CONTEXT_WARNING at 80% of CONTEXT_LIMIT |
FINAL_TURN_WARNING on next iteration after exceeding CONTEXT_LIMIT |
Soft warnings are advisory (“start wrapping up”) and keep tools available. The hard stop disables tools through the active adapter to mechanically prevent further tool calls, ensuring the model produces text.
The contents list in detail
The agent treats each entry in contents as opaque adapter content during
execution. Provider representations and canonical conversions are documented
in Hummingbird AI.
model.make_user_content() constructs the initial event turn and
model.make_tool_responses() appends results for dispatched calls. The agent
stores the provider response as raw_content and appends it unchanged.
Typical 8-iteration conversation shape
contents[0] = user: {"project":"org/repo","iid":42,...} # initial event
contents[1] = model: tool_call(gitlab_get_mr_details) # iter 1
contents[2] = tool: result(gitlab_get_mr_details) # iter 1
contents[3] = model: tool_call(fetch_batch_to_sandbox) # iter 2
contents[4] = tool: result(fetch_batch_to_sandbox) # iter 2
contents[5] = model: tool_call(sandbox_exec) # iter 3
contents[6] = tool: result(sandbox_exec) # iter 3
...
contents[13] = model: tool_call(sandbox_exec) # iter 7
contents[14] = tool: result(sandbox_exec) # iter 7
contents[15] = model: text("## Konflux Failure Analysis...") # iter 8 (final)
This entire list is persisted as canonical messages in context.json; the
adapter converts to and from its provider format at the persistence boundary.
See Hummingbird
AI for
details.
How tools are called
When the model’s response contains tool calls, _execute_tool_calls
iterates them sequentially:
tool_registry.execute(ToolCall(name, args))
match name:
"sandbox_exec" -> runs sh -c in container -> {exit_code, stdout, stderr}
"fetch_to_sandbox" -> calls data source func -> writes to sandbox file -> {saved_to, bytes}
"fetch_batch_to_sandbox" -> multiple fetch_to_sandbox in one call
any data source name -> calls func directly -> returns inline or auto-spills large output
Large output handling
When stdout or a data source response exceeds 4KB, it is automatically saved
to a sandbox file (/tmp/_out/N.txt) and only a preview (head + tail) is
returned to the model. The model gets the file path and can use sandbox_exec
with jq/grep/head to process it. This keeps the context window
manageable.
How a user reply integrates (session resumption)
When a user replies to an agent note on GitLab, the orchestrator resumes the conversation by restoring the previous state and appending the reply.
What gets restored from S3
context.json– the persisted conversation in canonical form; the active adapter converts it to the in-memory representation before the loop runssandbox.tar.gz–/tmp/_out/files (PipelineRun JSONs, test logs, jq output, etc.) restored into the new sandbox
Canonical storage allows sessions to resume with another provider; conversion details are in Hummingbird AI.
The resumed contents list
// Restored from context.json (previous run)
contents[0] = user: {"project":"org/repo","iid":42,...} # original event
contents[1] = model: tool_call(gitlab_get_mr_details) # iter 1
contents[2] = tool: result(gitlab_get_mr_details) # iter 1
... # all prior turns
contents[15] = model: text("## Konflux Failure Analysis...") # previous final text
// NEW: user reply appended
contents[16] = user: "Can you look at the clair-scan timeout more closely?"
The agent loop continues
ITERATION 1 (resumed):
-> Send: system_instruction (+ CONTINUATION_PROMPT) + contents[0..16] + tool_defs
<- Response: tool_call(sandbox_exec, {command: "grep -i timeout /tmp/data/..."})
^ using restored sandbox files
contents[17] = model: tool_call(sandbox_exec)
contents[18] = tool: result(sandbox_exec)
ITERATION 2 (resumed):
<- Response: text("The clair-scan timeout is caused by...")
contents[19] = model: text("The clair-scan timeout is caused by...")
BREAK
Cold start vs resumed – key differences
| Aspect | Cold start | Resumed |
|---|---|---|
| System prompt | BASE + tool_notes + workflow | BASE + tool_notes + workflow + CONTINUATION_PROMPT |
| First user message | {"project":..., "iid":...} |
{"project":..., "iid":...} (restored) |
| Conversation history | Empty (just the event) | Full prior conversation |
| Latest user message | – | "Can you look at the clair-scan timeout?" |
| Sandbox files | Empty | Restored from archive |
| Tool definitions | Same | Same |
| Session format | Adapter content in memory | Canonical on disk; adapter content after load |
The CONTINUATION_PROMPT
Without this, the workflow prompt (e.g. analyze-failures.md) tells the model
to follow a rigid workflow: Data.1, Data.2, Data.3, Analysis.1… The model
might try to re-run the entire analysis. The continuation prompt overrides
this:
## Continuation
This is a follow-up to a previous conversation. The conversation history
contains your prior analysis and tool calls. The user is replying with a
question or request about your previous work.
**Do NOT re-run the full workflow from scratch.** Instead:
- Respond directly to the user's question
- Use your tools to investigate further if needed (logs, data are still
available in the sandbox)
- Reference your previous findings where relevant
- Keep your response focused on what the user asked
The model now understands: “I already did the analysis (it’s all in the conversation history). The user has a specific question. Let me answer it.”
Iteration and token budget for resumed sessions
The resumed session gets a full fresh iteration budget. run_workflow
sets max_iterations from the workflow config (e.g. 50 for
analyze-failures), and the loop counter starts at 0 regardless of how many
iterations the previous session used. The model can make as many tool calls
as it needs to answer the follow-up.
The practical constraint is the context window, not iterations. The restored contents can be large – a typical 8-iteration cold start uses ~50-100K input tokens. Since the model re-counts the full history every iteration, the resumed session starts at roughly the token count where the previous session ended, plus the new user message. Each new tool call adds more. Token counts logged by the agent are billed tokens (cumulative across API calls; each call re-sends the full history, so these overlap).
The existing CONTEXT_LIMIT check still applies: at 80% of the limit, a
soft CONTEXT_WARNING tells the model to start wrapping up. If the limit is
exceeded, the next iteration becomes final (FINAL_TURN_WARNING with provider
tools disabled by the adapter).
For the typical case (user asks one focused follow-up, agent does 1-3 tool calls), this is fine. Multi-turn deep dives will eventually hit the context limit, at which point the agent wraps up – and the user can start a fresh conversation if needed.
Provider behavior
Provider wire formats, canonical message conversion, prompt caching, server-side
tools, and adapter continuations are owned by the
hummingbird-ai library.
The Hummingbird agent loop
remains responsible for tool dispatch, iteration/context budgets, and session
resumption.
4.5.20.2 - Hummingbird Agent Design
Architectural design document for the hummingbird-agent. Covers the reasoning behind every major design choice so that future changes can be made safely, without accidentally violating invariants that hold the system together.
For operational usage (CLI, config reference, deployment), see Hummingbird Agent. For agent turn flow and tool handling, see Agent Model Loop. Provider adapters and wire formats are documented in Hummingbird AI.
1. Design Philosophy
Five principles shaped the agent’s architecture. Every component traces back to at least one of these.
Security by default. The agent processes untrusted merge requests. All LLM-driven commands run inside isolated sandbox containers with no network and no credentials. Orchestrator write tokens and model read tokens live in separate code paths that never cross. No cluster-admin is required.
Config over code. Investigation logic lives in markdown workflow files that become the LLM system prompt verbatim – changing the analysis strategy requires no code change and no redeployment. Operational settings (data sources, project allowlists, iteration limits) live in a YAML config file that is auditable and committable. Secrets are the only thing in environment variables.
Bounded cost. Every output path has a size cap. Large tool outputs are auto-spilled to sandbox files with only a preview returned to the model. Each session has both an iteration limit and a per-call context token limit with two-tier escalation (soft warning, then hard stop with tool disabling). The relationship between these constants is documented and centralized.
Partial over nothing. When some data is unavailable (expired pod logs, unreachable Konflux cluster, Testing Farm 404), the agent continues with whatever data it has and notes the gap in its output. A partial report is more valuable than a crash.
Model-agnostic agent loop. The agent loop (agent.py) does not inspect
the internal structure of the contents list. It appends raw_content from
model responses and make_tool_responses() output without looking inside.
Each model backend owns its wire format. This makes it possible to add new
model backends (Claude, GPT) without touching the agent loop.
2. System Architecture
2.1. Component overview
flowchart TB
subgraph input [Event Sources]
SQS["SQS Queue<br/>(gitlab::pipeline, gitlab::note)"]
CLI["CLI<br/>(--event / --event-file)"]
end
subgraph orchestrator [Orchestrator Process]
Events["events.py<br/>SQS consumer"]
Runner["runner.py<br/>Event routing"]
Agent["agent.py<br/>Model loop"]
Tools["tools.py<br/>Tool registry"]
Actions["actions.py<br/>GitLab notes"]
Sessions["sessions.py<br/>Persistence"]
WfConfig["workflow_config.py<br/>YAML config"]
end
subgraph ai [Shared AI Library]
Models["hummingbird-ai<br/>Gemini / Anthropic adapters"]
end
subgraph sandbox [Sandbox Container]
SB["PodmanSandbox / K8sSandbox / KubeVirtSandbox<br/>jq, python3, yq"]
SpillFiles["/tmp/data/_out/ spill files"]
DataFiles["/tmp/data/ working files"]
end
subgraph dataSources [Data Source Modules]
GL["gitlab.py"]
KX["konflux.py"]
TF["testing_farm.py"]
end
subgraph external [External APIs]
GitLabAPI["GitLab API"]
KonfluxAPI["K8s + Kubearchive"]
TFAPI["Testing Farm API"]
end
subgraph storage [Storage]
S3["S3 Sessions"]
MRNote["GitLab MR Notes"]
end
SQS --> Events --> Runner
CLI --> Runner
Runner --> Agent
Agent --> Tools
Tools --> SB
Tools --> dataSources
dataSources --> external
Agent --> Models
Runner --> Actions --> MRNote
Runner --> Sessions --> S3
Runner --> WfConfig
2.2. Request lifecycle
A complete run proceeds through these stages:
-
Event arrival. An SQS message (production) or CLI invocation (dev) provides a GitLab project path and MR IID.
-
Config lookup.
workflow_config.pymaps the project to one or more workflows via theproject_index. Each match yields aWorkflowConfigandProjectEntrywith data source declarations and per-project settings. -
Rate-limit check and SHA dedup.
actions.scan_agent_threads()scans MR discussions in a single pass, parsing JSON session markers to determine the per-workflow thread count and whether the current SHA has already been reviewed. If the SHA was already reviewed, the workflow is skipped. If the thread count meets or exceedsmax_runs_per_mr, the workflow is skipped. Slash commands and replies bypass this check. -
Placeholder note.
actions.create_placeholder_note()posts a placeholder so the reviewer knows analysis is in progress. The note contains a JSON session marker (<!-- hummingbird-session: {"id":"UUID","wf":"name","sha":"abc"} -->) for rate limiting, SHA dedup, and session resumption. -
Sandbox start.
sandbox.create_sandbox()starts a Podman container or K8s pod./tmp/data/is pre-created for the model’s use. -
Data source registration.
data_sources.register_selected()resolves tokens and URLs from the config and registers tool functions on theToolRegistry. Only data sources declared in the workflow config are registered. -
Agent loop.
agent.run_agent_loop()runs the model loop: the workflow markdown becomes the system prompt, tool definitions are provided, and the model iterates calling tools and producing text until it emits a final response or hits a budget limit. -
Note update. The final text, wrapped with a session marker and reply prompt, replaces the placeholder note.
-
Session save. Conversation history (
contents), transcript, and sandbox archive are saved to S3 (production) or a local directory (dev). -
Sandbox cleanup. The container/pod is deleted. K8s pods also have a configurable
activeDeadlineSecondsbackstop (default 1800s) in case the orchestrator dies.
2.3. Architectural boundaries
The codebase is organized around four strict boundaries:
Runner (runner.py) is the orchestration layer. It owns event routing,
the placeholder/update note lifecycle, session save/load, and sandbox
lifecycle. It calls agent.run_agent_loop() but never reaches into the
agent’s internals.
Agent (agent.py) is the model loop. It knows about the model interface,
tool definitions, and the contents list, but nothing about GitLab, SQS,
sessions, or actions. It returns (text, usage, transcript, contents) and
is completely unaware of what happens with those values.
Tools (tools.py) bridge the agent and the sandbox/data-sources. The
agent calls tool_registry.execute(tool_call) and gets a dict back. It
never calls sandbox methods directly. This indirection is what enables
auto-spill: the tool registry can transparently save large outputs to
sandbox files and return previews.
Model providers are supplied by the shared
hummingbird-ai library,
which owns adapter construction,
provider formats, retries, and AI usage accounting. The agent loop uses the
library’s normalized adapter interface and treats conversation content as
opaque.
3. Module Architecture
3.1. Dependency graph
flowchart TD
main["__main__.py"]
config_mod["config.py"]
wf_config["workflow_config.py"]
events_mod["events.py"]
runner_mod["runner.py"]
agent_mod["agent.py"]
tools_mod["tools.py"]
sandbox_mod["sandbox.py"]
actions_mod["actions.py"]
sessions_mod["sessions.py"]
transcript_mod["transcript.py"]
ds_init["data_sources/__init__.py"]
ds_gitlab["data_sources/gitlab.py"]
ds_konflux["data_sources/konflux.py"]
ds_tf["data_sources/testing_farm.py"]
config_watch_mod["config_watch.py"]
ai_pkg["hummingbird-ai library"]
main --> config_mod
main --> config_watch_mod
main --> wf_config
main --> events_mod
main --> runner_mod
main --> sessions_mod
main --> transcript_mod
main --> actions_mod
config_watch_mod --> config_mod
config_watch_mod --> wf_config
runner_mod --> actions_mod
runner_mod --> agent_mod
runner_mod --> config_mod
runner_mod --> ds_init
runner_mod --> sandbox_mod
runner_mod --> sessions_mod
runner_mod --> tools_mod
runner_mod --> ai_pkg
runner_mod --> wf_config
runner_mod --> transcript_mod
agent_mod --> config_mod
agent_mod --> ai_pkg
agent_mod --> tools_mod
tools_mod --> config_mod
tools_mod --> ai_pkg
tools_mod --> sandbox_mod
ds_init --> tools_mod
ds_init --> wf_config
ds_init --> ds_gitlab
ds_init --> ds_konflux
ds_init --> ds_tf
ds_gitlab --> ai_pkg
ds_konflux --> ai_pkg
ds_tf --> ai_pkg
sessions_mod --> sandbox_mod
3.2. Module responsibilities
Each module has a single, well-defined responsibility. The boundary rules below are invariants – violating them breaks the security model or the separation of concerns.
config.py – Runtime configuration and constants
- Owns:
Configdataclass and agent tuning constants (DEFAULT_MAX_ITERATIONS,CONTEXT_LIMIT,OUTPUT_PREVIEW_BYTES, etc.) - Boundary: Pure data. No I/O except reading env vars in
Config.load(). No imports from other Hummingbird agent modules; model setup and pricing are owned byhummingbird-ai. - Invariant: All interdependent budget/spill constants must be defined here with their relationship documented in the comment block.
workflow_config.py – YAML config loader and token resolution
- Owns:
AgentConfig,WorkflowConfig(withtrigger,description,trigger_rules,ignore_users,ignore_branchesfields),TriggerRule,ProjectEntry,DataSourceConfigdataclasses;load(),validate_env(),evaluate_trigger_rules(),resolve_tool_token(),resolve_cluster_url(),get_prompt(). - Boundary: Reads YAML from disk and env vars. Never instantiates network clients or model objects.
- Invariant:
resolve_tool_token()resolves model tool tokens only. It must never return orchestrator tokens. The resolution chain is:project_entry.tokens[ds] -> workflow_cfg.data_sources[ds].token_env -> "" (empty).
events.py – SQS consumer
- Owns:
decode_sns_message()(SNS envelope decoding),poll_loop()(blocking SQS consumer with concurrency control). - Boundary: Knows about SQS/SNS wire formats. Calls a generic
EventHandlercallback. Does not know about GitLab, workflows, or the agent. - Invariant: Failed messages are not deleted from SQS (they go to the DLQ after visibility timeout expires). Successful messages are deleted after handler returns.
config_watch.py – Background config hot-reload
- Owns:
ConfigHolder(thread-safe config pair with atomic swap),start_watcher()(daemon thread that polls config file mtime),_watch_loop(). - Boundary: Knows about
config.Config.load()andworkflow_config.load(). Does not know about events, the agent, or any runtime state. - Invariant: On reload failure (parse error, missing env var), the previous config is kept and a warning is logged. The watcher never crashes the serve loop.
runner.py – Event routing and workflow execution
- Owns:
handle_event(),handle_pipeline(),handle_merge_request(),handle_note(),_execute_workflow(),_handle_reply(),run_workflow(),_acquire_sandbox(),_resolve_slash_workflow(),_format_help(),_is_ignored_user(),WorkflowRequest,WorkflowResult,SandboxOpts. - Boundary: Orchestrates everything: config lookup, rate limiting,
placeholder notes, sandbox lifecycle, agent invocation, session save.
This is the only module that touches both
actionsandagent. - Invariant:
run_workflow()either lingers the sandbox (success) or cleans it up (exception).
agent.py – Model-agnostic tool-calling loop
- Owns:
run_agent_loop(), system prompt constants (BASE_SYSTEM_PROMPT,CONTINUATION_PROMPT, warning/nudge strings), budget logic. - Boundary: Uses the
ModelAdapterinterface fromhummingbird-aiandtools(forexecute()). Does not know about GitLab, SQS, sessions, or actions, or which backend is in use. - Invariant: The
contentslist is treated as opaque. Items are appended viaresponse.raw_contentandmodel.make_tool_responses(). The agent never inspects, modifies, or deletes items in the list (except for empty response retries, whichpop()the last appended item before the model has seen any tool results for it).
tools.py – Tool registry
- Owns:
ToolRegistryclass withsandbox_exec,fetch_to_sandbox,fetch_batch_to_sandboxbuilt-in tools; data source registration and dispatch; auto-spill logic. - Boundary: Owns the sandbox reference and all tool execution. The agent never touches the sandbox directly.
- Invariant:
_spill_counteris shared across all spill paths (sandbox exec stdout, sandbox exec stderr, data source auto-spill) to prevent filename collisions in/tmp/data/_out/.
sandbox.py – Sandbox backends
- Owns:
Sandboxprotocol,PodmanSandbox,K8sSandbox,K8sPoolSandbox,SandboxPool,KubeVirtSandbox,VmiPool,KubeVirtPoolSandbox,create_sandbox(),ExecResult. - Boundary: Knows about container/pod/VM lifecycle and command execution. Does not know about tools, models, or the agent.
- Invariant: All backends must implement the same 8-method protocol
(
start,exec,write_file,stream_to_file,stream_exec,read_file_iter,cleanup,linger).exec()accepts optionalstdin_datafor piping raw bytes. All must pre-create/tmp/data/instart(). All must usestdinpiping forwrite_file()(never host volume mounts).
actions.py – GitLab note lifecycle
- Owns: Orchestrator token resolution (
resolve_orchestrator_token()), workflow action token client creation (_make_client()), bot user ID collection (collect_workflow_bot_ids()), note CRUD, JSON marker parsing (marker_tag,parse_marker),scan_agent_threads(per-workflow rate limiting + SHA dedup),post_simple_reply, member access checks, session-for-reply lookup,AgentThreadInfo,SessionRef. - Boundary: Functions are split by token tier:
- Orchestrator token (operational reads + infrastructure notes, use
_get_client):check_member_access,get_latest_push_author,get_head_pipeline_id,is_first_note_in_discussion,reply_access_denied,notify_rate_limit,post_simple_reply. - Workflow action token (per-workflow-attributable writes, take explicit
token: str):create_placeholder_discussion,post_discussion_note,resolve_agent_threads,scan_agent_threads. Never touches model tool tokens, the tool registry, or the agent.
- Orchestrator token (operational reads + infrastructure notes, use
- Invariant: Orchestrator tokens are resolved from
ORCHESTRATOR_*env vars only. Workflow action tokens are resolved fromaction_tokensin YAML and passed as explicit parameters. Neither tier is exposed to the model.
sessions.py – Session persistence
- Owns:
archive_sandbox(),save_local(),save_s3(),load_s3(),load_local(),restore_sandbox(),SessionData. - Boundary: Knows about sandbox (for archiving) and S3/filesystem (for
storage). Does not know about the agent, tools, or models. Load/save uses
canonical envelopes with
format_version: 1; unversioned legacy sessions are not supported. - Invariant: The sandbox archive is never extracted on the orchestrator.
It is created inside the sandbox (
tar czf), streamed out viaread_file_iter(), and restored inside a new sandbox viastream_exec("tar xzf -").
transcript.py – Transcript rendering
- Owns:
render_markdown(), per-tool rendering functions, truncation. - Boundary: Pure transformation from
TranscriptEntrylist to markdown. No I/O, no side effects.
data_sources/__init__.py – Data source registration
- Owns:
register_selected()– resolves tokens/URLs from config and calls each data source’sregister()function. - Boundary: Bridges
workflow_config(for token/URL resolution) andtools(for registration). Only registers data sources declared in the workflow config.
data_sources/gitlab.py, konflux.py, testing_farm.py
- Own:
register()function that creates tool definitions and closures over credentials, and registers them on theToolRegistry. - Boundary: Each module talks to one external API. Credentials are captured at registration time via closure, never stored globally.
4. Security Model
The agent runs in a shared OpenShift cluster without cluster-admin access, processing merge requests from repositories where external contributors can submit code. The security model addresses two threat vectors: (1) the LLM executing arbitrary commands chosen by an attacker-controlled MR, and (2) credential leakage between the model, the sandbox, and orchestrator actions.
4.1. Sandbox isolation
All commands generated by the LLM run inside an ephemeral container, never on the orchestrator host. The sandbox has no credentials, no network, and no visibility into the orchestrator.
flowchart LR
subgraph orchestrator [Orchestrator]
Agent["Agent Loop"]
Creds["Secrets<br/>(tokens, kubeconfig)"]
end
subgraph sandbox [Sandbox Container]
Shell["sh -c commands"]
Files["/tmp/data/<br/>(incl. _out/ spill dir)"]
end
Agent -->|"stdin pipe<br/>(write_file)"| sandbox
Agent -->|"exec command"| sandbox
sandbox -->|"stdout/stderr"| Agent
sandbox -.-x|"NO network"| Internet["Internet / K8s API"]
sandbox -.-x|"NO access"| Creds
Podman (local development):
--network=none– complete network isolation; curl, wget, pip install all fail--user 65532– fixed non-root UID; no privilege escalation- No host volume mounts – data enters only via
stdinpiping throughwrite_file()
Kubernetes (production):
Every field in the pod manifest is set explicitly for portability to vanilla
Kubernetes with Pod Security Admission (restricted level), not just reliance
on OpenShift’s restricted-v2 SCC admission:
automountServiceAccountToken: false– no K8s API access from sandboxrunAsNonRoot: true– enforced at pod levelseccompProfile: RuntimeDefault– required by restricted levelallowPrivilegeEscalation: false– container levelcapabilities.drop: ["ALL"]– container levelactiveDeadlineSeconds: 1800– pod self-terminates after 30 minutes even if the orchestrator crashes; prevents orphaned podsrestartPolicy: Never– pod does not restart on failureNetworkPolicyon the sandbox namespace blocks all egress from all pods in the namespace (podSelector: {},egress: [])
The pod does not set runAsUser explicitly. On OpenShift, the SCC assigns a
UID from the namespace range. On vanilla K8s, the image’s USER directive is
used.
4.2. Credential separation (four-tier token model)
Tokens are split into four tiers with strict code-path separation:
flowchart TB
subgraph tier1 [Tier 1: Orchestrator Tokens]
OT["ORCHESTRATOR_GITLAB_TOKEN_*<br/>Operational reads<br/>Env vars only, never in YAML"]
end
subgraph tier2 [Tier 2: Workflow Action Tokens]
AT["action_tokens in YAML<br/>Per-workflow bot identity<br/>Write scope (api)"]
end
subgraph tier3 [Tier 3: Model Tool Tokens]
MT["GITLAB_TOKEN_RO, etc.<br/>Read scope (read_api)<br/>Env var NAMES in YAML"]
end
subgraph tier4 [Tier 4: Sandbox]
SB["Zero credentials<br/>Zero network<br/>Zero SA token"]
end
OT -->|"used by"| OpsReads["actions.py<br/>(member access, push author,<br/>rate-limit notices)"]
AT -->|"used by"| WfWrites["actions.py<br/>(notes, thread resolution)"]
MT -->|"used by"| DataSources["data_sources/<br/>(GitLab, Konflux, TF)"]
SB -->|"used by"| SandboxExec["sandbox_exec<br/>(sh -c commands)"]
OpsReads -.-x|"NEVER"| DataSources
WfWrites -.-x|"NEVER"| DataSources
WfWrites -.-x|"NEVER"| SandboxExec
DataSources -.-x|"NEVER"| OpsReads
DataSources -.-x|"NEVER"| WfWrites
Tier 1 – Orchestrator tokens. Tokens used by actions.py for
operational reads (member access checks, push author lookup, head pipeline
queries) and infrastructure notes (access-denied replies, rate-limit
notices). Resolved by convention from env vars:
ORCHESTRATOR_GITLAB_TOKEN_<MANGLED_PROJECT> (per-project) or
ORCHESTRATOR_GITLAB_TOKEN (global fallback). The ORCHESTRATOR_ prefix is
a structural safeguard – these env var names can never appear in the YAML
config’s data_sources, tokens, or action_tokens sections. These tokens
can be scoped to read_api since all per-workflow writes moved to Tier 2.
Tier 2 – Workflow action tokens. Per-workflow bot identity tokens
(api scope) used by actions.py for all workflow-attributable GitLab
writes: placeholder notes, discussion notes, and thread resolution. Each
workflow gets a dedicated bot user per project, providing clear audit trails
for which agent produced each note. Declared in the YAML config as env var
names at the project level:
projects:
redhat/hummingbird/containers:
action_tokens:
gitlab: HUMMINGBIRD_AGENT_ACTION_CODE_REVIEW_GITLAB_TOKEN_CONTAINERS
Resolved by resolve_action_token(project_entry, ds_name) which reads the
env var name from action_tokens and resolves it from os.environ. These
tokens are passed as explicit token: str parameters to write functions
in actions.py. They NEVER enter the ToolRegistry or model tool calls.
At startup, collect_workflow_bot_ids() authenticates each unique workflow
token to build a set of all workflow bot user IDs. This set is used by
handle_note for self-filtering (skipping notes from any workflow bot)
and by find_session_for_reply for session marker scanning.
The naming convention groups tokens alphabetically:
HUMMINGBIRD_AGENT_ACTION_<WORKFLOW>_GITLAB_TOKEN_<PROJECT>.
Tier 3 – Model tool tokens. Read-only tokens (read_api scope for
GitLab) used by data source modules during tool execution. Declared in the
YAML config as env var names (not values):
data_sources:
gitlab:
token_env: GITLAB_TOKEN_RO # name of the env var
Per-project overrides are possible:
projects:
redhat/hummingbird/containers:
tokens:
gitlab: GITLAB_TOKEN_CONTAINERS_RO # overrides token_env for this project
The resolution chain in resolve_tool_token() is:
project_entry.tokens[ds] -> workflow_cfg.data_sources[ds].token_env -> ""
Storing env var names (not values) in YAML means the config file is safe to commit and audit. Actual secret values live in environment variables, injected via K8s Secrets at deployment time.
Tier 4 – Sandbox. The sandbox container has zero credentials, zero
network access, and automountServiceAccountToken: false (no K8s API
access). Data enters the sandbox only via write_file() (stdin piping).
The model cannot instruct the sandbox to reach external APIs – it must use
the orchestrator’s data source tools.
4.3. Namespace separation
Sandbox pods run in a dedicated namespace (hummingbird--agent-sandbox),
separate from the orchestrator namespace (hummingbird--internal). This
limits blast radius: even if a sandbox pod is compromised, it has no
visibility into the orchestrator’s Secrets, Pods, or ServiceAccount tokens.
RBAC setup:
The orchestrator’s ServiceAccount gets a Role in the sandbox namespace (not its own namespace) granting only:
pods: create, get, list, delete, patch– sandbox pod lifecycle and pool claimspods/exec: create– command execution viakubectl exec
No CRDs, no custom runtimes, no cluster-scoped resources. The orchestrator needs only namespace-scoped permissions, so it works with standard OpenShift RBAC without requesting cluster-admin.
Konflux data is fetched via bearer token from kubeconfig credentials, not from inside the cluster. The orchestrator’s ServiceAccount does not need access to Konflux namespaces.
NetworkPolicy:
A deny-all-egress NetworkPolicy in the sandbox namespace uses podSelector: {}
to match all pods and sets egress: []. The sandbox cannot reach the internet,
the K8s API, or other pods in the cluster.
4.4. Security invariants
These must hold for the security model to be effective. Any change that violates one of these is a security regression:
-
Orchestrator tokens must NEVER flow into
ToolRegistry,data_sources, or modelcontents. They are resolved inactions.pyonly. -
Workflow action tokens must NEVER flow into
ToolRegistry,data_sources, or modelcontents. They are passed as explicittoken: strparameters withinactions.pyandrunner.pyonly. -
Model tool tokens must NEVER flow into
actions.py. They are resolved inworkflow_config.pyand consumed indata_sources/. -
The sandbox must NEVER have network access or credentials. No host mounts, no SA token, no env vars with secrets.
-
The sandbox archive is NEVER extracted on the orchestrator. It is created inside one sandbox and restored inside another. The orchestrator only transports the bytes.
-
No cluster-admin required. Only namespace-scoped resources (Role, RoleBinding, Pod, NetworkPolicy) are used.
-
Agent-generated notes are skipped on re-processing. Notes containing
SESSION_MARKER_PREFIXare filtered out inhandle_note()to prevent infinite loops. Self-filtering checks against all workflow bot user IDs (collected at startup) plus the orchestrator bot ID. -
All auto-triggers require Developer access.
handle_pipeline(),handle_merge_request(), andhandle_note()each checkcheck_member_access()on the event’s user before executing a workflow. Events from non-developers are silently skipped (or replied to with an access-denied message for slash commands). This ensures that the target MR’s work products – its diff, description, CI logs, and commit messages – originate from a trusted author. Fork MRs from external contributors are blocked because the MR author lacks Developer+ on the target project.Scope limitation: this gate only covers the target MR. Once the agent is running, the model controls tool arguments and can direct tools at content beyond the target MR – other MRs, other refs, other job IDs, even other projects reachable by the read-only token. The current mitigations are: (a) workflow prompts instruct the model to use the event’s
{project, iid, sha}, (b) the model would need to be manipulated via prompt injection from already-trusted content to deviate, (c) the sandbox prevents the model from acting on manipulated reasoning beyond producing text output, and (d) the token is read-only with minimal scope. However, if a data source tool is added that returns user-authored prose from arbitrary resources (e.g. issue bodies, wiki pages), it should apply per-author trust filtering (invariant #10). -
Data source tools that accept URLs must validate them against an allowlist.
tf_get_test_logrestricts URLs to the Testing Farm artifacts prefix to prevent the LLM from directing the orchestrator to fetch arbitrary URLs (SSRF). -
Third-party commentary entering the model prompt must be trust-filtered. Data sources that feed text from users other than the event trigger into
contents(discussion comments, issue bodies) must gate on project membership at Developer+ level per author. Content from untrusted authors must be redacted to a fixed placeholder, never sanitized or escaped – there is no reliable way to escape adversarial text for an LLM prompt. Discussions where all notes are untrusted must be dropped entirely. See §9.1 for the reference implementation ingitlab_get_mr_discussions.
This invariant covers commentary (what other people said about the MR), not work products (the diff, CI logs, commit messages). Work products are inherently the input to the agent’s analysis and cannot be content-filtered without defeating the agent’s purpose. For the target MR, their trust comes from invariant #8 (the event trigger is Developer+). For content the model fetches beyond the target MR, trust depends on the tool: discussion tools must filter per-author (#10), while code/log/metadata tools rely on the mitigations described in #7.
5. Configuration System
5.1. Two sources, strict separation
Configuration comes from exactly two sources with no overlap:
- YAML config file (
CONFIG_PATH): operational settings, workflow definitions, project allowlists, data source declarations, and token env var names. This file is safe to commit, review, and audit. - Environment variables: secrets only (API keys, GitLab tokens, kubeconfig paths). These are injected at deployment time via K8s Secrets.
This split exists because YAML provides structure, validation, and audit trails, while secrets must stay out of version control.
5.2. Config loading
Two config objects are built at startup:
Config (from config.py): loaded by Config.load(settings), where
settings is the settings: section from the YAML file. Auth/bootstrap
fields come from env vars (GOOGLE_API_KEY, GOOGLE_CLOUD_PROJECT, etc.).
Operational fields come from the settings dict with sensible defaults.
AgentConfig (from workflow_config.py): loaded by
workflow_config.load(path). Contains all workflow definitions, project
entries, and a pre-built project_index.
The two are kept separate because Config is needed everywhere (model
construction, sandbox creation, session storage), while AgentConfig is
only needed for event routing and data source registration.
5.3. YAML config structure
settings:
gitlab_url: https://gitlab.com # operational, not a secret
sandbox:
image: quay.io/.../image:tag
namespace: my-namespace # K8s only
model: gemini-3.1-pro-preview
# model: claude-sonnet-4-5@20250929 # Anthropic via Vertex AI (alternative)
max_iterations: 30
max_runs_per_mr: 5
internal_notes: true
max_concurrent_agents: 4
sqs_queue_url: "" # empty = no SQS (local dev)
s3_session_bucket: "" # empty = no S3 (local dev)
trigger_prefix: /hummingbird # slash command prefix
session_marker_prefix: hummingbird-session # HTML comment marker ID
auto_trigger: true # auto-run on failed pipelines
workflows:
analyze-failures:
prompt: workflows/analyze-failures.md # relative to config file dir
action: post_gitlab_note
model: gemini-3.1-pro-preview # per-workflow override
max_iterations: 50 # per-workflow override
data_sources:
gitlab:
token_env: GITLAB_TOKEN_RO # env var name, not the value
konflux:
cluster_url: https://example.com:6443/ns/my-tenant
kubeconfig_env: KUBECONFIG
kubearchive_url: https://kubearchive-api-server-product-kubearchive.apps.example.com
testing_farm: {}
projects:
redhat/hummingbird/containers:
tokens: # per-project token overrides
gitlab: GITLAB_TOKEN_CONTAINERS_RO
Design choices in this structure:
Workflow-first organization. Each workflow owns its project list, not the other way around. This scopes data source permissions per workflow-project pair. A future code-review workflow can have different GitLab tokens (with different scopes) than the analyze-failures workflow, with no ambiguity.
Token env var names in YAML (not values). The YAML file is committed to
the repo. It contains token_env: GITLAB_TOKEN_RO (a name), not the actual
token. The actual secret value is resolved at runtime via
os.environ.get(env_var). This allows the config to be reviewed and
audited without exposing secrets.
Inline cluster_url only. The Konflux cluster URL is operational
configuration (it identifies which cluster to talk to), not a secret.
Putting it inline in YAML makes it visible and auditable. The URL must
include the namespace path (e.g. /ns/my-tenant).
5.4. Project index
At load time, _build_project_index() constructs a reverse lookup:
project_index: dict[str, list[tuple[str, WorkflowConfig, ProjectEntry]]]
# e.g. {"redhat/hummingbird/containers": [("analyze-failures", wf_cfg, proj_entry)]}
This provides O(1) lookup when a pipeline event arrives with a project
path. A single project can appear in multiple workflows (e.g. both
analyze-failures and a future code-review), and each matching workflow
will be triggered independently.
5.5. Token resolution chains
Model tool tokens (for data source API calls):
resolve_tool_token(project_entry, ds_name, workflow_cfg):
1. project_entry.tokens[ds_name] -> per-project override
2. workflow_cfg.data_sources[ds_name].token_env -> workflow default
3. "" (empty string) -> no token
Each step resolves the env var NAME, then reads os.environ[name].
Workflow action tokens (for per-workflow GitLab writes):
resolve_action_token(project_entry, ds_name):
1. project_entry.action_tokens[ds_name] -> env var NAME
2. os.environ[name] -> resolved token value
3. "" (empty) -> not configured
No fallback chain. Missing tokens cause validate_env to fail at startup.
Orchestrator tokens (for operational reads and infrastructure notes):
resolve_orchestrator_token(project_path):
1. ORCHESTRATOR_GITLAB_TOKEN_<MANGLED_PROJECT> -> per-project
2. ORCHESTRATOR_GITLAB_TOKEN -> global fallback
Mangling: "/" -> "_", "-" -> "_", uppercase.
e.g. "redhat/hummingbird/containers" -> ORCHESTRATOR_GITLAB_TOKEN_REDHAT_HUMMINGBIRD_CONTAINERS
Cluster URL (for Konflux):
resolve_cluster_url(ds_cfg):
-> ds_cfg.cluster_url (inline value in YAML)
5.6. Environment validation
validate_env(agent_cfg) checks at startup that all referenced env vars
exist. It walks every workflow’s data sources, project token overrides, and
project action token entries, collecting missing vars into a single error
message. This catches configuration errors early instead of failing mid-run
when a specific data source or workflow action is first used. Missing
action_tokens env vars are treated the same as missing model tool tokens:
startup fails unconditionally.
See Also
- Hummingbird Agent Design — Engine — Agent loop (§6), tool system (§7), sandbox backends (§8)
- Hummingbird Agent Design — Runtime — Data sources (§9), event pipeline (§10), sessions (§11), workflows (§12), deployment (§13)
- Hummingbird Agent Design — Decisions — Design Decision Registry
4.5.20.3 - Hummingbird Agent Design — Engine
Agent loop, tool system, and sandbox backends for the Hummingbird Agent. This page covers the execution engine: how the agent iterates, dispatches tools, and isolates execution. For foundations (design philosophy, system architecture, modules, security, config), see Hummingbird Agent Design.
6. Agent Loop Design
For the wire-format walkthrough (what bytes go to the model API, what comes back), see Agent Model Loop. This section covers the design rationale behind the loop.
6.1. Full history replay
The Gemini and Anthropic APIs are stateless. Every call sends the complete contents list
from the beginning of the conversation. This means every large tool output
sitting in history inflates every subsequent API call.
This property is fundamental to why auto-spill exists (section 7). Without
auto-spill, a single cat of a 32KB file early in the conversation adds
~8K tokens to every remaining API call. Over a 15-iteration run, that is
~120K wasted tokens.
The alternative – conversation compaction (replacing old tool results with summaries) – was considered and deferred. Each provider has strict requirements about content structure, and modifying history risks confusing the model or violating API constraints. Provider-specific format requirements are documented in Hummingbird AI. Auto-spill solves 90% of the problem with none of the risk.
6.2. Budget model: iterations + context limit
The agent uses a dual-limit approach rather than a cumulative token budget:
Iteration limit (max_iterations, default 30 per workflow config).
Hard cap on the number of model round-trips. This is the primary cost
control lever. With auto-spill keeping per-call context bounded, iteration
count is roughly proportional to cost.
Per-call context limit (CONTEXT_LIMIT, default 60,000 tokens).
Checked after each API call using response.usage.total_input_tokens. This is
a safety net for cases where auto-spill is not sufficient (e.g., many
small tool results that individually stay under the spill threshold but
cumulatively fill the context).
Why not a cumulative token budget? Because with full history replay, each API call re-sends everything. “Cumulative billed tokens” double-counts: call 1 sends 5K, call 2 sends 10K (including the 5K again), so billed total is 15K but actual new content is only 10K. Iteration count is a simpler and more predictable proxy for cost.
6.3. Two-tier budget escalation
Both limits use the same escalation pattern:
flowchart LR
Normal["Normal<br/>tools available"]
Soft["Soft Warning (80%)<br/>ITERATION_WARNING or<br/>CONTEXT_WARNING<br/>tools still available"]
Hard["Hard Stop (100%)<br/>FINAL_TURN_WARNING<br/>toolConfig: NONE"]
Normal -->|"80% reached"| Soft
Soft -->|"100% reached"| Hard
Soft warning (80%). An ephemeral user message (ITERATION_WARNING) is
injected into contents for that turn only, then popped before the
response is persisted. Tools remain available so the model can finish
in-progress work.
Hard stop (100%). FINAL_TURN_WARNING is injected as an ephemeral
user message AND tool_defs is set to [] (empty list) to physically
prevent further tool calls. The model must produce text. This is more
reliable than relying on provider-side tool disabling, which models may
ignore.
6.3.1 Ephemeral messages and provider normalization
All per-turn warnings use ephemeral user messages: they are appended to
contents before the API call and popped immediately after. This keeps
the system prompt stable across all iterations and prevents warnings from
polluting the conversation history saved in sessions.
Provider adapters normalize provider-specific conversation requirements while preserving the agent’s saved history; details are in Hummingbird AI.
6.4. Empty response handling
Models occasionally return unusable responses (no text and no valid tool calls). The
agent retries up to MAX_EMPTY_RETRIES (2) times:
- Pop the empty response from
contents. It adds nothing and may confuse the model on the next call. - Inject
EMPTY_RESPONSE_NUDGEas an ephemeral user message on the next turn: “Your previous response was empty. Please continue…” - Continue the loop (consuming an iteration).
The nudge is delivered as an ephemeral user message (injected before the API call and popped after). This avoids mutating the system prompt and keeps the conversation history clean for session persistence.
The empty_retries counter resets to 0 after any successful iteration
(one where tool calls were executed). This means the model gets fresh
retries if it produces empty responses at different points in the
conversation.
MALFORMED_FUNCTION_CALL handling. Models sometimes return a
finishReason of MALFORMED_FUNCTION_CALL (Gemini) with no usable tool calls.
This is treated as a special case of empty response: the retry mechanism
kicks in, but the nudge is replaced with MALFORMED_CALL_NUDGE which
tells the model to retry with simpler arguments and avoid large text
payloads in tool call arguments.
6.5. Error recovery
Model API and transport errors. The hummingbird-ai library classifies
provider and transport failures, applies the workflow’s retry budget, and
records AI usage and request metrics. See Hummingbird AI
for provider-specific behavior. The agent loop retains partial results when a
terminal error reaches it.
Unexpected loop errors. run_agent_loop wraps the
_generate_with_retry call in a try/except Exception that logs and
breaks instead of propagating. This ensures the function always returns
partial results (accumulated contents, transcript, sandbox state)
even when an unexpected exception occurs (e.g. JSONDecodeError from a
truncated API response). The caller saves the session normally –
conversation history and sandbox archive are preserved for resumption.
This follows the “partial over nothing” principle: 32 iterations of work
are more valuable than a crash.
No failure-path session save. _execute_workflow does not save a
session when run_workflow raises. With the agent loop catching
unexpected errors, the failure path only fires for infrastructure errors
(sandbox start, config) where there is no useful state. Not saving
avoids overwriting a previous good session when a reply attempt fails.
Tool execution errors. _execute_tool_calls() wraps each
tool_registry.execute() in a try/except. Unhandled exceptions are caught
and returned to the model as {"error": "Tool X failed: ..."}. This
prevents a single broken tool from crashing the entire session – the model
sees the error and can adapt.
Sandbox exec timeout. If a command exceeds EXEC_TIMEOUT (120s), the
TimeoutExpired exception is caught in the tool registry and returned as a
structured error dict. The model can retry with a different command or
proceed without the result.
6.6. Session resumption
When resuming from a previous session:
initial_contents(the fullcontentslist from the previous run) is prepended, followed by the user’s reply as a new user turn.CONTINUATION_PROMPTis appended to the system prompt.- The sandbox is restored from the archived tarball.
- A fresh iteration budget starts from 0.
Saved sessions use a versioned JSON envelope (format_version: 1) whose
contents are in a canonical message format independent of the active model.
Unversioned legacy sessions are not supported. On resume, run_workflow asks
the active adapter to load canonical history; after a successful run, it
converts native contents back to canonical form before persistence. The
library documents the canonical format and provider conversions in
Hummingbird AI.
CONTINUATION_PROMPT is critical. Without it, the workflow prompt (e.g.,
analyze-failures.md) tells the model to follow a rigid multi-phase
workflow: Data.1, Data.2, Analysis.1… The model would try to re-run the
entire analysis. The continuation prompt overrides this: “Do NOT re-run the
full workflow. Respond directly to the user’s question.”
The practical constraint on resumed sessions is the context window, not
iterations. The restored history from an 8-iteration cold start uses
~50-100K input tokens. Each new tool call adds more. The existing
CONTEXT_LIMIT check still applies and will force wrap-up if the context
grows too large.
Canonical session format (rationale)
Storing sessions in canonical form decouples persisted history from any one
provider’s JSON shape. The agent loop treats native contents as opaque; the
library owns provider conversion at load and save boundaries.
6.7. Provider caching and pricing
Provider-specific prompt caching and token-cost estimation are owned by
hummingbird-ai.
7. Tool System and Auto-Spill
7.1. Tool registry design
ToolRegistry is the single point of dispatch for all tool calls. The
agent calls registry.execute(tool_call) and gets a dict back. It never
calls sandbox methods or data source functions directly.
Three built-in tools are always available:
sandbox_exec– runssh -c <command>in the sandbox container. Returns{exit_code, stdout, stderr}, with auto-spill for large outputs.fetch_to_sandbox– calls a data source function and writes the result to a specified path in the sandbox. Returns metadata only (path, byte count, line count). Used when the model wants explicit path control.fetch_batch_to_sandbox– calls multiplefetch_to_sandboxin one tool call. Saves iterations vs. sequential calls (e.g., fetching both pipelineruns and taskruns in one round-trip).
Data source tools are registered dynamically per workflow config. Each data
source module provides a register() function that creates ToolDef
objects and closures over credentials, then calls
registry.register_data_source(tool_def, func, response_metadata).
7.2. Auto-spill architecture
Auto-spill is the key mechanism for keeping the LLM context bounded. Without
it, large outputs accumulate in the contents list and inflate every
subsequent API call (because both APIs replay the full history).
flowchart TD
Output["Tool produces output"]
SizeCheck{"size > OUTPUT_PREVIEW_BYTES<br/>(4 KB)?"}
Inline["Return full output inline"]
Spill["Write full output to<br/>/tmp/data/_out/N.txt"]
Preview["Return preview<br/>(head + tail) +<br/>file path metadata"]
Output --> SizeCheck
SizeCheck -->|"<= 4 KB"| Inline
SizeCheck -->|"> 4 KB"| Spill --> Preview
Three spill paths share the same _spill_counter to prevent filename
collisions:
sandbox_exec spill. When stdout or stderr exceeds
OUTPUT_PREVIEW_BYTES (4096 bytes), the full output is written to
/tmp/data/_out/{counter}.txt via _spill_field(). The model receives:
{
"exit_code": 0,
"stdout": "<first 4KB>",
"stdout_truncated": true,
"stdout_file": "/tmp/data/_out/0.txt",
"stdout_bytes": 32768,
"stdout_lines": 1024,
"stdout_tail": "<last 512 bytes>"
}
The preview (head + tail) gives the model enough context to decide whether
to process the full file with jq/grep/head.
Data source inline spill. When a direct data source call returns text
larger than MAX_INLINE_SIZE (4096 bytes), _spill_data_source() writes
it to /tmp/data/_out/{name}_{counter}.txt and returns:
{
"saved_to": "/tmp/data/_out/konflux_list_pipelineruns_1.txt",
"bytes": 85432,
"lines": 2100,
"preview": "<first 4KB>"
}
Streaming responses (e.g., gitlab_get_repo_archive which returns a
StreamingResponse with suffix .tar.gz) are written via
_spill_streaming(). Text streams (binary=False, the default) capture a
UTF-8 preview from the head while piping chunks to the sandbox. Binary
streams (binary=True) skip preview capture entirely and return only
{saved_to, bytes}, avoiding meaningless decoded output for formats like
tar.gz. The StreamingResponse.suffix field controls the file extension
(.jsonl, .tar.gz, .log).
Non-streaming binary responses are written to .bin files with no preview
via _spill_binary().
Sandbox-optional mode. ToolRegistry accepts sb=None for use cases
that do not require a sandbox (e.g. evaluation harnesses that only use
native model tools like Google Search grounding). When no sandbox is
available:
- Sandbox tools (
sandbox_exec,fetch_to_sandbox,fetch_batch_to_sandbox) are omitted fromget_tool_defs()so the model never sees them. - Text data sources (inline and streaming) degrade to truncated previews
(
OUTPUT_PREVIEW_BYTES). The model receives{preview, truncated: true}instead of{saved_to, preview}. Streaming text is consumed into a memory buffer up to the preview limit and the rest is discarded. - Binary data (streaming and non-streaming) returns an error with the
byte count (
{error, bytes}). Binary content cannot be meaningfully previewed as text.
fetch_to_sandbox spill. Always writes to the caller-specified path
(not /tmp/data/_out/). Returns metadata only (no preview). The model uses this
when it wants a specific filename for later processing.
7.3. Why auto-spill instead of rejecting large outputs
The earlier design rejected large data source responses with “use
fetch_to_sandbox instead.” This caused two wasted iterations per rejection:
the model makes the call, gets rejected, then has to repeat with
fetch_to_sandbox. With auto-spill, the data flows to a sandbox file
transparently. Validation showed ~10% fewer iterations with auto-spill.
7.4. Why fetch_to_sandbox still exists alongside auto-spill
Auto-spill handles the common case, but fetch_to_sandbox provides:
- Explicit path control. The model can choose meaningful filenames
(
/tmp/data/pipelineruns.json) instead of getting auto-generated names (/tmp/data/_out/konflux_list_pipelineruns_1.txt). - Batch fetching.
fetch_batch_to_sandboxcombines multiple fetches in one tool call, saving iterations. - No preview overhead.
fetch_to_sandboxreturns metadata only, which is useful when the model knows it will process the file withsandbox_execanyway.
7.5. ToolDef notes
ToolDef has an optional notes field for domain knowledge that belongs
in the system prompt but not in the tool’s JSON schema. Examples:
- Konflux notes explain dual K8s/Kubearchive fetch, UID deduplication, and the two label selectors (BUILD vs. TEST).
- Testing Farm notes explain XML result structure and usage patterns.
ToolRegistry.get_tool_notes() collects all non-None notes into a
## Data Source Notes section that is prepended to the workflow body in
the system prompt:
BASE_SYSTEM_PROMPT + tool_notes + workflow_body
This keeps domain knowledge close to the tool definitions (in the data
source module) rather than duplicated in every workflow .md file.
7.6. Response metadata
register_data_source() accepts optional response_metadata – a dict
that is merged into every response from that tool (inline, auto-spill, and
fetch_to_sandbox). Used for:
- Konflux:
{"konflux_ui": "https://..."}so the model can build reviewer-facing PipelineRun links. - Testing Farm:
{"artifacts_base": "https://..."}so the model can build artifact links.
This avoids having the model ask “what is the Konflux UI URL?” – the information arrives with every tool response.
8. Sandbox Architecture
8.1. The Sandbox protocol
All backends implement an 8-method typing.Protocol:
class Sandbox(Protocol):
def start(self) -> None # create container/pod, pre-create /tmp/data/
def exec(command, *, stdin_data: bytes | None) -> ExecResult # sh -c in sandbox
def write_file(path, data: bytes) -> None # stdin pipe: cat > path
def stream_to_file(path, chunks: Iterable[bytes]) -> tuple[int, int] # Popen stdin pipe, returns (bytes, lines)
def stream_exec(command, chunks: Iterable[bytes]) -> ExecResult # Popen with streaming stdin
def read_file_iter(path) -> Iterator[bytes] # Popen stdout pipe in chunks
def cleanup(self) -> None # rm -f container / delete pod
def linger(session_id) -> None # keep alive for reuse, or fall back to cleanup
ExecResult is (exit_code: int, stdout: str, stderr: str).
stream_to_file() and stream_exec() use subprocess.Popen with a stdin
pipe, writing chunks incrementally. This avoids buffering large payloads in
orchestrator memory (e.g., streaming a tar.gz archive into the sandbox).
read_file_iter() reads from a Popen stdout pipe in 64 KB chunks.
write_file() uses stdin piping (cat > path), never host volume mounts.
This is critical: it means data flows through the orchestrator process, not
through a shared filesystem. The sandbox has no host mounts.
8.2. Backend selection
Five sandbox backends are available:
| Backend | --sandbox |
When to use |
|---|---|---|
PodmanSandbox |
podman |
Local development (default for run) |
K8sSandbox |
k8s |
Direct pod creation on a K8s cluster |
K8sPoolSandbox |
k8spool |
Pre-warmed pool via Deployment (default for serve) |
KubeVirtSandbox |
kubevirt |
Direct VMI creation with SSH exec |
KubeVirtPoolSandbox |
kubevirtpool |
Pre-warmed VMI pool via VMIRS |
create_sandbox() factory handles podman, k8s, and kubevirt. Pool
backends (k8spool, kubevirtpool) are handled in __main__.py, which
creates a SandboxPool or VmiPool and passes it to run_workflow().
The sandbox backend can be set per-workflow via the sandbox: field in the
workflow config YAML. Resolution order: workflow.sandbox > CLI --sandbox
default. This allows different workflows to use different backends (e.g., lightweight code review uses pod pool, heavier analysis uses VM pool).
K8s is never auto-detected from the environment. It requires explicit CLI flags. This prevents accidental use of a K8s sandbox when developing locally.
For K8s namespace resolution (k8s and k8spool):
- If
--namespaceis provided, use it. - If
--contextis provided, extract the namespace from the kubeconfig context. If the context has no default namespace, raise an error. - If neither is provided (in-cluster), use
sandbox.namespacefrom the config file.
8.3. PodmanSandbox
podman run -d --name hb-sandbox-{uuid8} --network=none --user 65532 \
--workdir /tmp {image} sleep infinity
- Unique container name with UUID suffix prevents collisions
sleep infinitykeeps the container alive for repeatedexeccalls--network=noneprovides complete network isolation--user 65532is a fixed non-root UID (matchesnonrootin distroless)- Cleanup:
podman rm -f(force, in case exec is still running)
8.4. K8sSandbox: the hybrid approach
The K8s sandbox uses a hybrid of two tools:
- kubernetes Python library for pod lifecycle:
create_namespaced_pod,read_namespaced_pod(poll for Running),delete_namespaced_pod. kubectl execsubprocess for command execution.
Why not use the kubernetes Python library for exec too? Three problems discovered during development:
-
No stdin EOF signal in WebSocket v1-v4. The Kubernetes exec protocol uses WebSocket channels (stdin=0, stdout=1, stderr=2). Protocol versions 1-4 have no mechanism to signal “stdin is done.” Commands like
cat > /filehang forever waiting for more input. Python client v5 support does not exist. -
BrokenPipeErroron large stdin. Sending more than ~1MB through the WebSocketstream()API causes pipe errors, breakingwrite_file()for large data source responses. -
Unbounded memory from
stream(). Thestream()function accumulates all stdout/stderr data in memory with no streaming control. A command producing megabytes of output would consume unbounded memory.
kubectl exec as a subprocess avoids all three problems and provides the
same interface as podman exec – stdin via subprocess.PIPE, stdout/stderr
captured, exit code from return code. The implementation in exec() is
nearly identical between PodmanSandbox and K8sSandbox.
8.5. Pod manifest design
The K8s pod manifest is built by _build_pod_manifest():
metadata:
labels:
app.kubernetes.io/name: hummingbird-agent-sandbox
spec:
automountServiceAccountToken: false # no K8s API from sandbox
activeDeadlineSeconds: 1800 # 30min hard timeout, backstop
restartPolicy: Never
securityContext:
runAsNonRoot: true
seccompProfile: RuntimeDefault
containers:
- name: sandbox
command: ["sleep", "infinity"]
workingDir: /tmp
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
resources:
requests: { cpu: 100m, memory: 256Mi, ephemeral-storage: 256Mi }
limits: { cpu: "1", memory: 1Gi, ephemeral-storage: 2Gi }
runAsUser is deliberately omitted. On OpenShift, the restricted-v2 SCC
assigns a UID from the namespace UID range. On vanilla K8s, the image’s
USER directive is used. Setting an explicit UID would conflict with
OpenShift’s SCC admission.
8.6. Auth modes
- Local development:
--contextflag passes the kubeconfig context toconfig.new_client_from_config(context=...), creating a per-instanceApiClient. - Production (in-cluster):
context=Nonetriggersconfig.load_incluster_config(), using the pod’s ServiceAccount token.
The kubectl exec commands include --context when running locally but
omit it when in-cluster (kubectl uses the default in-cluster config).
8.7. Pre-created directories
Both backends run mkdir -p /tmp/data in start() after the
container/pod is up. This provides a working directory for the model
without consuming an iteration. /tmp/data/_out (the spill directory) is
created on demand by mkdir -p $(dirname ...) in write_file().
8.8. K8sPoolSandbox: Deployment-backed pool
On-demand pod creation adds 5-30 seconds of latency per workflow (image pull, scheduling, container start). For interactive use (slash commands, reply-based resumption), this delay is user-facing. The pool eliminates it.
Mechanism. A Kubernetes Deployment maintains a set of pods labelled
hummingbird/role: standby. When a sandbox is needed, SandboxPool.claim()
finds a Running standby pod, patches it to role: active, clears its
ownerReferences (detaching it from the ReplicaSet), and annotates it with
hummingbird/reap-by (an absolute UTC deadline). The Deployment controller
sees the ReplicaSet is under the desired replica count and creates a
replacement.
Pod lifecycle.
- Deployment creates pod ->
role: standby(managed by ReplicaSet) claim()patches ->role: active,ownerReferences: [],reap-by: now + max_active_seconds- Agent uses the pod via inherited
K8sSandbox.exec() - On success:
linger()patches ->reap-by: now + linger_seconds,session-id: <id>(pod stays alive) - On reply within linger window:
try_reclaim()patches ->reap-by: now + max_active_seconds, clearssession-id - On failure or linger expiry: pod is deleted (by
cleanup()or the reaper)
Pod lingering. After a successful workflow, pool sandboxes are kept
alive for linger_seconds (default 300) instead of being deleted
immediately. The pod is annotated with hummingbird/session-id to link it
back to the session. If a user reply arrives within the linger window,
try_reclaim() finds the pod by session-id, clears the annotation (marking
it as in-use), and resets reap-by. This skips pod creation and S3 archive
restoration. If no reply arrives, the reaper deletes the pod when its
reap-by deadline passes.
Concurrency safety. session-id is only present while lingering. Its
absence during active execution prevents concurrent replies from adopting
the same pod. try_reclaim() is serialized by a threading lock; the first
caller wins, others fall back to S3 restore with a fresh pod.
Reaping. reap_expired() is called once per workflow execution (after
sandbox acquisition, before the model loop). It performs a single sorted
pass over all active pods:
- Non-lingering pods (no
session-id) past theirreap-bydeadline are deleted immediately. - Lingering pods (with
session-id) are sorted byreap-byascending. The loop deletes pods that are either expired (now > reap-by) or exceedmax_lingering_pods(default 2, configurable), starting with those closest to their deadline. The loop stops when both conditions are satisfied (now <= reap-byand remaining count <= limit).
This replaces the previous design where _reap_expired() ran inside
claim() on every poll iteration. Moving it to a once-per-workflow call
reduces API load and centralizes cleanup. The max_lingering_pods cap
prevents unbounded pod accumulation from many short-lived workflows.
Indefinite wait. claim() polls until a pod is available or the
shutdown event fires. It logs at DEBUG level on each poll, escalating to
WARNING after ~2 minutes. This is normal behavior when the pool (Deployment
replicas) is smaller than max_concurrent_agents – the SQS semaphore caps
concurrency, so demand will not permanently exceed supply.
Config simplification. In pool mode, the pod image, resources, metadata,
and security context are defined solely in the Deployment template. The
agent config only needs sandbox.namespace and
sandbox.active_deadline_seconds. This eliminates duplication between the
agent configmap and the Deployment.
Class design. K8sPoolSandbox inherits from K8sSandbox to reuse
exec(), write_file(), read_file_iter(), and cleanup(). It overrides
__init__() (does not call super().__init__() since that resolves
kubeconfig), start() (claims from pool instead of creating a pod), and
linger() (patches annotations instead of deleting). It adds
start_from_existing() for the reclaim path. All sandbox backends
implement linger(session_id): non-pool backends fall back to cleanup().
8.9. KubeVirtSandbox: SSH-based VM execution
KubeVirtSandbox provides full VM isolation using KubeVirt
VirtualMachineInstances. It parallels K8sSandbox but uses
CustomObjectsApi for VMI lifecycle and ssh for command execution
instead of kubectl exec.
Why SSH? VMs have no kubectl exec equivalent. KubeVirt provides
virtctl console (serial) and VNC, but these are not suitable for
programmatic command execution. SSH provides a reliable, well-understood
exec channel with stdin/stdout/stderr piping.
SSH key injection. The VM sandbox image includes a custom
inject-ssh-keys.service that reads SSH public keys from a KubeVirt
Secret volume attached as a virtio disk with serial ssh-pubkeys
(visible at /dev/disk/by-id/virtio-ssh-pubkeys) and installs them to
/root/.ssh/authorized_keys before sshd starts. This avoids the need
for cloudInitNoCloud or qemu-guest-agent.
Single-VM lifecycle:
- Generate an ephemeral ed25519 key pair via
ssh-keygen - Create a K8s Secret with the public key
- Create the VMI referencing the containerDisk image and the SSH Secret
- Poll VMI status until Running with a pod network IP
- Poll SSH readiness (
ssh sandbox@ip true) - Execute workflow commands via SSH
- Cleanup: delete VMI, delete Secret, remove temp key files
Polling strategy. Two-phase wait: first poll status.phase via
get_namespaced_custom_object until Running (same interval as pod
polling), then poll SSH readiness with ConnectTimeout=5. Total timeout
matches POD_START_TIMEOUT (300s) for VMI startup plus
_SSH_READY_TIMEOUT (120s) for SSH.
8.10. KubeVirtPoolSandbox: VMIRS-backed pool
KubeVirtPoolSandbox + VmiPool parallel K8sPoolSandbox +
SandboxPool, operating on VMI custom resources instead of Pods.
VirtualMachineInstanceReplicaSet (VMIRS). KubeVirt’s native
controller maintains a desired number of identical VMIs. When a VMI is
claimed (label flipped to active, ownerReferences cleared), the VMIRS
automatically creates a replacement – the same pattern as a Deployment
replenishing claimed pods.
Shared SSH key. Pool mode uses a single key pair for all standby VMIs.
The public key is stored in a K8s Secret referenced by all VMIs in the
VMIRS template. The private key path is provided via
sandbox.vm_ssh_private_key in the config. On pool restart, a new key
pair can be generated; existing VMIs drain naturally with the old key.
Claim/reap/linger semantics. Identical to the pod pool: claim()
polls for Running VMIs with hummingbird/role=standby, patches to
active, clears ownerReferences. reap_expired() deletes VMIs past
their hummingbird/reap-by deadline. try_reclaim() finds lingering
VMIs by hummingbird/session-id. The only difference is the API: all
operations go through CustomObjectsApi with kubevirt.io/v1
virtualmachineinstances.
Class design. KubeVirtPoolSandbox inherits from KubeVirtSandbox
to reuse exec(), write_file(), read_file_iter() (all SSH-based).
It overrides __init__() (takes pool, skips key generation),
start() (claims from pool), cleanup() (only deletes VMI, not
Secret/keys which are pool-owned), and linger() (patches annotations).
4.5.20.4 - Hummingbird Agent Evals
Evaluation framework for scientific measurement of LLM output quality. For the agent itself, see Hummingbird Agent. For architecture details, see Hummingbird Agent Design.
What evals are
Evals measure the quality of LLM outputs under controlled conditions. They answer questions like “does prompt v5 produce better code reviews than v4?” or “can an LLM answer this question correctly using only public documentation?”
Evals are not unit tests. Unit tests verify deterministic behavior; evals quantify stochastic output quality through structured scoring.
General pattern
Evals generally follow a pattern of:
- Inputs – a set of test cases: golden dataset with scoring criteria, real MRs to review, or other structured inputs.
- Model execution – run one or more LLM configurations against the inputs, optionally with tools (web search, sandbox).
- Scoring – evaluate outputs via LLM-as-judge, comparison against reference answers, or human review.
- Iteration – adjust prompts or config, re-run, check for improvement.
Not all evals use every step. Baseline measurements may stop at step 3; comparison evals may skip the golden dataset entirely.
Shared infrastructure
hummingbird_agent.eval
The hummingbird_agent.eval module provides generic evaluation
primitives used across all evals. Key components:
EvalJudge– LLM-as-judge wrapper over Vertex AI with explicit temperature and token limit control. Supports both strict judging (low temperature) and open-ended analysis (higher temperature).- Snapshot management – saves timestamped evaluation results with
metadata, maintains a
latestsymlink. - Variant tracking – persists the best-performing prompt or config variant across iterations.
- Convergence detection – checks whether scores have plateaued across consecutive iterations.
- Holdout validation – compares holdout scores against dev scores using standard error of the mean.
See the module docstrings for the full API.
hummingbird_agent.evals.mr_helpers
MR-experiment-specific helpers built on top of hummingbird_agent.eval.
Provides GitLab auth, MR data I/O, review generation orchestration, and
CLI scaffolding. Used by the code-review and renovate-triage evals.
Running an eval
Prerequisites
-
GCP Application Default Credentials:
gcloud auth application-default login export GOOGLE_CLOUD_PROJECT=<your-gcp-project> -
The agent package installed in editable mode:
cd hummingbird-agent pip install -e .
Execution
Each eval is a Python module under hummingbird_agent/evals/. Run from
hummingbird-agent/:
python -m hummingbird_agent.evals.<name>.<script> [options]
Refer to each eval’s own README for specific options and configuration.
Existing evals
- Documentation Quality Evaluation – measures whether public documentation answers contributor questions
Writing a new eval
Create a new directory under hummingbird_agent/evals/ with:
README.md– what the eval measures and how to run it, linking back to this document for the general framework.- Golden dataset – inputs and scoring criteria. Can live outside the eval directory (e.g. in the documentation repo alongside the content being measured) and be passed via CLI.
run_*.py– orchestration script usinghummingbird_agent.evalprimitives (orhummingbird_agent.evals.mr_helpersfor MR-based evals).
The eval’s run script should be importable as a module
(python -m hummingbird_agent.evals.<name>.<script>). Add an __init__.py if needed.
Do not hardcode GCP project IDs; VertexAuth resolves them at
runtime from GOOGLE_CLOUD_PROJECT or ADC credentials.
Authentication
All evals use VertexAuth from hummingbird_ai.http, which
resolves credentials and project in this order:
- Explicit
project=parameter (if passed). GOOGLE_CLOUD_PROJECTenvironment variable.- ADC default project (from
gcloud auth application-default login).
Service account keys work automatically when
GOOGLE_APPLICATION_CREDENTIALS is set.
4.5.20.5 - Documentation Quality Evaluation
Reference for the documentation quality evaluation, which measures whether public documentation is complete enough for LLMs with web search to answer questions correctly. For the general evaluation framework, see Hummingbird Agent Evals.
Config file
A single YAML file combines evaluation settings and the golden dataset.
It is passed to the run script via -c:
python -m hummingbird_agent.evals.docs_eval.run_eval -c path/to/docs_eval.yml
The config file lives in the documentation repo alongside the content being measured.
Evaluation settings
| Key | Required | Default | Description |
|---|---|---|---|
models |
yes | – | List of model configurations |
models[].id |
yes | – | Vertex AI model identifier |
models[].region |
yes | – | Vertex AI region |
models[].google_search |
no | false |
Enable Google Search grounding (Gemini) |
models[].web_search |
no | false |
Enable web search tool (Claude) |
judge |
yes | – | Judge model configuration |
judge.model |
yes | – | Judge model identifier |
judge.region |
no | global |
Judge model region |
repetitions |
no | 3 |
Runs per (question, model) pair |
max_iterations |
no | 5 |
Agent loop iteration cap per question |
Golden dataset
The questions list defines what the evaluation measures:
| Field | Required | Default | Description |
|---|---|---|---|
id |
yes | – | Unique identifier (used in output paths) |
question |
yes | – | Question sent to each model |
facts |
yes | – | List of fact checks |
facts[].question |
yes | – | Yes/no question about the response |
facts[].weight |
no | 1 |
Relative importance |
coherence |
no | – | Criterion for narrative quality scoring |
coherence_weight |
no | 1.5 |
Weight of coherence in the final score |
threshold |
no | 0.4 |
Minimum weighted score to pass |
CLI options
| Option | Default | Description |
|---|---|---|
-c / --config |
required | Path to config YAML |
--phase |
all |
ask, judge, or all |
--results-dir |
results/ next to config |
Output directory |
--run-id |
UTC timestamp | Subdirectory under results |
Scoring
Each response is scored independently:
- Fact scoring – the judge answers each fact question with yes or no. Passing facts contribute their weight to the numerator.
- Coherence scoring – the judge rates narrative quality from 0.0 to 1.0. The coherence score is multiplied by its weight and added to the total.
- Weighted score –
sum(passed_fact_weights + coherence_contribution) / total_weight. A response passes ifweighted_score >= threshold.
The summary aggregates per-model averages across all repetitions.
4.5.20.6 - Hummingbird Agent Design — Runtime
Data sources, event pipeline, sessions, workflows, and deployment for the Hummingbird Agent. This page covers the runtime: how the agent connects to external systems, processes events, manages sessions, and is deployed. For the execution engine (agent loop, tool system, sandbox), see Hummingbird Agent Design — Engine.
9. Data Sources
Data sources are external APIs wrapped as tool-calling functions. The model invokes them by name; the orchestrator executes them and returns results (inline or auto-spilled). Each data source module follows the same pattern:
- Define
TOOL_DEFS– a list ofToolDefobjects with names, descriptions, parameter schemas, and optional notes. - Define implementation functions that take a pre-configured client as the first argument.
- Provide a
register()function that creates the client, wraps implementations withfunctools.partial, and callsregistry.register_data_source()for each tool.
Credentials are captured at registration time via functools.partial
closures. They are never stored as global state and never leak into tool
definitions or model contents.
9.1. GitLab
Uses python-gitlab library with retry_transient_errors=True for
automatic retry on transient HTTP errors.
Tools:
gitlab_get_mr_details– MR metadata (title, author, SHA, labels, URLs)gitlab_get_commit_statuses– all CI/CD statuses for a commit SHA (paginated automatically). Covers both Konflux external statuses and native GitLab CI job statuses. Each status includestarget_url(job page link containing the job ID) andallow_failure.gitlab_get_mr_diff– changed files in the MRgitlab_get_file_at_ref– raw file content at a git refgitlab_get_repo_archive– repository tar.gz viarepository_archive(iterator=True). Returns aStreamingResponsewith chunked binary data (suffix.tar.gz), so the archive streams to the sandbox without buffering in orchestrator memory.gitlab_get_job_log– job trace (log output) for a GitLab CI job. Useslazy=Trueon the job object andtrace(iterator=True)for streaming. Each chunk is decoded witherrors='replace'and ANSI escape codes are stripped per-chunk before re-encoding. Returns aStreamingResponse(suffix.log). Per-chunk ANSI stripping is safe because escape sequences are <20 bytes and chunks are 1024+ bytes.
No response_metadata is set because GitLab commit statuses already contain
target_url fields that the model uses for linking.
gitlab_get_mr_discussions – trust filtering and redaction.
MR discussions are the first data source that feeds user-generated free text into the model prompt. Unlike diffs and CI logs (which are code or machine output), discussion comments can contain arbitrary prose written by anyone who can post on the MR – including external contributors on public projects. This creates a prompt injection vector: an attacker posts a comment containing instructions that manipulate the model’s review output.
The discussions tool addresses this with a three-layer defence:
-
Author trust gate. Each note’s author is checked for Developer+ access (level >= 30) on the project via
members_all.get(author_id). Results are cached perauthor_idwithin a single call to avoid repeated API lookups. System notes (merge events, label changes) bypass the author check. Agent-authored notes are trusted via their author’s Developer+ access (the bot account holds Developer access on configured projects). Note: agent note detection for redaction purposes (stripping transcripts and footers) uses session marker presence, but trust is always based on the author’s access level, not the marker. This prevents marker injection from granting trust to non-member notes. -
Redaction, not sanitization. Untrusted notes in a discussion that also contains trusted notes are replaced with a fixed placeholder (
[redacted: non-member comment]). The placeholder preserves the conversation structure (the model sees that someone replied, but not what they said). Discussions where all notes are untrusted are dropped entirely. No attempt is made to sanitize or escape untrusted content – there is no reliable escaping mechanism for LLM prompts, so the only safe option is to withhold the content entirely. -
Agent note stripping. Agent-authored notes contain large
<details><summary>Agent transcript</summary>...</details>blocks (15K-90K chars of raw tool calls), metadata footers, and session markers. These are stripped before the note enters the model prompt, leaving only the review text. This serves dual purposes: token efficiency and avoiding feeding the model its own raw tool call history (which could cause degenerate self-referential loops).
Residual risks and scope limitations:
- A compromised Developer+ account can inject adversarial content. This is accepted as equivalent to the existing risk of a compromised developer pushing malicious code (which the agent would also process).
- The trust threshold is project-level (Developer role on the project), not MR-level. A developer on the project can influence any MR’s review.
- Comments posted after the discussions are fetched but before the review is posted are not seen. This is a TOCTOU gap but has no security impact (the model simply misses late comments).
9.2. Konflux
Uses raw requests against K8s and Kubearchive APIs (not the kubernetes
Python SDK). This avoids the heavy kubernetes client dependency for what is
essentially bearer-token HTTP with label selectors.
Dual-fetch architecture:
flowchart LR
subgraph fetch [Fetch Phase]
KA["Kubearchive<br/>(historical)"]
K8s["K8s API<br/>(live)"]
end
Combine["Combine items"]
Dedup["Deduplicate<br/>by metadata.uid"]
KA --> Combine
K8s --> Combine
Combine --> Dedup
For each resource type (PipelineRuns, TaskRuns), the client fetches from
both Kubearchive (completed/historical resources) and the live K8s API,
combines the results, and deduplicates by metadata.uid. This ensures no
resources are missed regardless of whether they have been archived yet.
Two label selectors. Konflux uses different labels for BUILD and TEST PipelineRuns:
- BUILD:
pipelinesascode.tekton.dev/sha=<commit_sha> - TEST:
pac.test.appstudio.openshift.io/sha=<commit_sha>
Each iter_pipelineruns()/iter_taskruns() call queries both selectors and
combines the results. _iter_resources(endpoint, label_selectors) is the
generic implementation: it creates one session, iterates each selector through
_iter_both_sources(), and deduplicates across selectors by UID.
Per-PLR TaskRun fetching. iter_taskruns() accepts either commit_sha
(two-selector path) or pipelinerun (single selector:
tekton.dev/pipelineRun=<name>). When pipelinerun is set, only the
~15-21 TaskRuns for that PipelineRun are fetched (~600 KB) instead of all
TaskRuns for the commit (~530 MB for large commits). This is server-side
filtering — the K8s API and Kubearchive both support the
tekton.dev/pipelineRun label selector.
Status filtering and streaming counts. _items_to_jsonl(items, status, stats) accepts an optional status: set[str] | None allowlist and a
_StreamStats accumulator. For each item it increments stats.by_reason
(keyed by reason) regardless of the filter (so counts reflect all items
seen), then skips items whose status.conditions[0].status is not in the
allowlist. The allowlist values are "True" (success), "False" (failure),
or "Unknown" (in-progress) — only 3 possible values, covering every Tekton
reason without enumeration. Items without conditions always pass. The
_finalize() generator wrapper exhausts the chunks iterator, then writes
counts, included, and excluded into the StreamingResponse.metadata
dict — after the sandbox file is fully written. The filtering happens O(1)
per item with no buffering.
StreamingResponse.metadata. StreamingResponse (in tools.py) has a
mutable metadata: dict[str, Any] field (default empty dict). Both
_exec_fetch_to_sandbox() and _spill_streaming() merge it into the response
dict after the iterator is exhausted. The Konflux code populates it with
counts, included, and excluded; other data sources can use it for their
own post-streaming summaries without touching tools.py.
Pod log fetching. get_pod_log() accepts an optional container
parameter. When specified, it fetches logs for that single container. When
omitted, it discovers all containers from the pod spec and concatenates
their logs with === container_name === headers. Each individual log
fetch tries Kubearchive first, then falls back to the live K8s API. Logs
may be unavailable from both sources if the pod has expired.
Streaming pagination. K8s list endpoints can return pages of 80+ MB
when a commit touches many components (e.g. 500 PipelineRuns per page).
iter_paginated() uses session.get(stream=True) and ijson.parse() to
stream-parse each page: items are yielded one at a time via
ObjectBuilder, and the metadata.continue pagination token is captured
from the same parse pass. resp.raw.decode_content = True is set so
urllib3 transparently decompresses gzip/deflate Content-Encoding inline
(Kubearchive returns gzip-compressed responses). Only one item is in
memory at a time – O(single_item) regardless of page size. Truncated or
malformed streams raise IncompleteJSONError, which is caught and treated
like a network error (log a warning, stop iterating that source).
Credential resolution. KonfluxClient.__init__() parses the kubeconfig
file to extract the API server URL and bearer token for the cluster. The
cluster domain is extracted from the cluster_url config value. This
approach avoids depending on kubectl or the kubernetes Python library for
API authentication.
Response metadata: Static metadata {"konflux_ui": "...", "konflux_pipelinerun_url_pattern": "..."} is merged into every response so the model can build reviewer-facing links.
Dynamic per-call metadata (counts, included, excluded) is populated
into StreamingResponse.metadata by _finalize() during streaming and
merged by tools.py after the iterator is exhausted.
9.3. Testing Farm
Uses requests.Session with a module-level retry adapter (429, 5xx) for
resilience against transient errors.
Tools:
tf_get_results– JUnit XML results for a request IDtf_get_test_log– individual test log by URL (from results.xml); restricted to theARTIFACTS_BASEprefix to prevent SSRFtf_get_request_status– request state, queue/run times
ToolDef notes on tf_get_results document the XML structure:
//testsuites/@overall-result, //testcase/@result, //testcase/logs/log
with @name and @href. This domain knowledge goes into the system prompt
so the model knows how to parse the XML with sandbox_exec using
python3 xml.etree.ElementTree.
Response metadata: {"artifacts_base": "https://artifacts.osci.redhat.com/testing-farm"}
is merged into every response so the model can build artifact links like
[{request_id}]({artifacts_base}/{request_id}/).
9.4. Registration flow
data_sources.register_selected() is the entry point called by runner.py.
It iterates the workflow config’s data_sources dict and registers only the
declared sources:
for ds_name, ds_cfg in wf_cfg.data_sources.items():
if ds_name == "gitlab":
token = resolve_tool_token(proj_entry, "gitlab", wf_cfg)
gitlab.register(registry, gitlab_url, token)
elif ds_name == "konflux":
cluster_url = resolve_cluster_url(ds_cfg)
kubeconfig_path = os.environ.get(ds_cfg.kubeconfig_env, "")
konflux.register(registry, cluster_url, kubeconfig_path, ds_cfg.kubearchive_url)
elif ds_name == "testing_farm":
testing_farm.register(registry)
This selective registration means a workflow with data_sources: {gitlab: {...}}
only exposes GitLab tools to the model. Konflux and Testing Farm tools do
not appear in the tool definitions, preventing the model from attempting to
use unconfigured data sources.
9.5. Data flow: streaming vs buffered
Data moves from external APIs through the orchestrator to the model (inline or auto-spilled to sandbox). The memory profile of each tool depends on whether the HTTP response is consumed incrementally or buffered entirely.
Streaming (preferred for large/unbounded responses). The HTTP response
is consumed incrementally – via ijson stream parsing, iterator=True
in python-gitlab, or lazy pagination. Only one chunk or item is in memory
at a time. Used by:
iter_paginated()(Konflux) –session.get(stream=True)+ijsonget_commit_statuses()(GitLab) –statuses.list(iterator=True)via_iter_commit_statuses(), yielding JSONL bytes one status at a timeget_repo_archive()(GitLab) –repository_archive(iterator=True), returns chunked tar.gz binary viaStreamingResponse(suffix=".tar.gz", binary=True)– no text preview is generatedget_job_log()(GitLab) –trace(iterator=True)with per-chunk ANSI stripping, returnsStreamingResponse(suffix=".log")
All produce StreamingResponse objects that flow through
stream_to_file() into the sandbox without accumulating in orchestrator
memory. StreamingResponse.suffix controls the auto-spill filename
extension (.jsonl, .tar.gz, .log). StreamingResponse.binary
controls whether a head preview is captured (False for text, True to
skip for opaque binary formats).
Buffered (acceptable for small/bounded responses). The full response is loaded into memory as a string or dict. This is fine when the response size is bounded and small (typically < 1 MB). Used by:
get_mr_details()(GitLab) – single MR object, < 10 KBget_file_at_ref()(GitLab) – single file, bounded by repo constraintstf_get_results(),tf_get_test_log(),tf_get_request_status()(Testing Farm) – XML/text/JSON, typically < 1 MB
Buffered with risk (candidates for future streaming). Same buffered pattern but the response size is not bounded by design. Auto-spill mitigates the context growth problem (large outputs are written to sandbox files), but the orchestrator still spikes RSS during the fetch:
get_pod_log()(Konflux) –resp.textper container, concatenated. Multi-container pods accumulate all logs.get_mr_diff()(GitLab) – full diff JSON, scales with MR size. GitLab truncates server-side but the result can still be large.
Invariant: paginated K8s list endpoints must always use streaming. Page sizes scale with the number of components in the commit – a single page can contain hundreds of PipelineRuns or TaskRuns (80+ MB JSON). Buffering these responses risks OOM under normal production workloads, especially with concurrent workflows.
10. Event Pipeline and Production Operations
10.1. Two-stage SQS pipeline (ingress router + FIFO worker)
Events flow through a two-stage pipeline to serialize per-session work:
flowchart LR STD["Standard SQS<br/>(ingress)"] R["Router"] FIFO["SQS FIFO<br/>(work queue)"] W["Worker<br/>(poll_loop)"] STD -->|ReceiveMessage| R R -->|"SendMessage<br/>MessageGroupId"| FIFO W -->|"continuation<br/>group=session_id"| FIFO FIFO -->|ReceiveMessage| W
Ingress router. events.router_loop() is a single-threaded loop
that long-polls the standard (ingress) queue, peeks into each SNS
envelope to assign a MessageGroupId, forwards the raw message body
to the FIFO queue, then deletes from the standard queue.
- Note events:
MessageGroupId = discussion_idfrom the webhook body. This serializes replies to the same agent session – FIFO delivers at most one in-flight message per group. - Pipeline / MR events:
MessageGroupId = SQS MessageId(unique per message). No serialization – each event is its own group. - MessageDeduplicationId: Always the SQS
MessageIdfrom the standard queue. Absorbs duplicate deliveries from standard SQS’s at-least-once semantics (5-minute dedup window).
If SendMessage to FIFO fails, the message is not deleted from the
standard queue and retries via visibility timeout.
FIFO worker. events.poll_loop() long-polls the FIFO queue,
decodes SNS envelopes, and dispatches to handle_event. The
max_concurrent_agents semaphore caps parallel groups being processed.
FIFO guarantees that within a group, only one message is in-flight.
Two-phase processing. When the FIFO worker picks up a webhook
(Phase 1), the handler validates the event, resolves the session_id,
then posts a continuation message back to the same FIFO with
MessageGroupId = session_id. The Phase 1 message is deleted quickly
(~1-5s). Phase 2 processes the continuation: loading session state,
building WorkflowRequest, and calling _execute_workflow. Because
Phase 2 continuations share a MessageGroupId per session, all work
for a given session is serialized – even if it spans multiple GitLab
discussion threads.
Uniform message format. Both webhook messages (from SNS) and
continuation messages use the same SNS-style envelope with gzip+base64
compression. encode_envelope() is the inverse of
decode_sns_message(). The consumer doesn’t need to distinguish
between webhook and continuation messages at the transport layer.
Graceful shutdown. SIGTERM/SIGINT set a shutdown_event. Both
loops exit. Messages in SQS become visible again after the visibility
timeout for other consumers.
Why two queues. The standard queue receives events from SNS (via subscription). The FIFO queue serializes per-session work. The gitlab-event-forwarder and SNS subscription are unchanged – the routing logic lives entirely in the agent codebase.
10.2. SNS envelope decoding
Messages arrive as SNS notifications with:
MessageAttributes.source– event source (e.g.,"gitlab")MessageAttributes.event_type– event type (e.g.,"pipeline","merge_request","note")MessageAttributes.content_encoding–"gzip+base64"for gitlab-event-forwarder, absent for kubernetes-event-forwarderMessage– the actual event body (JSON string, or gzip+base64 encoded)
decode_sns_message() handles both encoding formats transparently.
10.3. Event routing
runner.handle_event() dispatches on (source, event_type):
flowchart TD
Event["Event arrives"]
Check{"source/type?"}
Cont["handle_continuation()"]
Pipeline["handle_pipeline()"]
MRHandler["handle_merge_request()"]
Note["handle_note()"]
Ignore["Ignore"]
Event --> Check
Check -->|"agent/continuation"| Cont
Check -->|"gitlab/pipeline"| Pipeline
Check -->|"gitlab/merge_request"| MRHandler
Check -->|"gitlab/note"| Note
Check -->|"other"| Ignore
Pipeline --> FilterSource{"source = merge_request_event?"}
FilterSource -->|"yes"| AutoResolve["auto-resolve on success<br/>(unconditional)"]
FilterSource -->|"no"| Skip1["Skip"]
AutoResolve --> PipeRules["evaluate trigger_rules<br/>(status + user + branch + title)"]
PipeRules -->|"allow"| Enqueue1["_enqueue_continuation()"]
PipeRules -->|"deny"| Skip1a["Skip"]
MRHandler --> FilterMRAction{"action in open/update?<br/>draft = false?"}
FilterMRAction -->|"yes"| MRRules["evaluate trigger_rules<br/>(user + branch + title)"]
FilterMRAction -->|"no"| Skip1b["Skip"]
MRRules -->|"allow"| Enqueue2["_enqueue_continuation()"]
MRRules -->|"deny"| Skip1c["Skip"]
Note --> FilterMRNote{"MR note?<br/>action = create?"}
FilterMRNote -->|"yes"| NoteType
FilterMRNote -->|"no"| Skip3["Skip"]
NoteType{"Note type?"}
NoteType -->|"author_id == bot_id"| Skip4["Skip (own note)"]
NoteType -->|"/hummingbird help"| Help["post_simple_reply (help)"]
NoteType -->|"/hummingbird wf-name"| SlashCmd["_resolve_slash_workflow()<br/>→ _enqueue_continuation()"]
NoteType -->|"DiscussionNote reply"| FindSession["find_session_for_reply()"]
NoteType -->|"other"| Skip5["Skip"]
FindSession --> Found{"session found?"}
Found -->|"yes"| Enqueue3["_enqueue_continuation()<br/>(resume_session)"]
Found -->|"no"| Skip6["Skip"]
Cont --> ContType{"work_type?"}
ContType -->|"new_session"| ExecNew["_execute_workflow()"]
ContType -->|"resume_session"| LoadS3["load session from S3<br/>→ _execute_workflow()"]
Phase 1 handlers (handle_pipeline, handle_merge_request,
handle_note) perform validation, auth checks, and session resolution,
then post a WorkOrder continuation message via _enqueue_continuation().
They do not call _execute_workflow directly.
Phase 2 (handle_continuation) receives the WorkOrder, resolves
the workflow config, builds a WorkflowRequest, and calls
_execute_workflow. For resume_session orders, it loads the previous
session from S3. _execute_workflow handles SHA dedup and per-workflow
rate limiting via scan_agent_threads.
Slash commands dispatch to a single named workflow via
_resolve_slash_workflow (exact match, then prefix match). Help and error
replies are posted via post_simple_reply (no session marker, no rate
limit impact).
10.4. Pipeline trigger design
Why gitlab::pipeline instead of kubernetes::PipelineRun: A GitLab
pipeline event fires once when the pipeline completes. Since Konflux
external stages are attached to the pipeline, the event naturally waits for
all builds and tests to finish before triggering. This means the agent sees
the full picture in one event, without needing to deduplicate or wait for
stragglers.
Trigger filters:
statusin{failed}– only failed pipelines trigger analysissource == "merge_request_event"– only MR pipelines, not branch/tag
10.4a. Merge request trigger design
The handle_merge_request handler fires on MR open, reopen, and update
events (action in {"open", "reopen", "update"}). Draft MRs are skipped
(object_attributes.draft == True); marking a draft as ready triggers a
review since the event arrives with draft: false.
Event classification is deliberately simple: the handler does not inspect
oldrev or changes.draft fields. Instead, SHA-based deduplication in
_execute_workflow (via JSON session markers) ensures each code revision
is reviewed at most once. Metadata-only updates (title, label changes) on
an already-reviewed SHA are silently skipped.
10.5. Note trigger design
Two sub-flows:
Slash command (/hummingbird <workflow-name>): Triggers a specific
named workflow, bypassing rate limiting. _resolve_slash_workflow performs
exact-match lookup first, then falls back to unique prefix matching. If
the subcommand is missing or is “help”, _format_help returns a list of
available workflows with descriptions. Ambiguous or unknown subcommands
produce an error message via post_simple_reply. The note author must
have Developer+ access (level >= 30) on the project, checked via
check_member_access(). If denied, the agent replies in the same
discussion thread with a short access-denied message.
Reply to agent note: When a user replies to an existing agent note (which contains a JSON session marker):
find_session_for_reply()walks the discussion thread, filtering by bot author ID (get_bot_user_id()), and returns the first matching session marker. All bot markers in a thread share the same session ID.handle_noteposts aresume_sessioncontinuation to the FIFO (grouped bysession_id).handle_continuationloads the session from S3 (conversation history and sandbox archive), builds aWorkflowRequest, and calls_execute_workflow()to resume with the user’s reply.- If the session is not found (expired/deleted), the agent replies with a message explaining the session has expired and suggesting to start a new run.
Reply authors are also subject to the Developer+ access check. If denied, the agent replies in the discussion thread with the same access-denied message.
Non-agent-directed notes: Regular comments that are neither slash commands nor replies to agent threads are silently ignored (debug-level log). No access check is performed for these.
Self-note filtering: Notes where author_id matches the bot’s own
user ID (resolved via get_bot_user_id()) are skipped immediately. This
prevents infinite loops where the agent’s own output triggers another
agent run. The author-ID check replaces the previous substring check
(marker_prefix_str in note_body), which was vulnerable to
denial-of-service: a user including the marker prefix in their reply
would cause the bot to silently ignore it.
Threading logic for replies: If the project requires internal notes
(internal_notes: true) but the original discussion was public, the reply
is posted as a new top-level internal note instead of replying in the
public thread. This prevents leaking internal analysis into public threads.
10.6. Note lifecycle
1. create_placeholder_note() -> "Running the <workflow> workflow..."
+ JSON session marker (id, wf, sha)
2. run_workflow() -> agent loop
3. update_note() -> replace placeholder with result
+ reply prompt + JSON session marker
Reply notes (from session resumption) omit the reply prompt since the
user is already engaged in the conversation.
On failure, the placeholder is updated to “Hummingbird analysis failed.” The JSON session marker is always present so the note can be identified as agent-generated (for rate limiting, SHA dedup, and self-note filtering). The marker embeds the workflow name and commit SHA, enabling per-workflow rate limiting and SHA-based deduplication.
10.7. Rate limiting and SHA deduplication
scan_agent_threads() performs a single pass over MR discussions, parsing
JSON session markers (<!-- hummingbird-session: {"id":...,"wf":...,"sha":...} -->)
to determine:
- Per-workflow thread count – how many threads the given workflow has
created on this MR. If the count meets or exceeds
max_runs_per_mr, the workflow is skipped. - SHA dedup – whether the current commit SHA has already been reviewed by this workflow. If so, the workflow is skipped (prevents re-reviewing the same code on metadata-only MR updates).
Slash commands and replies bypass both checks entirely.
The rate limit is per-workflow per-MR: different workflows maintain
independent thread counts on the same MR. JSON markers are backward
compatible – old plain-UUID markers (<!-- hummingbird-session: UUID -->)
are parsed as {"id": "UUID"} with no workflow or SHA information, so they
are not counted toward any specific workflow’s limit.
11. Session System
Sessions enable conversation continuity: a user can reply to an agent note and the agent picks up where it left off, with full context and sandbox files restored.
11.1. What gets saved
Three artifacts are saved after each run:
context.json– the fullcontentslist from the agent loop in the canonical provider-independent format. It is the minimum state needed to resume the conversation; format conversion is handled by the AI library.transcript.md– a human-readable markdown rendering of the run for debugging and auditing. Not used for resumption.sandbox.tar.gz– an archive of/tmp/data/(the sandbox working directory, which includes the_out/spill subdirectory). Contains PipelineRun JSONs, test logs, jq output, and any other files the model created during the run. Restored into the new sandbox on resumption so the model can reference its previous work.
11.2. Storage backends
S3 (production):
s3://{bucket}/sessions/{session_id}/context.json
s3://{bucket}/sessions/{session_id}/transcript.md
s3://{bucket}/sessions/{session_id}/sandbox.tar.gz
Local directory (development, --save-session):
{directory}/context.json
{directory}/transcript.md
{directory}/sandbox.tar.gz
Both backends have the same interface. S3 save is best-effort: wrapped in try/except so a transient S3 error does not prevent the MR note from being posted. The note always gets delivered first.
11.3. Sandbox archive transport
The sandbox archive requires special handling because the orchestrator cannot directly access the sandbox filesystem (no volume mounts):
Archive (inside sandbox):
sb.exec("tar czf /tmp/_archive.tar.gz -C /tmp data")
Stream out (sandbox -> host temp file):
for chunk in sb.read_file_iter("/tmp/_archive.tar.gz"):
tmp.write(chunk) # 64 KB chunks, no base64
Restore (host temp file -> new sandbox):
sb.stream_exec("tar xzf - -C /tmp", file_chunks())
Both directions use streaming binary I/O via Popen stdin/stdout pipes.
No base64 encoding is needed: read_file_iter() yields raw bytes from
the sandbox via a Popen stdout pipe, and stream_exec() feeds raw bytes
into a Popen stdin pipe. The archive is never extracted on the
orchestrator – it exists only as opaque bytes being transported between
sandboxes. This is a security invariant: the orchestrator never parses or
inspects the archive contents.
11.4. Resumption flow
flowchart TD
Reply["User replies to agent note"]
FindSession["find_session_for_reply()<br/>walks discussion thread"]
LoadS3["load_s3(session_id)"]
NotFound{"session found?"}
Expired["Post session-expired reply"]
NewSandbox["Start new sandbox"]
Restore["restore_sandbox(archive)"]
BuildContents["contents = old_contents + user_reply"]
RunLoop["run_agent_loop(<br/>initial_contents, CONTINUATION_PROMPT)"]
Reply --> FindSession --> LoadS3
LoadS3 --> NotFound
NotFound -->|"no"| Expired
NotFound -->|"yes"| NewSandbox
NewSandbox --> Restore --> BuildContents --> RunLoop
Key aspects:
- Fresh iteration budget. The resumed session starts iteration 0 with
the full
max_iterationsbudget, regardless of how many iterations the previous session used. - Context window is the real limit. A typical 8-iteration cold start
uses ~50-100K input tokens. The restored history is sent in full on every
API call. The
CONTEXT_LIMITcheck applies and will force wrap-up if needed. - Session ID reuse. The resumed session keeps the original session ID. S3 state is overwritten in place with the updated conversation history. Since GitLab discussions are linear (not branching), there is no need for a tree of sessions – the conversation is a single sequential thread.
- Graceful degradation. If the S3 session is not found (expired, deleted), the agent posts a reply explaining the session has expired and suggesting to start a new run. It does not fall back to a cold start.
11.5. Session markers
Every agent note contains a hidden HTML comment with a JSON payload:
<!-- hummingbird-session: {"id":"UUID","wf":"code-review","sha":"abc123"} -->
The JSON payload contains:
id– session UUID (always present)wf– workflow name (present for new-format markers)sha– commit SHA at the time of review (present for auto-triggered runs)
This marker serves four purposes:
- Resumption:
find_session_for_reply()searches discussion threads for this marker to find the session ID. Only bot-authored notes are considered; the first matching marker wins. - Per-workflow rate limiting:
scan_agent_threads()counts discussion threads for a specific workflow using thewffield. Only bot-authored notes are scanned. - SHA deduplication:
scan_agent_threads()checks whether the current SHA has already been reviewed by the workflow using theshafield. - Self-filtering:
handle_note()compares the webhook event’sauthor_idagainst the bot’s own user ID (viaget_bot_user_id()) to prevent infinite loops.
All marker-scanning functions resolve the bot user ID via gl.auth()
(GET /user) on the orchestrator token. This ensures markers in
non-bot notes (user replies, external contributors) are never parsed,
preventing session hijacking via injected markers.
Old plain-UUID markers (<!-- hummingbird-session: UUID -->) are parsed
as {"id": "UUID"} for backward compatibility. They are counted for
resumption but not for per-workflow rate limiting or SHA dedup (no
wf/sha information).
12. Workflow System
12.1. Separation of prompt and metadata
A workflow has two parts that live in different places:
- Prompt (
.mdfile): the LLM system prompt, verbatim. This is the investigation strategy, tool usage guidance, data patterns, and output format. Pure text, no code. - Metadata (YAML config): operational settings –
action,model,max_iterations,data_sources,projects. This controls what the orchestrator does with the workflow, not what the model does.
This separation means changing the analysis strategy (e.g., adding a new investigation step) requires editing a markdown file. Changing operational parameters (e.g., which projects use this workflow, which model to use) requires editing the YAML config. Neither requires a code change.
12.2. System prompt layering
The system prompt sent to the model is built from three layers:
BASE_SYSTEM_PROMPT # agent.py: sandbox rules, tool usage tips
+ tool_notes # from ToolRegistry: per-data-source domain knowledge
+ workflow_body # full content of e.g. workflows/analyze-failures.md
On session resumption, a fourth layer is appended:
+ CONTINUATION_PROMPT # agent.py: "do not re-run the full workflow"
The system prompt is rebuilt every iteration (to allow warning suffixes to be appended), but the base content is stable. Warning suffixes are appended at the end so they override earlier instructions.
12.3. Design choice: strategy, not procedure
The workflow .md file describes strategy and guidance, not a rigid script.
The model decides when and how to use each tool based on what it sees.
This matters because merge request failures are diverse. A fixed procedure would either miss edge cases (e.g., build failures before tests ran) or waste iterations on steps that don’t apply. By giving the model a strategy (“identify failing tests, fetch details, analyze root causes, group by similarity”), it can adapt to whatever it encounters.
The workflow does structure the investigation into phases (Data Collection, Analysis, Output) for clarity, but these are guidelines, not enforced checkpoints.
12.4. Workflow file anatomy (analyze-failures.md)
The primary workflow is structured as:
- Scope and approach – what this workflow analyzes and what it ignores
- Input – event JSON format (
project,iid) - Data Collection Phase:
- Data.1: Fetch MR details (direct call, small response)
- Data.2: Fetch commit statuses, identify failures
- Data.3: Batch fetch PipelineRuns + TaskRuns (fetch_batch_to_sandbox)
- Data.4: Process with jq, extract Testing Farm data in bulk
- Analysis Phase:
- Analysis.1a: Investigate each failed PipelineRun individually
- Analysis.1b: Summarize and group by root cause
- Output – markdown template with root causes, collapsible details, clickable links (PipelineRun, Testing Farm, test logs)
- Error Handling – partial report philosophy
Design considerations embedded in the workflow:
- Efficient bulk fetching: PipelineRuns and TaskRuns are fetched in one
fetch_batch_to_sandboxcall, not individually. - Testing Farm data extracted in Data.4, analyzed in Analysis.1: All TF results.xml are fetched in bulk before analysis starts, enabling cross-failure pattern detection.
- Log fetching is selective: Only 1-2 representative logs per failure pattern, not all logs. This keeps iteration count bounded.
- Reviewer-facing URLs: The output template instructs the model to
include clickable links using
konflux_uiandartifacts_basefrom response metadata.
12.5. Prompt file resolution
get_prompt(workflow_cfg, config_dir) resolves the prompt: field relative
to the config file’s directory:
prompt_path = config_dir / workflow_cfg.prompt
# e.g. /app/config.yaml with prompt: workflows/analyze-failures.md
# -> /app/workflows/analyze-failures.md
In the container image, workflows are baked in at /app/workflows/. A
ConfigMap can override them at deploy time by mounting at the same path.
13. Deployment and Container
13.1. Container build strategy
The Containerfile uses an all-RPM builder+installroot+scratch pattern (no pip, no venv):
- Builder stage: Uses a Fedora-based builder image with dnf. Installs
all dependencies as RPMs into a clean
--installroot. - Application code: Copies
hummingbird_agent/andworkflows/into the installroot at/app/. - Final stage:
FROM scratch, copies the entire installroot. No package manager, no shell beyond what RPMs provide.
RPM dependencies: python3-boto3, python3-google-auth+requests,
python3-gitlab, python3-kubernetes, python3-pyyaml, python3-requests,
python3-sentry-sdk, kubernetes1.35-client (for kubectl).
This approach was chosen over pip because:
- All deps come from Fedora’s package repository – no PyPI supply chain risk
- Smaller image (~22 MB saved vs. google-genai SDK alone)
- Reproducible builds from known RPM versions
- No compilation step (no gcc/python3-devel in the image)
13.2. Container runtime properties
CMD ["python3", "-m", "hummingbird_agent", "serve"]
WORKDIR /app
USER 65532
The default CMD runs serve mode for production. Local development uses
run mode via explicit command override. USER 65532 matches the standard
nonroot UID used by distroless images and the Podman sandbox.
13.3. K8s deployment manifests
Located in hummingbird-agent/kubernetes/:
deployment.yaml:
- Single replica with rolling update (
maxSurge: 1,maxUnavailable: 0) terminationGracePeriodSeconds: 900(15 minutes) to allow in-flight agent runs to complete on shutdown- ServiceAccount:
hummingbird-agent - Secrets mounted from K8s Secret
hummingbird-agent - Commented-out mounts for workflow ConfigMap and custom CA trust
rbac.yaml (applied in the sandbox namespace):
- ServiceAccount
hummingbird-agent - Role with
pods: create, get, list, delete, patchandpods/exec: create - RoleBinding linking the SA to the Role
patchis required for pool mode (relabeling pods during claim)
sandbox-pool.yaml (applied in the sandbox namespace):
- Deployment with
replicas: 3(tuned to balance latency vs cost) - Pods labelled
hummingbird/role: standbyfor pool discovery - Same security context and resource limits as direct-creation pods
revisionHistoryLimit: 2, rolling update withmaxSurge: 1,maxUnavailable: 0
networkpolicy.yaml:
podSelector: {}– applies to all pods in the namespaceegress: []– deny all outbound traffic
secret.yaml:
- Template with all required env vars (API keys, tokens, URLs)
- Values must be populated per deployment
13.4. Workflow mounting
Workflows are baked into the image at /app/workflows/. To update workflows
without rebuilding the image:
- Create a ConfigMap from the workflows directory
- Uncomment the volume mount in the deployment YAML
- The ConfigMap mount replaces the baked-in directory (all-or-nothing)
This is useful for rapid iteration in staging without waiting for a new image build.
13.5. Custom CA trust
For clusters with internal CA certificates (common in enterprise environments), the deployment supports OpenShift’s CA injection:
- Create a ConfigMap with the
config.openshift.io/inject-trusted-cabundlelabel - Mount it at
/etc/pki/custom - Set
REQUESTS_CA_BUNDLE=/etc/pki/custom/ca-bundle.crt
OpenShift automatically injects the cluster’s CA bundle into the ConfigMap.
4.5.20.7 - Hummingbird Agent Design — Decisions
Design Decision Registry (DDR) for the Hummingbird Agent — every major architectural choice, the alternatives considered, why the chosen approach won, and what would break if reversed. For the full design context, see Hummingbird Agent Design. Provider adapter decisions are in Hummingbird AI Design.
Design Decisions
kubectl exec over kubernetes Python exec API
- Chosen:
kubectl execas subprocess for sandbox command execution. - Alternative: kubernetes Python client
stream()API. - Why: Three showstopper bugs in the Python client: (1) WebSocket
v1-v4 has no stdin EOF signal, so
cat > /filehangs forever; (2)BrokenPipeErroron stdin larger than ~1MB; (3)stream()accumulates all stdout in memory with no control.kubectlavoids all three and provides the same subprocess interface as Podman. - If reversed:
write_file()would hang or fail on large data. Pod log retrieval with large outputs would OOM. Sandbox reliability would drop significantly.
YAML config over env vars for operational settings
- Chosen: Single YAML file for workflows, projects, limits, data source declarations.
- Alternative: Everything in env vars (original design).
- Why: Env vars cannot express structured data (workflow-project mappings, per-project token overrides, data source config with multiple fields). YAML provides structure, validation, and audit trails.
- If reversed: Token scoping would be lost (no per-workflow-project token overrides). Project allowlists would be impossible. The config would be unauditable.
Token env var names in YAML, not values
- Chosen: YAML contains
token_env: GITLAB_TOKEN_RO(the env var name), not the actual token value. - Alternative: Inline secrets in YAML, or env vars for everything.
- Why: The YAML file can be committed, reviewed, and audited. Actual secrets stay in env vars (injected via K8s Secrets). Inline secrets would make the config file a secret itself.
- If reversed: The config file would become a secret, breaking audit trails and code review workflows.
Orchestrator token prefix convention (ORCHESTRATOR_*)
- Chosen: Orchestrator tokens use
ORCHESTRATOR_GITLAB_TOKEN_*env var names by convention. - Alternative: Same env var namespace as model tokens, distinguished by context.
- Why: Structural separation makes it impossible to accidentally pass
an orchestrator token to a model tool (or vice versa). The
ORCHESTRATOR_prefix is never used in YAMLtoken_envfields. - If reversed: A misconfiguration could leak write-capable tokens to the model, which could then expose them via tool calls.
Auto-spill over conversation compaction
- Chosen: Large outputs are saved to sandbox files with previews returned to the model.
- Alternative: Replace old tool results in
contentswith compact summaries (conversation compaction). - Why: Compaction requires modifying the
contentslist, which risks violating provider conversation constraints. Auto-spill achieves ~90% of the token reduction without modifying model history. See Hummingbird AI for adapter format requirements. - If reversed: Token usage would increase ~15-30%. Per-call context would grow unbounded. Sessions would hit the context limit much sooner. Conversation compaction could be added on top of auto-spill in the future, but is not needed with current workloads.
Iteration count over cumulative token budget
- Chosen:
max_iterationsas the primary cost control lever. - Alternative: Cumulative token budget (stop when total billed tokens exceed a threshold).
- Why: With full history replay, each API call re-sends everything. Cumulative billed tokens double-count: call 1 = 5K, call 2 = 10K (including 5K again), total = 15K billed but only 10K new content. With auto-spill keeping per-call size bounded, iteration count is a much simpler and more predictable proxy for actual cost.
- If reversed: The budget model would be confusing and inaccurate. Cost estimates would be wrong. The cumulative metric is still logged for observability, but it does not drive termination.
ToolDef notes in system prompt, not tool schema
- Chosen: Domain knowledge (Konflux dual-fetch, TF XML structure) goes
in
ToolDef.notes, injected into the system prompt. - Alternative: Put everything in the tool schema
description. - Why: Tool schemas have character limits and are sent in the
toolsfield of every API call. Long descriptions waste tokens on tool defs. System prompt notes are sent once and can be arbitrarily detailed. - If reversed: Tool descriptions would be bloated. Domain knowledge
would need to be duplicated in every workflow
.mdfile.
Full history replay (no compaction)
- Chosen: The
contentslist grows monotonically. Items are never removed or modified (except empty response retries). - Alternative: Compact old turns to reduce context size.
- Why: Provider APIs impose strict conversation structure. Modifying or removing items risks invalid history. Auto-spill handles the growth problem at the source (preventing large items from entering history).
- If reversed: Risk of provider API errors from malformed conversation structure and of confusing the model with summarized results.
fetch_to_sandbox kept alongside auto-spill
- Chosen: Both
fetch_to_sandboxand auto-spill coexist. - Alternative: Remove
fetch_to_sandboxsince auto-spill handles large outputs automatically. - Why:
fetch_to_sandboxprovides explicit path control (model can choose meaningful filenames), batch fetching (one tool call for multiple fetches), and no-preview responses (useful when the model will process with jq anyway). - If reversed: The model would lose path control and batch fetching would require multiple auto-spilled calls. The workflow would need more iterations to achieve the same result.
K8s library for lifecycle + kubectl for exec (hybrid)
- Chosen: Use the
kubernetesPython library for pod create/read/delete,kubectlsubprocess for exec. - Alternative: All-kubectl (subprocess for everything) or all-library.
- Why: The library provides typed pod status polling and clean error
handling for lifecycle.
kubectlprovides reliable exec (see “kubectl exec over kubernetes Python exec API” above). Using the library for exec would require solving the three WebSocket bugs. - If reversed: Either lifecycle management would be fragile (parsing kubectl JSON output for pod status) or exec would be unreliable.
Single-file YAML config over multiple config files
- Chosen: Everything in one
config.yamlwithsettingsandworkflowssections. - Alternative: Separate files per workflow, or settings.yaml + workflows.yaml.
- Why: Single source of truth. All project-workflow mappings visible in one place. Settings-level defaults flow down to all projects. No file discovery logic needed.
- If reversed: Token resolution would need cross-file lookups. Project index would need multi-file aggregation. Config validation would be more complex.
Workflow .md as pure prompt, metadata in YAML
- Chosen: The workflow
.mdfile is pure system prompt text. Metadata (action,model,max_iterations,data_sources,projects) lives in the YAML config. - Alternative: YAML frontmatter in the
.mdfile (original design). - Why: Separation of concerns. The
.mdfile is the model’s instructions – it should be readable and editable by anyone writing prompts. The YAML config is the orchestrator’s instructions – it controls routing, limits, and credentials. Mixing them in one file conflates two audiences. - If reversed: Prompt authors would need to understand YAML config
structure. Token/project config would be scattered across
.mdfiles instead of centralized.
Nudge via system prompt suffix, not user message
- Chosen: Empty response nudge and budget warnings are appended to the system prompt as suffixes.
- Alternative: Inject synthetic user messages into
contents. - Why: User messages in
contentsmust come from the actual user (the initial event, or a reply). Injecting synthetic messages pollutes the conversation history that is saved in sessions. System prompt suffixes are transient – they affect one API call without permanently modifying the conversation state. - If reversed: Saved sessions would contain synthetic user messages. Resumed conversations would be confusing. The model might respond to the synthetic messages instead of the user’s actual input.
Session ID reuse over new-ID-per-resume
- Chosen: Resumed sessions keep the original session ID. S3 state is overwritten in place.
- Alternative: Each resume generates a new UUID, saving alongside the original (append-only tree of sessions).
- Why: GitLab discussions are linear, not branching. There is no
scenario where two different sessions from the same thread are both
valid. A new UUID per resume creates orphaned S3 snapshots that are
never referenced again, since
find_session_for_reply()always picks the latest marker. Reusing the ID is simpler, uses less storage, and matches the linear conversation model. - If reversed: S3 would accumulate orphaned session snapshots. Each resume would need a new note marker, but the old markers would still be in the thread, creating confusion about which session is current.
Reply to unauthorized agent-directed notes instead of silent skip
- Chosen: When an unauthorized user sends a slash command or replies to an agent thread, the agent replies in the same discussion with a short access-denied message. Non-agent-directed notes are silently ignored (debug log, no access check).
- Alternative: Log a warning and skip silently for all unauthorized notes (the original behavior).
- Why: Silent skip gives no feedback to someone who intentionally tried to engage the agent, which is confusing. On the other hand, checking access and logging a warning for every random comment on a public project is noisy and pointless. Moving the access check after intent detection cleanly separates the two cases. The reply uses the discussions API so it automatically inherits confidentiality from the parent note/thread.
- If reversed: Users without sufficient access who try
/hummingbirdwould get no feedback. The agent log would be noisy with warnings for every comment on public MRs.
Deployment-backed pod pool over Python-thread pool
- Chosen: A Kubernetes Deployment maintains pre-warmed standby pods. The agent claims a pod by relabeling it; the Deployment replaces it.
- Alternative: A Python-side thread pool that pre-creates pods and queues them for use.
- Why: The Deployment provides self-healing (restart crashed pods),
native scaling (
kubectl scale), rolling updates (image changes), and monitoring via standard K8s tooling. A Python pool would need to reimplement all of these. - If reversed: Pod replenishment, crash recovery, and image updates would all need custom code. Scaling would require agent redeployment.
Pod isolation: never reuse sandbox pods
- Chosen: Each workflow run gets its own pod. After cleanup, the pod is deleted. Claimed pods are detached from the ReplicaSet.
- Alternative: Return used pods to the pool and reset them.
- Why: Residual state from a previous run (files, environment, running processes) could leak between MR investigations, creating security and correctness risks. Deletion is simple and foolproof.
- If reversed: Would need a reliable pod-reset mechanism and auditing that no state survives between runs.
Absolute reap-by deadline over relative claimed-at age
- Chosen: Active pods are annotated with
hummingbird/reap-by(an absolute UTC timestamp). The reaper deletes pods wherenow > reap-by. - Alternative: Annotate with
claimed-atand compute age relative tomax_active_seconds; or useactiveDeadlineSecondson the pod spec. - Why: An absolute deadline simplifies the reaper to a single
comparison. It also supports varying deadlines: claim sets
reap-by = now + max_active_seconds, while linger setsreap-by = now + linger_seconds. With a relative timestamp, the reaper would need to know which mode the pod is in.activeDeadlineSecondsapplies from pod creation, not from claim – standby pods would expire before being used. - If reversed: The reaper would need mode-aware age calculations. Lingering pods would require a separate annotation or reaper path.
Indefinite claim wait over timeout
- Chosen:
SandboxPool.claim()polls indefinitely (governed by shutdown event), logging at DEBUG then WARNING. - Alternative: Timeout after N seconds and raise an error.
- Why: The pool is typically smaller than
max_concurrent_agentsfor cost reasons. Waiting for a replacement pod is normal operational behavior, not an error. A timeout would cause spurious failures during burst traffic. The SQS semaphore already bounds concurrency. - If reversed: Burst traffic would cause avoidable failures instead of brief delays.
Pool config in Deployment only, not in agent configmap
- Chosen: In pool mode (
k8spool), the pod image, resources, labels, and security context are defined solely in the Deployment template. The agent config only needsnamespaceandactive_deadline_seconds. - Alternative: Keep image/resources/metadata in the agent configmap too
(as in
k8smode). - Why: Eliminates config duplication. The Deployment template is the single source of truth. Changes to pod resources or image only require updating one place and re-rolling the Deployment.
- If reversed: Config drift between the Deployment and the agent configmap would be a constant risk.
Pod lingering over immediate cleanup for reply latency
- Chosen: After a successful workflow, pool sandbox pods linger for
linger_seconds(default 300) with asession-idannotation. Replies within the window reclaim the pod viatry_reclaim(), skipping pod creation and S3 archive restoration. - Alternative: Always delete the pod immediately and restore from S3 on every reply.
- Why: Reply latency drops from seconds (pod claim + S3 restore) to
near-zero. The S3 archive is still saved as a fallback if the pod is
gone. The
reap-byannotation ensures lingering pods are cleaned up if no reply arrives. Thesession-idannotation doubles as a concurrency guard: it is only present while lingering, preventing active pods from being reclaimed. - If reversed: Every reply would pay full pod + restore latency, even for immediate follow-ups. User experience for conversational interactions would degrade noticeably.
linger() on Sandbox protocol over isinstance checks in runner
- Chosen: All sandbox backends implement
linger(session_id). Non-pool backends fall back tocleanup(). The runner callssb.linger()without type checks. - Alternative:
isinstance(sb, K8sPoolSandbox)inrun_workflow(). - Why: Keeps the runner backend-agnostic. Adding a new backend requires only implementing the protocol, not touching the runner. The fallback behavior is co-located with each backend.
- If reversed: The runner would need to know about every backend type and their linger capabilities.
GCP SA key over Workload Identity Federation
- Chosen: GCP service account key stored in Vault, rotated via cki-tools credential manager (prepare/switch/clean cycle).
- Alternative: Workload Identity Federation (WIF) with projected SA token. Two sub-options: (a) automatic OIDC discovery if the cluster issuer is public, (b) manual JWKS upload for internal issuers.
- Why: mpp-prod’s OIDC issuer is
https://kubernetes.default.svc(internal, not publicly reachable), so GCP STS cannot discover it automatically. Manual JWKS upload works but requires re-upload after SRE-triggered signing key rotations. SA key integrates with the existing credential manager rotation infrastructure (same pattern as AWS keys and GitLab tokens), needs no OIDC reachability, and enables automated validate/update viagoogle-authin CI. The application code (hummingbird_ai.http.VertexAuth/google.auth.default()) works identically with both approaches – switching to WIF later requires only infrastructure changes. - If reversed: Replace SA key with WIF projected token + credential
config ConfigMap. The
gcp_service_account_keytoken type in cki-tools would no longer be needed for this use case.
Developer+ trust filtering over unfiltered or sanitized discussion comments
- Chosen:
gitlab_get_mr_discussionschecks each note author’s project access level (Developer+ / >= 30). Untrusted notes are replaced with a fixed placeholder in mixed-trust discussions, or the entire discussion is dropped if all notes are untrusted. Agent notes are always trusted (detected by session marker) but have transcripts and metadata stripped. - Alternatives considered:
- (a) Include all comments unfiltered. Simplest, but any external contributor can craft comments that manipulate the model’s review output (prompt injection).
- (b) Sanitize/escape untrusted content (strip markdown, quote as code blocks, prefix with “[external]”). There is no reliable escaping mechanism for LLM prompts – the model interprets natural language regardless of formatting. Escaping gives a false sense of security.
- (c) Only include agent-authored notes (skip all human comments). Safe, but defeats the purpose: the model would never see developer responses to its own findings.
- (d) Include only discussions where the agent participated. Better, but still misses developer-initiated review threads that provide relevant context.
- Why: Developer+ is the same threshold used for pipeline trigger authorization (invariant #8) and slash command access. It matches the trust boundary already established: people who can push code and approve MRs are trusted to provide review context. The placeholder approach preserves discussion structure (the model sees that someone replied) without exposing the content. Full-drop for all-untrusted discussions avoids noise from discussions that contain zero useful context.
- If reversed: (a) opens a prompt injection vector on any project that accepts external MRs. (b) provides no actual protection. (c) makes follow-up reviews unable to see developer explanations, causing repeated false positives. (d) misses developer-initiated context.
Bot-author filtering for session marker parsing over unfiltered note scanning
- Chosen:
find_session_for_reply(),scan_agent_threads(), and thehandle_note()self-filter all resolve the bot user ID viaget_bot_user_id()(which callsgl.auth()on the orchestrator token) and only consider notes whereauthor.idmatches. Infind_session_for_reply(), the first matching marker wins. - Alternative: Parse markers from any note and keep the last match;
substring self-filter in
handle_note(previous behavior). - Why: The previous unfiltered approach allowed session hijacking: a
user reply or agent review prose containing an example marker
(
<!-- hummingbird-session: ... -->) was parsed as a real session, causing “session expired” errors in production. Forhandle_note, the substring check caused the reverse problem: a user embedding the marker prefix in a reply silently suppressed the event (denial of service). Thegl.auth()call is oneGET /userper invocation – negligible cost._get_client_with_bot_id()returns both the authenticated client and the bot user ID, avoiding redundant client construction. - If reversed: Any MR participant can inject a marker to hijack the session ID or inflate rate-limit counts. The agent’s own review prose containing example markers causes spurious “session expired” messages.
ijson streaming over buffered JSON for K8s list responses
- Chosen:
session.get(stream=True)+ijson.parse(resp.raw)foriter_paginated(). Items are yielded one at a time viaObjectBuilder; themetadata.continuepagination token is captured in the same parse pass.resp.raw.decode_content = Trueis required because Kubearchive returns gzip Content-Encoding; without itijsonsees compressed bytes. - Alternative:
session.get().json()loads the full page into memory (the original implementation). - Why: A single K8s list page can contain 500 PipelineRuns (80+ MB
JSON). Parsing this with
.json()creates a +247 MB RSS spike (raw bytes + decoded string + parsed dict coexist). With 4 concurrent workflows, this exceeds any reasonable pod memory limit.ijsonstream-parsing yields items one at a time, reducing the peak to +12 MB for the same data – a 95% reduction. The continue token appears in themetadataobject (before or afteritemsdepending on the server); the event-driven parser captures it regardless of order. - If reversed: Large MRs (100+ components) would OOM the agent pod. Concurrent workflows would multiply the problem. The pod memory limit would need to scale with the largest possible page size, which is unbounded.
Action token tier for per-workflow identity
- Chosen: A dedicated Tier 2 of workflow action tokens, separate from both orchestrator tokens (Tier 1) and model tool tokens (Tier 3). Each workflow gets a dedicated GitLab bot user per project for all write operations (notes, thread resolution).
- Alternative: Extend orchestrator tokens with a per-workflow fallback
chain (
ORCHESTRATOR_<WORKFLOW>_GITLAB_TOKEN_<PROJECT>→ORCHESTRATOR_GITLAB_TOKEN_<PROJECT>→ORCHESTRATOR_GITLAB_TOKEN). - Why: Orchestrator tokens serve a fundamentally different purpose
(operational reads + infrastructure notes) than workflow output (analysis
results, review comments). Mixing them in one fallback chain risks a
single bot identity posting notes on behalf of multiple workflows,
eliminating the audit trail. A separate tier with explicit
token: strparameters makes the code path unambiguous: callers must choose which tier they use. The naming convention (HUMMINGBIRD_AGENT_ACTION_<WORKFLOW>_GITLAB_TOKEN_<PROJECT>) groups tokens alphabetically by workflow, making env var auditing straightforward. - If reversed: All workflow notes would appear from the same bot user, making it impossible to distinguish which agent (code-review vs analyze-failures vs renovate-babysit) produced a given note in the MR timeline. GitLab’s audit log would attribute all actions to one identity.
Per-project-only action token config (no workflow-level default)
- Chosen:
action_tokensis defined at the project level only in YAML config. No workflow-levelaction_tokensdefault is supported. - Alternative: Allow workflow-level
action_tokensthat projects inherit by default, with per-project overrides. - Why: GitLab project access tokens are project-scoped. A workflow-level default would be misleading – a single token cannot write to multiple projects. Forcing per-project declaration makes the mapping explicit and prevents configuration errors where a cross-project default silently fails.
- If reversed: Operators could set a workflow-level token that only works for one project, causing silent 403 errors for other projects in the same workflow.
SSH over kubectl exec for KubeVirt VMs
- Chosen: SSH (
ssh -T sandbox@ip command) for VM command execution. - Alternative:
virtctl console(serial console), KubeVirt websocket API, orqemu-guest-agent exec. - Why: VMs have no
kubectl execequivalent.virtctl consoleprovides a serial terminal, not a programmable exec channel. The websocket API requires a VNC or serial connection.qemu-guest-agentis not installed in the VM image. SSH provides clean stdin/stdout/stderr separation, timeout control, and is already well-tested in subprocess pipelines. - If reversed: Would need
qemu-guest-agentin the image and theexecendpoint, which has poorer error reporting and no streaming stdin.
Secret volume for SSH key injection
- Chosen: Kubernetes Secret mounted as a virtio disk (serial
ssh-pubkeys) read by the VM’sinject-ssh-keys.serviceat boot. - Alternative:
cloudInitNoCloudoraccessCredentialswithqemu-guest-agent. - Why: The VM image already has
inject-ssh-keys.servicewhich reads from the virtio disk.cloudInitNoCloudrequires cloud-init in the image.accessCredentialsrequiresqemu-guest-agent. The Secret volume approach requires no additional software in the VM image. - If reversed: Would need cloud-init or guest agent in the bootc image, adding complexity and attack surface.
VMIRS for VM pool replenishment
- Chosen:
VirtualMachineInstanceReplicaSet(VMIRS) as the replenishment controller for the VM pool. - Alternative: Custom controller, or VirtualMachine with RunStrategy.
- Why: VMIRS is KubeVirt’s native equivalent of a Deployment’s ReplicaSet – it maintains a desired count of identical VMIs. When a VMI is claimed (ownerReferences cleared), the VMIRS creates a replacement automatically. Same pattern as the pod pool with Deployment.
- If reversed: Would need a custom controller to maintain the VMI pool, adding operational complexity.
Per-workflow sandbox backend
- Chosen:
sandbox:field in workflow config, resolved per-workflow with fallback to CLI--sandboxflag. - Alternative: Global-only sandbox selection via CLI.
- Why: Different workflows have different isolation requirements. Code review needs only a lightweight container, while failure analysis may need a full VM with root access. Per-workflow selection avoids running all workflows in VMs (wasteful) or all in containers (insufficient isolation).
- If reversed: All workflows would share the same sandbox backend, requiring either over-provisioning (VMs for everything) or accepting weaker isolation for some workflows.
4.5.21 - Red Hat Catalog
Web UI for browsing Hummingbird container images, security advisories, and API documentation.
Features
- Image Catalog - Browse container images with tags, architectures, and specifications
- Security Feed - Security advisory feed with CVE details
- API Documentation - Interactive API documentation browser
- Image Details - SBOM, provenance, and release history per image
- My image requests - Signed-in users can see current and historical Jira requests associated with their email, including newly submitted items in triage, with their current status, latest note, and last-updated date
- Analytics identity - After Red Hat SSO login, Amplitude receives a hashed
user_id(issuer + subject) plusis_authenticatedandemail_domain. Logout resets identity. Raw email is not sent.
Architecture
React 18 + PatternFly v6 SPA. Data fetching via TanStack Query against the container-catalog API (proxied through webpack-dev-server in development, direct in production). Webpack 5 build. Hosted on CloudFront + S3 (production) with MR previews on GitLab Pages.
Prerequisites
- Node.js 22+ (for host targets)
make+podman(for container targets)
Usage
Only make and podman are required for container targets. Host
variants (*-host) run without podman.
| Target | Description |
|---|---|
redhat-catalog/setup |
Install dependencies (container) |
redhat-catalog/check |
Type-check, lint, and test (container) |
redhat-catalog/build |
Production build (container) |
redhat-catalog/dev |
Dev server on port 9000 (container) |
redhat-catalog/setup-host |
Install dependencies (host) |
redhat-catalog/check-host |
Type-check, lint, and test (host) |
redhat-catalog/build-host |
Production build (host) |
redhat-catalog/dev-host |
Dev server (host) |
Set ASSET_PATH to control the base path for non-root deployments
(used by webpack output.publicPath and React Router basename).
Configuration
Build Variables
| Variable | Description |
|---|---|
ASSET_PATH |
Webpack public path / React Router basename |
API base URLs are configured in
app-config.ts.
SAM Parameters
| Parameter | Description |
|---|---|
ResourcePrefix |
Prefix for all resource names |
CatalogDomainName |
Custom domain (optional, leave empty for CloudFront default) |
Deployment
Production deployment uses a CloudFront + S3 stack defined in
template.yaml. The infrastructure post-deploy script builds
the SPA in a Node.js container, syncs it to S3, and creates a CloudFront
cache invalidation. Custom domain support is optional via the
CatalogDomainName SAM parameter. When set, an ACM certificate is
created in-template with DNS validation – the validation CNAME must
already exist in the external DNS zone before deployment.
MR previews are deployed to GitLab Pages with path_prefix per MR,
auto-cleaned on merge/close with a 1-week expiry.
Environments and CI promotion
Experimental, staging, and production URLs, MR vs main pipelines, deploy
triggers, and UAT sign-off are documented in
Red Hat Catalog Environment Promotion
(monorepo documentation/) and
Red Hat Catalog UAT Program.
Development
Detailed contributor documentation lives in-tree:
- Developer Guide – component patterns, state management, data fetching
- Testing Guide – Vitest + RTL patterns and commands
- UAT Checklist – human acceptance testing before prod promotion (HUM-2179)
- Architecture – system design, data flow, React Query patterns
- API Responses – observed API response shapes
- Settings Architecture – settings/theme system
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.
4.5.22 - Hummingbird Data Flow
Overview of all data sources, what is stored where, and how data moves between systems.
Data sources
| Source | What it provides |
|---|---|
| Jira REST API | CVE tracker ticket fields (status, timestamps, labels, custom fields) |
Jira whiteboard (customfield_10841) |
Per-ticket timestamp cache written back by the analysis tool |
Red Hat Pulp (packages.redhat.com) |
SRPM/RPM publish timestamps for Hummingbird repos |
| Hummingbird container catalog API | Image rebuild history (when a package first appeared in a rebuilt image) |
Red Hat CSAF VEX feed (security.access.redhat.com) |
Advisory fix/not-affected status per CVE |
OSIDB (osidb.prodsec.redhat.com) |
Subpackage-level affectedness; flaw created_dt and hummingbird-1 affect created_dt |
| NVD / CVE list | CVE publish dates, product/version data |
| GitHub / GitLab | Upstream fix commit / PR / release dates |
| Fedora updates | Fedora update availability timestamps |
| Konflux SNS/SQS events | Pipeline run, snapshot, release, MR, push events |
PostgreSQL databases
There are two Postgres databases.
1. hummingbird-status — Konflux pipeline events
Fed by SNS/SQS events via the hummingbird-status ingestor.
| Table | Primary key | What is stored |
|---|---|---|
gitlab_pushes |
(sha, repo, ref) |
Git push events — sha, branch, commit list |
components |
name |
Konflux component definitions and git context |
pipelineruns |
name |
Build/test PLR outcomes, status, start/completion times |
snapshots |
name |
Multi-component snapshots, source PLR, sha |
releases |
name |
Release outcomes, images, LLM analysis text |
gitlab_merge_requests |
(project, iid) |
Current MR state, merge commit sha |
gitlab_mr_versions |
(project, iid, sha) |
Per-commit history of each MR head |
The legacy tables above remain in public. The independent pipeline-timing
service consumes a second SQS copy of each SNS envelope and writes only to its
Alembic-managed pipeline_timing schema in the same database. The
pipeline-timing Pod’s init container creates that schema and runs migrations
before its worker container starts. The single-replica Deployment uses
Recreate for upgrades. Scaling it to zero leaves the queue untouched without
affecting legacy ingestion.
| Timeline table | Primary key | What is stored |
|---|---|---|
events |
SNS message ID | Source, kind, timestamp, exact S3 bucket and key |
merge_requests, commits, merge_request_commits |
GitLab project ID and source identities | MR lifecycle and head/merge commits |
gitlab_pipelines, gitlab_jobs |
GitLab project ID and pipeline/job ID | Execution status and timing per attempt |
pipeline_runs, task_runs, snapshots, releases |
Cluster and Kubernetes UID | Konflux execution, bridge lineage, and Release completion |
These tables store selected fields and event references, not raw event JSON. The
archive retains the raw SNS envelope. Canonical HTTPS repository URL plus SHA links
MR commits, GitLab pipelines, and Konflux PipelineRuns; project and pipeline IDs
link GitLab jobs. Kubernetes owner UIDs link TaskRuns to PipelineRuns, while the
Snapshot label and source run name connect build to test, and a Release names its
Snapshot and managed PipelineRun. Distinct run UIDs and job IDs preserve retries.
Consumers calculate elapsed intervals from the stored timestamps. The Dashboard
continues to read only the public tables.
2. hummingbird-dashboard — CVE lifecycle + dashboard overlays
| Table | Primary key / unique | What is stored |
|---|---|---|
cve_ticket_events |
(ticket_key, event_type) |
One row per lifecycle milestone per ticket. occurred_at = timestamp; metadata = JSONB (see below). The source of all R-Time computations. |
cve_analysis_log |
id |
Each analysis tool run: timestamp, ticket count, log output |
cve_ticket_claims |
ticket_key |
Claim/lock for deduplicating concurrent analysis runs |
cve_ticket_blame |
(ticket_key, category) unique |
Human-assigned R-Time blame categories (hummingbird, prodsec, konflux, testing, pulp, other); a ticket may have multiple |
dashboard_settings |
key |
Runtime flags: auto-rerun enabled, analysis enabled, etc. |
auto_rerun_log |
id |
History of automatic retrigger attempts |
blocked_error_patterns |
id |
Regex patterns that suppress auto-rerun |
blocked_snapshots |
snapshot_name |
Manually blocked snapshots |
analysis_log / analysis_costs |
id |
LLM failure analysis runs and token costs |
blocked_push_builds / push_build_rerun_log |
id |
Push build blocking and retry history |
package_lifecycle |
(package, event_type) |
Package-scoped lifecycle milestones (e.g. rpm_first_published). Distinct from cve_ticket_events — one row per package/milestone, not per ticket. |
cve_ticket_events lifecycle milestone event_type values
These are the canonical names after the HUM-5918 migration:
event_type |
Meaning |
|---|---|
cve_published |
CVE published date (NVD / CVE list) |
osidb_flaw_created |
OSIDB flaw created_dt (when ProdSec ingested the CVE) |
osidb_affect_created |
hummingbird-1 affect created_dt for this ticket’s pscomponent (when a HUM tracker could be filed) |
cve_embargo_lifted |
OSIDB flaw unembargo_dt, only when it postdates created_dt (i.e. the flaw was genuinely embargoed; omitted otherwise) |
osidb_flaw_public |
OSIDB flaw unembargo_dt, unconditionally – when OSIDB considers the flaw public, whether genuinely embargoed or not |
hum_ticket_created |
HUM Jira ticket creation date |
hum_ticket_closed |
HUM Jira ticket resolution date |
upstream_fix_merged |
Upstream fix commit / release merged |
fedora_update_available |
Fedora update containing the fix became available |
rpm_fix_published_to_pulp |
Fix RPM published to Hummingbird Pulp repo |
image_rebuilt_on_quay |
Hummingbird container image rebuilt with the fix |
vex_resolved |
Red Hat CSAF VEX advisory confirmed the resolution (HUM-5843) |
rpm_fix_published_to_pulp and image_rebuilt_on_quay are mutable — they
are updated when a newer delivery event supersedes the previous one. All other
event types are write-once: once set, occurred_at is never overwritten.
cve_ticket_events.metadata JSONB fields
Each row carries a metadata blob that reflects the ticket’s state at the time
of the most recent analysis run:
| Field | Source | Description |
|---|---|---|
computed_resolution |
analysis tool | Hummingbird’s computed fix status |
jira_status |
Jira status field |
Current Jira issue status |
labels |
Jira labels field |
All labels on the ticket |
fixed_in_build |
Jira customfield_10578 |
SRPM set manually in the “Fixed in Build” field |
detected_fixed_build |
Pulp / SRPM detection | SRPM filename auto-detected from Pulp repodata |
vex_status |
Red Hat VEX feed | fixed / known_not_affected / … |
vex_match_state |
reconciliation logic | matched / pending / mismatch |
vex_resolved |
VEX reconciliation | Timestamp when VEX first agreed with Jira resolution |
catalog_image_source |
collector catalog map | true when the package is a catalog image SBOM source (R-Time delivery is image publish); false means RPM publish. Missing on old rows: dashboard still requires image |
Jira whiteboard (customfield_10841)
The CVE analysis tool uses the Jira whiteboard field as a per-ticket timestamp cache. This is being phased out in favour of Postgres as the durable store (HUM-5917), but is still the source for a subset of fields.
The whiteboard holds compact JSON. The relevant sub-key is cve_cycle:
cve_cycle key |
Meaning | Mutable? |
|---|---|---|
cve_published |
CVE publish date | No — write-once |
jira_created |
HUM ticket creation date | No — write-once |
jira_closed |
HUM ticket close date | No — disabled (255-char limit) |
upstream_fix |
Upstream fix timestamp | No — write-once |
fedora_fix |
Fedora update timestamp | No — write-once |
hb_rpm_fix |
RPM published to Pulp | Yes — re-evaluated each run |
hb_image_fix |
Image rebuilt on Quay | Yes — re-evaluated each run |
The whiteboard is read and written only by the CVE analysis tool. The
dashboard never reads it directly; its authoritative source is always
cve_ticket_events.
Jira fields read by the CVE analysis tool
On each run the analysis tool fetches every open HUM CVE tracker ticket and reads the following fields from the Jira REST API:
| Jira field / custom field | Purpose |
|---|---|
summary |
Ticket title / package detection |
status, resolution, resolutiondate |
Ticket lifecycle state |
created |
Jira ticket creation date → hum_ticket_created |
description, comment |
CVE ID extraction, fix evidence |
labels |
cve-next-release, fix-in-progress, etc. |
security |
Embargo level |
assignee |
Ticket owner |
customfield_10578 (Fixed in Build) |
Manually set SRPM identifying the fix build |
customfield_10841 (Whiteboard) |
Cached cve_cycle timestamps (read + write) |
customfield_10667 (CVE ID) |
Structured CVE identifiers |
customfield_10860 (Embargo Status) |
Embargo flag |
customfield_10020 (Sprint) |
Sprint membership |
customfield_10014 (Epic Link) |
Parent epic |
Data flow
┌─────────────────────────────────────────────────────────────────┐
│ External data sources │
│ │
│ Jira REST API ─────────────────────────────────────┐ │
│ NVD / CVE list ────────────────────────────────────┤ │
│ GitHub / GitLab (upstream fix commits) ────────────┤ │
│ Fedora updates ────────────────────────────────────┤ │
│ Red Hat Pulp (RPM publish times) ─────────────────►│ │
│ Hummingbird catalog API (image rebuild times) ─────┤ │
│ Red Hat CSAF VEX feed ─────────────────────────────┤ │
│ OSIDB (subpackage affectedness) ───────────────────┘ │
└──────────────────────────────┬──────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────┐
│ CVE analysis tool (cron, hummingbird-cve-analysis) │
│ │
│ • Reads Jira ticket fields + whiteboard cache │
│ • Resolves fix SRPM from Pulp repodata │
│ • Resolves image rebuild timestamp from catalog API │
│ • Checks upstream PR/release dates (GitHub/GitLab) │
│ • Fetches CVE publish date (NVD/CVE list) │
│ • Reconciles VEX status (CSAF feed) │
│ • Checks subpackage affectedness (OSIDB) │
│ │
│ Writes back to: │
│ ├── Jira whiteboard (cve_cycle timestamp cache) │
│ ├── Jira labels / Fixed-in-Build field │
│ ├── Jira comments │
│ └── dashboard DB → cve_ticket_events (one row per milestone) │
│ cve_analysis_log (run record) │
└──────────────────────────────┬───────────────────────────────────┘
│
┌───────────────────┴───────────────────┐
│ │
▼ ▼
┌──────────────────────────┐ ┌───────────────────────────────┐
│ Jira whiteboard │ │ dashboard DB │
│ (per-ticket JSON cache) │ │ cve_ticket_events table │
│ cve_cycle timestamps │ │ one row per (ticket, │
│ ← being phased out │ │ event_type) milestone │
│ (HUM-5917) │ └───────────────┬───────────────┘
└──────────────────────────┘ │
▼
┌──────────────────────────────────┐
│ hummingbird-dashboard API │
│ │
│ Pivots cve_ticket_events rows │
│ into per-ticket dicts, computes:│
│ • entry-level R-Time fields │
│ (HUM-5920) │
│ • duration legs inside stages │
│ (HUM-5921) │
│ │
│ Serves JSON + Jinja templates │
└──────────────────────────────────┘
flowchart LR
EVENTS[GitLab pushes, pipelines, jobs, MRs<br/>Kubernetes PipelineRuns, TaskRuns, Snapshots, Releases]
SNS[(SNS topic)]
ARCHIVER[S3 archiver]
S3[(Deterministic archive)]
STATUS[hummingbird-status ingestor]
STATUS_DB[(hummingbird-status DB)]
DASHBOARD[Dashboard API]
TIMELINE[Pipeline timing queue and worker]
TIMELINE_DB[(pipeline_timing schema)]
EVENTS --> SNS
SNS --> ARCHIVER --> S3
SNS --> STATUS --> STATUS_DB --> DASHBOARD
SNS --> TIMELINE --> TIMELINE_DB
Kubernetes TaskRun transport
The Kubernetes event forwarder applies optional resource labelSelector and
fieldSelector values to both its initial LIST and subsequent Watch requests.
Field-selector support depends on the Kubernetes resource and API server; TaskRun
collection should use a confirmed stable RPM application label instead of a
status field selector.
The staging and production forwarders watch the configured resource kinds in
both hummingbird-tenant and rhtap-releng-tenant, including TaskRuns in each
namespace. Events go through each environment’s SNS topic and existing SQS
copies; its Pipeline Timing worker consumes its dedicated queue. Because the
resource list applies to each kubeconfig context, the managed context also
watches the configured PipelineRun, Snapshot, Release, and Component kinds
there. The TaskRun entry has no selector, so both forwarders list and watch all
TaskRuns in both namespaces until a stable RPM application label is confirmed or
namespace-wide volume is approved.
The forwarder publishes the logical kubeconfig context.cluster value, rather
than the API server URL, with each Kubernetes event. For TaskRuns, it removes
only metadata.managedFields before gzip/base64 encoding; identity, owner
references, labels, annotations, task specification, conditions, results, pod
name, and timing remain in the published and archived payload.
API field names
The dashboard API exposes these computed fields per R-Time entry:
Entry-level fields
| API field | Meaning |
|---|---|
cve_published_at |
Start of R-Time: CVE published, falling back to HUM created when missing (Include CVE-HUM); the later of HUM created and CVE published (Exclude, so an embargoed ticket also starts at disclosure); either way, then floored at Upstream Fix Merged so the clock never starts before a fix exists to backport |
fix_delivered_at |
Done timestamp: latest of delivery and Jira close (Exclude VEX), plus VEX when Include VEX is on; unset until the required gates exist. Delivery is image publish for catalog image sources, RPM publish otherwise |
cve_to_delivery_hours |
Completed R-Time only (cve_published_at → fix_delivered_at); None while still accumulating |
cve_to_hum_created_hours |
Filing lag after NVD: max(HUM created − NVD, 0); 0.0 when HUM is first |
advisory_to_vex_hours |
Completed ADV-VEX only (hum_ticket_closed → vex_resolved); None until VEX exists |
display_advisory_to_vex_hours |
ADV-VEX hours, or elapsed-to-now when the VEX feed has not updated yet |
fix_before_cve_published |
True when the delivery timestamp predates notification (informational) |
hum_ticket_open |
True until the selected done gates exist |
deferred_to_next_release |
True for cve-next-release tickets |
display_duration_hours |
R-Time hours, or elapsed-to-now when the ticket is not yet done |
Duration legs (stages dict)
| API field | Interval |
|---|---|
cve_published_to_osidb_flaw |
CVE publish → OSIDB flaw created |
osidb_flaw_to_osidb_affect |
OSIDB flaw created → hummingbird-1 affect created |
osidb_affect_to_hum_created |
hummingbird-1 affect created → HUM ticket opened |
cve_published_to_hum_created |
CVE publish → HUM ticket opened |
cve_published_to_upstream_fix |
CVE publish → upstream fix merged |
hum_created_to_upstream_fix |
HUM ticket → upstream fix merged |
upstream_fix_to_fedora_update |
Upstream fix → Fedora update available |
fedora_update_to_rpm_fix_published |
Fedora update → fix RPM in Pulp |
upstream_fix_to_rpm_fix_published |
Upstream fix → fix RPM in Pulp |
hum_created_to_rpm_fix_published |
HUM ticket → fix RPM in Pulp |
rpm_fix_published_to_image_rebuilt |
Fix RPM in Pulp → image rebuilt on Quay |
image_rebuilt_to_hum_closed |
Image rebuild → HUM ticket closed |
hum_closed_to_vex_resolved |
HUM ticket closed (advisory MR merge) → VEX feed update |
Derived flags (stages dict)
osidb_flaw_unavailable is computed in _build_stages(), not persisted to
cve_ticket_events: True only when osidb_affect_created is present but
osidb_flaw_created is absent – OSIDB demonstrably knows the CVE (an
affect was found) but has no public Flaw record for it. A ticket missing
all three OSIDB fields is ambiguous (the collector may predate these
fields, or OSIDB may have no data yet) and is not flagged. When true,
_timeline_events() prepends a synthetic osidb_flaw_public entry with no
real timestamp, displayed as “INTERNAL” and always first in the Stage
Breakdown’s timestamp grid on the Open, Blame, and R-Time pages – a flaw
must exist before any public OSIDB/CVE activity can reference it, so it
precedes every other milestone even without a known date. It is
intentionally excluded from the chronological bar graph (stages.timeline),
which only plots real timestamps.
osidb_data_missing is the weaker, mutually exclusive counterpart: True
when OSIDB has neither a Flaw nor an Affect record for the CVE –
stating a fact (no OSIDB data found), not a cause, since this can simply
mean OSIDB has not yet ingested a very recently published CVE. Rendered as
a small “osidb-missing” badge next to the ticket key in the collapsed table
row (the same spot and style as the R-Time page’s budget-excluded badge),
not inside the Stage Breakdown drawer, on the Open, Blame, and R-Time pages.
Aggregate stats
| API field | Meaning |
|---|---|
avg_hum_created_to_rpm_fix_hours |
Mean HUM ticket → RPM fix (skips missing legs) |
avg_rpm_fix_to_image_rebuilt_hours |
Mean RPM publish → image rebuild |
avg_advisory_to_vex_hours |
Mean Done-Errata close → VEX feed (completed ADV-VEX only) |
Package lifecycle data
rpm_first_published is the earliest timestamp at which any SRPM for a given
package appeared in the Hummingbird Pulp repo. It is package-scoped — one
row per package — as opposed to cve_ticket_events which is ticket-scoped.
Collection
scripts/collect_rpm_first_published --prod --rpms-repo /path/to/rpms
Flow:
- Loads the package list from
rpms_repo.load_package_map_from_metadata(). - For each package: calls
pulp.fetch_earliest_srpm_time(), which browsespackages.redhat.com/.../source/Packages/{letter}/, parses the HTML listing already used by the analysis tool, and returns the earliest upload timestamp across all SRPMs for that package. - POSTs all results in one request to
/api/cve-import(same endpoint used forcve_ticket_eventssync). The dashboard upserts withCOALESCE(LEAST(existing, incoming), existing, incoming)— re-runs only updateoccurred_atif the incoming timestamp is earlier, and NULLs are never stored over a real value.
Surfaced in dump_lifecycle
scripts/dump_lifecycle --prod HUM-1234
After fetching ticket milestones from cve_ticket_events, the script resolves
the package name from those rows and makes a second call to
/api/cve-export?section=package_lifecycle to fetch rpm_first_published for
that package. It is shown at the bottom of the text output and under
rpm_first_published in the JSON output.
4.5.23 - Kubernetes Event Forwarder
A Kubernetes deployment that watches resource changes (ADDED/MODIFIED/DELETED) across multiple clusters and forwards them to an SNS topic with structured metadata for filtering. Uses kubeconfig contexts as the source of truth for which clusters and namespaces to watch.
The full Kubernetes object JSON is forwarded as the SNS message body, compressed with gzip and base64-encoded.
The staging and production forwarders watch the configured resources in both
hummingbird-tenant and rhtap-releng-tenant on RH03. TaskRun collection is
enabled in both namespaces and environments.
For TaskRun events only, Kubernetes metadata.managedFields is removed before
compression. All other object data, including identity, owner references,
labels, annotations, task specification, conditions, results, pod name, and
timing is retained.
Features
- Multi-Cluster Support: Watch resources across multiple Kubernetes clusters using kubeconfig contexts
- Dynamic Resource Watching: Configure any namespaced resource type using
standard Kubernetes
apiVersionandkind - SNS Integration: Publish events with structured message attributes for precise filtering
- Compression: Events are compressed (gzip+base64) to reduce SNS message size
- Selectors: Resource-specific Kubernetes label and field selectors limit LIST and Watch requests
- Automatic Reconnection: Handles watch connection failures and reconnects automatically
Architecture
Threading Model
The forwarder uses a multi-threaded architecture:
-
Watcher threads (one per context + resource type): Each runs an independent LIST+Watch loop. Isolation ensures one slow/failing cluster doesn’t affect others.
-
Publisher thread (single, shared): Reads events from a queue and publishes to SNS. Decouples K8s API interaction from SNS latency (~150ms per publish), preventing watch loop stalls that could cause resourceVersion staleness.
Memory optimization: When SPOOL_DIR is set, messages are written to
temporary files instead of being held in memory. This reduces memory pressure
during event bursts (e.g., LIST operations or many resources created at once)
and enables recovery of unsent messages after restarts (e.g., after OOM kills).
LIST + Watch Pattern
The forwarder uses explicit LIST followed by Watch rather than
resource_version="0":
- LIST retrieves all current objects and a snapshot
resourceVersion - Watch starts from the LIST’s
resourceVersion(not object RVs) - On graceful watch timeout (300s), restart watch from last known RV (no re-list)
- On errors, fresh LIST to ensure current state
This is necessary because synthetic ADDED events from resource_version="0" have
unsorted, potentially-stale resourceVersions (they reflect when objects were last
modified). If the watch disconnects mid-stream, the tracked RV could be
arbitrarily old and may already be compacted by etcd, causing a 410 error
cascade.
Connection recovery
LIST and Watch requests use a 10-second client connection timeout. The client read timeout is 30 seconds longer than the server watch timeout. With the default 300-second watch timeout, a silent socket fails after 330 seconds without data. The server timeout alone cannot interrupt a client read on a broken connection. The transport does not retry requests internally. The LIST+Watch loop controls retries so that each request failure triggers resource-version recovery.
After a request error, the watcher closes the stream and starts a fresh LIST. This recovers the current state of resources that changed during the outage. Consumers must tolerate repeated events from this LIST. Objects deleted before the LIST cannot be recovered through this process.
The hummingbird_event_forwarder_watch_cycles_total metric records completed and
failed cycles. Labels identify the cluster, namespace, resource kind, and
outcome, which is completed or error. Normal watch completion also produces
an INFO log with the resource version. An idle watch completes normally and does
not require a resource change to show progress.
Prerequisites
- Target cluster: ServiceAccount with watch permissions on the resources you want to monitor
- Deployment cluster: Kubernetes cluster to run the forwarder
- AWS credentials: SNS publish permissions (optional - logs events if not configured)
Deployment
The container image is built via Konflux CI/CD and published to
quay.io/hummingbird-ci/kubernetes-event-forwarder:latest.
Example Kubernetes manifests are provided in the kubernetes/
directory:
rbac.yaml- ServiceAccount, Role, RoleBinding for the target clustersecret.yaml- Kubeconfig and AWS credentialsconfigmap.yaml- Resource watch configurationdeployment.yaml- Forwarder deployment
Quick Start
-
On the target cluster (the one you want to watch), apply RBAC and create a token:
kubectl apply -f kubernetes/rbac.yaml kubectl create token kubernetes-event-forwarder --duration=8760h -
Update the example manifests with your values:
secret.yaml: cluster URL, token, AWS credentialsconfigmap.yaml: resources to watchdeployment.yaml: SNS topic ARN, AWS region
-
Apply the manifests to your deployment cluster:
kubectl apply -f kubernetes/secret.yaml kubectl apply -f kubernetes/configmap.yaml kubectl apply -f kubernetes/deployment.yaml
Prerequisites: Deploy hummingbird-events-topic first to create the SNS topic, then deploy the AWS resources (see below).
AWS Resources
The SAM template (template.yaml) provisions IAM resources for SNS publishing:
- IAM User (
${ResourcePrefix}-user) - Service account for the forwarder - IAM Policy (
${ResourcePrefix}-policy) - Grantssns:Publishto the SNS topic
Deploy using containerized AWS SAM CLI:
cd kubernetes-event-forwarder
sam build
sam deploy --guided # First deployment (interactive)
sam deploy # Subsequent deployments
After deployment, create access keys for the IAM user and store them securely.
SAM Parameters
| Parameter | Description | Default |
|---|---|---|
ResourcePrefix |
Prefix for resources | myapp-prod |
SnsTopicArn |
SNS topic ARN | (required) |
Resource naming: IAM resources follow {ResourcePrefix}-{type} pattern
(e.g., myapp-prod-user, myapp-prod-policy).
Usage
Configure resources to watch in config.yaml:
resources:
- apiVersion: v1
kind: Pod
labelSelector: app=example
fieldSelector: status.phase=Running
- apiVersion: apps/v1
kind: Deployment
- apiVersion: v1
kind: ConfigMap
labelSelector and fieldSelector are optional per-resource settings and use
standard Kubernetes selector syntax. Field-selector support is resource and API
server dependent; do not use a TaskRun status field selector. Use a stable RPM
application label for TaskRuns only after confirming its propagation in the
target environment. The deployed resource list applies to every kubeconfig
context, so both forwarders watch all configured kinds in both namespaces. The
TaskRun entry has no selector and therefore watches all TaskRuns in both
namespaces. Confirm a stable RPM application label and representative payload
sizes; otherwise review and approve the namespace-wide event rate. HUM-7532
provides the managed namespace access, and HUM-7535 owns environment-specific
infrastructure.
SNS Subscription Filter Examples:
Pod events in a specific namespace:
{
"source": ["kubernetes"],
"kind": ["Pod"],
"namespace": ["production"]
}
All deployment changes:
{
"source": ["kubernetes"],
"kind": ["Deployment"]
}
Deleted resources across all clusters:
{
"source": ["kubernetes"],
"event_type": ["DELETED"]
}
Development
See the main README for development workflows.
make kubernetes-event-forwarder/setup # Install dependencies
make check # Lint code (ruff)
make fmt # Format code
make test # Run unit tests
make coverage # Run tests with coverage
Configuration
The deployment is configured via environment variables:
| Variable | Description |
|---|---|
CONFIG_PATH |
Path to config YAML file |
CONFIG |
Inline config YAML (alternative) |
SNS_TOPIC_ARN |
SNS topic ARN (optional - logs if unset) |
SPOOL_DIR |
Optional spool directory for file-based message queue |
AWS_ACCESS_KEY_ID |
AWS access key ID |
AWS_SECRET_ACCESS_KEY |
AWS secret access key |
AWS_DEFAULT_REGION |
AWS region |
METRICS_PORT |
Prometheus metrics port (default: 9090) |
SENTRY_DSN |
Optional Sentry DSN |
Event Metadata
The forwarder extracts metadata from Kubernetes events and adds them as SNS message attributes:
| Attribute | Description | Example |
|---|---|---|
source |
Always "kubernetes" |
kubernetes |
cluster |
Logical kubeconfig cluster entry name | konflux-rh03 |
namespace |
Object namespace | production |
api_version |
Resource API version | v1, apps/v1 |
kind |
Resource kind | Pod, Deployment |
event_type |
Event type | ADDED, MODIFIED, DELETED |
object_name |
Name of the object | nginx-7d8c4c9d6f |
content_encoding |
Message encoding | gzip+base64 |
cluster is the context.cluster value for the kubeconfig context that the
forwarder uses to watch the resource. It is not the API server URL.
Security & Limitations
Security:
- AWS credentials stored in Kubernetes Secret
- Kubeconfig credentials stored in Kubernetes Secret
- SNS topic follows least privilege principle (publish-only)
- Sentry integration for error tracking
Limitations:
- Only namespaced resources supported
- One thread per context + resource type combination
- SNS message size limit: 256 KB (after compression)
- SNS measures the UTF-8 gzip/base64 message plus attribute names, types, and
values against its 262,144-byte limit. The forwarder logs each computed size
and remaining headroom and exports
hummingbird_event_forwarder_sns_publish_size_bytes. Review representative RPM build, test, and tenant-release TaskRun measurements before enabling the source; do not remove further fields without an approved projection. - The source-tree fixtures in
kubernetes-event-forwarder/tests/fixtures/are synthetic, redacted RPM build, test, and tenant-release TaskRun regression fixtures. Their unit test measures the complete publish body and attributes and requires at least 32 KiB of headroom. HUM-7533 owns source-activation measurement review.
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.
4.5.24 - MR Artifact Promotion
Hummingbird builds and tests an RPM artifact during a merge request (MR). After merge, the push pipeline promotes that same artifact to release without building it again. The tested image digest is therefore the one consumed by the release pipeline. This flow is in use for RPMs; container promotion is separate work.
RPM flow
flowchart LR
A["MR build and tests<br/>source B, digest D"] --> B["Merge to main<br/>merge commit M"]
B --> C["Promotion PipelineRun<br/>finds passed MR Snapshot"]
C --> D["Component Snapshot<br/>same B and D"]
D --> E["Integration Service<br/>creates Release"]
E --> F["Release Service<br/>checks and publishes RPMs"]
- The promotion PipelineRun reads the two parents of merge commit
Mfrom Git. The second parent,B, is the tested MR head. The pipeline requires a two-parent merge commit; it does not need an MR number or GitLab API lookup. - The artifact lookup Task finds a passed MR Component Snapshot for
Bin live Konflux or KubeArchive. It verifies the linked build PipelineRun, target branch, source repository (including forks), test result, and image digestD. A passed/testor/retestbuild is eligible when the Snapshot and build record the same PaC event. Multiple passed Snapshots for one commit are normal — a/testor/retestrun creates another — and while RPM builds are non-reproducible (HUM-9785) their image digests may differ. Promotion uses the newest eligible Snapshot. - The promotion Task rechecks that evidence and creates a single-Component,
push-event Snapshot with the same source revision
Band image digestD. It recordsM, the source MR Snapshot and build, and the promotion PipelineRun. The Snapshot’stest.appstudio.openshift.io/pipelinerunstarttimeis the promotion PipelineRun’s start time in epoch milliseconds. - Integration Service processes the Component Snapshot and creates the automatic Release. Release Service runs the existing Conforma, signing, and publication checks. The promotion PipelineRun creates no Release or new build attestation. The released RPMs come from the tested OCI artifact.
The promotion Task checks that no IntegrationTestScenario would run on the new Snapshot, so this handoff does not repeat the MR tests. RPM promotion does not create an override Snapshot or update the Global Candidate List: RPMs do not use it for group tests.
For RPM operations, including push pipeline overrides, verifying a promotion, and recovery, see the RPM MR artifact promotion guide.
Ordering and recovery
Before creating a Snapshot, promotion checks whether a newer merge changed the same package. An unrelated package merge does not block it. The Component Snapshot start-time annotation also lets Integration Service apply its existing push Snapshot ordering check. These are the same ordering controls used by push builds; they do not promise a global serial release queue.
A retry uses the same Snapshot name derived from the Component and M. It
reuses an existing live Snapshot only when its identity, digest, provenance,
and original promotion start time match. It never replaces an archived Snapshot
or creates a second one to retry a failed Release.
Promotion stops if required MR evidence is unavailable in both live Konflux and KubeArchive, the source image has expired, a test failed, or a matching IntegrationTestScenario would run. If promotion cannot use the tested artifact, recovery requires an explicit new build and required tests. A failed automatic Release is retried through the release workflow. MR Snapshot retention and Quay image retention are independent; the source image must remain available through merge and release retries.
Relationship to RHEL on Konflux
RHEL on Konflux (RoK) also reuses an MR-tested RPM after merge. Its current MR artifact is an imported Koji draft build, and its release continues through Brew and Errata. Hummingbird creates its RPM artifact in Konflux and uses a Component Snapshot and automatic Release to publish to Pulp. The shared design point is release of the tested artifact without a post-merge rebuild; artifact storage and publication differ.
Container promotion
Container promotion remains in the container epic. Its group tests use the Global Candidate List, so it needs a separate, non-releasing override Snapshot as well as the Component Snapshot used for release. The container path must establish a no-retest handoff and prevent an older override from replacing a newer candidate before rollout.
MR container images also need tag-level expiry instead of the embedded
quay.expires-after label: that label would follow a promoted container image
into release. A set-tag-expiry task should expire the MR tag through Quay
without changing the image; its retention window, Quay access, and Conforma
trust remain open design questions. RPM releases publish the RPMs inside the
OCI artifact, so that label does not appear on released RPMs. The MR source
image must still survive until the RPM release completes.
For Task inputs, permissions, and failure handling, see the Hummingbird tools Task reference.
4.5.25 - Jira Image Requests
AWS Lambda proxy that serves Hummingbird Image Request issues from Jira (with DynamoDB caching) and creates Stories/Epics from the catalog request form.
Features
- Public list: GET
/image-requestsreturns all Ready Image Request issues, following Jira search pagination so requests are not hidden by its page limit - Personal request history: Authenticated GET
/my-image-requestsreturns current and historical requests whose Jira contact email matches the signed-in user - Form submit: POST
/image-requestscreates a version Story. An Epic is created only when a second request uses the same image name; later requests reuse that Epic. A first-time name is Story-only. Authenticated submissions immediately appear in My requests as In Triage. - Customer scoring: Writes the Customer criterion on the Epic from paid email domains (placeholder until subscriber login)
- Total scoring: Maintained by Jira Automation from manual triage fields
Prerequisites
- AWS CLI with permissions for Lambda, API Gateway, DynamoDB, SSM, CloudFormation
- Podman or Docker for SAM build
- Jira bot token in SSM (SecureString)
- HUM project custom fields for scoring (see below)
Deployment
make jira-image-requests/build
make jira-image-requests/deploy
Configuration
| Parameter / env | Description |
|---|---|
ResourcePrefix |
Prefix for AWS resource names |
JiraTokenParameter |
SSM parameter name for the Jira API token |
CacheTtlSeconds |
Cache TTL for the public list (default 300) |
CorsAllowOrigin |
Allowed CORS origin |
ScoringCustomerField |
Jira custom field id for Customer (customfield_…); empty skips write |
PaidCustomerDomains |
Comma-separated email domains scored as paid (e.g. acme.com,contoso.com) |
OidcUserinfoUrl |
OIDC userinfo endpoint used to verify the bearer token and resolve email |
SentryDsn |
Optional Sentry DSN |
My requests ownership
The browser sends the signed-in user’s access token when it loads or submits a
personal request. The Lambda resolves that token through OidcUserinfoUrl and
uses a SHA-256 digest of the normalized email as the DynamoDB partition key.
Raw email is not stored in the ownership records.
On the first authenticated load, the service searches every page of Image Request Stories in Jira, verifies the exact contact email from each description, and writes the matching records to DynamoDB. Later loads reconcile Jira again so status and notes remain current. Anonymous submissions remain supported and are associated with the contact email entered in the form.
Every request uses the same /image-requests?request=REQ-<number> URL throughout
its lifecycle. While a request is In Triage, the drawer resolves it only
from the authenticated owner’s personal history. After the team accepts it and
Jira includes it in the public Ready list, the same URL resolves from the public
list for signed-out visitors and other users as well.
Request status projection
The catalog projects Jira workflow states into the following customer-facing state machine. A newly submitted item starts in In Triage. Jira reopen or backlog changes provide the regression paths.
stateDiagram-v2
[*] --> InTriage: form submitted
InTriage --> Submitted: accepted for backlog
InTriage --> Declined: triage declines
Submitted --> InProgress: work starts
Submitted --> Declined: request declined
InProgress --> Delivered: work completes
InProgress --> Declined: work stops
InProgress --> Submitted: returned to backlog
Submitted --> InTriage: Jira resets to new
Delivered --> InProgress: Jira reopens
Declined --> InTriage: Jira reopens for triage
Each DynamoDB ownership record stores the request key, summary, projected
status, submitted and updated timestamps, resolution date, latest public note,
and whether the latest visible change was a status change, note change, or
both. The UI falls back to Request updated when the previous snapshot is not
available or another Jira field changed. Its keys are OWNER#<email digest>
and REQUEST#<Jira key>.
Image request scoring
Triage scores live on the Image Request Epic, not on customer-facing Stories.
Custom fields (HUM Epics)
Create (or map) these fields, then record each customfield_XXXXX id:
| Field | Type / values | Who sets it |
|---|---|---|
| IR Legal | Select: Y / N | Manual |
| IR Competitive | Number (typically 0–1) | Manual |
| IR Marketable | Number | Manual |
| IR Supportability | Number | Manual |
| IR Upstream Health | Number | Manual |
| IR Level of Effort | Number: 3 easy, 2 medium, 1 hard | Manual |
| IR Technical Feasibility | Number: 0 blocking, 1 feasible | Manual |
| IR Portfolio Conflicts | Number: 0 conflict, 1 good | Manual |
| IR Customer | Number: 0 or 1 | Lambda (ScoringCustomerField) |
| IR Total | Number | Jira Automation only |
Gate: Legal must be Y to continue scoring. Competitive, Marketable, and Supportability / Maturity are also expected to be 1 before a meaningful Total (team convention from the triage spreadsheet).
Lambda: Customer criterion
When a POST creates or reuses an Epic (second or later request for a name), the Lambda:
- Loads Stories under the Epic
- Sets Customer to 1 if any submitter email domain is in
PaidCustomerDomains, otherwise 0 - Writes the value to
ScoringCustomerFieldwhen configured
A first request for a name does not create an Epic, so Customer is not written until a second request for that name arrives.
When subscriber login lands, replace the domain list with real subscription status.
Jira Automation: Total and Legal flag
Configure two rules in Jira (Project settings → Automation). Use the real field names/ids from your instance.
Rule 1 — Sum Total when Legal is Y
- Trigger: Field value changed for any of: Competitive, Marketable, Supportability, Upstream Health, Level of Effort, Technical Feasibility, Portfolio Conflicts, Customer (and optionally Legal)
- Condition: Issue type = Epic AND component = Image Request AND IR Legal = Y
- Action: Edit issue → set IR Total to the sum of the numeric criteria (exclude Legal). Example smart value shape:
{{#=}}
{{issue.IR Competitive}} + {{issue.IR Marketable}} + {{issue.IR Supportability}} +
{{issue.IR Upstream Health}} + {{issue.IR Level of Effort}} +
{{issue.IR Technical Feasibility}} + {{issue.IR Portfolio Conflicts}} +
{{issue.IR Customer}}
{{/}}
Use your instance’s smart-value field keys (often customfield_XXXXX).
- Optional action: Remove label
ir-legal-blockedwhen Legal is Y
Rule 2 — Flag when Legal is not Y
- Trigger: IR Legal changed
- Condition: Issue type = Epic AND component = Image Request AND IR Legal is empty OR IR Legal = N
- Actions:
- Clear IR Total (or set to empty)
- Add label
ir-legal-blocked
Jira Automation recalculates Total whenever triage fields change — no webhook or Lambda rescoring path is required.
Development
make jira-image-requests/setup
make test
License
GPL-3.0-or-later
4.5.26 - PAC Trigger
A CLI tool to manually trigger Konflux PAC (Pipelines-as-Code) pipelines for a specific component and branch. Useful for debugging pipeline issues or re-running builds without pushing a new commit.
Usage
# Trigger pipeline for a component (uses branch HEAD)
pac-trigger --gitlab-project-url https://gitlab.com/org/group/project \
--component myimage--default--main \
--cluster-url https://konflux-ui.apps.cluster.example.com
# Trigger with specific commit
pac-trigger --gitlab-project-url https://gitlab.com/org/group/project \
--component myimage--default--main \
--commit abc123def456 \
--cluster-url https://konflux-ui.apps.cluster.example.com
# Dry run (preview without creating)
pac-trigger --gitlab-project-url https://gitlab.com/org/group/project \
--component myimage--default--main \
--cluster-url https://konflux-ui.apps.cluster.example.com \
--dry-run
Options
| Option | Description | Default |
|---|---|---|
--gitlab-project-url |
GitLab project URL (required) | - |
--component |
Component name (required) | - |
--branch |
Branch name | main |
--commit |
Specific commit SHA | HEAD |
--cluster-url |
Konflux cluster URL (required) | - |
--dry-run |
Preview PipelineRun without creating | false |
-v, --verbose |
Enable debug logging | false |
Installation
cd pac-trigger
pip install -e .
Prerequisites
- Kubeconfig: Context with access to the target Konflux cluster/namespace
- Repository resource: PAC Repository must exist for the GitLab project
- Push template: Component must have a push template in
.tekton/*.yaml
How It Works
- Fetch commit: Gets HEAD commit SHA from GitLab (or uses provided commit)
- Fetch template: Downloads
.tekton/*.yamlfiles via GitLab API, finds push template matching the component - Get namespace: Extracts namespace from template metadata
- Find credentials: Looks up PAC Repository resource to find git secret
- Create git-auth secret: Creates ephemeral
pac-trigger-gitauth-*secret with git credentials - Create PipelineRun: Substitutes template variables and creates the PipelineRun
- Link secret: Sets ownerReference on secret for garbage collection
- Print URL: Outputs Konflux UI URL for the PipelineRun
Template Variables
The following PAC template variables are supported:
| Variable | Substituted with |
|---|---|
{{revision}} |
Commit SHA |
{{target_branch}} |
Branch name |
{{repo_url}} |
GitLab project URL |
{{git_auth_secret}} |
Created secret name |
Unsupported template variables will cause an error.
Labels Added
| Label | Value | Purpose |
|---|---|---|
pac-trigger/manual |
true |
Identifies manual triggers |
Features
- Template-based: Fetches PipelineRun templates from
.tekton/*.yamlfiles via anonymous GitLab API - Push-only: Only triggers push templates (filters by CEL expression)
- Automatic credentials: Creates ephemeral git-auth secrets from existing PAC Repository secrets
- Garbage collection: Secrets are linked to PipelineRun via ownerReference
- Kubeconfig-based: Uses local kubeconfig for cluster authentication
Development
See the main README for development workflows.
make pac-trigger/setup # Install dependencies
make check # Lint code (ruff)
make fmt # Format code
make test # Run unit tests
make coverage # Run tests with coverage
Comparison with PAC-triggered Runs
Manually triggered PipelineRuns differ from PAC-triggered ones:
Present in both:
appstudio.openshift.io/applicationappstudio.openshift.io/componentpipelines.appstudio.openshift.io/type: buildbuild.appstudio.redhat.com/commit_shabuild.appstudio.redhat.com/target_branch
Unique to pac-trigger:
pac-trigger/manual: truelabel
Missing (PAC internal metadata):
pipelinesascode.tekton.dev/event-typepipelinesascode.tekton.dev/sha- GitLab status reporting annotations
These differences don’t affect pipeline execution—they’re used for PAC’s internal tracking and GitLab commit status updates.
Limitations
- Only push templates supported (not pull-request)
- Only GitLab repositories supported
- Requires existing PAC Repository resource
- No GitLab commit status reporting
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.
4.5.27 - Hummingbird Status
SQS worker and database management for ingesting Konflux pipeline events.
Features
- SQS Worker - Real-time database updates from SNS pipeline events
- GitLab Event Ingestion - Push events and merge request events with version tracking
- Database Init - Populate from pg_dump, S3 archive, or local filesystem mirror
- Prometheus Metrics - Built-in metrics endpoint for monitoring
- MR Reconciliation - Fix stale MR state by checking GitLab API
- Schema Management - Idempotent database schema creation
Pipeline timing and lineage are ingested by the separate Pipeline Timing
service into a dedicated schema in the same events database. The Status worker
ignores pipeline, job, and TaskRun events it does not store in public.
Prerequisites
- Python 3.11+
- PostgreSQL 16
- AWS credentials (for SQS worker and S3 initialization)
Installation
cd hummingbird-status
pip install -e .
Usage
Local Development
cd hummingbird-status
# Database management
./dev.sh db-start # Start PostgreSQL container
./dev.sh db-init # Initialize from available source
./dev.sh db-shell # PostgreSQL interactive shell
./dev.sh db-dump # Create pg_dump file
./dev.sh db-stop # Stop PostgreSQL
./dev.sh db-reset # Stop and delete volume
# With data source
SNS_MIRROR=/path/to/mirror ./dev.sh db-init # From filesystem
S3_BUCKET=bucket-name ./dev.sh db-init # From S3
# SQS worker (requires credentials)
SQS_QUEUE_URL=https://... ./dev.sh worker
Container Deployment
The container runs the SQS worker by default:
podman build -f Containerfile -t hummingbird-status .
podman run -e DATABASE_URL=... -e SQS_QUEUE_URL=... hummingbird-status
For database initialization:
podman run -e DATABASE_URL=... -e S3_BUCKET=... \
hummingbird-status python3 -m hummingbird_status.worker.init_db
MR Reconciliation
MR state is updated via GitLab webhooks. If a webhook is lost, an MR can appear as “opened” in the dashboard when it is actually merged or closed. The reconcile command checks all open MRs against the GitLab API and corrects stale entries:
# Using environment variables
GITLAB_TOKEN=glpat-... python -m hummingbird_status.reconcile
# Using CLI arguments
python -m hummingbird_status.reconcile \
--database-url postgresql://... \
--gitlab-token glpat-...
In a container:
podman run -e DATABASE_URL=... -e GITLAB_TOKEN=... \
hummingbird-status python3 -m hummingbird_status.reconcile
Configuration
Environment Variables
| Variable | Default | Description |
|---|---|---|
DATABASE_URL |
- | PostgreSQL connection URL |
SQS_QUEUE_URL |
- | SQS queue URL (worker) |
S3_BUCKET |
- | S3 bucket for init |
S3_PREFIX |
sns/ |
S3 key prefix |
LOCAL_MIRROR_PATH |
/data/sns-mirror |
Local S3 mirror path |
INIT_DUMP_PATH |
/data/init/dump.sql |
pg_dump file path |
GITLAB_URL |
https://gitlab.com |
GitLab instance URL (reconcile) |
GITLAB_TOKEN |
- | GitLab API token (reconcile) |
METRICS_PORT |
9090 |
Prometheus metrics port |
Database Initialization Priority
- pg_dump file -
$INIT_DUMP_PATHif exists - Local S3 mirror -
$LOCAL_MIRROR_PATH/sns/*.json.gzif exists - S3 bucket -
$S3_BUCKET/$S3_PREFIXwith AWS credentials
Database Schema
The database stores Konflux pipeline events and GitLab notifications (pushes and MRs).
flowchart TD
push[["<b>gitlab_pushes</b><br/>commit sha, changed files"]]
mr[["<b>gitlab_merge_requests</b><br/>current MR state"]]
mrv[["<b>gitlab_mr_versions</b><br/>head commit history"]]
comp[["<b>components</b><br/>git_context → component mapping"]]
build[["<b>pipelineruns</b> (type=build)<br/>one per affected component"]]
promotion[["<b>pipelineruns</b> (type=promotion)<br/>reuses the tested MR image"]]
snap[["<b>snapshots</b><br/>image digest, links via source_plr"]]
promoted["<b>promotion Snapshot</b><br/>source_plr from Hummingbird annotation"]
test[["<b>pipelineruns</b> (type=test)<br/>integration tests per snapshot"]]
rel[["<b>releases</b><br/>publish to registry, links to snapshot"]]
relplr[["<b>pipelineruns</b> (type=release)<br/>executes release, linked via release_plr"]]
push -- "+ affected" --> build
push -- "promotion-enabled RPM" --> promotion
mr -- "sha" --> mrv
mrv -- "sha joins" --> build
comp -- "components" --> build
build -- "success<br/>creates" --> snap
snap -. "tested digest" .-> promotion
promotion -- "creates" --> promoted
snap -- "triggers" --> test
test -- "success<br/>creates" --> rel
promoted -- "auto-release<br/>without retest" --> rel
rel -- "managed<br/>by" --> relplr
RPM promotion Snapshots identify their creating PipelineRun with the
hummingbird-project.io/promotion-pipelinerun annotation. The ingest stores
that name as source_plr. Historical Snapshot records without source_plr
are linked through the promotion PipelineRun’s COMPONENT_SNAPSHOT result.
Tables
| Table | Primary Key | Description |
|---|---|---|
gitlab_pushes |
sha,repo,ref | GitLab push events to main branch |
gitlab_merge_requests |
project,iid | Current state of merge requests |
gitlab_mr_versions |
project,iid,sha | Head commit history for each MR |
components |
name | Konflux Component resources |
pipelineruns |
name | Build, test, and release pipelines |
snapshots |
name | Image snapshots after successful builds |
releases |
name | Published releases to target registry |
Merge Request Tracking
The MR tables enable tracking build status across MR versions:
-
gitlab_merge_requests- Stores current MR metadata (title, state, branches, author, latest head SHA). Updated viaON CONFLICT ... WHERE updated_at <to keep the most recent state.author_namecaptures the webhookuser.namealongsideauthor(the username) so the dashboard can show a bot/access-token account’s real GitLab name (e.g. “chore-mr”) instead of its generated username – likeauthor, it’s set once and preserved viaCOALESCEon conflict, so it isn’t backfilled for MRs that predate this column. -
gitlab_mr_versions- Records each unique head commit SHA for an MR. When force-pushing the same SHA,created_atis updated to the latest event timestamp viaGREATEST(), ensuring correct ordering even after force pushes.
Development
Running Tests
cd hummingbird-status
pip install -e ".[dev]"
pytest
Project Structure
hummingbird-status/
├── Containerfile
├── dev.sh
├── template.yaml # SAM template for AWS resources
├── hummingbird_status/
│ ├── db.py # Database schema and utilities
│ ├── ingest.py # SNS event parsing and ingestion
│ ├── reconcile.py # MR state reconciliation with GitLab
│ └── worker/
│ ├── sqs.py # SQS consumer
│ └── init_db.py # Database initializer
└── tests/
AWS Resources
Deploy the SQS queue using SAM:
cd hummingbird-status
make build # Build SAM application
make deploy # First deployment (guided)
make redeploy # Subsequent deployments
Parameters
| Parameter | Description | Default |
|---|---|---|
ResourcePrefix |
Prefix for all resource names | myapp-prod |
SnsTopicArn |
ARN of the SNS topic to subscribe | (required) |
Prerequisites: Requires an existing SNS topic. Deploy hummingbird-events-topic first.
See the main README for development workflows.
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.
4.5.28 - Lambda S3 Cache
An AWS Lambda setup that caches files from public URLs in S3, supporting both HTTP file caching (RPMs, repository metadata) and OCI Distribution API v2 (container image manifests and blobs). When a URL is requested, the service returns a 302 redirect to either the cached S3 copy (via presigned URL) or the original source. Cache misses trigger asynchronous downloads to S3, ensuring future requests are served from the cache.
Caching uses the original URL’s host and path as the S3 key for HTTP content,
and a flat content-addressed layout (cache/oci/<digest>) for OCI artifacts.
Frequently accessed content automatically extends its cache lifetime on each
access.
Note: Cached content behavior depends on file type:
- Immutable content (matching immutable extension filter, e.g.,
.rpm): Changes at origin won’t be reflected until cache expires - Mutable content (non-matching extension, e.g.,
repomd.xml): Cache is revalidated on each request via HEAD check; stale behavior is configurable
Cache Revalidation: Mutable content uses HTTP ETags for efficient validation.
The uploader stores the origin’s ETag in S3 metadata when caching content. On
subsequent requests, the handler sends a HEAD request with If-None-Match header
containing the stored ETag. If the origin responds with 304 Not Modified, the
cache is fresh and served directly. If the ETag differs, the cache is stale and
an async refresh is triggered; response behavior depends on StaleCacheBehavior.
Note: If the origin doesn’t provide ETags, the cache cannot validate freshness. In this case, mutable content always redirects to origin without triggering cache updates (caching would be ineffective since every request would redirect anyway).
Features
- Streaming Upload: Handles files efficiently by streaming directly from source to S3 without loading into memory
- Presigned S3 URLs: Returns short-lived signed URLs for cached content
- Automatic Expiration: Cached content expires after a certain time, with lifetime extended on each access
- URL Prefix Allowlist: Only caches content matching explicitly allowed host+path prefixes
- Immutable Extension Filter: Identifies immutable content that doesn’t require revalidation
- Cache Revalidation: Mutable content (non-matching extensions) is validated on each request; stale cache behavior is configurable (origin or cache)
- OCI Image Caching: Caches OCI container image manifests and blobs via the OCI Distribution API v2, protecting against upstream digest garbage collection
- Multi-Registry Support: OCI caching supports multiple upstream registries by encoding the registry hostname in the request path
- Custom Domain: Optional custom domain with automatic TLS certificate management via ACM and Route53
Architecture
Three Lambda functions handle the caching workflow:
- Handler - API Gateway endpoint that checks cache, returns 302 redirects, and triggers async operations. Implements touch cooldown to prevent S3 throttling by only touching objects after a configurable time period has elapsed since last modification.
- Uploader - Downloads from origin and streams to S3 on cache misses (invoked asynchronously)
- Touch - Updates S3 object timestamps to extend cache lifetime on cache hits (invoked asynchronously only when cooldown period has elapsed)
Request Flow
flowchart TD
Start([Request]) --> CheckURL{URL matches<br/>AllowedPrefixes?}
CheckURL -->|No| RedirectOrigin
CheckURL -->|Yes| HeadS3[HEAD S3]:::network
HeadS3 --> CheckCache{Cache exists?}
CheckCache -->|No: Cache Miss| InvokeUploaderMiss[Invoke Uploader async]:::async
CheckCache -->|Yes: Cache Hit| CheckImmutable{Immutable extension?}
CheckImmutable -->|No: Mutable| HeadOrigin[HEAD Origin<br/>If-None-Match: stored ETag]:::network
HeadOrigin --> CheckChanged{Origin changed?}
CheckChanged -->|No ETag from origin| RedirectOrigin
CheckChanged -->|ETag differs| InvokeUploaderStale[Invoke Uploader async]:::async
CheckChanged -->|ETag matches| CheckCooldown
CheckChanged -->|304 Not Modified| CheckCooldown
CheckChanged -->|"Error/timeout"| CheckCooldown
CheckImmutable -->|Yes| CheckCooldown
CheckCooldown{Touch cooldown<br/>elapsed?} -->|Yes| InvokeTouch[Invoke Touch async]:::async
InvokeUploaderMiss --> RedirectOrigin
InvokeUploaderStale -->|"STALE_CACHE_BEHAVIOR=origin"| RedirectOrigin
InvokeUploaderStale -->|"STALE_CACHE_BEHAVIOR=cache"| RedirectCache
CheckCooldown -->|No| RedirectCache
InvokeTouch --> RedirectCache
subgraph redirectGroup [ ]
RedirectOrigin[302 to Origin URL]:::redirect
RedirectCache[302 to S3 Presigned URL]:::redirect
end
style redirectGroup fill:none,stroke:none
classDef network fill:#10b981,color:#000
classDef async fill:#ff9900,color:#000
classDef redirect fill:#3b82f6,color:#fff
Legend: Green = network call, Orange = async Lambda invocation, Blue = 302 redirect response
Prerequisites
- AWS CLI configured with appropriate credentials (IAM permissions for Lambda, API Gateway, S3, CloudFormation, CloudWatch Logs, and optionally Route53/ACM for custom domain)
- Podman or Docker (for containerized SAM build/deploy)
- Python 3.11 or later (for development)
Deployment
Build and deploy using containerized AWS SAM CLI:
make lambda-s3-cache/build # Build Lambda package
make lambda-s3-cache/deploy # First deployment (interactive/guided)
make lambda-s3-cache/redeploy # Subsequent deployments (non-interactive)
Deployment output: ApiEndpoint - the API Gateway URL to use for requests
Custom Domain
Optional custom domain with automatic TLS certificate management (ACM +
Route53). Requires a Route53 hosted zone. Deploy with CustomDomainName and
HostedZoneId parameters - CloudFormation handles certificate creation, DNS
validation, and configuration. Certificate validation takes 5-30 minutes; allow
up to 1 hour for DNS propagation.
Parameters
| Parameter | Description | Default |
|---|---|---|
ResourcePrefix |
Prefix for all resource names | myapp-prod |
PresignedUrlExpiration |
Presigned URL expiration (seconds) | 3600 |
AllowedPrefixes |
Whitespace-separated list of allowed URL prefixes | example.com/path/ |
ImmutableExtensions |
File extensions for immutable content (no revalidation) | .rpm |
CacheExpirationDays |
Days to keep cached content (minimum: 1) | 14 |
TouchCooldownMinutes |
Minimum minutes between touch operations | 60 |
StaleCacheBehavior |
When cache is stale: origin or cache |
origin |
OciAllowedPrefixes |
Whitespace-separated OCI image prefixes (registry/name) |
`` |
OciOriginHeadTimeout |
Timeout (seconds) for HEAD requests to OCI registries | 5 |
CustomDomainName |
Optional custom domain name | `` |
HostedZoneId |
Route53 hosted zone ID (required if custom domain) | `` |
SentryDsn |
Optional Sentry DSN for error tracking | `` |
Resource naming: All AWS resources follow {ResourcePrefix}-{type}-{name}
pattern (e.g., myapp-prod-bucket, myapp-prod-lambda-handler).
Usage
Make GET requests to the API endpoint (or custom domain if configured). The URL
to cache is encoded in the path (without the https:// prefix):
curl -L "https://<api-endpoint>/example.com/path/to/file.rpm"
Behavior:
- First request (cache miss): Redirects to original URL while triggering async S3 upload
- Subsequent requests (cache hit):
- Immutable content (matching immutable extension filter): Redirects to presigned S3 URL and resets cache expiration
- Mutable content (non-matching extension): HEAD request validates cache
freshness (10s timeout). If stale, behavior depends on
StaleCacheBehavior; if fresh, serves from cache
- Disallowed prefixes: URLs not matching allowed prefixes are transparently redirected to the original URL (302 pass-through)
Immutable Extension Filter Behavior
The ImmutableExtensions parameter determines caching behavior:
-
Matching extension (e.g.,
.rpm): Treated as immutable content. Cached without revalidation - changes at origin won’t be reflected until cache expires. Served directly from cache on all requests. -
Non-matching extension (e.g.,
.xml,.gz): Treated as mutable content. Cached with revalidation - HEAD request on each cache hit validates freshness using ETags. If stale, behavior depends onStaleCacheBehavior:origin(default): Redirects to origin for fresh content, async refreshcache: Serves stale cache immediately (faster), async refresh in background
Important: All files matching AllowedPrefixes are cached, regardless of
extension. The extension filter only determines whether revalidation is performed.
Usage as Repository Proxy
The cache can be used as a DNF/yum baseurl for RPM repositories. Configure
ImmutableExtensions to include .rpm:
.rpmfiles (matching extension): Cached as immutable - fast, no revalidation- Metadata files (non-matching extension, e.g.,
repomd.xml,primary.xml.gz): Cached with revalidation - ensures fresh metadata while benefiting from cache
[myrepo]
name=My Repository
baseurl=https://koji-s3-cache.example.com/download.example.org/pub/repo/$basearch/
enabled=1
This provides caching benefits for RPM downloads while ensuring repository metadata stays fresh.
OCI Image Caching
The cache also supports the OCI Distribution API v2 for caching container image manifests and blobs. This prevents deployment failures caused by upstream registries garbage-collecting digest-pinned images when tags move.
Requests with a v2/ path prefix are handled as OCI requests. The upstream
registry is extracted from the first path segment after v2/, enabling
multi-registry support through a single cache endpoint.
OCI request format:
GET /v2/<registry>/<image-name>/manifests/<digest>
GET /v2/<registry>/<image-name>/blobs/<digest>
Example:
# Health check
curl https://cache.example.com/v2/
# Manifest request (quay.io/openshift/origin-oauth-proxy)
curl -L https://cache.example.com/v2/quay.io/openshift/origin-oauth-proxy/manifests/sha256:504c036d...
OCI cache strategy (origin-first):
The OCI cache uses an origin-first strategy, matching the proven CKI dependency-archive pattern. This minimizes S3 egress — the cache is only consulted when the upstream registry is unavailable:
- HEAD upstream registry — check if the digest is still available
- Origin available + not cached → redirect to upstream, trigger async S3 upload
- Origin available + cached → redirect to upstream (touch if cooldown elapsed)
- Origin unavailable + cached manifest → serve inline from S3 (200 with
body,
Content-Typefrom metadata,Docker-Content-Digestheader) - Origin unavailable + cached blob → 302 to presigned S3 URL
- Origin unavailable + not cached → redirect to upstream as last resort
- Any error → redirect to upstream (fail-open)
S3 key layout: cache/oci/sha256:<digest> — flat, content-addressed. The
same blob shared by different images is stored only once.
OCI request flow:
flowchart TD
Start([OCI Request]) --> CheckPath{Path starts<br/>with v2/?}
CheckPath -->|No| RPMHandler[RPM cache handler]
CheckPath -->|Yes| V2Health{v2 or v2/?}
V2Health -->|Yes| Return200[200 OK<br/>registry/2.0]:::redirect
V2Health -->|No| ParsePath[Parse path:<br/>registry/name/kind/digest]
ParsePath --> ValidDigest{Valid digest?<br/>manifests or blobs?}
ValidDigest -->|No| ForwardUpstream[302 to upstream]:::redirect
ValidDigest -->|Yes| CheckAllow{In OCI<br/>AllowedPrefixes?}
CheckAllow -->|No| ForwardUpstream
CheckAllow -->|Yes| HeadOrigin[HEAD upstream registry]:::network
HeadOrigin --> OriginUp{Origin available?}
OriginUp -->|Yes| CheckOciCache{Cached in S3?}
CheckOciCache -->|No| InvokeUploader[Invoke Uploader async]:::async
CheckOciCache -->|Yes| TouchCheck{Touch cooldown?}
TouchCheck -->|Elapsed| InvokeTouchOci[Invoke Touch async]:::async
TouchCheck -->|Not elapsed| RedirectUpstream
InvokeTouchOci --> RedirectUpstream[302 to upstream]:::redirect
InvokeUploader --> RedirectUpstream
OriginUp -->|No| CheckFallback{Cached in S3?}
CheckFallback -->|Yes, manifest| ServeInline[200 inline<br/>from S3]:::redirect
CheckFallback -->|Yes, blob| PresignedRedirect[302 to<br/>presigned S3]:::redirect
CheckFallback -->|No| ForwardUpstream
classDef network fill:#10b981,color:#000
classDef async fill:#ff9900,color:#000
classDef redirect fill:#3b82f6,color:#fff
Note: Tag-based pulls (/manifests/latest), tag listings
(/tags/list), and images not in OciAllowedPrefixes are transparently
forwarded to the upstream registry without caching.
Development
See the main README for development workflows.
make lambda-s3-cache/setup # Install dependencies
make check # Lint code
make fmt # Format code
make test # Run unit tests
make coverage # Run tests with coverage
Configuration
Lambda functions receive configuration via environment variables (automatically set by CloudFormation):
| Variable | Handler | Uploader | Touch | Description |
|---|---|---|---|---|
S3_BUCKET_NAME |
✅ | ✅ | ✅ | S3 bucket name |
PRESIGNED_URL_EXPIRATION |
✅ | Presigned URL expiration (sec) | ||
UPLOADER_LAMBDA_ARN |
✅ | Uploader Lambda ARN | ||
TOUCH_LAMBDA_ARN |
✅ | Touch Lambda ARN | ||
TOUCH_COOLDOWN_MINUTES |
✅ | Min minutes between touch ops | ||
ALLOWED_PREFIXES |
✅ | Allowed URL prefixes | ||
IMMUTABLE_EXTENSIONS |
✅ | Extensions for immutable content | ||
STALE_CACHE_BEHAVIOR |
✅ | Stale behavior: origin or cache | ||
OCI_ALLOWED_PREFIXES |
✅ | OCI image prefixes to cache | ||
ORIGIN_HEAD_TIMEOUT |
✅ | OCI origin HEAD timeout (sec) | ||
SENTRY_DSN |
✅ | ✅ | ✅ | Optional Sentry DSN |
Security & Limitations
Security:
- S3 bucket has public access blocked; all objects encrypted at rest (AES256)
- Presigned URLs expire after a certain time
- IAM policies follow least privilege principle
- URL prefix allowlist prevents caching arbitrary URLs
- OCI image allowlist (
OCI_ALLOWED_PREFIXES) controls which images are cached - Immutable extension filter distinguishes immutable vs mutable content for revalidation
- OCI caching only handles public images (no auth forwarding)
Limitations:
- Lambda timeout: 15 min (uploader), 30 sec (handler, touch)
- Lambda memory: 1024 MB (uploader), 256 MB (handler, touch)
- S3 object size: Up to 5 TB (AWS limit)
- API Gateway 6 MB response limit for inline manifest responses (manifests are typically under 100 KB)
- OCI caching only supports digest-based pulls (
sha256:...); tag-based pulls are forwarded to the upstream registry
Design Decisions Record
DDR-1: S3 Lambda OCI cache vs ECR pull-through cache
Decision: Build the OCI cache as a new code path in the existing lambda-s3-cache Lambda, rather than using AWS ECR pull-through cache.
Context: Third-party container images pinned by digest vanish when
upstream registries garbage-collect old manifests after tag re-pushes
(e.g., quay.io/openshift/origin-oauth-proxy:5.1 digests disappear
within hours). This causes ImagePullBackOff failures on pod restarts.
Alternatives: (a) ECR pull-through cache — managed service, no custom
code, retains cached digests when upstream is unavailable. (b)
images.paas.redhat.com — Red Hat internal registry with existing
pull-through proxy orgs for some registries. (c) Artifactory — supports
any upstream registry as a remote repository.
Rationale: ECR pull-through cache requires private ECR registries,
which need IAM authentication — tokens expire every 12 hours, requiring a
CronJob for credential rotation on OCP pods. The S3 Lambda approach
serves content via a public HTTPS URL with zero authentication overhead.
images.paas.redhat.com was ruled out due to dependency on Red Hat IT
(rug-pulling risk if they decommission the proxy orgs). Artifactory was
not in use and would add a new service dependency. The S3 Lambda approach
reuses existing infrastructure (same Lambda, API Gateway, S3 bucket, custom
domain) and adds ~100 lines of OCI handler code ported from the proven CKI
dependency-archive.
DDR-2: Origin-first (HEAD-first) strategy vs cache-first
Decision: Use an origin-first strategy: always check the upstream registry first via HEAD request, redirect to upstream when available, and serve from S3 only when upstream returns 404 or errors.
Context: The existing RPM cache uses a cache-first strategy (check S3 first, revalidate with origin). For OCI artifacts, the design choice affects S3 egress costs and cache freshness.
Alternatives: (a) Cache-first — check S3 first, serve from cache if present. (b) Origin-first — check upstream first, fall back to S3.
Rationale: OCI digests are content-addressed and immutable — there is no staleness concern (unlike RPM repo metadata). The origin-first approach minimizes S3 egress costs: when upstream is healthy (the common case), every pull redirects directly to the upstream registry with zero S3 data transfer. S3 is only read when upstream has garbage-collected the digest. This matches the CKI dependency-archive pattern, which is proven in production. The cache populates asynchronously on first access and serves as a safety net, not a performance layer.
DDR-3: Registry hostname in URL path vs per-registry subdomains
Decision: Encode the upstream registry hostname as the first path
segment after v2/ (e.g.,
koji-s3-cache.example.com/quay.io/openshift/origin-oauth-proxy@sha256:...).
Context: The CKI dependency-archive assumes a single upstream registry
(OCI_REGISTRY env var). Hummingbird pulls from 4+ registries (quay.io,
docker.io, ghcr.io, registry.gitlab.com).
Alternatives: (a) Per-registry subdomains
(quay-io.cache.example.com/...) — standard OCI URL format but requires
wildcard TLS certificates and DNS records, potentially separate Lambda
deployments. (b) Registry hostname in the URL path — single endpoint,
handler extracts the registry from the first path segment.
Rationale: Encoding the registry in the path allows a single Lambda,
single API Gateway, single custom domain, and single TLS certificate to
serve all registries. The /{path+} catch-all route already captures
everything. The v2/ prefix naturally separates OCI traffic from RPM
cache traffic. The kubelet constructs the correct path automatically from
the image reference. This approach requires zero new infrastructure beyond
the existing lambda-s3-cache stack.
DDR-4: Jinja macro rewrite vs registries.conf
Decision: Rewrite image URLs at Jinja template render time using a
cached_image() macro, rather than configuring registries.conf on OCP
nodes.
Context: OCP pods need to pull from the cache instead of directly from
upstream registries. The standard approach is registries.conf mirror
configuration on the cluster nodes.
Alternatives: (a) Node-level registries.conf via MachineConfig. (b)
Jinja macro rewriting image refs at deploy time.
Rationale: Hummingbird has namespace-admin access only on OCP
clusters — no MachineConfig, no node-level changes. The Jinja macro
approach works entirely at the deployment manifest level:
vars.d/*.yml files continue to contain upstream refs (Renovate keeps
working unchanged), and the macro rewrites them to cache URLs at render
time. This is transparent to Renovate, requires no cluster-admin
privileges, and can be disabled with a single variable change.
DDR-5: Scope limited to third-party images
Decision: Only cache third-party images. Hummingbird-owned images
(quay.io/hummingbird-ci/* and registry.access.redhat.com/hi/*) are
excluded.
Context: The cache exists to protect against upstream registries garbage-collecting digest-pinned images.
Alternatives: Cache all images uniformly.
Rationale: Hummingbird-owned images have additional sha-based tags
(commit SHA, build ID) that persist independently of the floating
:latest tag. Old digests are not garbage-collected because these
immutable tags keep them reachable. Caching them would add S3 storage
and Lambda invocations with no benefit. The cached_image() Jinja macro
skips rewriting for quay.io/hummingbird-ci/ prefixes.
DDR-6: Separate OCI handler function vs extending RPM cache logic
Decision: Implement _handle_oci_cacheable() as a separate function
from the existing RPM cache logic (lambda_handler → _handle_rpm).
Context: Both OCI and RPM caching share the same S3 bucket, uploader Lambda, and touch mechanism. The code could be unified.
Alternatives: Extend the existing RPM handler to handle OCI requests by adding OCI-specific branches.
Rationale: The RPM cache has ETag-based revalidation and configurable stale-cache behavior (origin vs cache) that do not apply to OCI — digests are immutable by definition. Keeping the handlers separate avoids accidentally changing RPM behavior and keeps each code path focused. The OCI handler was ported from the CKI dependency-archive, which is proven in production; staying close to the CKI code structure reduces the risk of introducing untested edge cases.
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.
4.5.29 - RPM CVE Count
A CLI tool to count the number of known CVEs for a given list of RPM packages by querying the Red Hat OSIDB database.
Features
- Batch processing: Query multiple packages from a file
- Impact filtering: Filter by CVE severity (CRITICAL, IMPORTANT, MODERATE, LOW)
- Date filtering: Count only CVEs created after a specific date
- CSV output: Easy to import into spreadsheets or process with other tools
Prerequisites
- Red Hat VPN: Must be connected to access the OSIDB database
- Go 1.21+: For building from source
Installation
Install the latest version:
go install gitlab.com/redhat/hummingbird/tools/rpm-cve-count@latest
Or build from source:
git clone https://gitlab.com/redhat/hummingbird/tools.git
cd tools/rpm-cve-count
go build
Usage
rpm-cve-count -file <package-file> [-after <date>] [-impact <level>]
Options
| Option | Description | Required |
|---|---|---|
-file |
Read packages from file (one per line) | Yes |
-after |
Count CVEs created after date (YYYY-MM-DD) | No |
-impact |
Filter by impact: CRITICAL, IMPORTANT, MODERATE, LOW | No |
Examples
Create a file with package names (one per line):
# packages.txt
kernel
systemd
openssl
glibc
Count all CVEs for the packages:
$ rpm-cve-count -file packages.txt
kernel,342
systemd,87
openssl,156
glibc,234
Count only CRITICAL CVEs:
$ rpm-cve-count -file packages.txt -impact CRITICAL
kernel,23
systemd,5
openssl,18
glibc,12
Count CVEs created after a specific date:
$ rpm-cve-count -file packages.txt -after 2024-01-01
kernel,45
systemd,12
openssl,28
glibc,31
Combine filters to count CRITICAL CVEs from the last year:
$ rpm-cve-count -file packages.txt -impact CRITICAL -after 2024-01-01
kernel,8
systemd,2
openssl,5
glibc,3
Save results to CSV:
rpm-cve-count -file packages.txt > results.csv
Output Format
CSV format with two columns:
- Package name
- CVE count
Development
# Build
go build
# Run tests
go test ./...
# Install locally
go install
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.
4.5.30 - Hummingbird Dashboard
Web dashboard and CLI for monitoring Konflux build pipeline status.
Features
- Web Dashboard - Real-time view of build, test, and release status
- Commits View - Detailed pipeline status per commit with expandable releases
- Components View - Latest status per component grouped by state
- Merge Requests View - Track build status across MR versions
- CLI Tool - Command-line access to pipeline data in YAML/JSON/table formats
- Failed Releases - View and manage failed releases with LLM-powered analysis
- Failed Push Builds - Detect and auto-retry failed on-push Konflux builds
- Auto-Rerun - Automatically retry transient release and push build failures
- Blocked Snapshots - Block/unblock snapshots from auto-rerun with error pattern matching
- Smart Triggering - Application-aware rules for determining affected components
- Error Budget Alerts - Slack notifications when CVE R-Time error budget hits 20%, 5%, or 0% remaining
- Blame Page -
/cve/blameview listing late unblamed tickets with per-category breakdown and per-ticket stage detail - ProdSec CVE-HUM SLA Auto-Blame - Auto-locks ProdSec blame when CVE-HUM time exceeds 6h
- Image Pull Availability - Estimated SLI-UE1-B availability from manifest and sampled checks
- RPM Tracker API - Read-only snapshot of tracker-owned tables in the shared database
- RPM Tracker Views - Package currency, upstream staleness, merge based rollout lag, and stuck update MRs
Prerequisites
- Python 3.11+
- PostgreSQL database (via hummingbird-status or port-forward)
Installation
cd hummingbird-dashboard
pip install -e .
Usage
Local Development
cd hummingbird-dashboard
# Option 1: Port-forward to production database
./dev.sh port-forward # In terminal 1
DATABASE_URL=postgresql://postgres@localhost:15432/events ./dev.sh start # In terminal 2
# Option 2: Use local database (via hummingbird-status)
cd ../hummingbird-status && ./dev.sh db-start && ./dev.sh db-init
cd ../hummingbird-dashboard
DATABASE_URL=postgresql://postgres:dev@localhost:5432/events ./dev.sh start
The dashboard runs at http://localhost:8080 with live reload.
The dashboard’s Alembic migrations do not create the components table: it
belongs to hummingbird-status. With a fresh local database created by
dev.sh db-start alone, every page that lists applications fails with
relation "components" does not exist, so run
hummingbird-status/dev.sh db-init first (or hummingbird_status.db.create_schema()
when the venv lacks boto3).
Web UI
| Endpoint | Description |
|---|---|
/ |
Dashboard with all applications |
/apps/{app}/commits |
Commits with build or promotion PipelineRuns and releases |
/apps/{app}/components |
Component status from builds, promotions, and releases |
/mrs |
Merge requests overview (filterable) |
/mr/{project}/{iid} |
MR builds and the post-merge build or promotion |
/releases/failed |
Failed releases with analysis |
/releases/blocked |
Blocked snapshots management |
/releases/analysis-log |
LLM analysis run history |
/cve/status |
CVE ticket status dashboard |
/cve/run-log |
CVE analysis run history |
/cve/open |
Open CVE trackers with SLO status |
/cve/closed |
Closed tickets with VEX reconciliation |
/cve/r-time |
CVE R-Time (notified to done) metrics |
/cve/blame |
Late unblamed tickets and per-category blame breakdown |
/rpm/rollout |
RPM currency, staleness, lifecycle, and update MRs |
/rpm/stuck-mrs |
Stuck RPM update MRs, including unknown ages |
/availability |
Estimated image pull availability (SLI-UE1-B) |
/health |
Health check endpoint |
/metrics |
Prometheus metrics |
For RPMs configured for MR artifact promotion, the post-merge PipelineRun promotes
the tested MR image. The commits and MR detail views label that run as a
promotion and follow its Component Snapshot to the automatic Release. Component
status counts a successful promotion and Release as current. If a retry reuses
the Component Snapshot, the dashboard also links it through the retry’s
COMPONENT_SNAPSHOT result. Failed promotions offer the manual retrigger
button: the clone keeps the merge SHA, gets a fresh git-auth secret, and the
promotion Task reuses the deterministic Component Snapshot when its evidence
still matches. Build throughput and failed push-build auto-rerun continue to
count actual builds only.
Snapshot lookups use the indexed source_plr column or the Snapshot name from
the promotion PipelineRun result. This also covers historical promotion
Snapshots that were ingested before their annotation was stored in source_plr.
The dashboard summary validates the newest successful run per component first,
then checks older runs in one batch when a required test failed or a promotion
has no successful Release.
Each CVE tracker page (/cve/status, /cve/open, /cve/closed,
/cve/r-time) has a “Download CSV” button at the bottom of the page that downloads
the current view (same filters as the page) as a CSV file, via a matching
/cve/{tab}/csv route (for example /cve/open/csv?slo=24). A “Show CVE ID”
toggle in the header reveals the CVE ID next to each HUM ticket key
(HUM-1234/CVE-2026-56789); the preference persists via localStorage.
On /cve/status, a “Show no-CVE” toggle next to it reveals HUM tickets that
have no assigned CVE ID in the category drawers. They are hidden by default,
and the preference persists via localStorage.
/cve/r-time/csv appends one column per Stage Breakdown milestone (HUM Ticket Created, CVE Published, OSIDB Flaw Created, OSIDB Flaw Public,
OSIDB Affect Created, Upstream Fix Merged, Fedora Update Available,
RPM Published to Pulp, Image Rebuilt on Quay, HUM Ticket Closed, VEX Updated, CVE Embargo Lifted) after the summary columns, in the same
chronological order used in the per-ticket Stage Breakdown panel. Each cell
is an ISO 8601 timestamp, or blank when that milestone has no data for the
ticket.
On the CVE Tracker Status, Open, and Closed pages, a compact outline icon beside
a HUM ticket indicates the CVE Agent’s final Jira verdict label, not a confirmed
Jira resolution. Hover or focus the ticket for the recommendation; the “Agent
verdicts” help control explains the icons. One policy table per verdict drives
the icon, the expected closure, the automatic review and the failure handling. It
mirrors hummingbird_cve_agent/labels.py and
hummingbird_cve_analysis/lib/states.py without adding a build dependency.
Closed-tab scoring
| Verdict | Expected resolution | Treatment |
|---|---|---|
needs_change, already_fixed |
Done-Errata |
Matching closure agrees; contradictory closure disagrees |
not_affected |
Not a Bug |
Matching closure agrees; contradictory closure disagrees |
needs_human, error |
— | Failure |
no_upstream_fix |
— | Reviewable, excluded from closure scoring |
Accuracy is matches divided by comparable closures plus failures, within the
selected period. A current saved review overrides Jira closure for both the row
label and accuracy, so a reviewed no_upstream_fix counts while it stays
current; stale reviews fall back to Jira closure. Tickets without a verdict and
incomparable resolutions without a review (such as Won't Do or a missing
resolution) are excluded; no score is shown without scorable results. Closure
agreement establishes neither decision correctness nor MR quality.
Reviews
| Review | Behavior |
|---|---|
| Automatic | Derived without database writes, for failures and for not_affected closed as Not a Bug with no linked agent MR and no VEX mismatch |
| Saved | Takes precedence over an automatic review; a non-stale decision overrules the closure comparison |
| No upstream fix | Can confirm the historical judgment; recorded upstream_fix_merged evidence marks the verdict outdated and its review stale |
The agent never re-investigates a recorded verdict, and saving another review does not clear the no-fix stale flag. Automatic re-investigation is not implemented.
RPM Tracker Views
The Rollout view supports package search, status and packaged stream lifecycle filters, and sortable columns. Summary counts describe the full inventory before filters. By default, All packages lists current and behind packages first, with unmonitored and unknown packages last. Each group is sorted alphabetically. Explicit column sorting overrides this default grouping. Unavailable sort values remain last in either direction.
Each row shows the packaged and eligible upstream versions, currency, staleness, merge based rollout lag, packaged stream lifecycle, and a selected open update MR. Staleness starts at the first newer eligible upstream release, not the most recent one. Rollout lag is available only for current packages whose version equals the selected upstream version and has a resolved merge time. Missing or inconsistent dates remain unavailable. The merge timestamp is a proxy, not proof of successful publication. A fallback MR is labeled Package candidate only, not a verified update to the selected upstream version. Unsupported streams flag support risk without changing downstream CVE policy.
The Stuck MRs view lists every open update MR older than 24 hours, plus MRs with unknown creation based ages. It includes multiple MRs for the same package and MRs whose packages are absent from the inventory. The view also supports listing all open update MRs and searching by package or MR title. Unknown ages are not shown as zero or silently excluded.
Both views read the rpm_tracker schema through this dashboard deployment’s
existing DATABASE_URL configuration. The preprod dashboard reads the preprod
Status database, and the production dashboard reads the production Status
database. No separate tracker URL, data copying, or environment selector is
required. The dashboard does not migrate or write tracker tables.
Rollout reads are limited to rollouts for currently packaged versions. The MR
view and listing API do not read package inventory or rollout history. Retained
historical rollouts remain available through GET /api/rpm-upstream-state.
The pages show the last successful collection and independent MR reconciliation time. Collection older than eight hours displays a warning while retaining the last successful inventory. Package rows show their check time, upstream monitor scan state, last successful scan, and scan errors separately. A missing or incompatible schema produces a readable HTTP 503 page, rather than an empty inventory. Database connection failures and pool timeouts also return HTTP 503, with a temporary unavailability message rather than a schema warning. These failures are logged without exposing connection details on the page. A schema with no successful collection or MR reconciliation shows the corresponding pending state. See RPM Tracker for collection, reconciliation, and migration details.
Examples:
/rpm/rollout?status=behind&search=openssl
/rpm/rollout?status=current&lifecycle=active&sort=rollout_lag_hours&order=desc
/rpm/stuck-mrs?stuck_only=false&search=openssl
Competitive Trends
On /competitive, click trend for an image to compare Hummingbird and
Chainguard CVE counts over the past 30 days. Distributor series share a
chronological time axis and value scale, with distinct colors and legend labels.
Hover over a point to see its distributor, collection date and time, and CVE count.
Missing observations are not filled with zero, and explicit null counts leave
gaps in the line. Points with invalid collection timestamps are omitted. Opening
another image clears the previous chart immediately, so a failed request cannot
leave another image’s data displayed. Data collection and CVE calculations are
unchanged.
Competitive CVE Diff
On /competitive, click the Delta value on any row to expand a drawer
listing the CVE IDs that differ between the Hummingbird and Chainguard scans
for that image. A + marks a CVE ID reported only by Hummingbird; a -
marks a CVE ID reported only by Chainguard. This is a scan result
difference between the two tools, not a confirmed affectedness
determination — reasons a CVE shows up on only one side include differing
scanner databases, matching heuristics, and false positives/negatives, not
necessarily that the other side is truly unaffected. CVE IDs both sides
report are shared and excluded from the diff. Each CVE ID links to its Red
Hat CVE page; when a
HUM Jira ticket tracks it, the ticket is shown and linked alongside it
(HUM-5678/CVE-2026-1234). If a scan predates CVE ID collection, the drawer
shows a notice instead of a misleading empty diff.
GET /api/competitive/comparison (the page’s main listing) returns only
per-image counts and delta — not the CVE ID arrays, HUM ticket links, or
even whether per-CVE detail is available — with hundreds of tracked images,
computing and shipping that detail for every row up front would be wasted
work for the rows no one expands. A row’s drawer content is fetched from
GET /api/competitive/diff?image_name=...&tag=... the first time it’s
opened, then cached client-side so reopening it doesn’t refetch; that
response’s own cve_detail_available decides whether to render the diff or
a “check back later” notice.
Both sides of a scan can report a native GHSA or Go vulnerability DB ID
instead of a CVE for some ecosystems (e.g. Go modules in an image like
cosign): Chainguard’s own Grype scan and the Hummingbird catalog API’s
Grype-based scan alike. The collector resolves these against grype’s local
database before storage (see chainguard-comparison.md), so by the time data
reaches the dashboard, only IDs with no CVE anywhere in that database
remain unresolved. These are shown as plain text with no link and no HUM
ticket match attempted, rather than resolved here: that would require an
OSV.dev network call per unresolved ID, which is unreliable (many
environments, including this one, can’t reach it) and, done sequentially,
slow enough to hang the drawer open for tens of seconds on an image with
several unresolved IDs.
A ticket is indexed under every CVE ID in its cve_ids field (from
hummingbird_cve_analysis’s dedicated CVE ID Jira field — see
hummingbird-cve-analysis.md), since a single ticket can legitimately track
more than one CVE; tickets collected before that field existed fall back to
the single first CVE-format label found. The Competitive page compares
whole images, but HUM tickets are filed per affected RPM. A CVE in a single
bundled dependency (e.g. openssh) can affect several unrelated images
(e.g. both git and curl) with no per-image ticket at all. Tickets
resolved known_not_affected (VEX “Not a Bug”) are excluded from matching
entirely — a ticket confirming its own
package isn’t actually impacted must never be suggested as if it were
tracking the CVE for some other image. Of the remaining candidates, each
one’s catalog_images field (every {image}:{stream} its package actually
appears in, from hummingbird_cve_analysis’s SBOM-derived catalog source
map — see hummingbird-cve-analysis.md) is checked first: an exact match
there is ground truth, so it’s used directly. Older tickets collected before
that field existed fall back to a package-name heuristic (an exact
{image}{stream} match, the Red Hat packaging convention for versioned
streams like mariadb11.8, then substring overlap). When nothing
disambiguates to a single ticket either way, every remaining candidate is
linked rather than guessing one or hiding all of them.
The image name/tag checked against catalog_images is the catalog’s own
identity (chainguard_scans.catalog_image_name /
catalog_stream, populated by chainguard-comparison.md’s collector — see
below), not the Competitive page’s “common” display name or parsed
version: a few images are shown under a cross-vendor alias (Hummingbird’s
valkey image is displayed as redis, to pair with Chainguard’s image of
that name) and the page’s parsed version does not always match the
catalog’s own stream label (e.g. a canonical tag 8.23.0 parses to 8.23
here while the catalog calls that stream 8). Comparing the display
name/version against catalog_images would spuriously reject tickets that
actually apply. Scan rows collected before catalog_image_name /
catalog_stream existed fall back to the display name/tag, matching the
pre-migration behavior exactly.
A candidate is only excluded by a non-matching catalog_images when that
image/stream was actually checked during the ticket’s collection run.
hummingbird_cve_analysis’s catalog source map silently skips an
image/stream whose tag listing or SBOM fetch failed (see
hummingbird-cve-analysis.md), so a nonempty catalog_images list is not
necessarily complete; each ticket’s catalog_unverified_images field lists
exactly which image/stream pairs that run couldn’t verify, and a candidate
matching one of those is kept as plausible (falling through to the
package-name heuristics) rather than excluded.
Only the most-recently-collected tag is shown per image; if a tracked
image’s version changes between collection runs (e.g. curl 8.22 -> 8.23),
the superseded tag’s row is dropped rather than lingering alongside the
current one.
CVE Search API
GET /api/cve-runs/search — search CVE analysis run history.
| Parameter | Type | Default | Description |
|---|---|---|---|
package |
string | — | Filter by package name (substring match) |
ticket |
string | — | Filter by ticket key (substring match) |
q |
string | — | Free-text search across ticket details |
log |
string | — | Search CronJob log output |
days |
int | 30 | Time window in days (1–365) |
limit |
int | 500 | Max results (1–2000) |
At least one of package, ticket, q, or log is required.
The response includes two result sets:
results— per-ticket matches from the JSONB details columnlog_matches— per-run matches from CronJob log output, each with alog_excerptshowing the matching lines in context
Example:
# Find runs whose logs mention "merge train" — show excerpts
curl -s '/api/cve-runs/search?log=merge+train' | jq '.log_matches[] | {recorded_at, log_excerpt}'
# Find tickets for a package, also searching logs
curl -s '/api/cve-runs/search?package=openssl&log=advisory+failed&days=7' \
| jq '{tickets: [.results[].detail.key], log_hits: .log_matches | length}'
# List all ticket keys matching a free-text query
curl -s '/api/cve-runs/search?q=needs-attention&days=14' | jq '[.results[].detail.key] | unique'
CLI
# View latest commit status
hummingbird-dashboard --application myapp --format table commit
# View specific commit
hummingbird-dashboard --application myapp commit abc1234
# View multiple commits
hummingbird-dashboard --application myapp --format table commit --limit 10
# Component status overview
hummingbird-dashboard --application myapp component
# JSON output
hummingbird-dashboard --format json commit | jq .
# Auto-rerun failed releases
hummingbird-dashboard auto-rerun
# Analyze failed releases via LLM
hummingbird-dashboard analyze-failures
# Probe catalog image pull availability (HUM-721)
hummingbird-dashboard check-image-pulls
hummingbird-dashboard check-image-pulls --dry-run
hummingbird-dashboard check-image-pulls --force
REST API Reference
All endpoints return JSON. Read-only GET endpoints are unauthenticated.
Interactive OpenAPI/Swagger documentation is available at /docs.
RPM Tracker
GET /api/rpm-upstream-state returns the collection generation and checkout
SHA with upstream, package status, rollout reasons, and all open update MRs
from the independently migrated rpm_tracker schema. It also exposes the MR
reconciliation scope and freshness and one selected blocking_mr per package
row. GET /api/rpm-update-mrs lists every open update MR independently of that
selection, including multiple MRs for one package. Add ?stuck_only=true for
MRs older than 24 hours and MRs with unknown creation-based ages; unknown ages
remain null rather than disappearing. The dashboard does not create or write these
tables. Run the tracker initializer before deploying this API; until a compatible
tracker schema is present, this endpoint returns HTTP 503. See
RPM Tracker for migration and storage details.
Tier 1 — Core Pipeline Status
GET /api/apps/{app_name}/commits
Pipeline status for commits in an application.
| Parameter | Type | Default | Description |
|---|---|---|---|
sha |
string | — | Filter by commit SHA (prefix) |
component |
string | — | Filter by component name |
limit |
int | 10 | Max commits to return (max 100) |
curl -s 'http://localhost:8080/api/apps/rpms/commits?limit=5' | jq .
curl -s 'http://localhost:8080/api/apps/rpms/commits?sha=abc123&component=openssl' | jq .
GET /api/apps/{app_name}/components
Latest build status per component, grouped by state.
| Parameter | Type | Default | Description |
|---|---|---|---|
search |
string | — | Filter components by name (substring) |
curl -s 'http://localhost:8080/api/apps/rpms/components' | jq .
curl -s 'http://localhost:8080/api/apps/rpms/components?search=openssl' | jq .
Tier 2 — Operational Visibility
GET /api/dashboard
Full overview across all applications: aggregate build counts, failed releases, component staleness, and recent activity.
curl -s 'http://localhost:8080/api/dashboard' | jq .
GET /api/releases/failed
List of currently failed releases with analysis results.
| Parameter | Type | Default | Description |
|---|---|---|---|
application |
string | — | Filter by application name |
curl -s 'http://localhost:8080/api/releases/failed' | jq .
curl -s 'http://localhost:8080/api/releases/failed?application=rpms' | jq .
GET /api/releases/{name}/analysis
LLM failure analysis for a specific release (root cause, classification, recommendation).
curl -s 'http://localhost:8080/api/releases/my-release-abc/analysis' | jq .
GET /api/releases/auto-rerun-log
History of automatic rerun attempts.
| Parameter | Type | Default | Description |
|---|---|---|---|
limit |
int | 50 | Max entries |
curl -s 'http://localhost:8080/api/releases/auto-rerun-log?limit=10' | jq .
GET /api/releases/analysis-log
History of LLM analysis runs.
| Parameter | Type | Default | Description |
|---|---|---|---|
limit |
int | 50 | Max entries |
curl -s 'http://localhost:8080/api/releases/analysis-log?limit=10' | jq .
GET /api/releases/analysis-costs
Cumulative token usage and estimated cost for LLM analysis runs.
curl -s 'http://localhost:8080/api/releases/analysis-costs' | jq .
GET /api/mrs
Merge requests across monitored repositories.
| Parameter | Type | Default | Description |
|---|---|---|---|
state |
string | opened |
MR state: opened, merged, closed, all |
project |
string | — | Filter by project (repository) name |
curl -s 'http://localhost:8080/api/mrs' | jq .
curl -s 'http://localhost:8080/api/mrs?state=merged&project=rpms' | jq .
GET /api/mrs/projects
List of all projects (repositories) with tracked merge requests.
curl -s 'http://localhost:8080/api/mrs/projects' | jq .
GET /api/mr/{project}/{iid}
Detail for a single merge request, including all versions and per-version build status.
curl -s 'http://localhost:8080/api/mr/rpms/42' | jq .
GET /api/cve/status
Current CVE ticket status across all tracked packages.
curl -s 'http://localhost:8080/api/cve/status' | jq .
GET /api/cve/open
Open HUM CVE tracker tickets with SLO status. Notified is the later of HUM ticket created and fix available (upstream or Fedora). SLO is green PASS unless a fix is available and elapsed time is within 8h of the selected SLO (yellow AT RISK) or past it (red FAIL).
| Parameter | Type | Default | Description |
|---|---|---|---|
slo |
float | 24 | SLO threshold hours (24, 72, or 168) |
include_next_release |
bool | true | Include cve-next-release tickets |
curl -s 'http://localhost:8080/api/cve/open?slo=24' | jq .
GET /api/cve/closed
Closed HUM CVE tracker tickets with Red Hat CSAF VEX reconciliation
(HUM-5843). Analysis stores package-scoped vex_status and Jira
resolution; the dashboard computes MATCH from those facts
(Done-Errata → fixed, Not a Bug → known_not_affected,
package_not_listed, or fixed – a downstream package rebuilt by an
unrelated dependency’s fix is not a contradiction of a Not a Bug verdict).
Shows Hummingbird vex_status, match state
(matched / pending / mismatch), and vex_resolved (first
scan time where VEX agreed with the Jira resolution). Event metadata is
merged across a ticket’s rows in occurred_at order (HUM-6091): later
non-empty fields overlay earlier ones, so Close/VEX facts are not stuck
on the first cve_published row. The Resolution column uses Jira
Closed / {resolution} when stored analysis text is not already Closed.
The Closed tab shows the VEX timestamp as a link to the CVE’s Red Hat
CSAF VEX document when the package is named in CSAF. package_not_listed
(Not a Bug, package absent from the document) shows N/A with no link:
this ticket did not produce a VEX change.
Tickets without a VEX match status are counted below the summary cards. The
awaiting-vex label count overlaps the matched, pending, mismatch, and unknown
groups; it is not an additional category to add to the total.
The HTML page renders 100 tickets per page (page query, 1-based) while the
summary cards, CVE Agent performance and CSV export cover the whole period.
| Parameter | Type | Default | Description |
|---|---|---|---|
days |
int | 28 | Time window (0 = all time) |
curl -s 'http://localhost:8080/api/cve/closed?days=30' | jq .
Entries carry the agent verdict, closure agreement, linked MRs, any recorded
ticket cost (agent_cost, with a null estimated_cost_usd when the analysis
predates cost reporting, and unpriced_calls when some calls had no pricing)
and any saved review; agent_stats aggregates them as described above,
including cost_total, cost_tickets, cost_unknown and cost_partial.
PUT /api/retrigger/cve-agent-review/{ticket_key}
Save a reviewer’s needed and mr_outcome, plus an optional note, for a
closed CVE ticket. Requires login; the reviewed verdict, resolution and MR
identities/head SHAs are snapshotted server-side. Returns 422 when the ticket
has no agent verdict. The response returns status, ticket_key, the ticket’s
updated agent_outcome_label, agent_outcome_tone and agent_outcome_note,
and the reviewed_text byline, so the page updates the row and review panel in
place without reloading. The save reads only the reviewed ticket, not the whole
Closed tab; period statistics refresh on the next page load.
curl -s -X PUT 'http://localhost:8080/api/retrigger/cve-agent-review/HUM-123' \
-H 'Content-Type: application/json' \
-d '{"needed": "backport", "mr_outcome": "edits", "note": "test needed fixing"}'
GET /api/cve/r-time
CVE R-Time (notified to done) for Done-Errata tickets measures time spent on a
fixable CVE, so its start never predates the upstream fix in either mode.
Include CVE-HUM starts at CVE publication (NVD datePublished /
cve_published), falling back to HUM ticket creation only when CVE
publication is missing – CVE publication wins even when the HUM ticket
predates it, so an embargoed ticket’s R-Time starts at public disclosure
rather than at ticket creation (which can predate disclosure by days while
the CVE is still privately embargoed). Exclude CVE-HUM starts at the later of
HUM created and CVE publication – same as HUM created for a normal ticket
(Exclude’s purpose is to ignore filing lag after disclosure), but an
embargoed ticket’s clock must never start before public disclosure in either
mode, so it also starts at CVE publication there. Whichever of the two the
mode picks, Upstream Fix Merged is then applied as a floor on top: the final
notified timestamp is the later of that value and Upstream Fix Merged, so
neither mode ever counts time before a fix existed to backport. Delivery is
the catalog image publish when the package is a catalog image source,
otherwise the RPM publish. Include VEX requires delivery, VEX updated, and
Closed / Done-Errata; Exclude VEX requires delivery and close only. The done
timestamp is the latest of the required gates, and only when all of them
exist. Incomplete tickets stay on the table with elapsed time until now; the
days window filters by completion (fallback: Done-Errata close, then
notified), so a period means completed in that window – the Closed tab
filters by Jira close date instead. CVE-HUM is filing lag after NVD
(max(HUM created − NVD, 0)); HUM-first tickets show 0.0h. ADV-VEX is
Done-Errata close to VEX feed update (max(VEX − close, 0)); close is
recorded when the advisory MR merges. Fedora is not start or end; Upstream
Fix Merged only ever gates R-Time’s start, never its end. Pre-built (delivery
before notification) is informational only. Rows without
catalog_image_source in metadata still require an image until the collector
restamps them. Each entry includes package onboarding timestamps and delay
analysis (rpm_first_published_at, pkg_lag_hours, and our_delay_hours).
Default excludes cve-next-release, CVE-HUM time, and VEX time.
| Parameter | Type | Default | Description |
|---|---|---|---|
days |
int | 28 | Time window (0 = all time) |
slo |
float | 168 | SLO threshold hours (24, 72, or 168) |
include_next_release |
bool | false | Include cve-next-release tickets |
include_hum_cve_time |
bool | false | Include NVD-to-HUM filing lag in R-Time |
include_vex_time |
bool | false | Require VEX feed update before R-Time ends |
curl -s 'http://localhost:8080/api/cve/r-time?days=10&slo=168' | jq .
GET /api/cve/run-log
History of CVE analysis CronJob runs.
| Parameter | Type | Default | Description |
|---|---|---|---|
limit |
int | 100 | Max entries |
curl -s 'http://localhost:8080/api/cve/run-log?limit=10' | jq .
POST /api/cve/agent-cost-keys
Report which of the posted tickets already have a recorded CVE agent cost, so
the collector downloads only the remaining analysis attachments. The check
returns keys only and uses the unique ticket-key index. Requires the same
bearer token as /api/cve-report, which accepts the costs as an agent_costs
list (ticket_key, input_tokens, output_tokens, cache_read_tokens,
cache_creation_tokens, unpriced_calls, estimated_cost_usd, where a null
cost marks a pre-cost attachment) and rejects the whole report when a row is
malformed. The cache counters are subsets of input_tokens, and a non-zero
unpriced_calls makes the cost a lower bound.
curl -s -X POST 'http://localhost:8080/api/cve/agent-cost-keys' \
-H "Authorization: Bearer $CVE_REPORT_TOKEN" \
-H 'Content-Type: application/json' \
-d '{"tickets": ["HUM-123", "HUM-124"]}' | jq .
GET /api/cve-export
Export CVE sync data as JSON for prod-to-preprod replication. Valid section
values are cve_analysis_log, cve_ticket_events, package_lifecycle, and
cve_ticket_blame.
Optional pagination parameters:
| Parameter | Type | Default | Description |
|---|---|---|---|
section |
string | — | Section to export |
limit |
int | 2000 | Rows per page |
offset |
int | 0 | Pagination offset |
curl -s -H "Authorization: Bearer $CVE_REPORT_TOKEN" \
'http://localhost:8080/api/cve-export' | jq '.cve_ticket_events | length'
curl -s -H "Authorization: Bearer $CVE_REPORT_TOKEN" \
'http://localhost:8080/api/cve-export?section=cve_ticket_events&limit=1000&offset=0' \
| jq '.count,.has_more'
# Package-scoped lifecycle milestones (rpm_first_published, etc.)
curl -s -H "Authorization: Bearer $CVE_REPORT_TOKEN" \
'http://localhost:8080/api/cve-export?section=package_lifecycle&limit=10000&offset=0' \
| jq '.rows[] | select(.event_type == "rpm_first_published")'
Event type validation. The import path (POST /api/cve-import) validates
every event_type value against a canonical allowlist defined in
hummingbird_dashboard/sources.py:
CANONICAL_TICKET_EVENT_TYPES— values accepted forcve_ticket_events:cve_published,osidb_flaw_created,osidb_affect_created,cve_embargo_lifted,osidb_flaw_public,hum_ticket_created,hum_ticket_closed,upstream_fix_merged,fedora_update_available,rpm_fix_published_to_pulp,image_rebuilt_on_quay,vex_resolved.CANONICAL_PACKAGE_EVENT_TYPES— values accepted forpackage_lifecycle:rpm_first_published.
Legacy names (e.g. jira_created, delivered_in_rpm, …) are
mapped to their canonical equivalents via _LEGACY_EVENT_KEY_MAP.
Unknown names are rejected with HTTP 400.
POST /api/cve-import
Replace local CVE sync data from a payload previously returned by
/api/cve-export.
curl -s -X POST \
-H "Authorization: Bearer $CVE_REPORT_TOKEN" \
-H "Content-Type: application/json" \
--data @cve-export.json \
'http://localhost:8080/api/cve-import' | jq .
curl -s -X POST \
-H "Authorization: Bearer $CVE_REPORT_TOKEN" \
-H "Content-Type: application/json" \
--data '{"section":"cve_ticket_events","rows":[],"reset":true}' \
'http://localhost:8080/api/cve-import' | jq .
Ticket-event types such as osidb_flaw_created, osidb_affect_created,
cve_embargo_lifted, and osidb_flaw_public import with the rest of
cve_ticket_events because they are on CANONICAL_TICKET_EVENT_TYPES.
Analysis-log exports include run_started_at when available. Paginated imports
ignore rows whose non-null identity is already present, making retries safe;
legacy rows without an identity remain append-only. The ingest timestamp is
exported as recorded_at; imports still accept the legacy run_at key, but a
sync target older than this release cannot read the new key.
occurred_at conflict resolution on import matches the live collector path
(HUM-6860): rpm_fix_published_to_pulp and image_rebuilt_on_quay are
mutable delivery timestamps where a non-null incoming value wins, so a sync
can update them, without letting a null incoming value erase an
already-recorded delivery timestamp; every other event_type is write-once,
keeping the existing timestamp once set so a sync or backfill re-run cannot
overwrite an established immutable milestone.
Bulk prod-to-preprod replication of cve_analysis_log, cve_ticket_events,
package_lifecycle, and cve_ticket_blame no longer goes through this API —
cve_analysis_log rows can carry up to 512 KiB of log_output, so paginating
that table meant hundreds of small requests that aborted entirely on the
first 5xx. See copy_prod_to_preprod_db in
CVE Analysis
Reference
for the
direct pg_dump/pg_restore sync tool that replaced it. These endpoints
remain in use for the smaller, targeted writes collect_rpm_first_published
and backfill_osidb_timestamps.py make directly against prod or preprod.
Tier 3 — Analytics & SRE Metrics
GET /api/stats/build-throughput
Build counts over time (successful, failed, total).
| Parameter | Type | Default | Description |
|---|---|---|---|
days |
int | 30 | Lookback window in days |
curl -s 'http://localhost:8080/api/stats/build-throughput?days=7' | jq .
GET /api/stats/release-success-rate
Release success/failure ratios over time.
| Parameter | Type | Default | Description |
|---|---|---|---|
days |
int | 30 | Lookback window in days |
application |
string | — | Filter by application |
curl -s 'http://localhost:8080/api/stats/release-success-rate?days=14&application=rpms' | jq .
GET /api/stats/auto-rerun-effectiveness
Success rate and time-to-resolution for automatic reruns.
| Parameter | Type | Default | Description |
|---|---|---|---|
days |
int | 30 | Lookback window in days |
curl -s 'http://localhost:8080/api/stats/auto-rerun-effectiveness?days=7' | jq .
GET /api/components/staleness
Components ranked by time since last successful build.
curl -s 'http://localhost:8080/api/components/staleness' | jq .
GET /api/releases/similar-failures
Find releases with error messages similar to a given string.
| Parameter | Type | Default | Description |
|---|---|---|---|
error |
string | — | Error text to match against |
curl -s 'http://localhost:8080/api/releases/similar-failures?error=timeout+connecting' | jq .
GET /api/apps/{app_name}/last-successful
Timestamp and SHA of the last fully successful pipeline per component.
curl -s 'http://localhost:8080/api/apps/rpms/last-successful' | jq .
Tier 4 — System
GET /api/service-status/json
Health of upstream services the dashboard depends on (database, Konflux API, KubeArchive, etc.).
curl -s 'http://localhost:8080/api/service-status/json' | jq .
GET /api/settings
Current runtime settings (auto-rerun enabled, analysis enabled, blocked patterns, etc.).
curl -s 'http://localhost:8080/api/settings' | jq .
GET /health
Enriched health check returning service version, uptime, database connectivity, and dependency status.
curl -s 'http://localhost:8080/health' | jq .
Configuration
Environment Variables
| Variable | Default | Description |
|---|---|---|
DATABASE_URL |
postgresql://...localhost:5432/ |
PostgreSQL connection URL |
PORT |
8080 |
Web server port |
Authentication Variables
These control the retrigger functionality (requires OAuth proxy in production):
| Variable | Default | Description |
|---|---|---|
TRIGGER_AUTH_MODE |
oauth |
oauth (production) or local |
TRIGGER_AUTH_GROUP |
konflux-hummingbird-admin-access |
OpenShift group required to retrigger |
TRIGGER_LOCAL_USER |
local-dev |
Username when TRIGGER_AUTH_MODE=local |
CVE_REPORT_TOKEN |
Bearer token for CVE API auth |
The dashboard checks TRIGGER_AUTH_GROUP against the user’s groups on the
OpenShift cluster hosting the dashboard. Access on the Konflux cluster alone
does not grant retrigger permission. Membership in the default
konflux-hummingbird-admin-access group is managed in Rover and synchronized
to the dashboard cluster by the infrastructure configuration. Claiming a CVE
ticket requires only a dashboard login, so the user menu can show Claim ✅ and
Retrigger ❌. Successful group lookups are cached for five minutes; failed
lookups are retried on the next request.
For local development with retrigger enabled:
TRIGGER_AUTH_MODE=local DATABASE_URL=... ./dev.sh start
Auto-Rerun Variables
These control the auto-rerun cronjob and failure analysis:
| Variable | Default | Description |
|---|---|---|
AUTO_RERUN_MIN_AGE_MINUTES |
30 |
Minimum failure age before retrying |
AUTO_RERUN_MAX_RETRIES |
3 |
Max rerun attempts per snapshot+plan |
KONFLUX_KUBECONFIG_PATH |
Path to Konflux kubeconfig file | |
RELEASE_NAMESPACE |
Namespace where releases are created | |
MANAGED_NAMESPACE |
Namespace for fetching release PLRs | |
KUBEARCHIVE_URL |
KubeArchive API URL for archived resources | |
GOOGLE_APPLICATION_CREDENTIALS |
Path to GCP SA key for Vertex AI | |
GOOGLE_CLOUD_PROJECT |
GCP project ID for Vertex AI | |
ANALYSIS_MODEL_API_KEY |
Gemini API key (fallback if no GCP creds) | |
ANALYSIS_MODEL |
gemini-2.5-flash |
LLM model for failure analysis |
ANALYSIS_MODEL_REGION |
global |
Vertex AI region |
MAX_ANALYSES_PER_CYCLE |
5 |
Max releases to analyze per cycle |
SLACK_WEBHOOK_URL |
Slack webhook for rerun/analysis notifications | |
DASHBOARD_URL |
Dashboard base URL for Slack links | |
GITLAB_SLACK_MAP_PATH |
/etc/hummingbird/gitlab-slack-map.yaml |
GitLab→Slack map for author @-mentions (infra-mounted; missing omits mention) |
PUSH_RERUN_MIN_AGE_MINUTES |
10 |
Min push build failure age before retrying |
PUSH_RERUN_MAX_RETRIES |
3 |
Max retry attempts per component+sha |
Error Budget Alert Variables
The dashboard pod periodically evaluates CVE R-Time error budget (168h SLO, 30-day window) and posts to Slack on worsening transitions only:
| Remaining budget | Slack alert |
|---|---|
| ≤20% | Warning |
| ≤5% | Critical |
| 0% | Exhausted (sprint-stop) |
Incoming webhooks always post to the channel they were created for; the
payload cannot select #team-hummingbird. Create the webhook for that
channel and set ERROR_BUDGET_SLACK_WEBHOOK_URL (or point the shared
SLACK_WEBHOOK_URL at that webhook).
State is stored in dashboard_settings (error_budget_rtime_alert_state) so
the same threshold is not re-alerted. The stored state is claimed with a
compare-and-set before Slack is called, and rolled back if the post fails, so
a transient Slack error is retried on the next interval and concurrent
evaluators cannot double-post the same transition. Recovery updates the
stored state silently so a later re-worsening can alert again. No message is
sent when the webhook URL is unset, or when alerts are disabled via the
R-Time page toggle (error_budget_alerts_enabled in dashboard_settings;
default enabled).
| Variable | Default | Description |
|---|---|---|
ERROR_BUDGET_SLACK_WEBHOOK_URL |
SLACK_WEBHOOK_URL |
Incoming webhook created for the alert channel (typically #team-hummingbird) |
SLACK_WEBHOOK_URL |
Fallback webhook if the error-budget URL is unset | |
DASHBOARD_URL |
Base URL for the R-Time deep link in alerts | |
ERROR_BUDGET_ALERT_INTERVAL_SECONDS |
900 |
Seconds between background checks |
Failures
The failures section is accessible via the “Failures” nav item and provides two views selectable by tab: Releases and Push Builds.
Failed Releases
The /failures/releases page shows all failed releases with:
- LLM Analysis — Each failure is analyzed by Gemini with root cause, classification, and recommendation
- Failure Classification — Transient, Configuration, Code, External Service, or Unknown
- Rerun History — Past rerun attempts and outcomes per snapshot
- Error Pattern Matching — Auto-block snapshots matching known error patterns
- Auto-Rerun — Automatically retry transient failures (configurable via dashboard toggle)
- Analysis Toggle — Enable/disable LLM analysis from the dashboard
CLI Subcommands
The auto-rerun subcommand retries eligible failed releases:
hummingbird-dashboard auto-rerun --application myapp
The auto-rerun-push subcommand retries eligible failed push builds:
hummingbird-dashboard auto-rerun-push --application myapp
Before creating a retry PipelineRun, the command records an atomic claim for
its component and commit in push_build_rerun_claims. Later CronJob runs skip
that pair until the claimed PipelineRun is recorded as failed in the dashboard
database. A successful or still-running retry prevents another attempt, and
PUSH_RERUN_MAX_RETRIES also applies to claims that are not yet ingested.
If the create request has an uncertain outcome, the claim is retained to avoid
accidentally creating a duplicate. Check the claimed plr_name in Konflux
before manually clearing such a claim.
The analyze-failures subcommand runs LLM analysis on unanalyzed failures:
hummingbird-dashboard analyze-failures --managed-namespace rhtap-releng-tenant
All three are designed to run as Kubernetes CronJobs. Slack messages for auto-rerun, push-retry, analysis, and manual UI reruns include a GitLab MR link (and author @-mention when mapped) when the failure SHA resolves to an MR. Release success messages follow reruns tracked by the previous auto-rerun run, so delayed status ingestion does not skip a success. Failed Slack posts are recorded as pending in the auto-rerun log and retried on the next run.
Image Pull Availability
The /availability page tracks HUM-721 (SLI-UE1-B): can a customer actually
pull the images we publish. Continuously pulling every catalog image on a
tight schedule is not feasible (terabytes of registry traffic), so the check
is split into three tiers, run by check-image-pulls:
- Manifest HEAD (exhaustive) — a cheap HEAD request, no image bytes
transferred, against every
latest*image:variant tag in the catalog on every run. This is the headline “Estimated Availability” number. A digest cache skips the live HEAD when the tag’s digest is unchanged, the previous check succeeded, and it was confirmed within the last 6 hours. - Blob sample (sampled) — a real HEAD against every blob (config + layers) of a small random sample of currently-available images each run, confirming referenced layers still exist in storage.
- Full-pull sample (sampled) — a real GET of one image’s config blob (a few KB, following the registry’s CDN redirect) each run — genuine end-to-end retrieval, not just a HEAD.
Both sample tiers prioritize tags published in the last hour over older ones, since “did the thing we just shipped actually land” is the highest-value failure mode. The resulting availability % is estimated: the manifest tier is exhaustive, but the blob/full-pull tiers are sampled, not exhaustive — the page shows all three separately rather than blending them into one number.
- SLO: 95% target, 28-day rolling window, based on the manifest tier
- Error Budget: manifest-check misses allowed within the window
hummingbird-dashboard check-image-pulls
hummingbird-dashboard check-image-pulls --dry-run
hummingbird-dashboard check-image-pulls --force
| Option | Description |
|---|---|
--dry-run |
Probe without writing results to the database |
--force |
Force live manifest HEAD checks (ignore digest cache) |
--catalog-api-base |
Catalog API base URL (default: CATALOG_API_BASE or built-in) |
Designed to run as a Kubernetes CronJob (every 15 minutes). Checks can be
disabled from the dashboard via the image_pull_checks_enabled setting.
Merge Requests
The /mrs page shows merge requests across all monitored repositories with:
- State filter - Open, merged, closed, or all MRs
- Project filter - Filter by specific repository
- Build status - Aggregate status across all components
The /mr/{project}/{iid} detail page shows:
- All MR versions - Each head commit SHA that was pushed to the MR
- Build status per version - Full pipeline status (build, snapshot, test, release)
- Links to Konflux UI - Direct links to PipelineRuns and Snapshots
Versions are ordered by latest event timestamp, so force-pushed commits appear in the correct position even if they reuse an earlier SHA.
Component Status
The dashboard tracks component build status with these states:
| Status | Icon | Description |
|---|---|---|
| Success | ✅ | Build passed for expected commit |
| Superseded | 🔄 | Expected commit not built, but newer succeeded |
| Failed | ❌ | Build failed for expected commit |
| Stale | ⚠️ | Build not triggered for expected commit |
| Running | ⏳ | Build in progress |
| Missing | ❓ | No build found |
Components are grouped by status: Failed/Stale → Running → OK → Missing.
Trigger Rules
The dashboard uses application-specific rules to determine which components are affected by a push:
- containers: Changes in
images/{component}trigger builds - rpms: Changes in
rpms/{component}ormock/mock.cfgtrigger builds - tools: Changes in
{component}directory trigger builds
Certain files are excluded from triggering (README, templates, etc.).
Development
See the main README for development workflows.
Reusable text filtering
hummingbird_dashboard/web/static/text-filter.mjs exports initTextFilter(root).
It is independent of RPM data and can enhance other dashboard tables.
Mark the input with data-text-filter-input and each searchable row with
data-filter-text. The row attribute contains the text to search, such as a
package name or a combination of ticket key, package, and CVE ID. The optional
count, table wrapper, and empty message use the attributes shown below.
Matching ignores case and surrounding whitespace, and leaves table order intact.
<div data-text-filter>
<form method="get">
<label for="package-search">Package</label>
<input
id="package-search"
name="search"
type="search"
data-text-filter-input
/>
<button type="submit">Apply filters</button>
</form>
<p aria-live="polite">Showing <span data-text-filter-count>1</span></p>
<div data-text-filter-table>
<table>
<tbody>
<tr data-filter-text="example-package">
<td>example-package</td>
</tr>
</tbody>
</table>
</div>
<p data-text-filter-empty hidden>No matches.</p>
</div>
<script type="module">
import { initTextFilter } from "/static/text-filter.mjs";
document.querySelectorAll("[data-text-filter]").forEach(initTextFilter);
</script>
The input’s name is the URL query parameter. Mark sort or filter links with
data-text-filter-link to preserve the typed text when navigating. An unnamed
input filters rows without changing URLs. Components are scoped to their root,
and a form is optional. Use a dedicated GET filter form when server controls are
present. Text alone does not submit a request, but changing another form field
still submits normally.
For expandable tickets, put data-filter-text on a shared row group, such as a
tbody, rather than only the main row. The group hides together while its
details keep their expansion state.
Render all rows allowed by the other filters, not just the initial text matches.
Use hidden on initial nonmatching rows and set the initial count and empty
message on the server. Paginated tables should keep server filtering instead.
RPM Tracker uses this module now; adding it to CVE Tracker is separate work.
Running Tests
cd hummingbird-dashboard
pip install -e ".[dev]"
pytest
node --test tests/*.mjs
Browser logic tests use the Node.js built in test runner and are also included
in make node-test-host from the repository root.
The PostgreSQL migration regression test uses the CI PostgreSQL sidecar via
DATABASE_URL. Locally, set DATABASE_URL to a disposable PostgreSQL database
to run it; without one, the test is skipped.
Building Container Image
cd hummingbird-dashboard
podman build -f Containerfile -t hummingbird-dashboard .
Database Migrations
The dashboard uses Alembic to manage its schema.
Migrations run automatically on every startup via alembic upgrade head.
On first startup against an existing environment, the dashboard detects that the
schema is already present (via the blocked_snapshots sentinel table) and
automatically stamps the baseline migration as applied before upgrading. No
manual intervention is required.
Local development — migrations run automatically via ./dev.sh db-seed:
./dev.sh db-start
./dev.sh db-seed # runs alembic upgrade head then seeds test data
./dev.sh start
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.
4.5.30.1 - Hummingbird Dashboard: RPM upstream rollout tracker
Problem
We lack a clear view of which RPMs are behind upstream, how long they have been behind, and which update MRs are stuck. Staleness is found reactively.
Background
The upstream monitor service discovers, normalizes and
classifies upstream releases. It exposes a released_at timestamp per release,
groups versions into streams, and carries support lifecycle state sourced from
endoflife.date. The timestamp’s meaning depends on the upstream source. It
currently monitors about 200 projects.
The rpms repo package metadata carries upstream_repo and
sometimes track_upstream, upstream_branch, upstream_version_transform, or
version_suffix_strip. The packaged version comes from the spec file. The
existing ci/check_upstream_versions.py reads it with rpmspec, which resolves
macros. A scheduled upstream_version_update CI job bumps specs, fetches
sources, and opens update MRs.
hummingbird-status initializes the shared events PostgreSQL database and
ingests GitLab MRs into its public.gitlab_merge_requests table. The dashboard
reads that table and owns additional tables for its own features. The CVE
Tracker uses a collector POSTing to a token-authenticated endpoint. The RPM
tracker instead follows Pipeline Timing: an independent
schema owner writes directly to its schema in the shared database, and the
dashboard reads the resulting state.
Findings
- All 557 packages carry
upstream_repoin metadata. Read their packaged versions from the specs with the existingparse_spec_versionhelper rather than treating the metadataversionfield as authoritative. - The upstream monitor reports
released_atand the release source. The tracker does not need to fetch release dates from multiple upstream APIs, although a reported date is not always the instant a source became public. - The monitor has about 200 projects, not necessarily 200 of our packages. Package coverage must be counted after matching packages to projects. Its published API already exposes project definitions, paginated release history, stream lifecycle, and project scan status.
- Stream lifecycle (
active,security,extended,unsupported) is exposed per stream. An unsupported upstream stream is a support-risk signal. It does not change our downstream CVE policy on its own. - V1 can use an update MR’s actual merge time as a proxy for rollout when it is available. The planned Phase 2 definition is successful Release completion, confirmed with Benjamin Sherman (HUM-5820). The timeline data model and its resolution path are not yet implemented.
- A pipeline timeline alone cannot identify the target package and version of an update MR. That identity must be resolved from the rpms repo’s update MRs.
Proposal
Mirror upstream state from the monitor into the shared events database,
compare it against our packaged versions, and render an RPM Tracker tab. A
collector maintains the upstream and version rows through tracker-owned
transactional storage. The dashboard reads these rows and resolves open update
MRs from the existing Status data.
Data
RPM tracker owns and migrates its tables in a dedicated
rpm_tracker schema within the shared events database. Table names below
refer to this schema unless qualified otherwise. The original event tables in
public, including public.gitlab_merge_requests, are owned by
hummingbird-status. Keeping these use cases in distinct schemas preserves
table ownership while allowing the dashboard to join tracker and MR data
across schemas in one query. Run the tracker initializer before exposing the
dashboard read API and before each collector rollout; the collector checks the
schema revision before collecting or writing. A failed migration or revision
check stops collection without changing Status ingestion. Postgres can filter
and sort computed staleness alongside the existing MR data.
rpm_upstream_state is a tracker-owned read model. The monitor remains the
authority for upstream releases. Anything else that needs upstream state
consumes the monitor directly rather than reading our mirror.
rpm_upstream_state: Latest eligible upstream state and packaged stream
lifecycle per mapped package, computed from the monitor on every collector run.
Unmonitored packages have no upstream row and appear through a left join with
rpm_version_status.
| Column | Meaning |
|---|---|
package |
rpms repo package name (primary key) |
project_id |
Verified monitor project identifier |
upstream_version |
Latest eligible upstream version |
upstream_released_at |
Monitor timestamp for that release |
upstream_source |
Source reported for the upstream release |
packaged_stream |
Stream containing the packaged version, if known |
lifecycle_status |
Lifecycle of packaged_stream, or unknown |
monitor_last_success_at |
Last successful upstream scan for this project |
monitor_state |
Current upstream monitor scan state |
monitor_last_error |
Latest upstream monitor scan error, if any |
updated_at |
When we last wrote this row |
Map each package’s upstream_repo to a monitor project using its project
definition’s source and a verified repository mapping. If that mapping is not
unique, require an explicit upstream_monitor_project_id in package metadata.
A project ID is not derivable from upstream_repo alone.
Use track_upstream to select eligible releases as the existing update job
does. A version prefix restricts candidates to that line. latest considers
all streams. For a package without track_upstream, use an explicit
upstream_stream metadata override or the unique stream matching its packaged
version. An ambiguous choice yields unknown, not the project’s global latest.
When track_upstream is set, an upstream_stream override can identify the
packaged stream for lifecycle but cannot silently change the update target.
Do not treat upstream_branch as a stream name without a verified mapping.
Resolve packaged_stream independently of the update target so lifecycle
describes the version we package even when we track a newer stream. Respect
the monitor’s prerelease policy and the package’s version_suffix_strip and
upstream_version_transform when comparing with RPM versions. The unrelated
version_transform metadata field is for CVE version matching. If the monitor
already normalized a version in the same way, do not apply a transform twice.
Source artifact availability may delay update automation, but does not change
whether an upstream release is visible in this tracker.
rpm_version_status: One row per package, holding our side of the comparison
and the derived state. Upstream version and release date are read from
rpm_upstream_state via JOIN, and rollout timing from rpm_version_rollouts,
so neither is duplicated here.
| Column | Meaning |
|---|---|
package |
Package name (primary key) |
current_version |
Version currently packaged, from the spec file |
status |
current, behind, unmonitored, unknown |
first_behind_at |
Earliest newer upstream release date, if behind |
checked_at |
When the collector last successfully refreshed this row |
The dashboard resolves the blocking MR from public.gitlab_merge_requests for
display; MR identity and URL are not part of the replaceable inventory. MR
creation time is not stored in that table today, so the Stuck MRs view must
obtain it from GitLab or the MR-owning feed.
current means the packaged spec version is equal to or ahead of the latest
eligible upstream version. behind means an eligible upstream version is newer.
unmonitored means there is no verified monitor project mapping. unknown
means a mapped project exists, but its update target, release history, spec
version, or comparison is unusable.
rpm_sync_state: One row holding the last successful collection time, the rpms
checkout SHA, and a generation number. A successful full inventory replaces
current tracker rows and increments the generation in the same database
transaction. A failed run leaves the previous rows and generation intact.
| Column | Meaning |
|---|---|
id |
Primary key fixed to 1 for the singleton row |
generation |
Integer incremented after each successful ingest |
rpms_checkout_sha |
Default branch commit inspected by the collector |
last_successful_at |
Completion time of the last successful collection |
rpm_version_rollouts: Resolved rollout times, one row per
(package, version). A later tracker-owned reconciliation step can populate
them from the status-owned public.gitlab_merge_requests table. Keeping these
rows separate from the replaceable inventory preserves historical rollouts;
the dashboard reads them but does not write them.
| Column | Meaning |
|---|---|
package |
Package name (composite primary key) |
version |
The version that was rolled out (composite primary key) |
rolled_out_at |
Actual merge time of the matched MR (v1 proxy for publication) |
mr_iid |
IID of the matched MR |
mr_project |
Project of the matched MR |
resolved_at |
When this entry was resolved |
Match merged update MRs in public.gitlab_merge_requests by rpms project,
package, target version, and target branch. The update job uses a source
branch containing the package name and an MR title containing both package and
version. A branch match alone does not identify a version. Use GitLab’s actual
merged_at from the webhook payload when present. Obtain it from GitLab for
older MRs that lack it. Do not substitute the mutable MR updated_at. If the
version or merge time cannot be established, or multiple MRs still match, leave
rolled_out_at null and show rollout lag as unavailable. Manual bumps without
an identifiable package and version remain unresolved rather than silently
matching a different MR. Re-resolve unresolved rows as MR data arrives.
Collection and reconciliation
Every four hours, collect a fresh inventory from the rpms checkout and the
monitor’s existing REST API. GET /v1/projects lists projects.
GET /v1/projects/{projectID}/definition supplies the source configuration
used to verify repository mappings. For each distinct mapped project,
GET /v1/projects/{projectID}/status reports scan health,
GET /v1/projects/{projectID}/streams provides lifecycle, and
GET /v1/projects/{projectID}/releases provides dated release history with
X-Next-Cursor pagination. Fetch every release page and
GET /v1/projects/{projectID} for snapshot releases and observations, including
undated versions omitted from pagination. Merge by source and release ID, with
snapshot records taking precedence, including dates corrected to unknown. Treat
the monitor’s JSON null empty-stream representation as an empty list. Group
releases by project and reuse the results for packages sharing a project. Choose
the latest eligible version with RPM comparison rather than relying on API page
order. The combined releases give the first newer upstream release date; an
eligible newer release with no date makes staleness unavailable. This full read
also sees changed dates, stream lifecycle changes, and
new project mappings without an event cursor or a second database of historical
releases. The monitor’s historical catalog preserves observed releases after
they disappear upstream; a full read alone cannot identify withdrawals. Excluding
withdrawn releases requires an authoritative monitor eligibility signal.
The collector is implemented in rpm-tracker/rpm_tracker/collect.py, with monitor
reads in monitor.py and package selection in policy.py. It reads the expected
generation before cloning the default branch, reuses the checkout’s spec parser
and named transforms, and writes only through store.ingest. See
RPM Tracker for runtime
prerequisites and usage. Deployment
as an OpenShift CronJob remains a subsequent step.
The monitor’s per-project endpoints do not promise one snapshot across all
pages or projects. If any required request or page fails, do not replace the
last successful inventory. If releases change during collection,
the next full run reconciles them. Do not claim that each four-hour run is an
atomic view of all upstream projects. Expose the monitor’s project scan state
and last_success_at alongside the collector’s own freshness so a successful
API read does not conceal failed upstream polling. Prevent overlapping scheduled
runs and check the expected rpm_sync_state generation on ingest, including
runs with no release changes.
Metrics
- Currency (behind / current): compare the spec version against the latest
eligible upstream version with RPM semantics.
behindmeans upstream is newer. An equal or ahead packaged version iscurrent. If versions cannot be compared or the update target cannot be identified, useunknownrather than assumingcurrent. These are source repository statuses, not proof of publication. - Staleness (behind packages): Time since
first_behind_at, the earliest monitorreleased_atamong eligible releases newer than the spec version. A newer upstream release does not reset the clock. This is based on the monitor’s reported dates and does not measure security exposure. If history or its date is missing, show unavailable, not zero. - Rollout lag (current packages whose version equals
upstream_version): Time betweenupstream_released_atandrolled_out_at. An ahead packaged version is stillcurrent, but its lag is unavailable. In v1,rolled_out_atis the matched update MR’s merge time, not proof of successful publication. Historical rollout rows remain stored, but the v1 Rollout view stops showing their lag when a newer upstream release makes the packagebehindagain. - Lifecycle: flag an unsupported upstream stream as support risk. Whether
to fix or close CVEs follows the downstream
fix_statuspolicy in package metadata, not this upstream flag. - Stuck update MRs: all open update MRs older than 24 hours by GitLab MR creation time, independently of the single MR shown on a package row.
Approach
- Deploy the collector as a Kubernetes CronJob on OpenShift. Its
batch/v1schedule starts a Job every four hours, independently ofupstream_version_update. SetconcurrencyPolicy: Forbidto prevent overlapping scheduled runs. Run the tracker schema initializer before collection starts. Clone the rpms repo’s default branch read-only at the start of each run and record the checked commit SHA. - Read
metadata/*.jsonfrom that checkout and parse each package’s spec version with the existingparse_spec_versionhelper. If a spec cannot be parsed, mark that packageunknownrather than trusting a possibly stale metadata version. Map packages to projects using verified monitor definitions, fetch all needed project status, streams, and release history, and compute upstream state andfirst_behind_atwithrpm.labelCompare(provided by the collector image). Create a SQLAlchemy engine and read the generation withstore.get_generation(engine)before cloning or collecting external data. - Pass the complete package inventory, checked checkout SHA, and expected
generation to
store.ingest(engine, request). It rejects an outdated generation withGenerationConflictError, atomically replaces upstream and status rows, and increments the generation. Packages removed from a successfully read default branch checkout disappear. An incomplete fetch must not call ingest; a failed database write leaves the previous inventory unchanged. A conflict fails the Job visibly, and the next run collects a fresh inventory. Grant the CronJob database access limited to tracker-owned tables; use a separate migration role for schema changes. The dashboard remains a read-only consumer of tracker data. - A later tracker-owned reconciliation step resolves
rolled_out_atfor a package/version from matching merged update MRs. The dashboard may show one open update MR on a package row, but the Stuck MRs view queries all open rpms update MRs independently. Obtain MR creation time from the MR-owning feed or GitLab when missing; MRs without a known creation time appear asunknown age, not silently filtered out. - Render an RPM Tracker tab with a Rollout view (version, status, staleness
or rollout lag, lifecycle, blocking MR) and a Stuck MRs view, mirroring
the CVE Tracker, plus JSON API endpoints. Show
checked_at, the rpms checkout SHA, and the last successful collection time. Warn when collection exceeds eight hours without success. Show monitor scan failures and each project’s last successful upstream check separately.unmonitoredmeans no verified monitor project.unknownmeans a mapped project without a usable update target, release, or comparison. On collection failure, retain the last successful state and show its age. - Keep the existing Slack and failure-file output for update automation. Monitor failed or missed collector Jobs in OpenShift and show the last successful collection time in the dashboard.
MR reconciliation refinements
- Add
rpm_update_mrsin the tracker schema, keyed by(project, iid), as the dashboard’s MR read model. Reconciliation reads the Status table and stores package/version identity, MR state and links, explicit creation/merge times, and uncertainty reasons. Preserve merged identity evidence across later edits and retain known timestamps when later payloads omit them. Only currently qualifying MRs are rollout candidates. - Extend
rpm_sync_statewithmr_project,mr_target_branch, andlast_mr_reconciled_at. Fix project/branch scope on the first successful reconciliation and track MR freshness independently of inventory generation and collection freshness. Commit MR and rollout changes under the sync-row lock without changing the inventory generation. - Add
unavailable_reasontorpm_version_rolloutsand defineresolved_atas the time the resolution result last changed. Revisit historical keys after inventory advances and invalidate results when new ambiguity appears. - The initial reconciliation step uses only explicit timestamps in stored Status webhook payloads. The planned GitLab backfill remains follow-up work; missing timestamps stay unavailable until reliable evidence is supplied.
- Expose the independent open-MR listing through
GET /api/rpm-update-mrs, including itsstuck_onlyfilter, and include MR information inGET /api/rpm-upstream-state. Select the single package-row MR from the read model without restricting the independent listing.
Dependencies
- Package registration and mapping. Count actual package coverage by
verifying metadata against monitor definitions. Provision missing projects
separately.
POST /v1/projectsrequires a project ID, name, homepage, source provider configuration, version policy, and stream policy. Anupstream_repoURL alone is not sufficient to register a project. A package without a verified project mapping remainsunmonitored. - Monitor load. The existing API provides paginated release history, project streams, and project scan status. Confirm that a complete collection of mapped projects fits within the four-hour schedule at a bounded request rate. No new monitor endpoint or cursor protocol is required for v1.
- Collector runtime. The CronJob image needs Git,
rpmspec, the Python RPM bindings, and the dependencies needed to use the version parsing helper from the rpms repo. Install the tracker package with its bundled migrations, run the initializer before the collector, and give the CronJob schema-scoped database credentials. Provide a read-only GitLab credential if the repo requires one. - MR timestamps. MR reconciliation needs actual
merged_atand creation times from the MR-owning feed or GitLab for missing historical data. It cannot infer these times from mutableupdated_at.
Acceptance Criteria
- The Rollout view lists every package with its status. Behind packages show time since the first newer upstream release and the blocking update MR. Current packages show merge-time rollout lag only when their version equals the selected upstream version and has a matching MR. Unsupported upstream streams are flagged without changing CVE policy.
- Packages not registered with the monitor are shown as
unmonitored, not omitted or assumed current. Ambiguous stream mappings areunknown. A packaged version equal to or ahead of the latest eligible upstream version iscurrent. A package pinned withtrack_upstreamignores releases outside its pinned line. A package trackinglatestconsiders eligible releases in all streams while lifecycle reflects the packaged stream. - The Stuck MRs view lists every open update MR older than 24 hours with links and creation-based ages, including older MRs hidden by a newer MR on the same package. MRs with unknown creation time remain visible as such.
- The collector runs as a separate CronJob on OpenShift every four hours. It
recomputes every package from the current specs and complete upstream
history without slowing
upstream_version_update. Failed and missed runs are visible through OpenShift and dashboard freshness. - A release arriving during a run appears in that or the next complete run. Failed pagination or ingest retains the last successful inventory. Later runs pick up corrected release dates, lifecycle changes, and newly registered projects without replaying a release feed. Withdrawals require a monitor eligibility signal rather than disappearance from upstream. An outdated ingest generation is rejected even when upstream releases did not change.
- A newer upstream release does not reset staleness. An MR for a different version or with no reliable merge time does not acquire a rollout timestamp.
- Failed collection retains the last successful data with visible age. Monitor scan failures remain visible even after a successful collector run. Removed packages disappear only after a complete inventory run.
- Existing dashboard behavior, the update loop, and existing reporting are unchanged.
Phase 2
Future work, contingent on the independent pipeline-timing service from !1261
providing tested lineage and sufficient data in its pipeline_timing schema.
The joins needed by this tracker remain to be verified. !1261 preserves the
existing public tables and Dashboard reads. There is no planned removal of
public.gitlab_merge_requests that requires this phase.
What changes
Rollout timestamp: Replace the v1 merge-time proxy with completion time of
the first successful publication for the matched package and version. The
future model must support an exact component match, MR merge commit → build
PipelineRun → Snapshot → Release lineage, and identification of the intended
ReleasePlan/destination. A SHA alone or an arbitrary single run is insufficient.
If several builds or releases qualify, choose the earliest successful completion
for that destination. Failed releases and releases to other destinations must
not count. If the lineage or destination is ambiguous, leave rolled_out_at
null. Retry unresolved rows as new timeline events arrive. A version shipped
by a later repair commit needs a documented version-to-release association
rather than being attributed to the original MR by SHA alone. Use
pipeline_timing completion times only where the lineage is verified.
Backfill existing rollout rows and compare results with the v1 merge-time proxy before switching the timestamp. Retain the matched run and Release identifiers for traceability. Settle their column names against the implemented schema.
What stays the same
The collector CronJob, upstream mirror, and public.gitlab_merge_requests as
the source for versioned MR matching and the Stuck MRs view. The bot-convention
matching, rpm_version_status columns, and UI also stay the same. Phase 2 is a
rollout timestamp upgrade with no planned UI restructure.
4.5.31 - Pipeline Timing
Independent SQS worker for RPM and container execution timing and cross-system lineage.
It shares the Status events database but owns only the pipeline_timing schema.
Features
- Current state, timestamps, and separate attempts for GitLab jobs and Konflux PipelineRuns
- Commit, merge request, PipelineRun, TaskRun, Snapshot, and Release lineage
- Source-derived execution creation, start, and completion times
- Alembic-managed schema, dedicated queue, and worker metrics
Prerequisites
- Python 3.11+ and PostgreSQL 16
- A dedicated timing SQS queue subscribed to the environment’s SNS topic with
RawMessageDelivery: false
The existing Status worker and event forwarders keep their own images and queues.
Infrastructure provisions the timeline queues, IAM, and alerts separately from
the Status worker.
Each environment has a separate queue named
arr-hummingbird-<env>-pipeline-timing-sqs and a matching -sqs-dlq queue.
The source queue retains messages for 4 days; its DLQ retains them for 14 days
from the original enqueue time when SQS automatically moves a message to the
DLQ. Operator redrive from the DLQ resets the enqueue timestamp. The source queue
moves a message to its DLQ after 3 receives and uses a 300-second visibility
timeout, matching Status.
The SNS subscription preserves the SNS envelope (RawMessageDelivery: false)
and does not filter events. See Hummingbird Events Topic
for queue policy and redrive details.
Provision both environments’ queues and SNS subscriptions before deploying the
workers. Messages collected before a worker starts may expire after four days;
this pre-launch loss is accepted. A queue without a consumer does not move
messages to its DLQ because no receives occur. Validate the staging worker
before deploying the production worker or enabling new production event sources.
Installation
cd pipeline-timing
pip install -e ".[dev]"
The pipeline-timing Containerfile builds the separate
quay.io/hummingbird-ci/pipeline-timing image. Both the initializer and worker
run from this image.
Usage
Schema initialization
Run the initializer as the init container of the single-replica timing worker
Deployment. Use a Recreate strategy so Kubernetes stops the old worker before
starting a Pod whose init container may upgrade the schema:
DATABASE_URL="$DATABASE_URL" python3 -m pipeline_timing.worker.init_db
The initializer creates the pipeline_timing schema and runs
alembic upgrade head each time a Pod starts. This is a no-op at the expected
revision; a new image can apply a forward migration during rollout. If migration
fails, the initializer exits nonzero and Kubernetes does not start the worker.
The worker checks the revision again before polling. Fix migration errors with
a forward revision; do not downgrade or drop the schema during rollback.
Staging deployment from a merge request
Wait for the tools MR’s tests and Konflux image build to pass, then run the
manual deploy_pipeline_timing_staging job. The job does not wait for those
checks automatically. It triggers the infrastructure pipeline with
ONLY_JOB_NAME=pipeline-timing and ONLY_DEPLOYMENT_ENVIRONMENT=staging, using
the MR commit’s image for both the migration init container and worker.
Production is not deployed by this job.
Verify the init container completed, the worker accepts the new Alembic revision, and its queue is catching up before promoting the published image to production. Normal infrastructure pipelines use a shared image pin for both environments. Coordinate those deployments during MR validation so an older pinned image does not replace the MR image. The worker requires an exact Alembic revision match; an older image will refuse a newer revision even when the table layout is unchanged.
Execution-only schema and catalogue removal
Under HUM-10942, revision 003_execution_records added nullable created_at to
TaskRuns and PipelineRuns, stopped catalogue writes, and preserved old recovery
pointers temporarily. Historical creation-timestamp recovery completed in staging
and production using a one-time local tool. The tool was intentionally not retained.
Revision 004_drop_event_catalogue removes the legacy latest_event_id foreign
keys and columns from the reporting tables, then drops pipeline_timing.events
and its remaining indexes. It does not change reporting rows, their lifecycle
fields, parentage, or ordering metadata. The database named events and legacy
Status tables in public remain unchanged. Original migrations remain intact so
fresh initialization follows the same revision chain.
Before deployment, confirm recovery is complete or its gaps accepted in both
environments, no external consumer needs the catalogue, and the normal backup is
available. Record the catalogue allocation and PVC free space, then verify the
space recovery and continued ingestion in staging before production promotion.
The table/index allocation is released when the migration commits without a bulk
delete, table rewrite, or VACUUM FULL. Dropped pointer columns may retain small
amounts of physical row space; that does not justify a rewrite.
The migration limits lock waits to five seconds. A conflicting reader or unexpected dependency makes initialization fail and the transactional DDL roll back. Resolve contention or the dependency and retry with the compatible image. Downgrade is explicitly unsupported: recreating empty history would not restore the deleted catalogue or pointers. Use a compatible forward fix or the backup recovery procedure.
Kubernetes creation time comes from metadata.creationTimestamp; start and
completion come from status.startTime and status.completionTime. Missing
source timestamps remain null, not inferred from SNS publication time. Pending
conditions are distinct from running, and failure reasons retain cancellation
details. GitLab jobs and pipelines retain their source lifecycle fields and status.
Only execution state and lineage are maintained. There is no event-history array, per-step timing, new archive-location table, or permanent S3 pointer on each row. Resource deletion preserves the execution’s result, timings, and parentage.
Timing worker
Deploy one worker replica with the environment’s dedicated queue. Kubernetes starts it only after the init container succeeds:
SQS_QUEUE_URL="$TIMING_QUEUE_URL" python3 -m pipeline_timing.worker
The worker polls when it starts. Accepted notifications update reporting rows in
one committed transaction before SQS acknowledgement, without event-table writes. Malformed
events remain for retry and eventual dead-letter-queue redrive; database or
revision failures stop polling without affecting legacy Status. Scale the timing
worker to zero replicas to pause ingestion. After launch, monitor queue age
against the four-day source retention limit during a pause to avoid losing
events. SQS preserves the original enqueue time for automatic DLQ transfer, and
an operator redrive resets it. A new rollout uses Recreate: the old worker
stops, Alembic upgrades, and the new worker starts. The S3 archiver receives its
own SNS copy and remains independent of timing ingestion.
The worker follows Status’s polling pattern: a 20-second long poll receives up to 10 messages, processed sequentially with boto3’s default request retries. Each accepted event commits in its own transaction. At the end of each receive, one SQS batch-delete request acknowledges committed events and ignored messages. Short receives flush immediately; failed processing never adds a receipt to the delete request. Receive errors are logged and polling continues. Partial delete failures and uncertain request outcomes leave committed events available for idempotent redelivery, and polling continues. If a database failure interrupts a receive, the worker first attempts to acknowledge earlier confirmed commits. Database connection or schema failures stop the worker for Kubernetes to restart it. SQS visibility controls when an unacknowledged message becomes available again.
The worker caches up to 128 UPSERT statement shapes by table and ordered columns. Each execution binds fresh event values; the cache does not retain payloads, connections, or cursors. Source update timestamps order messages when both the incoming and existing row have them; SNS publication timestamps break ties or provide the fallback. Kubernetes condition transition times supply source ordering. Late pending/running snapshots cannot regress a completed Kubernetes execution. Sparse payloads preserve existing values, and creation time stays immutable once known.
Queue age and DLQ depth are monitored independently from the Status queue. Fix malformed messages before using the documented redrive procedure. If the worker is paused or not yet deployed, remove only a known test message after confirming that redrive returned it to the timing queue.
Configuration
| Variable | Default | Description |
|---|---|---|
DATABASE_URL |
- | Existing Status events database URL |
SQS_QUEUE_URL |
- | Environment-specific timing queue URL |
SQS_WAIT_TIME |
20 |
SQS long-poll interval in seconds |
METRICS_PORT |
9090 |
Prometheus port when the worker is enabled |
Infrastructure supplies queue values for each environment:
| Environment | Queue URL variable |
|---|---|
| Staging | ARR_HUMMINGBIRD_STAGING_PIPELINE_TIMING_SQS_QUEUE_URL |
| Production | ARR_HUMMINGBIRD_PROD_PIPELINE_TIMING_SQS_QUEUE_URL |
S3_BUCKET is no longer required or used by the worker. Remove the old deployment
variable only after the compatible worker is deployed in both environments.
The independent S3 archiver and its configuration remain unchanged.
POSTGRES_URL is also accepted when DATABASE_URL is not set. For local
development only, the database URL defaults to
postgresql://postgres:dev@localhost:5432/events.
Schema and lineage
| Table | Identity | Purpose |
|---|---|---|
merge_requests |
GitLab project ID and MR IID | Current state and lifecycle times |
commits |
GitLab project ID and SHA | Commit metadata |
merge_request_commits |
Project ID, IID, SHA, relation | Head and merge-commit links |
gitlab_pipelines |
Project ID and pipeline ID | Pipeline status and timing |
gitlab_jobs |
Project ID and job ID | Individual job attempts |
pipeline_runs |
Cluster and Kubernetes UID | Build, test, and Release attempts |
task_runs |
Cluster and Kubernetes UID | Task timing and parent UID |
snapshots |
Cluster and Kubernetes UID | Build-to-test bridge |
releases |
Cluster and Kubernetes UID | Snapshot-to-Release bridge |
Source timestamps accept ISO 8601 and GitLab’s YYYY-MM-DD HH:MM:SS UTC format
and are normalized to UTC. Timestamps without a timezone are interpreted as UTC.
latest_observed_at and source_updated_at order updates without a notification
catalogue or event foreign keys. The schema stores selected reporting fields,
not raw event JSON. Join MR head and merge
commits to GitLab pipelines by canonical HTTPS repository URL and SHA; join
jobs by project and pipeline ID. Konflux PipelineRuns use the same repository
URL and SHA, then fan out by component and execution UID. TaskRuns link through
PipelineRun owner UIDs; Snapshot source-run names, test-run Snapshot labels,
and Release references connect the remaining phases. The Release completion
time closes the current timeline. Distinct job IDs and PipelineRun UIDs retain
each retry’s status and timing.
PipelineRun SHAs come from supported labels and annotations, including
pac.test.appstudio.openshift.io/sha, before falling back to the revision
parameter.
Promotion Snapshots populate source_pipeline_run_name from
hummingbird-project.io/promotion-pipelinerun. This annotation takes precedence
over the usual pac.test.appstudio.openshift.io/log-url, so the parent is the
promotion run rather than the original MR build. Other Snapshots keep the
log-URL behavior. This connects the promotion run to its Snapshot and Releases
without changing the schema. References to the original MR build and source
Snapshot in promotion annotations are not stored separately.
Existing promotion Snapshots need a separate backfill if no newer notification arrives after this extraction change. Deploying the worker does not rescan stored Snapshots or automatically repair historical links.
Finding archived notifications
The independent S3 archiver still receives its own SNS copy. The timing database does not store a direct archive pointer or a complete notification history.
For Kubernetes executions, use the environment’s archive bucket and
latest_observed_at as a UTC SNS timestamp with milliseconds to narrow the object
prefix, for example sns/2026/01/02/03/04/2026-01-02T03:04:05.678Z#. Match source,
kind, cluster, namespace, and resource name in candidate filenames, then verify the
UID in the downloaded payload. For GitLab, verify the project and execution/commit
identity in candidate payloads.
This is candidate discovery, not a guaranteed unique or per-field provenance link. Several notifications may share a timestamp, and sparse updates can retain facts from earlier payloads. Lookup depends on archive delivery, naming, and retention; it does not recreate the removed database catalogue.
Metrics include hummingbird_pipeline_timing_records_total{kind,result},
hummingbird_pipeline_timing_failures_total{reason},
hummingbird_pipeline_timing_missing_lineage_total{kind}, and
hummingbird_pipeline_timing_worker_stopped.
success and ignored record results count only confirmed SQS acknowledgements,
including duplicates. Each failed or uncertain batch entry increments the sqs
failure counter. These counters do not measure unique event rows. Monitor queue age and
dead-letter-queue depth separately from the legacy Status queue. Fix malformed
events before redriving them. Identity-based upserts and ordering guards keep
redelivery idempotent; there is no database row for every SNS message.
Development
Run tests only against the disposable PostgreSQL service provided in CI or a
disposable local database. Tests truncate timing data and exercise schema upgrades
and historical downgrades; never use a staging or production database URL.
Migration tests create uniquely named disposable databases and need CREATEDB on
that test server. Older-revision tests stop at revision 003; they do not downgrade
through irreversible catalogue removal:
DATABASE_URL=postgresql://postgres:dev@localhost:5432/events \
python3 -m unittest discover -s tests -t .
Tests also load shared publisher payload fixtures from
sns-s3-archiver/tests/fixtures/archive-key-contract.json; timing ingestion no
longer calculates or persists archive keys. CI provides
DATABASE_URL for the PostgreSQL integration tests. The timing image has a
Kubernetes import smoke test in tests-k8s.yml.
License
This project is licensed under the GNU General Public License v3.0 or later. See the LICENSE.
4.5.31.1 - RPM pipeline timing design
Problem
We lack a clear way to see where time is spent in the RPM build pipeline.
Background
References: HUM-5819 - Create a diagram that describes all phases and steps of various pipelines from start to finish. Then review with team. HUM-5820 - Time breakdown from MR creation to completion of Release pipeline.
During the F44 rebuild, RPM builds had problems whose causes were not immediately clear. The sprint retrospective and a later F44-specific retrospective created HUM-5819 and HUM-5820 to improve visibility into pipeline behavior and time spent in each phase.
An initial pass through HUM-5819 identified key areas for time tracking. HUM-5820 records those areas.
Findings
The team initially considered a small addition to the tools/hummingbird-status database. A draft
MR explored that approach.
Findings from the draft MR:
- The existing Status events database is a suitable home because the system forwards GitLab and Konflux events through SNS, archives them in S3, and updates PostgreSQL. The timing worker runs as an independent service with its own image and deployment lifecycle.
- Some
hummingbird-dashboardfunctions do directly query therawJSON. - The original purpose of
hummingbird-statuswas to diagnose Konflux event issues, not long-term system diagnosis. - The
rawevent JSON for Konflux PipelineRuns consumes significant space, about 97% of the 47 GB database, despite redundant event storage in S3. hummingbird-statusPVC monitoring is now in place; the draft MR highlighted its absence.- The solution to our problem statement is not to mimic the existing approach.
Proposal
Track required event data in new tables in the existing database and update event processing to populate them without changing existing tables or Dashboard behavior.
Events Already Agreed Upon for Tracking
HUM-5820 outlines the events already agreed upon for tracking.
- GitLab MRs and commits
- GitLab pipelines
- GitLab pipeline jobs
- Konflux PipelineRuns
- Konflux Tekton TaskRuns
The model must support reporting of:
- Status
- Relevant start and end times, including MR creation and merge times
- Event timestamps
- Event lineage across GitLab and Konflux, sufficient to reconstruct a single pipeline timeline
This is not a database design. It identifies the reporting requirements that detailed design must meet.
Approach
- Preserve existing event ingestion, database tables, and Dashboard behavior. Run timing ingestion
in a separate
pipeline-timingservice and image while retaining the existing publishers, archiver, and events database. - Create a new schema in the existing PostgreSQL database.
- The schema provides logical isolation so existing consumers do not accidentally use the new tables.
- Create tables that hold the data needed to meet the agreed reporting requirements without duplicating raw event JSON.
- Treat the S3 archive as the authoritative raw-event record for the new schema.
- Store a reference that lets consumers retrieve the archived event for detail, display, or later data-model improvements.
- Configure GitLab webhooks and the Kubernetes event forwarder to include the agreed event types, then route them through the existing SNS event pipeline.
- Ensure the S3 archiver can archive and identify every agreed event type.
Acceptance Criteria
- Existing Status database tables and Dashboard behavior remain unchanged.
- The new PostgreSQL schema provides the agreed event coverage.
- Representative data demonstrates an RPM pipeline timeline from MR creation through completion of the Release pipeline.
- Consumers can calculate time spent between tracked pipeline phases without querying raw event JSON.
- Consumers can trace lineage across all agreed event types and retrieve the corresponding authoritative S3 event when needed.
- Tests verify the new data model, lineage, and S3 references.
Out of Scope
- Creating a UI for visualization is out of scope.
- The data may later support a new application, Dashboard changes, or Grafana. UX should be involved in that decision.
- Tracking the containers repository pipeline is out of scope.
- That pipeline builds OCI images from RPMs produced by the RPM pipeline and is a potential follow-up effort.
4.5.31.2 - Pipeline timing implementation plan
Purpose
The original RPM pipeline timing design records the HUM-5820 design and remains unchanged. This implementation plan records the HUM-10942 revisions to retain execution timing and lineage for both RPM and container pipelines, not a permanent notification catalogue or permanent S3 references on reporting rows. Its execution-only requirements supersede the original design’s archive-reference and RPM-only scope decisions.
The Status events database remains the storage location, but pipeline-timing is a separate
project, image, and worker with its own deployment lifecycle and resource quotas. Hummingbird Status
continues legacy ingestion; the GitLab and Kubernetes event forwarders remain independent
publishers. This service split follows HUM-7529 MR
1261 review without changing the
event contracts or lineage model.
Prior Exploration
Closed tools MR 1165 and closed infrastructure MR 1381 informed this plan. They are not implementation vehicles for the Stories above. New focused merge requests should reference the Story that owns their work.
Current execution-only transition
The original HUM-5820 delivery stories are complete. HUM-10942 owns the revised execution-only requirements and supersedes the old archive-reference and RPM-only scope decisions. Existing Status tables, publishers, the archiver, and queue contracts remain unchanged.
- Release one removes the unused composite S3 lookup index.
- Release two, revision
003_execution_records, adds nullable creation times to TaskRuns and PipelineRuns and makes the reporting tables’latest_event_idnullable. The worker stops inserting events and never changes old pointers. New rows use null pointers; the old foreign keys remain valid temporarily. - Operator-run local recovery filled historical creation timestamps through those frozen pointers without replaying state. Both environments were verified with zero missing TaskRun/PipelineRun creation times or creation-time ordering anomalies. The one-time tooling was intentionally removed rather than retained in the repo.
- Release three, revision
004_drop_event_catalogue, explicitly removes the legacy foreign keys andlatest_event_idcolumns, then dropseventsand its remaining indexes. All reporting values and ordering metadata are preserved.
Catalogue removal is transactional, uses five-second lock waits, and has no
supported downgrade. Confirm recovery/backup readiness and no remaining external
consumer, validate staging ingestion and physical space recovery, then use the
existing image promotion path for production. Do not recreate empty history as a
rollback. The events database and legacy public schema are not removed.
The worker does not require S3_BUCKET or calculate archive keys. Remove unused
infrastructure bucket configuration only after the compatible app is deployed in
both environments. Do not change the worker’s IAM permissions for local recovery.
Before each promotion, verify migration revision, execution timing/lineage, commit-before-acknowledgement, queue catch-up, and legacy isolation. During MR validation, prevent ordinary infrastructure pipelines from restoring the older shared image pin. Merge can lead to automatic Konflux publication, Renovate image pin updates, and deployment to both environments; it is not a recovery hold.
Test fresh initialization and populated-schema upgrades, preserving all reporting values and relation files while removing only the catalogue and legacy references. Keep revision-001/002/003 regression tests in isolated disposable databases that stop before irreversible removal. Verify a forced cleanup failure rolls back all constraint/column drops, and an attempted downgrade cannot fabricate history. Continue testing lifecycle times, states, lineage, redelivery, and ingestion with the catalogue absent. No change to the worker’s ingestion logic is required.
Implementation Plan
Scope Decisions
- The timeline tracks through completion of the Konflux
Releaseresource.- The Release completion timestamp is the final tracked milestone.
- PipelineRun and TaskRun detail in
rhtap-releng-tenantrequires separate provisioning and is required before HUM-5820 is complete. - PipelineRuns and TaskRuns in
hummingbird-tenantremain in scope.
SnapshotandReleaseevents are required as lineage bridges even though they were omitted from the initial agreed-event list.- A Snapshot links a build PipelineRun to test PipelineRuns.
- A Release links a Snapshot to Release pipeline completion and, where visible, its PipelineRun.
- The new schema stores only their reporting and lineage fields, not raw JSON.
- The new model stores the latest known state of each execution and its relevant timestamps. It
does not duplicate every status transition as a domain row.
- A retry is a new PipelineRun and has its own UID, row, status, start time, and completion time. It does not replace the failed PipelineRun.
- For example, a PipelineRun that fails after four hours and a retry that succeeds in ten minutes remain two separate records. A report must not show only the ten-minute retry as the time for both attempts.
- Both RPM and container pipeline executions are in scope. Retain the existing source mix rather than introducing repository filters during this transition.
- Full event replay and per-step timing are out of scope. Historical recovery is limited to best-effort creation timestamps and runs outside startup migrations.
- Each environment needs a dedicated timing SQS queue, dead-letter queue, and
pipeline-timingworker deployment using its own image. The existing Status queue, image, and worker continue legacy ingestion unchanged.
PostgreSQL Schema
Create a pipeline_timing schema in the existing Status events database. Do not add columns,
indexes, constraints, or triggers to existing tables in the public schema. The Dashboard continues
to query only the existing tables.
Domain tables contain selected scalar fields, source timestamps, lineage, and
ordering metadata, not a raw payload, history array, or catalogue foreign key.
| Table | Identity columns | Purpose |
|---|---|---|
pipeline_timing.merge_requests |
gitlab_project_id, iid |
Current MR state and lifecycle timestamps |
pipeline_timing.commits |
gitlab_project_id, sha |
Commit identity and available commit metadata |
pipeline_timing.merge_request_commits |
gitlab_project_id, iid, sha, relationship |
MR head and merge-commit lineage |
pipeline_timing.gitlab_pipelines |
gitlab_project_id, pipeline_id |
Pipeline status, source, ref, SHA, and execution timestamps |
pipeline_timing.gitlab_jobs |
gitlab_project_id, job_id |
Job attempt status, stage, timing, failure reason, and retry information |
pipeline_timing.pipeline_runs |
cluster, uid |
Konflux PipelineRun state, source identity, timing, and Snapshot lineage |
pipeline_timing.task_runs |
cluster, uid |
Tekton task state, timing, and parent PipelineRun UID |
pipeline_timing.snapshots |
cluster, uid |
Build PipelineRun to Snapshot lineage |
pipeline_timing.releases |
cluster, uid |
Snapshot to Release completion and PipelineRun lineage |
Store both GitLab project ID and a canonical HTTPS repository URL where available. GitLab project ID is the stable GitLab identity. The repository URL is needed because Konflux resources contain source URLs rather than GitLab project IDs.
004_drop_event_catalogue removes the temporary recovery catalogue and pointers
after recovery completion. latest_observed_at and source_updated_at remain on
all reporting tables for ordering, independently of notification-history storage.
The S3 archiver remains an independent subscriber; execution fields require no
archive lookup during normal ingestion or reporting.
Schema Management
Use Alembic (a database schema migration tool already used by
Hummingbird’s workqueue service) to manage only the pipeline_timing schema. Existing tables in the
public schema remain under the Hummingbird Status db.create_schema() path; adopting Alembic for
those tables is separate work.
- Add Alembic configuration and revisions to the
pipeline-timingproject and image. - Add
pipeline_timing.worker.init_dbfor the worker Deployment’s init container. It executesCREATE SCHEMA IF NOT EXISTS pipeline_timingdirectly and commits before invoking Alembic. Do not rely on Alembic to create the schema. Alembic manages every object inside the schema after that bootstrap. - Set Alembic’s
version_table_schema="pipeline_timing"and qualify every migration object with the schema. The version table and all managed objects must remain outsidepublic. - Create the tables, constraints, and required indexes in the initial revision.
- Run
alembic upgrade headunconditionally frompipeline_timing.worker.init_dbin the single-replica Deployment’s init container. UseRecreateso the old worker stops before a new image migrates the schema. Alembic is a no-op when the schema is already at head. - On migration failure, log the exception and expected revision at ERROR level and exit nonzero. Kubernetes must not start the worker container after a failed init container; the separate Hummingbird Status Deployment continues unchanged.
- Have the timing worker independently verify that the database is at the expected Alembic revision before polling its queue. If the revision check fails, log the expected and observed revision at ERROR level and stop the timing worker. The legacy worker continues unchanged.
- Deploy one replica with the timing image and init container once the queue and archiver are ready. To pause polling, scale the Deployment to zero and monitor queue age against the source queue’s 4-day retention. SQS preserves a standard message’s enqueue timestamp when automatically moving it to the DLQ, whose 14-day retention is a maximum total age. Operator redrive from the DLQ resets the enqueue timestamp.
- Do not automatically downgrade or drop the schema during application rollback. Stop the timeline worker and use a forward migration to correct schema problems.
Selected Fields
The detailed schema should include at least the following reporting fields. These are proposed database column names, not source-payload field names.
| Record type | Required columns |
|---|---|
| Merge request | gitlab_project_id, repository_url, iid, state, source_branch, target_branch, head_sha, merge_commit_sha, created_at, updated_at, merged_at, closed_at, web_url |
| Commit | gitlab_project_id, repository_url, sha, title, message, committed_at, web_url |
| MR commit link | gitlab_project_id, repository_url, iid, sha, relationship |
| GitLab pipeline | gitlab_project_id, repository_url, pipeline_id, mr_iid, sha, ref, source, status, created_at, updated_at, started_at, completed_at, web_url |
| GitLab job | gitlab_project_id, repository_url, job_id, pipeline_id, sha, name, stage, status, failure_reason, allow_failure, created_at, started_at, completed_at, duration_seconds, queued_duration_seconds, retry_count, web_url |
| PipelineRun | cluster, namespace, uid, name, application, component, pipeline_type, pipeline_name, trigger_type, repository_url, sha, mr_iid, snapshot, status, reason, created_at, started_at, completed_at |
| TaskRun | cluster, namespace, uid, name, pipeline_run_uid, pipeline_run_name, pipeline_task, task_reference, status, reason, pod_name, created_at, started_at, completed_at |
| Snapshot | cluster, namespace, uid, name, application, component, sha, source_pipeline_run_name, created_at |
| Release | cluster, namespace, uid, name, snapshot, status, pipeline_run_namespace, pipeline_run_name, completed_at |
TaskRun and PipelineRun creation time is metadata.creationTimestamp; start and
completion are the resource’s own status timestamps. Missing values remain null.
Unknown conditions with a pending reason report Pending, not Running.
Cancellation remains distinguishable through the failure reason. GitLab execution
timestamps and status remain source-derived. Non-execution entities retain their
meaningful lifecycle milestones rather than fabricated start/completion fields.
GitLab job retries have distinct job IDs and therefore distinct rows. Store retry metadata when GitLab supplies it, but do not invent a parent job identity. Consumers can group attempts by project, pipeline, stage, and job name.
PipelineRun retries also have distinct rows because each PipelineRun has a distinct Kubernetes UID. Keep the timing and result for every attempt. This plan does not define a retry-group column or a single combined retry duration. Consumers that need an end-to-end duration across retries must keep the duration of each attempt visible and calculate the full elapsed time from the first attempt’s start to the final attempt’s completion.
GitLab Job Hook payloads do not provide a dedicated job URL. Derive web_url without an API request
by appending /-/jobs/{build_id} to project.web_url.
merged_at and closed_at are new nullable TIMESTAMPTZ columns on
pipeline_timing.merge_requests; they are not additions to the existing
public.gitlab_merge_requests table. Use explicit GitLab merge and close timestamps when the
webhook supplies them. Otherwise, derive them from the first event that reports the corresponding
action or state, preferring the source updated_at timestamp and falling back to observed_at.
Preserve the first derived terminal timestamp across later sparse events. This makes MR
creation-to-merge timing available without requiring a GitLab API lookup.
Repository URL Normalization
Normalize repository identity to an HTTPS URL without a trailing slash, .git suffix, or GitLab
resource path. Prefer GitLab’s project.web_url when it is available. Accept HTTPS, SSH, SCP-style
Git, and GitLab tree URLs from Konflux metadata.
These source values:
https://gitlab.com/redhat/hummingbird/rpms.gitgit@gitlab.com:redhat/hummingbird/rpms.githttps://gitlab.com/redhat/hummingbird/rpms/-/tree/main
all normalize to:
https://gitlab.com/redhat/hummingbird/rpms
Use repository_url consistently in the new schema. Do not use a bare project path or
source-specific Git URL as the cross-system identity.
Identity and Update Ordering
- Use
(cluster, uid)as the primary execution identity for Kubernetes resources.- Names are not sufficient because Kubernetes can recreate a resource with the same name.
- Store namespace and name for display and fallback joins.
- Use source IDs for GitLab records and retain
repository_urlfor cross-system joins. - Store the latest accepted SNS timestamp as
latest_observed_aton reporting rows. - Use source update timestamps from GitLab and Kubernetes condition transition times.
- Prevent an older event delivered later by SQS from replacing newer state.
- Compare source timestamps only when both records have them; use SNS publication timestamps as the fallback and tie-breaker, including migration of legacy rows.
- Late non-terminal snapshots must not regress completed Kubernetes executions.
- Preserve source fields absent from sparse updates and keep known creation time immutable. Ordering metadata advances independently of notification history.
- Do not require foreign keys for cross-source lineage that may arrive out of order. Use indexed identity columns and late joins. There are no legacy event foreign keys or pointer columns after catalogue removal.
Required Indexes
The schema must provide indexes for the known lineage and timeline queries. Exact index names and partial predicates are implementation details, but the following column order is required as a baseline:
| Table | Columns | Supports |
|---|---|---|
merge_request_commits |
gitlab_project_id, iid |
commits for one MR |
merge_request_commits |
repository_url, sha |
MR lookup from a commit |
gitlab_pipelines |
repository_url, sha |
pipelines for a commit |
gitlab_jobs |
gitlab_project_id, pipeline_id |
jobs for a pipeline |
pipeline_runs |
repository_url, sha |
Konflux runs for a commit |
pipeline_runs |
cluster, namespace, name |
name-based fallback lineage |
pipeline_runs |
cluster, namespace, snapshot |
test PipelineRuns for a Snapshot |
task_runs |
cluster, pipeline_run_uid |
tasks for a PipelineRun |
snapshots |
cluster, namespace, source_pipeline_run_name |
Snapshot produced by a build PipelineRun |
releases |
cluster, namespace, snapshot |
Releases for a Snapshot |
Primary keys provide the identity indexes listed in the schema table and do not need duplicate
secondary indexes. Validate the final indexes with the representative acceptance query and
PostgreSQL EXPLAIN before source enablement.
The initial design required a reverse archive lookup index on s3_bucket, s3_key.
The staging storage investigation on October 22, 2026 found that this index occupied
about 8.1 GiB, roughly 40% of the event table’s total allocation including indexes,
with zero recorded scans. No application query using the reverse lookup was found.
The temporary legacy recovery used frozen latest_event_id values to read the
archived creation metadata. Runtime ingestion never needed that reverse index.
Revision 002_drop_events_s3_index removes this unused application index to reclaim
disk space and avoid maintaining it for every new event. It preserves event rows,
columns, and archive references. Scan statistics do not prove that no external
consumer needs the index; confirm that before deployment. Deploy and verify index
space recovery in staging and production before the execution-only transition.
Revision 004_drop_event_catalogue drops the remaining catalogue indexes with the
table. No execution-reporting index is removed or rebuilt.
Lineage Model
Consumers reconstruct a timeline with the following relationships:
- A merge request links to its head commits and merge commit through
merge_request_commits. - A commit links to GitLab pipelines by
repository_urland SHA. A direct pipeline MR IID is used when the webhook supplies it. - GitLab jobs link to their pipeline by project and pipeline ID.
- Konflux PipelineRuns link to a commit by
repository_urland SHA.- Extract the repository from Pipelines as Code annotations first, then the
git-urlparameter. - Extract SHA and MR IID from the current Pipelines as Code and build-service labels and annotations.
- Extract the repository from Pipelines as Code annotations first, then the
- TaskRuns link to PipelineRuns by Kubernetes owner UID. Parent name is a fallback for malformed or older resources without an owner UID.
- A Snapshot links to its source build PipelineRun by name and cluster/namespace context.
- Test PipelineRuns link to a Snapshot through the Snapshot label.
- A Release links to its Snapshot and to any referenced Release PipelineRun.
- In the current scope, the Release completion timestamp closes the timeline even when its managed PipelineRun is in an inaccessible namespace.
The primary cross-system correlation key within one source pipeline is canonical repository_url
plus commit SHA, not SHA alone. One commit can produce several component PipelineRuns, so the
resulting timeline fans out by component and execution rather than assuming one build per commit.
Event Processing and Failure Isolation
Subscribe a dedicated timing queue to the SNS topic already used by the existing Status queue. The
timing queue receives its own SQS copy of each SNS envelope. The existing Status queue and worker
remain unchanged. The pipeline-timing worker parses and upserts only pipeline_timing records.
For each timeline message:
- Start polling only after the Pod’s migration init container succeeds and the worker verifies the expected revision. Scaling the single-replica Deployment to zero leaves the queue untouched.
- Decode the SNS envelope. A valid event outside the accepted timeline event types is ignored and acknowledged.
- For an accepted notification, parse and upsert the reporting records in one transaction. Do not append a notification-history row or require a catalogue link.
- Commit the transaction, then delete the SQS message.
- On malformed data for an accepted event, roll back and leave the message on the queue. SQS retries it and moves it to the timeline dead-letter queue after the configured receive limit.
- On a database connection, schema, or migration failure, roll back, stop polling, and leave the queue intact for recovery. Do not send valid events to the dead-letter queue during an infrastructure outage.
A timeline failure must not delay, retry, or change legacy ingestion. Operators fix a malformed event or parser error, then redrive messages from the timeline dead-letter queue. They fix an infrastructure failure, then restart the worker to resume polling. Configure the source queue with 4-day retention and its DLQ with 14-day retention, a finite receive limit, and a visibility timeout at least twice the worker’s per-message processing budget. Automatic source-to-DLQ transfer preserves a standard message’s original enqueue timestamp; operator redrive from the DLQ resets it. Add queue-age and dead-letter-queue alerts, and document and test the redrive procedure.
Completed recovery and archive inspection
S3 archival remains independent and unchanged. The timing worker neither computes archive keys nor needs bucket configuration or S3 read permissions. The operator-run recovery read immutable creation timestamps from archived snapshots, updated only null creation fields, and finished before catalogue removal. It was not part of a migration, and its tooling is not retained.
latest_observed_at and resource identity can narrow an archive search while
objects are retained. Validate candidate payload identities; this is not guaranteed
unique lookup or per-field provenance because sparse updates can merge older facts.
See finding archived notifications.
No permanent per-execution S3 pointer or replacement history table is added.
Original HUM-5820 delivery and source rollout
The remaining delivery-story sections record the original rollout. Their permanent archive-reference, bucket-configuration, and RPM-only assumptions are historical and superseded by HUM-10942 and the execution-only transition above. The publishers and independent archiver retain their existing event contracts.
Tools Repository: Publisher and Archiver Changes
This section describes code, tests, and documentation in the tools repository. Environment-specific source infrastructure is owned by HUM-7535 and activated under HUM-7533.
GitLab Event Forwarder
The forwarder already publishes arbitrary GitLab webhook bodies and derives event_type from
object_kind. No event-specific forwarding implementation is required.
Add fixtures and tests for:
- Pipeline Hooks with
object_kind: pipeline - Job Hooks with
object_kind: build project_pathandevent_typeSNS message attributes- compressed payload round trips
Kubernetes Event Forwarder
- Add
TaskRunto the configured resources only after the deployed image understands its payload requirements. - Extend the resource configuration schema with optional
labelSelectorandfieldSelectorkeys. - Pass configured selectors to the existing
ResourceWatcherlabel_selectorandfield_selectorparameters. - Remove
metadata.managedFieldsfrom TaskRun messages before SNS publication because it is transport metadata and can dominate payload size. - Keep status, results, owner references, task identity, and timing data needed for extraction and future diagnosis.
- Measure compressed and base64-encoded TaskRun payloads from representative RPM build, test, and tenant Release PipelineRuns.
- Do not enable the source if representative payloads approach the SNS 256 KiB limit. Define and review any additional projection before rollout rather than silently dropping fields.
- Preserve spooled events on publish failure so a transient SNS error does not discard them.
Use a TaskRun label selector for the RPM application if production samples confirm that Tekton propagates a stable application label. If no reliable selector exists, estimate and approve the namespace-wide event rate before enabling the watch because every matching TaskRun will also be archived to S3.
S3 Archiver
The archiver is already generic, but its resource identity logic must change for the new event
types. Extend _extract_resource_name() and _build_s3_key() so GitLab pipeline and job keys
include their numeric pipeline or job ID and Kubernetes keys include logical cluster name,
namespace, and name. URL-valued cluster attributes from older forwarders retain name-only
identities. Cover those identities with the canonical contract fixture.
Continue archiving the complete event as received from SNS. The Kubernetes forwarder’s documented
removal of managedFields occurs before that boundary. Keep explicit test coverage for PipelineRun,
TaskRun, Snapshot, Release, merge request, commit/push, pipeline, and job events.
Pipeline Timing and Hummingbird Status Changes
- Add a
pipeline-timing/project, package, image, and Alembic configuration with an initialpipeline_timingrevision. Generate its independent Konflux build resources. - Add
pipeline_timing/timeline.pyfor extraction, ordering rules, and upserts. - Keep Hummingbird Status extraction, schema creation, upsert functions, dependencies, and image
responsible only for the
publicschema. - Add timing dispatch for GitLab
push,merge_request,pipeline, andbuildevents and KubernetesPipelineRun,TaskRun,Snapshot, andReleaseevents. - Mark the newly handled event types as known to the legacy Status parser so expected timing events do not generate unknown-kind warnings; do not route them to legacy tables.
- Add
pipeline_timing.worker.init_dbto the timing Deployment as an init container. Leavehummingbird_status.worker.init_dband legacyworker/sqs.pyresponsible only forpublic. - Deploy the independent timing image to poll only the timing queue and write only the
pipeline_timingschema in the shared events database. - Add archive bucket configuration to the timing worker deployment.
- Start worker polling only after its init container succeeds; retain the worker’s own startup revision check before queue access.
- Keep Alembic migration and revision-check failures isolated to the timing worker so they cannot prevent legacy worker startup.
- Provide separate
pipeline_timingdatabase counts without changing existing Status statistics queries.
Observability
Add metrics and structured logs for:
- timeline records processed by kind and result
- timeline extraction and database failures
- events missing expected lineage fields
- oversized Kubernetes payloads and SNS publish failures
- timeline queue age and dead-letter-queue depth
- timeline worker stopped because of a migration or revision-check failure
Failure logs must include the SNS message ID, source, kind, resource identity, and S3 location without logging the full raw event. Continue monitoring legacy queue age and DLQ metrics, archiver failures, PostgreSQL availability, and PVC usage.
Documentation Updates
Update the existing documentation for every component changed by implementation. Do not create a separate documentation Story. Each delivery Story owns the pages for the behavior it changes:
| Story | Required documentation updates |
|---|---|
| HUM-7529 | New pipeline-timing.md, legacy isolation in hummingbird-status.md, and the timing schema and lineage sections of data-flow.md |
| HUM-7530 | gitlab-event-forwarder.md, kubernetes-event-forwarder.md, sns-s3-archiver.md, and event-flow portions of data-flow.md |
| HUM-7531 | staging worker deployment, monitoring, and recovery details in pipeline-timing.md |
| HUM-7532 | managed Release namespace access in kubernetes-event-forwarder.md and data-flow.md |
| HUM-7533 | final documentation audit across all affected pages before HUM-5820 closes |
| HUM-7534 | production worker deployment, monitoring, and recovery details in pipeline-timing.md |
| HUM-7535 | both environments’ timeline queues, dead-letter queues, subscriptions, IAM, and alerts in hummingbird-events-topic.md and infrastructure monitoring documentation |
HUM-8015 already updated message-bus.md for the independent staging message bus. HUM-7535
clarifies that timing queue subscriptions can precede worker deployment.
Testing
Unit Tests
Cover:
- every accepted GitLab and Kubernetes event type
- status and timestamp extraction
- explicit and derived MR merge and close timestamps
- repository URL normalization across HTTPS, SSH, SCP-style Git, and tree URLs
- GitLab job URL derivation from
project.web_urlandbuild_id - MR head and merge-commit lineage
- GitLab job retries as distinct job IDs
- a failed PipelineRun followed by a successful retry, with separate statuses and timings for both attempts
- PipelineRun-to-TaskRun owner UID lineage
- Snapshot and Release bridge extraction
- sparse and out-of-order updates
- deterministic archive keys
- missing optional lineage without rejecting otherwise valid records
- TaskRun serialization without
managedFields - init-container migration gating and zero-replica pause behavior
- Alembic upgrade failure that blocks worker startup, and startup revision failure that stops only the timing worker
- timeline acknowledgement only after a committed timeline transaction
- timeline retry and dead-letter-queue handling for parsing, data, and database failures
- an archive key from a timeline queue copy that matches the archiver’s key for the same SNS event
- ignored acknowledgement of valid events outside the accepted timeline event types
- stopped polling on database, schema, or migration failure
PostgreSQL Integration Tests
Use the PostgreSQL service already available in repository CI to:
- Apply the initial Alembic revision to an empty database and to a database with populated legacy tables.
- Verify Alembic reports the expected head revision and that its version table exists in
pipeline_timing, notpublic. - Confirm the migration does not alter existing
publictable definitions or representative legacy queries. - Ingest representative events in and out of order.
- Verify idempotent current-state upserts.
- Verify that a timeline failure leaves its message available for retry without affecting legacy ingestion.
- Query one RPM timeline from MR creation through Release completion and calculate each available interval without raw JSON.
- Verify all domain rows resolve to event records with the expected S3 references.
- Inspect the new schema to confirm it has no raw JSON columns.
- Verify that timeline redrive produces idempotent rows and retains the same S3 reference.
End-to-End Acceptance Test
After source enablement, use a controlled RPM merge request that runs GitLab CI, Konflux build and test PipelineRuns, and a Release. Record the MR and component selected for the test.
The acceptance query must demonstrate:
- MR creation and merge timestamps
- GitLab pipeline and job timing
- derived GitLab job URLs that match
project.web_urlandbuild_id - pre-merge and post-merge commit lineage
- component build PipelineRun and TaskRun timing
- a failed PipelineRun and successful retry shown as separate attempts, without losing the failed attempt’s timing
- Snapshot and test PipelineRun lineage
- Release completion
- calculable durations between phases
- retrievable S3 objects for every row used by the timeline
- paused timeline processing, recovery, and dead-letter-queue redrive without affecting legacy Status
Run make test, make coverage, make check, and SAM template validation for the affected projects.
Infrastructure Implementation
HUM-8015 delivered the independent staging message bus: its own SNS topic, GitLab and Kubernetes publishers, S3 archiver and bucket, and consumer cutover. HUM-7535 adds only the timeline queue, dead-letter queue, subscription, IAM, and alerts in staging and production.
Staging Event Path
Use the staging message bus delivered by HUM-8015 before enabling the new sources in production:
- Verify that the HUM-8015 staging topic, publishers, archiver, bucket, and existing Status
consumer are deployed and that
message-bus.mddescribes them. - Under HUM-7535, subscribe dedicated staging and production timeline queues to their respective
topics with
RawMessageDelivery: false. Configure each source queue with a finite receive limit, a visibility timeout at least twice the worker’s per-message processing budget, its own dead-letter queue, and least-privilege worker credentials. Production queue provisioning does not depend on staging worker acceptance. - Configure 4-day retention for both source queues and 14-day retention for their dead-letter queues, then add queue-age and dead-letter-queue alerts. Messages collected before worker deployment may expire after four days; this pre-launch loss is accepted. Without receives, messages do not automatically move to the DLQ. Verify redrive before source enablement.
- Under HUM-7531, deploy one staging
pipeline-timingreplica withRecreatestrategy, its migration init container, timing queue, resource limits, and explicit stagingS3_BUCKET; do not use the production bucket. Verify the init container reached the expected revision and the worker is polling before enabling new event sources. Once the worker runs, monitor queue age to prevent loss during pauses. - Keep new production event sources disabled until staging acceptance succeeds.
The staging Kubernetes event forwarder may observe production Konflux resources, but it publishes only to the staging topic. Staging and production forwarders must not publish the same watch stream to the same topic.
Production Event Path
- Keep Pipeline Hooks already enabled by the shared Hummingbird webhook configuration.
- Add RPM Job Hooks only after the production Status and archiver images understand Job Hook events.
- Add PipelineRun and TaskRun watches only after compatible forwarder and Status images are deployed.
- Use the production queue, subscription, dead-letter queue, IAM, retention, and alerts provisioned
under HUM-7535. After staging worker acceptance, deploy one
pipeline-timingreplica withRecreatestrategy, a migration init container, resource limits, and an explicit productionS3_BUCKET. Pre-launch queue messages may have expired; no historical backfill is required. Verify the revision and worker before activating new event sources, and monitor queue age after launch. - Verify the production redrive procedure before enabling new sources.
- Enable new production event sources only through HUM-7533 after staging acceptance; the timing worker starts under HUM-7534.
Managed Release Pipeline Access
HUM-7532 owns the access needed for rhtap-releng-tenant and is required for Epic completion.
- Obtain read-only
get,list, andwatchpermissions for PipelineRuns and TaskRuns in the managed namespace. - Add managed-namespace contexts to the staging and production Kubernetes event forwarders.
- Verify each event-forwarder service account with
auth can-ibefore enabling its watches. - Link the managed PipelineRun to the Release record through its namespaced reference.
- Include signing and Pulp publication TaskRuns in the final acceptance timeline.
Infrastructure Verification Gates
- Render and resolve each final RPM webhook configuration before deployment.
- The production endpoint must retain its token and existing push, merge request, note, confidential note, and pipeline events while adding job events.
- If same-URL inheritance does not deep-merge those settings, use an explicit webhook block containing the complete configuration.
- Deploy compatible Hummingbird Status, Pipeline Timing, event-forwarder, and S3 archiver images before enabling Job Hook or TaskRun events in that environment.
- Pass the canonical archive-key contract tests in both the archiver and Pipeline Timing projects.
- Verify representative TaskRun payload size after removing
managedFields. - Use an RPM application label selector if production samples confirm Tekton propagates a stable label. Otherwise, document approval of the namespace-wide event rate before enabling the watch.
- Use a single replica and
Recreatestrategy; verify the migration init container succeeded and the worker checked the expected Alembic revision before it polls. - Verify the source queue has 4-day retention and the dead-letter queue has 14-day retention, with alerts and a tested redrive procedure.
Rollout
- Complete the independent Pipeline Timing image and the legacy Status parser compatibility change in HUM-7529, and event-bus support in HUM-7530.
- Verify the HUM-8015 staging message bus and provision both environments’ timing queues, subscriptions, IAM, and alerts in HUM-7535, independent of staging worker acceptance. Pre-launch messages may expire without reaching the DLQ; this loss is accepted.
- Under HUM-7531, deploy the single-replica staging timing worker with
Recreatestrategy and its migration init container. Verify the new schema and legacy isolation before enabling new sources. - Complete
rhtap-releng-tenantprovisioning in HUM-7532 and add it to the staging event forwarder. - Under HUM-7533, activate staging Job Hook, PipelineRun, and TaskRun sources after validating the running timing worker.
- Run a controlled RPM merge request and verify staging lineage, duration calculations, managed Release tasks, and S3 references.
- Under HUM-7534, deploy the single-replica production timing worker with
Recreatestrategy and its migration init container after staging worker validation. - Verify the production Alembic revision, legacy ingestion, timeline queue alerts, and metrics.
- Add the managed namespace to the production event forwarder, then activate new production sources under HUM-7533 after verifying the timing worker is running.
- Expect each Kubernetes watcher to emit an initial
ADDEDevent for every existing matching resource. Schedule source activation during a monitored window. - Monitor legacy and timeline queue age, dead-letter-queue depth, worker and archiver errors, forwarder spool growth, database size, and PVC free space.
- Run the final controlled RPM acceptance merge request in production before closing HUM-5820.
Rollback does not require dropping or downgrading the schema. Scale the timing worker to zero and disable the new TaskRun and Job Hook sources in the affected environment. Legacy ingestion continues. After launch, monitor queue age while paused: messages expire after four days unless the worker resumes in time. Malformed messages move to the DLQ after failed receives and can be redriven after repair.
Epic Completion Criteria
The approved design’s acceptance criteria remain authoritative. HUM-5820 is complete when:
- HUM-7529 through HUM-7535 are complete.
- The design’s acceptance criteria pass in both staging and production.
- Representative timelines include managed PipelineRun and TaskRun timing from
rhtap-releng-tenant. - The staging-first source rollout and final production acceptance run are documented under HUM-7533.
- HUM-7533 has verified that each Story’s required documentation update is complete.
Out of Scope
- Creating a UI for visualization is out of scope.
- The data may later support a new application, Dashboard changes, or Grafana. UX should be involved in that decision.
- Tracking the containers repository pipeline is out of scope.
- The pipeline-neutral schema, event contracts, and execution identities are intended to support its GitLab and Konflux events in a future follow-up without creating a parallel storage model.
- The RPM and containers pipelines do not connect directly. A separate scheduled process
discovers updated RPMs and commits changes to
rpms.lock.yamlfiles in the containers repository.
- Timeline replay and historical backfill are out of scope.
- Existing Hummingbird Status S3 ingestion bootstraps the
publicschema and collapses Kubernetes history, so it is not suitable for timeline recovery. - Future work must define scoped S3 read permissions and chronological ingestion into the
Alembic-managed
pipeline_timingschema. Events that were never collected cannot be reconstructed.
- Existing Hummingbird Status S3 ingestion bootstraps the
Delivery Stories
| Story | Responsibility |
|---|---|
| HUM-7529 Make RPM pipeline timing data queryable | Independent Pipeline Timing image and schema, Alembic, worker, ingestion, lineage, tests, and legacy Status compatibility |
| HUM-7530 Carry RPM pipeline timing events through the event bus | GitLab and Kubernetes forwarders, S3 archiver event identities, payload handling, and tests |
| HUM-7531 Validate RPM pipeline timeline storage in staging | Staging Pipeline Timing deployment, Alembic verification, legacy isolation, and representative queries |
| HUM-7534 Make RPM pipeline timeline storage available in production | Production Pipeline Timing deployment with event sources still disabled |
| HUM-7535 Support environment-specific RPM pipeline event collection | Both environments’ timeline queues, dead-letter queues, subscriptions, IAM, alerts, and configuration before worker rollout |
| HUM-7532 Expose managed Release pipeline timing | Read-only rhtap-releng-tenant access and managed PipelineRun and TaskRun event collection |
| HUM-7533 Roll out RPM pipeline event collection | Staging source activation, production source activation, monitoring, recovery, redrive, and end-to-end validation |
Dependency Order
- HUM-7529 and HUM-7530 can proceed in parallel.
- HUM-8015 is the completed staging message-bus prerequisite. HUM-7535 depends on HUM-7530’s event contracts and provisions timeline-specific infrastructure in both environments without waiting for staging worker acceptance.
- HUM-7531 depends on HUM-7529, HUM-7530, and the staging path from HUM-7535.
- HUM-7534 depends on successful staging validation in HUM-7531.
- HUM-7532 depends on the event support from HUM-7530 and the environment-specific infrastructure from HUM-7535. Managed namespace access can proceed alongside the Pipeline Timing deployments, but completion gates final acceptance under HUM-7533.
- HUM-7533 depends on HUM-7531, HUM-7534, HUM-7535, and HUM-7532. HUM-5820 remains open until the final rollout is verified.
4.5.32 - RPM Tracker
RPM Tracker owns the rpm_tracker schema in the shared events PostgreSQL
database. It stores upstream state and packaged version status for the dashboard
to read. See the system proposal for the collection and UI design.
Features
- Independent Alembic migration chain with an initializer and revision gate.
- Transactional replacement of the current inventory with a generation check.
- Read-only dashboard access to tracker data in the shared database.
- Full RPM spec and upstream monitor collection with currency, staleness, packaged-stream lifecycle, and independent upstream scan health.
- Database-only update MR reconciliation with merge-time rollouts, independent open-MR listing, and creation-based ages with explicit uncertainty.
- A standalone Konflux image with native RPM support and independent collection and MR-reconciliation CronJob templates for staging and production.
Prerequisites
Python 3.11 or newer and access to the shared PostgreSQL events database.
The migration user needs permission to create the rpm_tracker schema.
Collection also requires Git, rpmspec, the system Python RPM bindings, and
read access to the rpms repository and upstream monitor. On Fedora, install
git-core, rpm-build, and python3-rpm. The Python dependencies needed to
load the rpms version helper are included in the tracker package. Do not run
collection in a virtual environment that hides the system RPM bindings.
The helper also needs the packaged specfile dependency to include sudo’s
upstream patchlevel from its extraver macro.
MR reconciliation requires read access to public.gitlab_merge_requests and
write access to tracker tables. It needs neither GitLab API credentials nor
native RPM bindings.
Installation
cd rpm-tracker
pip install -e .
DATABASE_URL=postgresql://postgres@localhost:5432/events python -m rpm_tracker.init_db
The Alembic chain uses rpm_tracker.alembic_version, separate from other
services’ migration histories and outside public. The migration files are
bundled in the installed rpm_tracker package. The initializer creates the
schema and applies pending migrations on each invocation; it is a no-op at
the current revision. Run it before deploying the dashboard API, and run it as
an init step before the collector starts. A failed migration must prevent
collection. The dashboard read endpoint returns HTTP 503 until a migrated
schema compatible with its response model is available. Later tracker revisions
remain readable when they preserve the fields used by the dashboard.
Online initialization takes a PostgreSQL transaction advisory lock, so collection
and reconciliation init containers can start together without racing migrations.
The initializer delegates this database work to rpm_tracker.db.initialize().
That helper creates the schema, opens the transaction, and passes its connection
to Alembic before applying runtime grants. Alembic’s migrations/env.py only
configures offline or online migration execution and the tracker ORM metadata.
When RPM_TRACKER_RUNTIME_PASSWORD is supplied, initialization also creates or
updates the rpm_tracker login role. This requires the existing Status database
administrator credential. The role receives tracker-table reads and writes,
read-only access to the tracker revision and public.gitlab_merge_requests, and
no schema-creation or administrator privileges. Role/password updates and
migrations commit together. Omit this variable for migration-only initialization.
Standalone Alembic commands do not create the schema or configure runtime roles. Run the initializer first. Offline SQL likewise assumes that the tracker schema has already been provisioned.
Container image
The rpm-tracker/Containerfile installs Git, rpm-build, python3-rpm, CA
certificates, and the tracker package from pyproject.toml. The installed package
includes its migrations; the image runs as the base image’s non-root user.
Konflux discovers the Containerfile through the existing tools renderer and
builds both supported architectures. The release must be pinned by digest in the
infrastructure repository’s vars.d/rpm-tracker.yml before deployment.
podman build -t localhost/rpm-tracker:dev rpm-tracker
podman run --rm localhost/rpm-tracker:dev rpm-tracker-collect --help
The infrastructure kubernetes/rpm-tracker/ templates initialize the database
before each run, then use the restricted runtime role. Collection is scheduled at
0 */4 * * * UTC; reconciliation is scheduled independently every ten minutes.
The staging and production variants use their own Status database and runtime
password. The infrastructure operating runbook documents rollout and manual checks.
Usage
Collect an inventory
DATABASE_URL=postgresql://postgres@localhost:5432/events \
python -m rpm_tracker.collect \
--rpms-repository https://gitlab.com/org/group/rpms.git \
--monitor-url https://upstream-monitor.example.org
The installed rpm-tracker-collect command accepts the same arguments. Each run
checks the migrated schema and reads the expected generation before cloning the
repository’s default branch into a temporary directory. It reads every package
directory, queries spec versions through that checkout’s parse_spec_version
helper, and records the checked commit SHA. Metadata version is not a fallback.
Missing or malformed metadata and failed checkout enumeration stop collection;
an unparseable spec does not omit a package.
Clone failures log Git’s captured stderr before re-raising the original
CalledProcessError, which retains the diagnostics in its stderr attribute.
The collector reads every monitor project-catalog and release-history page,
following X-Next-Cursor even on an empty page. History pagination omits releases
without dates, so collection also reads GET /v1/projects/{id} and merges its
releases and observed_releases into the history by source and release ID.
Snapshot records take precedence for duplicates, including corrected or newly
unknown dates. Undated newer releases therefore affect currency and make
staleness unavailable instead of disappearing from the comparison. Anitya’s
latest-only date coverage is supported. A JSON null stream list is treated as
empty; malformed non-list responses remain errors. A project without eligible
releases produces unknown without preventing other packages from refreshing.
Requests have a 30-second timeout and are paced sequentially with a 0.2-second delay. All monitor redirects are rejected, including same-origin redirects, so bearer tokens are never forwarded to a redirect destination. Configure the canonical monitor API base URL; a redirect fails collection and retains the previous inventory. Definitions verify repository matches; shared projects are fetched only once. HTTP, response validation, or pagination failures, including a failed snapshot request, prevent any inventory write. Monitor APIs do not promise a single snapshot across requests; a later complete run reconciles changes.
Currency uses native RPM version comparison with no packaging or lexical
fallback. A version equal to or ahead of the eligible upstream maximum is
current; an eligible newer release makes it behind. No verified project
mapping means unmonitored. A mapped package with an unusable spec, update
target, or comparison is unknown; its reason is logged and monitor health is
retained. Pinned lines use component-boundary matching. With no track_upstream,
use upstream_stream or a uniquely matched packaged stream, never global latest.
An explicit stream does not change a pinned or latest update target.
Monitor scheme names are trimmed and lowercased during definition validation, matching the monitor registry. Mixed-case or whitespace-padded names therefore use the same normalization and pinned-target filtering as their canonical names.
first_behind_at is recomputed as the earliest date of an eligible release newer
than the packaged version. Later releases do not reset it. If any eligible newer
release lacks a date, staleness is unavailable. Lifecycle always describes the
packaged stream, independently of the update target. Non-numeric/custom stream
names require an exact release match or an explicit upstream_stream override.
Package suffix stripping and named transforms reuse the rpms helper; already
normalized OpenJDK versions are not transformed twice. After explicit package
transforms, semantic +build metadata is removed and prerelease hyphens are
represented as RPM tildes. Build-only differences do not create upgrades or
staleness, while OpenJDK’s build number is retained by its explicit transform.
For monitor schemes pep440 and python,
PEP 440 alpha, beta, and release-candidate versions are normalized to RPM tilde
notation (3.15.0rc2 becomes 3.15.0~rc2), so equivalent packaged prereleases
compare equal and final releases sort later. The standard packaging parser
handles PEP 440 aliases and casing. Nonzero epochs, development releases,
post-releases, local versions, and invalid PEP 440 values produce unknown
rather than an unsupported comparison. CVE version_transform and
source-artifact availability checks do not affect tracker currency.
Repository URLs are matched across HTTPS, SSH, and Git syntax with .git and
trailing slashes removed. Multiple matching projects require the metadata field
upstream_monitor_project_id. This is a monitor identifier, not an Anitya
release_monitoring_project_id. Providers whose repository cannot be recovered
from their definition, such as Anitya or numeric GitLab IDs, require that explicit
verified mapping. A resolvable source repository must agree with upstream_repo.
A monitor status reporting a failed upstream scan is preserved even if readable
history permits successful collection. Dashboard consumers can distinguish old
collection data from failed scanning using sync timestamps and the separate
monitor_state, monitor_last_error, and monitor_last_success_at fields.
Write failures and generation conflicts propagate and leave the last successful
inventory unchanged; they require a fresh collection rather than retrying an old
payload. The collector does not migrate the database; the CronJob’s blocking
init container runs the initializer first.
Reconcile update MRs
DATABASE_URL=postgresql://postgres@localhost:5432/events \
python -m rpm_tracker.reconcile \
--rpms-project org/group/rpms --target-branch main
The installed rpm-tracker-reconcile command accepts the same arguments. Run
the initializer before reconciliation and schedule reconciliation periodically,
independently of monitor collection. The first successful reconciliation stores
the exact project and target branch in rpm_sync_state; subsequent runs reject
a different scope to prevent historical rollout records being reassigned.
The reconciler reads the existing Status MR rows in the shared database. It
recognizes chore(rpms): Update <package> to <version> and
Update <package> to <version>, including Draft:, WIP:, and STALE PATCH:
prefixes. Both chore/upstream-update-<package> and
chore/upstream-update-failure-<package> corroborate package identity, including
diagnostic MRs retitled to valid update titles, but never establish a version.
Package names and versions are exact, not substring or prefix matches.
Unsupported RPM version syntax and conflicting
branch/title packages remain uncertain. Diagnostic failure MRs with title
Draft: investigate failed upstream update for <package> remain visible as
package-level candidates without a target version.
rpm_update_mrs preserves every qualifying MR by (project, iid), including
multiple open MRs for one package and packages absent from the current inventory.
Each successful run refreshes the candidate snapshot from the Status rows;
closed and no-longer-qualifying MRs disappear from the open list. Previously
observed merged identities survive unrecognized titles, changes of target
branch, and temporarily missing Status rows. This retained evidence can reject
a later version change, but only currently qualifying MRs can resolve a
rollout. The tracker stores creation and merge timestamps from the raw webhook’s
object_attributes.created_at and object_attributes.merged_at. Timestamps
must include an offset, Z, or GitLab’s space-separated UTC suffix and are normalized to
UTC. Neither mutable updated_at, observation time, nor MR-head-version dates
are substitutes. Previously reliable timestamps survive later omitted fields.
Conflicting timestamps remain unavailable, with their reason retained for
investigation.
Rollouts are keyed by (package, version). Exactly one verified merged MR in
the configured scope with an explicit reliable merge timestamp resolves a
rollout. Missing timestamps, conflicting identities, and multiple matching
merged MRs leave rolled_out_at null with an unavailable_reason. Merge times
before creation or in the future are unavailable. Changing a verified version
identity after merge cannot move its rollout to another version. The timestamp
is a v1 merge-time proxy, not proof of successful publication.
Every run revisits current packaged versions, verified merged update targets,
and historical rollout keys. Later MR evidence can therefore resolve an
unavailable historical version after the inventory has advanced. New ambiguity
invalidates an earlier resolution. A rollout’s resolved_at changes only when
its resolution result changes. The MR snapshot, rollout changes, and
last_mr_reconciled_at commit atomically under the existing sync-row lock;
failure retains the previous snapshot and freshness timestamp. MR reconciliation
does not change the inventory generation or collection freshness.
Read MR information
The dashboard exposes:
GET /api/rpm-upstream-state: inventory, rollout reasons, reconciliation scope/freshness, allopen_update_mrs, and oneblocking_mrper package row.GET /api/rpm-update-mrs: all qualifying open MRs independently of package rows, pluslast_mr_reconciled_at.GET /api/rpm-update-mrs?stuck_only=true: MRs older than 24 hours and MRs with unknown ages. Exactly 24 hours is not older than the threshold.
MR ages are computed from creation time at read time. Missing, conflicting, or
future creation times produce null age_seconds and null is_stuck, never
zero or omission. created_at_issue explains invalid or conflicting evidence.
The package-row MR prefers a verified match to the selected upstream version,
then the newest known creation time, source update time, project, and IID for
deterministic selection. A fallback MR link is a package-level candidate, not
evidence of a target-version match. Selecting one package-row MR never removes
other MRs from either listing. A null reconciliation timestamp means no MR
snapshot has been collected yet.
Transactional storage
Create a SQLAlchemy engine with rpm_tracker.db.get_engine() for the storage
module, and dispose of it when collection finishes. The tracker-owned ORM
models mirror the Alembic tables; only the initializer migrates them. Call
store.get_generation(engine) before collecting; both it and
store.ingest(engine, request) require the bundled schema revision. Fetch the
complete RPM checkout, every required monitor release page, and each mapped
project’s snapshot before calling
ingest() with one status per package and the checked checkout SHA. A failed
fetch must not submit a partial inventory. ingest() locks the singleton sync
row, checks the expected generation, and replaces upstream and status rows in
one transaction. A failed write or conflicting generation leaves the previous
inventory unchanged. Stored rollout times and MR information are separate from
inventory replacement. The dashboard reads the
sync, upstream, status, rollout, and MR tables in one read-only snapshot; it does
not migrate or write tracker tables.
Configuration
| Variable | Description |
|---|---|
DATABASE_URL |
Shared events PostgreSQL connection URL |
POSTGRES_URL |
Connection URL fallback |
RPMS_REPOSITORY_URL |
Repository URL, alternatively supplied by --rpms-repository |
UPSTREAM_MONITOR_URL |
Monitor base URL, alternatively supplied by --monitor-url |
UPSTREAM_MONITOR_TOKEN |
Optional bearer token for monitor API reads |
RPM_TRACKER_RUNTIME_PASSWORD |
Initializer-only password for the restricted runtime role |
PGPASSWORD |
PostgreSQL password supplied separately from the connection URL |
SSL_CERT_FILE |
Trusted CA bundle for the monitor client’s stdlib HTTPS requests |
GIT_SSL_CAINFO |
Trusted CA bundle for Git HTTPS requests |
REQUESTS_CA_BUNDLE |
Trusted CA bundle for the rpms helper’s Requests dependency |
| Reconciliation option | Description |
|---|---|
--rpms-project |
Required exact GitLab project path, e.g. org/group/rpms |
--target-branch |
Exact inventory target branch; defaults to main |
Development
cd rpm-tracker
pip install -e '.[dev]'
pytest
To run the PostgreSQL migration, reconciliation, and atomic replacement tests, set
DATABASE_URL to a disposable PostgreSQL database before running the tracker
and dashboard tests. CI supplies test_events; the database-backed tests are
skipped when DATABASE_URL is unset. The tracker tests refuse to replace an
existing nonempty tracker inventory.
The initializer privilege test requires no existing rpm_tracker login role and
uses a disposable database. It verifies concurrent initialization, quoted
passwords, allowed tracker writes, and rejected public/revision/schema writes.
Image smoke checks in rpm-tracker/tests-k8s.yml verify executable CLIs, native
RPM ordering, spec parsing, specfile macro expansion, and bundled migrations
through the existing Konflux K8s integration scenario.
Native RPM ordering and collector database integration checks require the system Python RPM bindings. Policy and collection-failure unit tests can run without them, but production collection fails immediately when the bindings are unavailable.
Check model and migration drift
After initializing a disposable database, compare its migrated schema with the
tracker ORM models from the rpm-tracker/ directory:
DATABASE_URL=postgresql://postgres@localhost:5432/events \
python -m alembic -c rpm_tracker/alembic.ini check
Alembic receives models.Base.metadata and reflects only the rpm_tracker
schema. Supported schema differences, including unexpected tracker columns and
tables, fail the check. Status tables in public and other service schemas are
excluded. Checking does not run role provisioning or change the database.
License
This project is licensed under the GNU General Public License v3.0 or later. See the LICENSE.
4.5.33 - Work Queue Service
Generic work queue service for Hummingbird. Manages work items through type-specific lifecycles with a shared processing framework. Currently handles MR creation and updates; designed to support CVE remediation, advisory tracking, and other work item types.
For design decisions and architectural rationale, see Work Queue Service Design.
Architecture
The service runs as a Kubernetes deployment sharing the same PostgreSQL database as hummingbird-status and hummingbird-dashboard (with a service-specific Alembic migration chain).
flowchart TD
clients["Internal services"] -- "POST /work-items/mr" --> svc["Work Queue Service"]
gl_events["GitLab\n(events)"] -- webhook --> gl_fwd["gitlab-event-forwarder"]
gl_fwd --> sns["SNS"]
k8s_events["Konflux K8s\n(PipelineRun events)"] -- watch --> k8s_fwd["k8s-event-forwarder"]
k8s_fwd --> sns
sns -->|"filter: source=gitlab,\nevent_type=[merge_request, build]"| sqs["SQS Queue"]
sns -->|"filter: source=kubernetes,\nkind=PipelineRun"| sqs
sqs --> svc
svc -- "Commits API\n(create + update)\nJobs API\n(trace + retry)" --> gl["GitLab\n(API)"]
svc -- "PLR fallback fetch\n(label selector on SHA)" --> k8s_api["Konflux K8s API"]
svc -- "archived PLR fetch" --> ka["Kubearchive"]
svc -- "shared DB" --> pg[("PostgreSQL")]
The SQS consumer thread receives GitLab merge_request and pipeline
events and Kubernetes PipelineRun events from the SNS topic via one
filtered SQS subscription with a two-branch $or filter policy covering
both source=gitlab and source=kubernetes. Events are stored in the
pending_events table and processed by existing work-item threads,
which map MR state changes to phase transitions, track Konflux build
status, and trigger CI job retries on pipeline failure.
Design
Generic + detail tables
The schema separates queue machinery from type-specific data:
work_items— generic: id, type, phase, actionable, attempts, error, timestamps. Shared by all work item types.mr_details— MR-specific: package, project, file_actions, commit_message, content_hash, mr_iid, etc. Joined to work_items via foreign key.work_item_statuses— independent readiness dimensions with composite key (work_item_id, dimension, variant). Includespending_untilfor deferred re-evaluation.
New work item types add their own detail table (e.g. cve_details) —
no changes to the generic layer. SQLAlchemy joined table inheritance
loads the correct subclass automatically via polymorphic_load="selectin".
Phases
Four generic phases track coarse lifecycle status:
| Phase | Description |
|---|---|
active |
Being processed — the only mutable phase |
completed |
All lifecycle steps done (terminal) |
failed |
Error occurred (terminal, retryable) |
cancelled |
Cancelled (terminal, retryable) |
Items start as active. Only active items accept event-driven phase
transitions and executor actions. The three terminal phases
(completed, failed, cancelled) are immutable to events — recovery
requires explicit action via the retry API. GitLab merge_request events
drive transitions from active: merge -> completed, close ->
cancelled.
Dimensions
Independent readiness signals stored in work_item_statuses. All
dimensions follow a uniform needs_* trigger model: status is
"true" (action needed) or "false" (handled). Absent dimensions
are treated as actionable by the decider (!= "false"). Each
dimension has an optional variant for multi-instance signals (e.g.
per-component Konflux builds) and an optional pending_until
timestamp for deferred re-evaluation (see
Deferred dimensions below).
Dimensions are set by executors or event handlers via
EventResult.dimensions; the generic handle_event layer applies
event-driven dimensions and emits audit entries.
Exception —
needs_plr: Unlike other dimensions,needs_plrrequires explicit arming to"true"to be actionable. Absent or"false"means inactive (not awaiting PLR events). The decider checksdims.get("needs_plr") == "true", so a missing row is treated as inactive, not actionable.
Dimension schema
work_item_statuses:
work_item_id: UUID (PK, FK)
dimension: Text (PK)
variant: Text (PK, default="")
status: Text (NOT NULL) -- "true" or "false"
updated_at: DateTime(tz)
detail: JSONB (nullable)
pending_until: DateTime(tz, nullable)
Unified state diagram
All dimensions share the same state machine:
stateDiagram-v2
[*] --> absent : item created
absent --> true : decider treats absent as actionable
true --> true_deferred : executor (defer for later)
true --> false : executor (done, no further checks)
true_deferred --> true : sweep notifies (pending_until expired)
true_deferred --> true : webhook/handler (clears pending_until)
false --> true : webhook/handler (re-activates)
note right of absent : no dimension row exists
note right of true : status="true", pending_until=NULL
note right of true_deferred : status="true", pending_until=future
note right of false : status="false", decider ignores
Per-dimension transitions
| Dimension | On success (→ "false") |
On error / re-trigger (→ "true") |
Deferred? |
|---|---|---|---|
needs_mr |
MR created or already exists | — | No |
needs_expected_plrs |
Expected components fetched | GitLab API error, unexpected error (malformed manifest permanently completes with error: "parse_error") |
Yes — 5 min error backoff |
needs_plr |
All PLRs terminal or fetched | PLR event or PLR fallback fetch (non-terminal) | Yes — 30 min fallback poll |
needs_dashboard |
Dashboard note posted or updated | PLR event, PLR fallback fetch, transient GitLab 5xx | No — always set “false”; re-armed by PLR handler/executor |
needs_reconcile |
Reconciler completed | Webhook (approved, pipeline, update), sweep expiry | Yes — 30 min cycle / 5 min error backoff / 15 s→5 min recheck backoff for unchecked/checking/unknown |
needs_empty_check |
Empty-diff check completed | Sweep expiry | Yes — 30 min cycle / 5 min error backoff |
needs_ci_retry |
All jobs processed | Build event (job failed), sweep expiry | Yes — per-job not_before delay |
Dimension detail schemas
The detail JSONB column carries executor-specific state. Dimensions
without detail (needs_mr, needs_empty_check) store null;
needs_reconcile stores null except during a merge-recheck streak.
Frozen dataclass definitions live in
workqueue_service/mr/dimension_types.py; typed accessor functions
(get_plr_detail, get_dashboard_detail, get_expected_plrs_detail)
deserialize the stored JSONB into the dataclass, or return None when
absent. JSON arrays round-trip as tuples — NeedsCiRetryDetail.jobs
and NeedsExpectedPlrsDetail.components are immutable.
Detail preservation. DimensionUpdate.detail defaults to a
_PRESERVE sentinel. Event handlers that return a DimensionUpdate
without explicit detail automatically keep whatever is stored in the
database (via SQL COALESCE). Pass detail=None explicitly to clear,
or a detail dataclass to replace. See Dimension update
helpers below.
Updating detail. Event handlers update the existing detail instead
of rebuilding it: read it via the typed accessor, then apply a mutation
helper — NeedsPlrDetail.with_component() adds or replaces one
component entry, NeedsCiRetryDetail.with_job() appends one job.
Both return a new frozen instance; the original is untouched. Handlers
pass the dataclass to DimensionUpdate, and executors that snapshot
authoritative state (ensure_plr_status, ensure_expected_plrs,
retry_ci_jobs) construct a fresh dataclass. crud.upsert_dimension
serializes dataclasses via dataclasses.asdict() before the DB write,
so callers never handle raw dicts.
needs_expected_plrs detail
Stored after fetching the Konflux trigger manifest for a commit SHA.
Set by ensure_expected_plrs; read by ensure_plr_status,
ensure_dashboard_note, and the PLR event handler.
| Field | Type | Required | Description |
|---|---|---|---|
sha |
str | null |
yes | Commit SHA the manifest was fetched for. null for items without a SHA. |
components |
list[str] |
yes | Expected component names (e.g. "caddy--hummingbird--default"). Empty when the repo has no trigger manifest (404). |
error |
str |
no | Set on parse failure; empty components list accompanies it. |
needs_plr detail
Tracks observed PipelineRun state per component. Updated by the PLR
SNS handler (merge per event) and ensure_plr_status (full snapshot
from K8s/Kubearchive). Read by ensure_dashboard_note.
| Field | Type | Required | Description |
|---|---|---|---|
components |
dict[str, PlrComponentInfo] |
yes | Keyed by component name. Empty dict on init. |
Each PlrComponentInfo value:
| Field | Type | Required | Description |
|---|---|---|---|
status |
str |
yes | PLR condition: "Running", "Succeeded", "Failed", "Unknown" |
plr_name |
str |
yes | Kubernetes PipelineRun resource name |
start_time |
str |
yes | ISO 8601 timestamp from status.startTime (pipeline start, not resource creation) |
needs_dashboard detail
Stored after posting or updating the MR dashboard note. Set by
ensure_dashboard_note; read on subsequent runs to decide whether
to update or post. The PLR handler and ensure_plr_status re-arm
needs_dashboard to "true" without passing detail, which preserves
this detail.
| Field | Type | Required | Description |
|---|---|---|---|
note_id |
int |
yes | GitLab note ID for in-place updates |
url |
str |
yes | Dashboard URL included in the note body |
note_body |
str |
no | Last rendered note body; compared to skip no-op GitLab API calls |
needs_reconcile detail
Optional — only present during a merge-recheck streak. When
reconcile_mr observes detailed_merge_status in the fast family
(unchecked, checking, unknown — recheck in flight), it re-arms the
dimension at now + MERGE_RECHECK_SCHEDULE.delay(retry_step) (15 s
doubling to a 5-minute cap) and bumps the step via advance_retry. The
converge outcomes (merge attempted, real blocker, idle) clear the
detail, so a fresh streak starts at 15 s again; the early exits (no MR
yet, GitLab fetch error) leave the stored step untouched. Webhook
re-arms preserve the stored step through the _PRESERVE sentinel.
| Field | Type | Required | Description |
|---|---|---|---|
retry_step |
int |
yes | Consecutive fast-family recheck count (starts at 1) |
needs_ci_retry detail
Accumulates failed CI jobs eligible for retry. Set by the build event
handler (appends per job) and retry_ci_jobs executor (removes
processed jobs). Each entry represents one failed GitLab CI job.
| Field | Type | Required | Description |
|---|---|---|---|
jobs |
list[CiRetryJobEntry] |
yes | List of job entries (see below) |
Each job entry:
| Field | Type | Required | Description |
|---|---|---|---|
build_id |
int |
yes | GitLab job ID |
build_name |
str |
yes | Job name (e.g. "test", "check") |
build_failure_reason |
str |
yes | GitLab failure reason (e.g. "script_failure") |
build_finished_at |
str |
yes | ISO 8601 timestamp when the job finished |
retries_count |
int |
yes | Number of retries already attempted |
pipeline_id |
int |
yes | GitLab pipeline ID the job belongs to |
matcher_name |
str |
no | Name of the retry matcher rule that matched this job |
not_before |
str |
no | ISO 8601 timestamp — earliest time the retry may be attempted |
Deferred dimensions
When an executor or handler sets pending_until to a future timestamp
alongside status="true", the decider treats the dimension as "false"
(not actionable) until that time. The periodic sweep finds items with
expired pending_until and notifies them — no synthetic events are
inserted. The sweep wakes exactly at the earliest armed deadline — or
after ~1 s when a deferred timer is already due, letting the next
pass’s expired scan pick it up (see
Processing loop) — so even short timers are honored
precisely rather than rounded up to a sweep tick.
status |
pending_until |
Decider sees | Meaning |
|---|---|---|---|
| (absent) | — | actionable | No row — needs action |
"true" |
NULL |
actionable | Needs action now |
"true" |
past | actionable | Sweep notified, ready to act |
"true" |
future | not actionable | Deferred — sweep will notify |
"false" |
any | not actionable | Done |
Dimension update helpers
crud.upsert_dimension provides safe-by-default mutation patterns via
its detail parameter:
- Omitted — keep existing detail (SQL
COALESCEon conflict); e.g. re-triggeringneeds_dashboardafter a PLR event without clobberingnote_id - Detail dataclass — replace with a full snapshot; e.g. initialising a dimension or completing an executor action
None— clear the stored detail explicitly
Partial detail updates do not use a merge helper — event handlers
apply the frozen dataclass mutation helpers (with_component(),
with_job()) to the existing detail instead, and executors construct a
fresh dataclass for a full snapshot. crud.upsert_dimension takes a
detail dataclass (or None) and serializes it via dataclasses.asdict()
internally; callers never handle raw dicts.
See Dimension detail schemas for the
_PRESERVE sentinel, frozen dataclass definitions, and typed accessors.
Dimension state (status plus optional deferred re-arm) is expressed via the
DimensionState enum in workqueue_service/mr/constants.py. Passing a bare status string
or timedelta to the state parameter is a type error; always use an enum member:
| Member | Status | Delay | Used for |
|---|---|---|---|
DONE |
false |
— | Dimension fulfilled, no action needed |
PROCESS_IMMEDIATELY |
true |
— | Needs action now |
PROCESS_AFTER_TIMEOUT |
true |
30 min | PLR silence fallback, reconcile/empty-check cycle |
PROCESS_AFTER_ERROR |
true |
5 min | Transient API error backoff, CI job not_before delay |
For computed absolute times (e.g. the CI retry not_before), use
DimensionState.process_at(when). It returns a ProcessAt wrapper that the CRUD layer
resolves to status="true" with pending_until set to the given time.
Decider
A pure function registered per type that derives the next action from
the work item’s current state. The decider reads the selectin-loaded
item.statuses list (in-memory, no DB call) to check dimensions,
pre-filtering deferred dimensions (where pending_until is in the
future). See
workqueue_service/mr/decider.py
for the MR decider implementation.
The decider scans dimensions in priority order
(DDR-45, DDR-48).
Absent dimensions are treated as actionable (!= "false"). Deferred
dimensions (pending_until in the future) are excluded from the
scan. Adding a new concern = new dimension + handler, no decider
changes.
The decider determines the next action to execute — it does not set phases or modify state.
MR upsert semantics
The MR submission endpoint uses upsert logic keyed on
(project, source_branch). Callers always submit the current desired
state; the service decides what to do based on existing work items:
stateDiagram-v2
[*] --> created : no active item
[*] --> unchanged : active, same hash
[*] --> updated : active, different hash
[*] --> retried : failed item
created --> active : new work item
unchanged --> active : no-op
updated --> active : refresh fields
retried --> active : reset phase
| Upsert action | Condition | HTTP status | Effect |
|---|---|---|---|
created |
No active/failed item for (project, branch) | 201 | New work item, actionable = true |
unchanged |
Active item exists, same content_hash |
200 | No-op, return existing item |
updated |
Active item exists, different content_hash |
200 | Refresh file_actions/title/description, set actionable = true |
retried |
Failed item exists for (project, branch) | 200 | Reset to active, clear error, update content, set actionable = true |
Content hash. A SHA-256 of the normalized file_actions JSON
(sorted by file_path + action, compact separators, sorted keys).
Enables cheap same-content detection without comparing the full payload.
Idempotency key. Derived server-side as
mr:{project}:{source_branch}:{generation} where generation is a
counter per (project, source_branch) pair. A partial unique index
ensures uniqueness only among non-terminal phases — completed and
cancelled items can share a key prefix without blocking new items for
the same branch.
Race handling. If two concurrent inserts race for the same
idempotency key, the loser catches the IntegrityError, rolls back,
and retries the full upsert — the winner is now visible and the
correct action (unchanged/updated) is taken.
MR action chain
The MR work item lifecycle chains executor actions. After upsert, the processing loop calls the decider to select the first action; subsequent actions are also selected by the decider after each executor completes:
ensure_mr— creates or updates the MR. Ifmr_iidis not set, creates the branch, commit, and MR via the GitLab Commits API and applies themanaged-by::workqueuescoped label. Ifmr_iidis already set, force-pushes updated content to the existing MR branch and syncs MR metadata (title, description). Detects empty diffs (before creation) or closes an empty MR after force-push. Captures the commit SHA ascurrent_sha, resets theneeds_plrdimension (clearing stale component state), and re-armsneeds_expected_plrs. Setsneeds_mrto"false"on success.ensure_expected_plrs— fetches the Konflux trigger manifest (.tekton/trigger-manifest-pull-request.json) from the target project via the GitLab repository files API, parses the component-to-path mappings, and derives the set of affected components from the MR’s changed file paths. Stores the component list in theneeds_expected_plrsdimension detail. If the manifest is missing (404), stores an empty list. Retries on transient API errors with a 5-minute backoff.ensure_plr_status— fallback executor that fetches current PLR state from the Konflux K8s API (with Kubearchive fallback) using label selectors on the commit SHA. Fires on 30-minutepending_untilre-arm when PipelineRun events have been silent. Writes the complete component snapshot into theneeds_plrdimension detail and re-armsneeds_dashboardto trigger a note update. Per component, the newest PipelineRun bystatus.startTimewins — after a retrigger, the older archived run does not overwrite the newer live one. If any component is still non-terminal, re-armsneeds_plrwith another 30-minute delay. RequiresKONFLUX_KUBECONFIG_PATHand optionallyKUBEARCHIVE_URLenvironment variables. The KubeArchive client reuses the bearer token from the kubeconfig’s auth settings.ensure_dashboard_note— posts or updates an internal note on the MR with the Hummingbird dashboard link and a per-component Konflux build status table. The note uses the:robot: **Hummingbird Status**header shared with the dashboard-mr-linker Lambda. On first run, it creates the note and storesnote_idin the dimension detail. On subsequent runs (triggered by PipelineRun events or the PLR fallback fetch), updates the existing note in place if the rendered body differs from the storednote_body. Always setsneeds_dashboardto"false"on completion — re-arming is handled by the PLR SNS handler and theensure_plr_statusexecutor, which setneeds_dashboard = "true"whenever PLR state changes.check_empty_diff— checks whether an active MR’s diff became empty (target branch moved). Triggered by theneeds_empty_checkdimension viapending_untilexpiry (~every 30 minutes per item). If the diff is empty, closes the MR with an explanatory note, deletes the branch, and cancels the item. Setsneeds_empty_checkto"false"on completion. Transient GitLab API errors are logged but do not fail the item.reconcile_mr— reconciles desired vs actual MR state. Triggered by theneeds_reconciledimension, set by webhooks (approved, unapproved, update, pipeline success/failure) andpending_untilexpiry. Merges directly whendetailed_merge_statusismergeableandauto_mergeis enabled. Syncs title and description if drifted. Re-armsneeds_reconcileaccording to the merge outcome: a failed merge PUT retries immediately;unchecked/checking/unknown(a mergeability recheck in flight, completing seconds later) retry on an exponential schedule — 15 s doubling to a 5-minute cap, with the step tracked in the dimension detail; every other status keeps the 30-minute cycle.
Draft state. The draft submission option controls initial MR
creation only. After creation, GitLab owns Ready/Draft state;
update_mr and reconcile_mr preserve the current GitLab state while
synchronizing title text. The details.draft response field reports the
initial creation setting, not the live GitLab status.
After all actions complete, the decider returns None and the item
sits in active phase waiting for event-driven lifecycle tracking
(merge → completed, close → cancelled).
Label coordination with dashboard-mr-linker. The managed-by::workqueue
label causes the dashboard-mr-linker Lambda to skip the MR, so the
workqueue-service creates the dashboard note for its MRs. When
dashboard_base_url is not configured, ensure_dashboard_note returns
immediately without posting a note.
head_pipeline_id on mr_details
The current head pipeline ID is stored as a typed integer column
(head_pipeline_id) on the mr_details table rather than as a
dimension. Build events and the reconciler update it directly. The CI
retry executor uses it to filter stale jobs from previous pipelines.
Processing loop
The ProcessingManager uses a bounded ThreadPoolExecutor (10 workers):
- One loop per item — a loop takes a session-level advisory lock
pg_try_advisory_lock(0x5751, item_id & 0x7FFFFFFF)on a pinned connection and holds it for its whole lifetime. Another loop (another pod, the sweep, or the local rerun path) returns immediately when the lock is held; duplicate submissions are cheap. Losers incrementhummingbird_workqueue_processing_lock_skipped_total, labelled by item type. The key is the item UUID’s low 31 bits, so two unrelated items can rarely collide; the skipped item is re-spawned by the sweep once the winner finishes — delayed, never lost. - Serial per item — one action at a time per work item, across pods.
- Parallel across items — independent items processed concurrently.
- Per-item deadline — processing loops exit after 60 seconds to prevent stuck items from holding pool workers.
- Executors apply state updates directly (phase, MR fields) and must
not call
session.commit()orsession.rollback(). - After execution, the loop re-derives the next action via the decider.
- Crash recovery — the lock lives on a pinned database connection;
if the pod or connection dies, PostgreSQL releases it. A death at any
loop step (inside an executor or at the post-execution state write)
aborts the loop; it never continues on a reconnected connection
without the lock. The item stays
actionable, so the periodic sweep re-spawns it. There is no lease, heartbeat, or TTL. - The periodic sweep submits actionable items and items with pending
events or due
pending_untildimensions (no synthetic events). It is deadline-driven: after each pass it sleeps until the earliest armedpending_untildeadline (plus ~1 s so the deadline is already expired on wake), or for ~1 s when a deferred timer is already due (a single query returns the earliest future deadline and a past-due flag over one snapshot, so a timer expiring between the sweep’s scans cannot be missed by both), keeping the former 5-minute interval as a backstop cap for the un-timed legs — actionable crash recovery, pending events, and deadlines armed by other pods. A wake event cuts a long wait short whenever an enqueue or an executor commit arms an earlier deadline in-process; sweep-initiated submissions do not wake it, as they arm no timer themselves.
Connection pinning. Each loop checks out one connection and binds
every session it opens to that connection, so the session lock survives
the per-action commits. The engine pool is sized to the worker count
with overflow headroom, and libpq keepalives plus tcp_user_timeout
detect a dead server instead of leaving a lock on a half-open socket.
The loop releases the lock explicitly before returning the connection
to the pool; if the release fails, it invalidates the DBAPI connection,
because a pooled close() alone would keep the session lock alive.
MR labeling
The executor always adds the managed-by::workqueue scoped label to
every MR it creates, merged with any caller-provided labels. This label
is a coordination signal: other services (e.g.
dashboard-mr-linker)
check for this label and
skip managed MRs, since the workqueue-service handles dashboard note
creation itself via the ensure_dashboard_note action.
Event processing
The SQS consumer thread receives events from the SNS topic — GitLab
webhooks (merge_request, build) and Kubernetes resource events
(PipelineRun) — and stores them in the pending_events table.
Work-item threads drain and process these events as part of their main
loop.
Consumer thread. A single daemon thread (poll_loop) long-polls SQS
with MaxNumberOfMessages=10 and WaitTimeSeconds=20. For each
message it decodes the SNS envelope (handles gzip+base64), resolves
matching work items via the resolver registry, INSERTs into
pending_events, commits, deletes the SQS message, and calls
manager.spawn_loop() for each affected work item. The consumer is started
only when WORKQUEUE_SERVICE_EVENTS_QUEUE_URL is set.
State mapping. MR state changes are mapped to phases:
| GitLab state | Phase | Notes |
|---|---|---|
merged |
completed |
Terminal |
closed |
cancelled |
Terminal (recover via retry API) |
opened |
— | Ignored (only active items accept events) |
locked |
— | Ignored (transient during merge) |
Only items in active phase accept event-driven transitions. Events
for terminal items (completed, failed, cancelled) are silently
discarded.
PipelineRun events. Kubernetes PipelineRun events (forwarded via
the event forwarder → SNS) are resolved to MR work items by matching
the commit SHA in the PipelineRun’s labels against
mr_details.current_sha. Only type=build PipelineRuns are
processed; test PipelineRuns are ignored. The handler also rejects
events whose SHA label no longer matches the item’s current_sha
(stale SNS delivery after a force-push). Each accepted event merges
the component’s status into the needs_plr dimension’s JSONB detail
and re-arms needs_dashboard to trigger a note update. Events are
attributed to a PipelineRun by metadata.name: every event for the
same PipelineRun carries the same status.startTime, so matching
names always apply the newer status (a terminal run never regresses
to a non-terminal one). For a different PipelineRun — a retrigger —
the event is accepted only when its status.startTime is strictly
newer (compared as parsed instants, so fractional-second formats
order correctly). An event without a startTime cannot overwrite a
component already recorded as Succeeded or Failed, nor an
in-progress component that already has a stored startTime.
Konflux SHA labels. Build and test PipelineRuns carry different SHA labels — using the wrong label returns zero results:
| PLR type | SHA label | Type label value |
|---|---|---|
| Build | pipelinesascode.tekton.dev/sha |
build |
| Test | pac.test.appstudio.openshift.io/sha |
test |
The event resolver checks pipelinesascode.tekton.dev/sha first
(build PLRs), falling back to pac.test.appstudio.openshift.io/sha
for edge cases. The K8s/Kubearchive fallback fetcher
(list_build_plrs_for_sha) queries only
pipelinesascode.tekton.dev/sha with type=build, matching the
label that build PLRs actually carry.
Out-of-order guard. The last_event_at column on work_items
rejects events with updated_at older than the last processed event.
Same-timestamp events pass the <= guard (processing is idempotent).
Exit coordination. The work-item exit decision runs under a
SELECT ... FOR UPDATE row lock on work_items, so an enqueue that
updates the item in the same transaction is either visible to the exit
or blocks until the exit commits and then leaves actionable = true.
The former pg_advisory_xact_lock insert/exit handshake was removed;
the row lock covers every writer, including the API enqueue paths the
handshake never covered. See
DDR-51.
After a no-work exit commits, the loop releases the session lock and
re-checks pending_events and the decider on the pinned connection.
If a writer committed in that gap, the loop re-acquires the session
lock and continues instead of returning. A writer that arrives after
the re-check is caught by the sweep; the item remains actionable.
Sweep extension. The deadline-driven periodic sweep submits
actionable items and items with unprocessed pending_events, providing
startup recovery and a safety net behind the row-lock exit and the
unlock/re-check. It wakes exactly at the earliest armed pending_until
deadline; the 5-minute backstop interval covers everything un-timed
(see DDR-52).
Concurrent drain tolerance. In steady state the per-item session
lock prevents two pods from draining the same item. During the rolling
update that introduced the session lock, an old pod still used
leased_until while a new pod used the session lock, so both could
briefly drain the same item. The drain therefore reads each listed
event’s (id, event_type, event_body) before processing the first one
and deletes by id — a row another pod already deleted is a no-op. An
event may be applied by both pods; application is idempotent (dimension
upserts, last_event_at guard), matching the at-least-once semantics
of crash recovery. See
DDR-50.
The same coexistence applied to executor actions, not just the drain:
the two-key session lock deliberately could not conflict with the old
single-key/lease path, so an old pod’s leased loop and a new pod’s
locked loop could both run the same action for one item during the
overlap. Read-and-upsert actions (reconcile_mr, ensure_plr_status,
ensure_expected_plrs) converge on the same state. Creation actions
(ensure_mr, ensure_dashboard_note) and retry_ci_jobs check their
result field (e.g. mr_iid) before acting, so a duplicate requires
both loops to pass that check before either commits — possible in the
worst case, producing one duplicate external mutation (a second MR or
note, or a second CI retry) for an item that was mid-action during the
pod replacement. New pods deliberately ignored leased_until rather
than skipping rows with an unexpired value: a lease written by a pod
that died mid-rollout must not hide the item from the sweep. That
coexistence window ended when the locking redesign finished rolling
out; the contract migration drops the leased_until
column, so the lease mechanism no longer exists. See
DDR-51.
Adding a new work item type
- Create a detail table and SQLAlchemy model (e.g.
CveWorkItem) - Register a decider function:
decider.register("cve", _cve_decide) - Register executor functions for each action
- Add a submission endpoint:
POST /api/work-items/cve
No changes needed to the processing loop, claiming, or generic API.
API
Endpoints
| Method | Path | Description |
|---|---|---|
POST |
/api/work-items/mr |
Upsert an MR work item (create, update, or skip) |
GET |
/api/work-items/mr |
List MR items (source, project, etc.) |
GET |
/api/work-items |
List all items (type, phase filters) |
GET |
/api/work-items/{id} |
Get item with nested details |
POST |
/api/work-items/{id}/retry |
Retry a failed item |
GET |
/api/whoami |
Introspect caller identity and permissions |
GET |
/oauth/start |
Initiate OAuth login (handled by oauth-proxy) |
GET |
/oauth/callback |
OAuth provider callback (handled by oauth-proxy) |
POST |
/oauth/token |
Exchange OAuth session cookie for Bearer token |
GET |
/healthz |
Health check |
GET |
/metrics |
Prometheus metrics |
Responses include a nested details dict with type-specific fields:
id: "..."
type: mr
phase: active
details:
source: rpms-ci
mr_iid: 42
mr_url: https://gitlab.com/org/group/project/-/merge_requests/42
Authentication
All callers authenticate via Authorization: Bearer tokens. The
service validates each token in one of two ways:
| Caller | Token type | Validation |
|---|---|---|
| K8s SA | JWT (projected) | JWKS signature verification |
| GitLab CI | JWT (OIDC) | JWKS signature verification |
| Human (web/CLI) | sha256~... (OCP) |
OCP users/~ API |
| Local dev | none | WORKQUEUE_SERVICE_LOCAL_AUTH_USER env var |
X-Forwarded-* headers are never trusted on API paths. The proxy
protects only the OAuth login flow (/oauth/start, /oauth/callback)
and the POST /oauth/token endpoint, which bridges cookie sessions to Bearer
tokens for browser and CLI users. The /oauth/token endpoint uses POST (not
GET) to prevent CSRF — browser-initiated GETs via <img> or
<script> tags cannot trigger it.
OAuth proxy deployment requirements
The oauth-proxy sidecar handles only the OAuth login flow. All /api/
paths bypass the proxy — the app handles auth itself. The proxy must be
configured with these flags:
| Flag | Purpose |
|---|---|
--bypass-auth-except-for=^/oauth/ |
Proxy only protects the OAuth flow (login + token exchange) |
--pass-access-token |
Sets X-Forwarded-Access-Token on cookie-authenticated requests (required for /oauth/token). Cookie secret must be exactly 16, 24, or 32 bytes (AES key size) |
--pass-user-headers |
Sets X-Forwarded-User on protected paths |
--cookie-refresh=1h |
Refreshes the OCP token inside the cookie before it expires (see below) |
--cookie-expire=24h |
Hard session limit, forces re-login |
--cookie-samesite=lax |
CSRF defense-in-depth for /oauth/token (POST already blocks the main vectors) |
Why --cookie-refresh is critical
The proxy manages two independent lifetimes that are easy to confuse:
- Session cookie — controlled by the proxy (
--cookie-expire, defaults to browser session). Contains the encrypted OCP access token. - OCP access token — controlled by OCP (default 24h). The actual
credential the app uses to call
users/~.
The proxy does not know when the OCP token inside its cookie expires.
It only checks whether the cookie is valid (signature, expiry).
Without --cookie-refresh, the cookie outlives the token:
| Time | What happens |
|---|---|
| T+0h | User logs in. Proxy creates cookie with fresh OCP token (expires T+24h) |
| T+24h | OCP token expires. Cookie still valid. |
| T+25h | User makes request. Proxy sets X-Forwarded-User (valid cookie) and X-Forwarded-Access-Token (expired token). App calls users/~ with expired token → 401 → anonymous → “not authorized”. |
--cookie-refresh=1h tells the proxy to silently refresh the OCP token
inside the cookie every hour, keeping it well within the 24h expiry.
The workqueue-service also uses a TTL-bounded cache (5 minutes) for
users/~ results, so expired-token failures are never permanent.
Authorization
Access rules are defined in the config file. Each rule maps a remote
identity to permissions (read, retry, submit), optionally scoped
to work item types. Rules use either groups (OCP group membership) or
issuer+claims (JWT bearer). Claim matching follows Vault’s
bound_claims pattern: all conditions AND, list values OR.
All permissions (including read) must be granted by a matching rule.
Authenticated callers with no matching rules get 403. Multiple matching
rules are merged (union of permissions).
See Configuration for the config file format.
Whoami introspection
GET /api/whoami returns the caller’s identity and effective
permissions. Example response:
authentication:
tier: ocp-user
identity: admin-user
groups:
- konflux-hummingbird-admin-access
authorization:
effective_permissions:
mr:
- read
- retry
- submit
matching_rules:
- groups:
- konflux-hummingbird-admin-access
permissions:
- read
- submit
- retry
item_types: []
API Documentation
Interactive API documentation available without authentication:
/docs (Swagger UI), /redoc (ReDoc), /openapi.json.
Audit Log
State-changing operations (processing, API calls, auth decisions)
emit audit entries. Every entry is logged at INFO level. When
WORKQUEUE_SERVICE_AUDIT_S3_BUCKET is set, entries are additionally
archived to S3 as gzipped JSON objects for long-term querying via
Athena.
Audited events
| Event type | Source | Description |
|---|---|---|
action_started |
Worker | Action execution began |
action_completed |
Worker | Action finished (success or failure) |
action_decided |
Decider | Decider assigned next action |
action_no_op |
Decider | Decider returned no next action |
phase_change |
Worker / API / Events | Phase transition (max attempts, submit, event) |
item_updated |
API | Work item content updated via upsert |
dimension_change |
Events / Executor | Dimension upsert (needs_dashboard posted, etc.) |
retry_requested |
API | Retry endpoint reset a failed item |
auth_failed |
Auth | Anonymous request rejected (401) |
auth_denied |
Auth | Authenticated caller lacks permission (403) |
S3 key format
When S3 archival is enabled, entries are stored as gzipped JSON with an Athena-partitionable key layout:
{prefix}/YYYY/MM/DD/HH/MM/{timestamp}#{work_item_id}::{event_type}.json.gz
The ContentEncoding: gzip header is set so S3 API consumers get
transparent decompression.
Field conventions
Follow these conventions when adding new audit calls:
actor: caller identity for API/auth events; thread name for worker events;"system"for decider/system-triggered transitions.old_state: state before the change. Includephaseandactionwhen both are relevant. Omit on creation (no prior state) and on auth events (no work item state).new_state: state after the change. Same keys asold_state. Omit fields that areNone(filtered byto_dict()).trigger: what caused this event. Always includesource("api","worker", or"decider"). Addactionfor deliberate operations,reasonfor error/system transitions,errorfor failure details.metadata: context orthogonal to the state transition — request info on auth events, type-specific data on submit. Not for duplicating state already inold_state/new_state.
CI Job Retry
When a CI pipeline on a managed MR fails, the service can automatically retry jobs that match known transient failure patterns. This is ported from the CKI pipeline-herder’s proven regex-based job log matching engine.
How it works
- GitLab sends a
build(job) webhook when a CI job finishes - The event handler appends the failed job to the
needs_ci_retrydimension’s jobs list and sets status to"true" - The decider selects the
retry_ci_jobsexecutor - The executor processes all jobs in the list, matching each against the config tree and retrying matching jobs via the GitLab Jobs API
- Jobs with non-zero delay set
pending_untilto the earliest per-jobnot_beforetimestamp — the sweep notifies when it expires - Multiple failed jobs in the same pipeline accumulate in the list
needs_ci_retry dimension
The needs_ci_retry dimension follows the unified state machine (see
Unified state diagram). Jobs with non-zero
delay use the deferred path: status="true" with pending_until set
to the earliest per-job not_before timestamp. The sweep notifies
when any job’s delay elapses.
The true → true self-loop fires when another job in the same
pipeline fails — the build event handler appends the new job to the
existing list (clearing pending_until so the decider picks it up
immediately).
The detail JSONB stores a jobs list. Each entry carries build_id,
build_name, build_failure_reason, build_finished_at,
retries_count, and pipeline_id — all from the build webhook
payload. The executor deduplicates by build_id and skips jobs whose
pipeline_id does not match the MR’s current head_pipeline_id.
When a job has a non-zero delay, the executor stamps a per-job
not_before ISO timestamp based on build_finished_at + retry_delays[retries_count]. The dimension’s pending_until is set
to the earliest per-job not_before so the sweep fires as soon as
any job is ready.
Matcher configuration
CI job retry is configured per type under handled_types.mr.job_retry.
The config uses a recursive matcher tree where parent fields are
inherited by children:
handled_types:
mr:
job_retry:
retry_delays: ["0", "5m", "10m"]
retry_limit: 5
matchers:
- name: runner-system-failure
description: Job failed with runner_system_failure
failure_reason: runner_system_failure
- name: git-clone
description: Git clone failure
messages:
- "fatal: unable to access"
- '/error: RPC failed; curl 92 HTTP\/2 stream \d+/'
Matcher fields
| Field | Type | Default | Description |
|---|---|---|---|
name |
string | "" |
Matcher name (used in metrics and audit) |
description |
string | "" |
Human-readable description |
failure_reason |
string | "" |
Exact match on GitLab job failure_reason |
job_name |
string | "" |
Prefix match on job name |
messages |
list[str] | [] |
Literal substring or /regex/ on job trace |
tail_lines |
int | 300 |
Number of trace lines to scan (from the end) |
retry_delays |
list[str] | ["5m"] |
Escalating delays indexed by attempt ("0", "5m") |
retry_limit |
int | 3 |
Maximum retry count per job name |
matchers |
list | [] |
Nested child matchers (recursive) |
Message matching
- Literal:
"fatal: unable to access"→ substring match per line - Regex:
'/error: RPC failed; curl \d+/'→re.searchon joined lines (slash-delimited)
Retry safety
Before retrying a job, the executor checks:
- No newer job with the same name exists (prevents double retries)
- Retry limit not exceeded (count of jobs with the same name)
Retry timing
The retry_delays list is indexed by the job’s attempt count
(0-based). When the index exceeds the list length, the last entry
repeats. A delay of "0" means immediate retry; non-zero delays set
pending_until to the computed not_before timestamp. The sweep
notifies the item when pending_until expires, waking exactly at the
earliest armed deadline.
Prometheus Metrics
The service exposes Prometheus metrics at /metrics on the same port
as the API (no sidecar). Metrics follow the repo naming convention:
generic operational metrics use the shared hummingbird_ prefix,
service-specific metrics use hummingbird_workqueue_.
Generic operational metrics
| Metric | Type | Labels | Description |
|---|---|---|---|
hummingbird_http_requests_total |
Counter | method, path, status | Total HTTP requests |
hummingbird_http_request_duration_seconds |
Histogram | method, path | HTTP request latency |
hummingbird_sqs_messages_received_total |
Counter | — | SQS messages pulled |
hummingbird_sqs_messages_processed_total |
Counter | status | SQS messages processed |
Service-specific metrics
| Metric | Type | Labels | Description |
|---|---|---|---|
hummingbird_workqueue_work_items_created_total |
Counter | item_type | Work items submitted via API |
hummingbird_workqueue_work_items_completed_total |
Counter | item_type, result | Action executions completed (success/failed) |
hummingbird_workqueue_work_items_retried_total |
Counter | item_type | Work items retried via API |
hummingbird_workqueue_action_duration_seconds |
Histogram | action | Executor action duration |
hummingbird_workqueue_active_processing_threads |
Gauge | — | Running work item threads |
hummingbird_workqueue_processing_lock_skipped_total |
Counter | item_type | Loops that lost the per-item session lock |
hummingbird_workqueue_pending_events_inserted_total |
Counter | — | Pending events inserted into DB |
hummingbird_workqueue_ci_jobs_retried_total |
Counter | project, matcher_name | CI jobs retried by the herder |
hummingbird_workqueue_ci_retry_skipped_total |
Counter | project, matcher_name, reason | CI job retries skipped |
hummingbird_workqueue_merge_rechecks_total |
Counter | merge_status | Fast-family merge recheck cycles by status |
UUID path segments in HTTP metrics are normalized to /{id} to limit
label cardinality. The /healthz and /metrics paths are excluded
from HTTP metrics.
CLI
export WORKQUEUE_SERVICE_URL=https://mr.apps.cluster.example.com
export WORKQUEUE_SERVICE_TOKEN=$(oc whoami -t)
# Update an existing file (local_path:repo_path)
workqueue-service submit \
--package kernel \
--project org/group/project \
--change-kind sync \
--file /path/to/sources.spec:sources.spec \
--commit-message "Sync sources from upstream" \
--source-branch mr-service/sync-1 --title "Sync sources"
# Create a new file
workqueue-service submit \
--package kernel \
--project org/group/project \
--change-kind sync \
--create /path/to/new-file.txt:new-file.txt \
--commit-message "Add new file" \
--source-branch mr-service/add-1 --title "Add file"
# Check status
workqueue-service status --id <uuid>
# List active MR items
workqueue-service status --phase active
# Show identity and permissions
workqueue-service whoami
# Raw JSON output
workqueue-service whoami --json
--url overrides WORKQUEUE_SERVICE_URL. --no-auth skips all
credential loading and sends unauthenticated requests (like
aws --no-sign-request). Useful with whoami to verify anonymous
access.
CLI authentication
The CLI resolves credentials in this order:
WORKQUEUE_SERVICE_TOKENenv var — used directly as a Bearer token. Recommended for CI (OIDC/projected SA tokens).- Cached token — if a valid token exists in the per-server cache, it is reused automatically.
- OAuth/Kerberos login — when neither env var nor cache is
available, the CLI performs a synchronous headless SPNEGO login
against the oauth-proxy, exchanges the session cookie for an OCP
Bearer token via
POST /oauth/token, and caches it for subsequent calls. Requiresrequests-gssapiand a valid Kerberos ticket (kinit).
Tokens are cached per-server under
$XDG_CACHE_HOME/workqueue-service/tokens/ (defaults to
~/.cache/workqueue-service/tokens/) with 0600 permissions and a
20-hour TTL. If a cached token is rejected (HTTP 401), the CLI
invalidates the cache and retries once with a fresh OAuth login.
Configuration
Config file
The service reads its configuration from a YAML file at startup. The
path defaults to /etc/workqueue/config.yml and can be overridden with
WORKQUEUE_SERVICE_CONFIG. The service refuses to start if the file is missing
or invalid (fail-closed).
authorization:
rules:
# Any in-cluster SA with the right audience can read
- issuer: k8s
claims:
aud: workqueue-service
permissions: [read]
# Human users -- OCP group membership
- groups:
- konflux-hummingbird-admin-access
permissions: [read, submit, retry]
# Specific K8s SA for CI automation
- issuer: k8s
claims:
aud: workqueue-service
sub: system:serviceaccount:hummingbird--internal:rpms-ci
permissions: [submit, retry]
item_types: [mr]
# GitLab CI OIDC
- issuer: https://gitlab.com
claims:
aud: workqueue-service
project_path: redhat/hummingbird/rpms
permissions: [read, submit, retry]
item_types: [mr]
handled_types:
mr:
gitlab_url: https://gitlab.com
projects:
redhat/hummingbird/rpms:
token_env: GITLAB_TOKEN_RPMS
redhat/hummingbird/containers:
token_env: GITLAB_TOKEN_CONTAINERS
redhat/rhel/src:
token_env: GITLAB_TOKEN_RHEL
Authorization rules
Each rule has either groups (OCP group membership from users/~
response) or issuer+claims (JWT bearer), not both.
issuerdetermines which JWKS keys validate the token signature. Usek8sas shorthand for the auto-detected cluster OIDC issuer.claimsmatches JWT payload fields. All conditions must match (AND); list values match any (OR). Values are compared case-insensitively with string coercion (handles numericproject_id, booleanref_protected).permissionsgrantsread,retry, and/orsubmit.item_typesoptionally restricts the rule to specific work item types. Omit to apply to all types.- Every issuer rule must include
audinclaimsto prevent cross-service token reuse.
Handled types
handled_types declares which work item types this instance serves.
Only listed types get their routers and executors registered. The
processing sweep only picks up items matching these types. Empty or
omitted = handle nothing.
Each type can have per-type operational config:
| Type | Field | Description |
|---|---|---|
mr |
gitlab_url |
GitLab instance URL (default: https://gitlab.com) |
mr |
dashboard_base_url |
Dashboard URL for note creation (when unset, notes are not created) |
mr |
job_retry |
CI job retry matcher config (when unset, retries are disabled) |
mr |
projects |
Per-project/group token and URL configuration |
Per-project token configuration
The projects map under handled_types.mr controls which GitLab API
token the executor uses for each target project. Keys are GitLab
project paths or group prefixes. Every project submitted to the service
must match an entry – there is no default fallback token.
Matching order:
- Exact match on the full project path
- Longest group-prefix match on a
/boundary (e.g.redhat/rhel/srcmatchesredhat/rhel/src/kernel)
Each entry requires token_env (the name of the environment variable
holding the GitLab API token) and optionally overrides gitlab_url:
handled_types:
mr:
gitlab_url: https://gitlab.com
projects:
redhat/hummingbird/rpms:
token_env: GITLAB_TOKEN_RPMS
redhat/hummingbird/containers:
token_env: GITLAB_TOKEN_CONTAINERS
gitlab_url: https://gitlab.example.com
redhat/rhel/src:
token_env: GITLAB_TOKEN_RHEL
At startup, the service validates that all referenced token_env
variables exist in the process environment and refuses to start if any
are missing.
Adding a new issuer
Add a rule with the issuer’s OIDC discovery URL. The service discovers the JWKS endpoint lazily on the first token from that issuer:
- issuer: https://token.actions.githubusercontent.com
claims:
aud: workqueue-service
repository: org/repo
permissions: [submit]
types: [mr]
Environment variables
| Variable | Default | Description |
|---|---|---|
DATABASE_URL |
— | PostgreSQL connection URL (required) |
SENTRY_DSN |
— | Sentry DSN for error reporting |
WORKQUEUE_SERVICE_CONFIG |
/etc/workqueue/config.yml |
Config file path |
WORKQUEUE_SERVICE_LOCAL_AUTH_USER |
— | Dev-only auth bypass (warns at startup) |
WORKQUEUE_SERVICE_URL |
— | CLI: service URL |
WORKQUEUE_SERVICE_TOKEN |
— | CLI: Bearer token |
(per-project token_env) |
— | GitLab API token(s) for MR executor (set in config projects) |
WORKQUEUE_SERVICE_EVENTS_QUEUE_URL |
— | SQS queue URL for events (when unset, event consumer is disabled) |
KONFLUX_KUBECONFIG_PATH |
— | Path to Konflux kubeconfig for PLR fallback fetch |
KUBEARCHIVE_URL |
— | Kubearchive API URL for historical PLR lookups |
WORKQUEUE_SERVICE_AUDIT_S3_BUCKET |
— | S3 bucket for audit log (production) |
WORKQUEUE_SERVICE_AUDIT_S3_PREFIX |
workqueue-service-audit |
Key prefix within the audit S3 bucket |
AWS_ACCESS_KEY_ID |
— | AWS credentials for S3 audit archival |
AWS_SECRET_ACCESS_KEY |
— | AWS credentials for S3 audit archival |
AWS_DEFAULT_REGION |
— | AWS region for S3 audit archival |
Development
Local Development
cd workqueue-service
./dev.sh db-start # Start local PostgreSQL
./dev.sh db-migrate # Run Alembic migrations
./dev.sh start # Start the service
./dev.sh db-shell # PostgreSQL shell
./dev.sh db-stop # Stop PostgreSQL
./dev.sh db-reset # Stop and delete volume
Running Tests
cd workqueue-service
pip install -e ".[dev]"
python -m unittest discover tests
Tests use testcontainers[postgres] locally (auto-detects Podman) or a
CI-provided PostgreSQL sidecar via DATABASE_URL.
License
This project is licensed under the GNU General Public License v3.0 or later — see the LICENSE file for details.
4.5.33.1 - Work Queue Service Design
Architectural design document for workqueue-service. Covers the
reasoning behind every major design choice so that future changes can be
made safely. For operational usage, see
Work Queue Service.
1. Design Philosophy
Four principles shaped the service’s architecture:
Security boundary. The service sits between untrusted callers (AI agents, CI jobs, automation scripts) and trusted write operations (GitLab pushes, MR creation). All changes are inspectable structured data, not opaque blobs. Path blocklists and action validation run before any GitLab write.
Type-agnostic queue, type-specific behaviour. The processing loop and per-item locking (a loop-lifetime session advisory lock) are generic. Type-specific logic (MR creation, CVE remediation) lives in pluggable subpackages that register deciders and executors at import time. Adding a new work item type requires no changes to the generic layer.
Single pending action. Only one action per work item at a time. After each action, the decider re-evaluates with current state. This avoids queuing actions that assume a future state which may not hold.
Phase + dimensions, not a linear state machine. The coarse lifecycle (active/completed/failed/cancelled) is separate from independent readiness signals (CI, approval, Konflux build). Dimensions can regress (force-push resets CI), which a linear state machine cannot express.
2. Architecture
Callers submit work items via REST API. The ProcessingManager drains
pending actions in per-item threads, dispatching to type-specific
executors. Executors create and update GitLab MRs via the Commits API.
flowchart LR
callers["Callers\n(CI, CLI, agents)"] --> API["REST API\n+ Auth"]
API --> DB[("PostgreSQL")]
Processing["ProcessingManager"] --> DB
Processing --> GL["GitLab\nCommits API"]
API -.-> Processing
3. Design Decisions Record
See Work Queue Service Design Decisions for all DDR entries covering data model, auth, processing, event pipeline, and MR lifecycle decisions.
4. Future Work
Event-driven dimension updates
The dimension mechanism is implemented with all MR dimensions following
the uniform needs_* model (DDR-48). MR state events drive phase
transitions (merge → completed, close → cancelled). Konflux
PipelineRun events merge per-component state into the needs_plr
dimension’s JSONB detail and re-arm needs_dashboard to trigger
dashboard note updates. A 30-minute fallback executor
(ensure_plr_status) fetches current PLR state from K8s/Kubearchive
to catch missed events.
Non-managed MR tracking
Design how the service handles MRs it did not create (MRs without the
managed-by::workqueue label). Requires subscribing to note events
so the service can discover dashboard notes posted by the Lambda and
adopt them as needs_dashboard dimensions. Prerequisite for retiring
the Lambda.
Status rendering
The ensure_dashboard_note executor now renders a per-component
Konflux build status table (using needs_plr dimension JSONB detail
and the expected components from needs_expected_plrs) and updates
the existing note in place via note_id.
CVE work item type
Add cve/ subpackage with CveWorkItem, CVE-specific decider
(analyze → label → create advisory MR → check VEX → close), and
CVE-specific executors. No changes to the generic layer.
Content validation and size limits
Add content-level inspection to the pre-commit checks: maximum file size, binary detection, and content pattern matching. Currently only path blocklists and action validation are enforced.
Reconciliation sweep
Note: The reconcile_mr executor (DDR-46) now provides per-item
reconciliation. A full sweep (querying GitLab for all active items in
bulk) remains a future optimization for catching items that fall
through individual reconciliation.
Service rename
The service was renamed from hummingbird-mr-service to
workqueue-service to reflect its generalized purpose. A further rename
to hummingbird-work-queue or similar is deferred until a second work
item type is implemented.
4.5.33.2 - Work Queue Service Design Decisions
Design Decision Registry (DDR) for workqueue-service — every major design
choice, the alternatives considered, and why the chosen approach won. For the
design overview, see
Work Queue Service Design. For operational
usage, see Work Queue Service.
DDR-1: Phase + dimensions vs linear state machine
Decision: Replace the 12-state linear state machine with a phase + dimensions model.
Context: The initial design used a single state column:
pending → in_progress → created → mr_ci → approved → merged → post_merge_ci → building → releasing → completed. This broke because
post-creation concerns (CI, approval, Konflux build) are independent —
CI can pass before approval, approval can be revoked, force-push resets
CI but not approval.
Alternatives: (a) Composite states (ci_passed_approval_pending) —
exponential explosion. (b) Multiple boolean columns — rigid, schema
migration for each new signal.
Rationale: Phase tracks coarse lifecycle; independent dimensions in a separate table track readiness signals with variant support for multi-instance cases (per-component Konflux builds). Dimensions can regress naturally. New dimension types need no schema migration.
DDR-2: Four generic phases (active/completed/failed/cancelled)
Decision: Narrow from 6 MR-specific phases to 4 generic phases.
Context: The intermediate design had pending, active, merged,
completed, failed, closed — mixing generic concepts with
MR-specific milestones.
Alternatives: Keep MR-specific phases and add more for each type.
Rationale: Phase answers “is this item done, and how?” Type-specific
milestones (MR merged, CI passed) are tracked via dimensions and
detail-table columns. Items start as active — there is no pending
phase because the decider immediately assigns an action at creation time.
DDR-3: Dimensions in a separate table with variant support
Decision: Store dimensions in work_item_statuses table, not as
columns on work_items.
Context: Initially considered a column per dimension
(ci_status, approval_status, etc.).
Alternatives: (a) Column per dimension — schema migration for each
new one. (b) JSONB column — flexible but loses CHECK constraints and
complicates queries. (c) Separate table with composite PK
(work_item_id, dimension, variant).
Rationale: Option (c). New dimensions need no migration. Variants handle multi-instance signals (multiple Konflux components). Each work item only has rows for dimensions that apply to it. The “are all dimensions satisfied?” query is a natural GROUP BY/HAVING.
DDR-4: Decider as pure function, not state machine transitions
Decision: The decider is a pure function
decide(item) → action | None registered per type. It only determines
the next action to execute.
Context: The original design had a TRANSITIONS dict mapping
state → frozenset[state] with validate_transition().
Alternatives: (a) Transition table — rigid, can’t express “depends on which dimensions passed”. (b) Decider function — flexible, testable.
Rationale: The decider is trivially testable (given state, assert action). State transitions come from executors (they apply phase changes directly) and events (they update dimensions). The decider just picks the next action based on current state.
DDR-5: Single pending action, not a queue
Decision: One pending action per work item at a time.
Context: Considered queuing multiple actions (“merge, then trigger Konflux build”).
Alternatives: Action queue table per work item.
Rationale: Queued actions assume future state that may not hold. If
“merge” fails, the queued “trigger Konflux build” is nonsense. After
each action, the decider re-evaluates with current state. The attempts
counter tracks consecutive failures, not total actions.
DDR-6: Claiming via leased_until (reusable)
Superseded by DDR-51. Work items are owned by a loop-lifetime session advisory lock, and the contract migration drops the
leased_untilcolumn. See DDR-51 for the design and its motivation.
DDR-7: Generic + detail tables (joined table inheritance)
Decision: Split work_items into generic queue table + per-type
detail tables (mr_details, future cve_details).
Context: The initial design had all MR-specific columns on the
work_items table.
Alternatives: (a) Single table with nullable type-specific columns — sparse, doesn’t enforce type constraints. (b) JSONB payload column — flexible but loses type safety. (c) Joined table inheritance.
Rationale: Option (c). SQLAlchemy’s polymorphic_load="selectin"
automatically loads the correct subclass. New types add their own detail
table — no changes to the generic layer. Type-specific columns have
proper types and constraints.
DDR-8: Type-dispatched decider/executor registries
Decision: Registry pattern for deciders and executors. Each type
registers at import time via decider.register("mr", _mr_decide).
Context: Needed a way to dispatch to type-specific logic without the generic layer knowing about types.
Alternatives: (a) if/elif chains in the processing loop. (b) Class-based dispatch (strategy pattern). (c) Registry functions.
Rationale: Option (c). Simple, explicit, no class hierarchy needed.
The mr/__init__.py import triggers registration. New types just add
their own register() calls.
DDR-9: Structured file actions vs git patch_data
Decision: Replace opaque patch_data (BYTEA, git format-patch
output) with structured file_actions (JSONB) and commit_message.
Context: The MR service sits on the security boundary — it writes to GitLab repos on behalf of untrusted callers. HUM-851 and the AI Investigation design rules require pre-commit inspection (path blocklists, content validation).
Alternatives: (a) Keep patch_data, clone repo, apply patch, then
inspect diff — inspection happens after creating a working tree with
credentials. (b) Parse patches with unidiff library — gives diffs not
full content, still needs original files. (c) Structured file actions.
Rationale: Option (c). Inspection is a pure function over data the
service already has — no clone, no git binary, no temp dirs. Path
blocklists and action validation run before any API call. The
file_actions JSONB column is also a queryable audit record.
DDR-10: GitLab Commits API vs git subprocess
Decision: Use the GitLab Commits API for branch creation and file
commits. Delete git_ops.py.
Context: The original executor shelled out to git (clone, am,
push), requiring: git binary in the container, subprocess timeout
handling, GIT_ASKPASS credential management, temp directory lifecycle.
Alternatives: (a) Keep git subprocess with GIT_ASKPASS and timeout. (b) GitLab Commits API.
Rationale: Option (b). gitlab_sync.py in the same repo already
proves the pattern. The Commits API’s start_branch parameter creates
the branch automatically. No subprocess, no credentials in URLs, no temp
dirs. Callers have full file contents (not patches), so git am
semantics aren’t needed.
DDR-11: OIDC issuer from discovery, not hardcoded
Decision: Read the OIDC issuer from the /.well-known/openid-configuration
discovery response. Rewrite the JWKS URI to use kubernetes.default.svc.
Context: The initial implementation hardcoded
issuer = "https://kubernetes.default.svc". This failed on EaaS clusters
where --service-account-issuer is
https://oidc.op1.openshiftapps.com/.... The JWKS URI from discovery
pointed to an IP:6443 that was unreachable from pods.
Alternatives: (a) Environment variable override
(KUBE_OIDC_ISSUER). (b) Read from discovery.
Rationale: Option (b). No configuration needed — works on any cluster
automatically. The JWKS fetch uses the path from discovery but rewrites
the host to kubernetes.default.svc (always reachable on port 443). SA
bearer token authenticates both requests (OpenShift blocks anonymous
JWKS access).
DDR-12: item_type not type
Decision: Name the discriminator column item_type everywhere (DB,
ORM, API, schemas).
Context: type is a Python builtin. Using it as a function
parameter triggers ruff A002.
Alternatives: (a) Use type with alias in different contexts.
(b) Rename to item_type everywhere.
Rationale: Option (b). Clean rename, no aliases, no special casing. Consistent at every layer.
DDR-13: source field removed
Decision: Remove the source column from mr_details. The
caller_identity field (set automatically from authentication) serves
the same purpose.
Context: source was a user-declared label (“rpms-ci”,
“dependency-updater”) while caller_identity is the authenticated
identity. Having both was redundant.
Alternatives: Keep both for cases where a shared SA submits on behalf of different logical sources.
Rationale: Removed until needed. caller_identity captures who
submitted. If per-source filtering is needed later, it can be
re-introduced as a field on MRWorkItemCreate.
DDR-14: attempts counts consecutive failures, not total actions
Decision: attempts increments only on executor failure and resets
to 0 on success.
Context: The initial implementation incremented attempts before
every action execution. A work item chaining 4 actions (ensure_mr →
merge → trigger_build → complete) would exhaust MAX_ATTEMPTS on the
third action.
Alternatives: (a) Separate total_actions counter. (b) Only count
failures.
Rationale: Option (b). attempts answers “how many consecutive
times has this action failed?” — the signal for escalation. Total
actions executed is an audit concern, not a processing concern.
DDR-15: Executor applies state directly; decider only picks next action
Decision: Executors directly modify item.phase, item.mr_iid,
etc. The decider only returns the next action to execute.
Context: Initially the processing loop set the phase based on
ExecutorResult.success. This mixed two concerns.
Alternatives: (a) Processing loop sets phase. (b) Decider sets phase. (c) Executor sets phase.
Rationale: Option (c). The executor knows what happened (“I created an MR, so phase stays active”). Events that update dimensions also directly set state. The pattern is the same: something happens → state is updated → decider evaluates for the next action. Symmetry with event-driven dimension updates.
DDR-16: Items stay active after MR creation
Decision: After the ensure_mr executor succeeds, the item remains
in active phase with no further actions pending.
Context: After MR creation, the decider returns None (no action).
The item sits in active indefinitely.
Alternatives: (a) Set phase to completed after MR creation.
(b) Keep active for future event-driven lifecycle tracking.
Rationale: Option (b). The active phase covers the entire lifecycle
— from submission through CI, approval, merge, and post-merge tracking.
The decider returns no actions until event-driven dimension updates
(future SQS consumer) trigger the next step.
DDR-17: Audit log to S3, not database
Decision: Service-level audit (state transitions, decider decisions, actions taken) goes to S3 as immutable objects.
Context: Raw webhook events are already archived by
sns-s3-archiver. The service needs its own audit trail for state
transitions and decider evaluations.
Alternatives: (a) Database table (work_item_events). (b) S3
immutable objects. (c) Both.
Rationale: Option (b). S3 is immutable and cheap. The operational
DB stays lean. Raw events are already in S3 via sns-s3-archiver;
service-level audit captures the service’s reaction to those events.
DDR-18: Agent direction — dumber, not smarter
Decision: The hummingbird-agent should become a pure compute engine. The work queue service is the MR lifecycle orchestrator. The agent never calls the work queue service; the work queue service calls the agent.
Context: HUM-857 proposed adding create_merge_request as a new
action in the agent’s actions.py. This was the wrong direction.
Alternatives: (a) Agent gains more actions (push, merge, MR management). (b) Agent becomes stateless compute; work queue service orchestrates.
Rationale: Option (b). The agent keeps LLM loop, tool registry, and sandbox — nothing else. Event routing, rate limiting, placeholder notes, result posting, session management, and all GitLab writes move to the work queue service. HUM-857 closed as Obsolete.
DDR-19: Shared database
Decision: The work queue service uses the same events PostgreSQL
database as hummingbird-status and hummingbird-dashboard.
Context: Cross-service joins (e.g. work_items JOIN
gitlab_merge_requests JOIN pipelineruns) are useful for the dashboard.
Alternatives: Separate database per service.
Rationale: Shared DB with service-specific Alembic migration chains
(alembic_version_mr_service). Each service manages its own tables
independently. The migration chain is isolated so one service can evolve
its schema without affecting others.
DDR-20: Pydantic config over cki-lib/jsonschema
Decision: Use Pydantic for config validation. Load config from a YAML file at startup.
Context: Needed structured config for authorization rules and per-type operational settings. Alternatives: cki-lib config helpers, jsonschema, or plain dicts.
Alternatives: (a) cki-lib — adds a dependency, the service stays self-contained without it. (b) jsonschema — separate schema file, no type-safe access. (c) Pydantic — already a dependency (API schemas).
Rationale: Option (c). Pydantic provides type-safe attribute access, defaults, cross-field validators (e.g. mutual exclusivity of groups vs issuer+claims), and optional JSON Schema export. No new dependencies.
DDR-21: Claim-based rules (Vault bound_claims pattern)
Decision: Authorization rules match on arbitrary JWT claims, not
just sub. All claim conditions AND; list values OR.
Context: Studied JWT trust/authz config from AWS API Gateway, HashiCorp Vault, Kubernetes Structured Auth, Istio, Traefik Hub, and oauth2-proxy. All support matching on arbitrary claims.
Alternatives: (a) Subject allowlist — rigid, requires exact sub
values. (b) Claim-based matching — flexible, supports project_path,
ref, groups, etc.
Rationale: Option (b). Vault’s bound_claims pattern is proven and
simple. Values are coerced to lowercase strings before comparison to
handle int/string mismatches (GitLab’s numeric project_id) and
bool/string mismatches (ref_protected: true).
DDR-22: Issuer per rule, audience as claim
Decision: issuer is the only special field on a token rule — it
determines which JWKS keys to use. All other JWT conditions (including
aud) are standard claim matches in the claims dict.
Context: Considered a separate jwt.issuers section and/or treating
aud as a first-class field alongside issuer.
Alternatives: (a) Global jwt section with trusted issuers +
separate rules section. (b) aud as a dedicated field on each rule.
(c) Issuer per rule, audience as a claim.
Rationale: Option (c). Keeps each rule self-contained and
self-documenting. The Pydantic validator enforces that every issuer rule
includes aud in claims to prevent cross-service token reuse.
DDR-23: Unified rule model for groups and tokens
Decision: A single rules list where every rule maps a remote
identity (groups or issuer+claims) to local permissions.
Context: The original design had admin_groups/retry_groups
(local permission → remote groups) alongside rules (remote identity →
local permissions) — opposite mapping directions.
Alternatives: Keep separate group and token rule formats.
Rationale: Unified model. Every rule follows the same structure: remote identity → local permissions. Pydantic cross-field validators enforce mutual exclusivity.
DDR-24: Bearer-only auth with users/~ self-authentication
Decision: All API auth uses Authorization: Bearer tokens. Opaque
OCP tokens (sha256~...) are validated by calling the OCP users/~
API with the user’s own token (self-authenticating).
Context: The oauth-proxy cannot simultaneously forward cookie-session
tokens AND pass through non-OCP Bearer tokens on the same path. The
TokenReview API would be cleaner but requires system:auth-delegator
ClusterRoleBinding — a cluster-scoped permission not available on
managed OCP clusters.
Alternatives: (a) Trust X-Forwarded-User from the proxy on all
paths — injection risk on bypassed paths. (b) TokenReview API —
requires cluster RBAC. (c) users/~ self-authentication.
Rationale: Option (c). No special RBAC needed. The users/~
endpoint returns username and groups. Results are cached with a
thread-safe TTL cache (5 minutes). The /me endpoint bridges OAuth
cookie sessions to Bearer tokens.
DDR-25: handled_types as explicit opt-in
Decision: handled_types declares which work item types this
instance serves. Empty means handle nothing.
Context: Different deployments may serve different types from the
same container image. Types carry per-type config (e.g. gitlab_url
for MR).
Alternatives: (a) Handle all types by default. (b) Explicit opt-in.
Rationale: Option (b). Prevents accidental catch-all deployments.
The sweep returns early when handled_types is empty — no items stolen
from other instances in a shared-database deployment.
DDR-26: k8s shorthand skipped when not in-cluster
Decision: Rules with issuer: k8s are silently skipped when the
in-cluster CA bundle is absent.
Context: The K8s OIDC issuer URL varies by cluster and is auto-detected at runtime. Local development doesn’t have a cluster.
Alternatives: (a) Error on startup if k8s issuer can’t be
resolved. (b) Skip silently.
Rationale: Option (b). Allows using a production config file for local development and testing without startup errors.
DDR-27: OAuth proxy as login-only bridge with /me endpoint and cookie refresh
Decision: The oauth-proxy sidecar handles only the OAuth login flow.
API endpoints bypass the proxy. A proxy-protected /me endpoint
bridges cookie sessions to Bearer tokens. The proxy must be configured
with --cookie-refresh=1h to prevent stale-token failures.
Context: The OCP oauth-proxy cannot simultaneously forward
cookie-session tokens AND pass through non-OCP Bearer tokens on the
same path. Protected paths validate Bearer tokens via TokenReview
(requires cluster RBAC we cannot obtain) or reject non-OCP tokens.
Bypassed paths pass all Bearer tokens through but ignore cookies.
Additionally, the OCP access token embedded in the proxy’s session
cookie has its own expiry (default 24h), independent of the cookie
lifetime. Without --cookie-refresh, the cookie outlives the token,
causing the proxy to forward expired credentials while still reporting
the user as authenticated — the “logged in but not authorized” failure
observed in the hummingbird-dashboard deployment.
Alternatives: (a) All paths protected — blocks non-OCP Bearer
tokens. (b) --openshift-delegate-urls — requires
system:auth-delegator ClusterRoleBinding. (c) App reads proxy cookies
directly — tight coupling to proxy internals. (d) Login-only proxy
with /me bridge.
Rationale: Option (d). The /me endpoint is the only
proxy-protected API path. It reads X-Forwarded-Access-Token (set by
the proxy from the cookie session) and returns the OCP access token to
the caller. Subsequent API calls use Authorization: Bearer sha256~...
on bypassed paths, where the app validates via the OCP users/~ API.
--cookie-refresh=1h keeps the embedded token fresh. TTL-bounded
caching (5 minutes) on users/~ results ensures expired-token failures
are never permanent (unlike @functools.cache which caches failures
indefinitely).
DDR-28: Audit log with logging + optional S3 archival
Decision: Always log audit entries via Python logging at INFO
level, and optionally archive to S3 as gzipped JSON objects when
WORKQUEUE_SERVICE_AUDIT_S3_BUCKET is set.
Context: Needed an immutable record of state transitions, action outcomes, and auth decisions for compliance and debugging.
Alternatives: (a) Database table — bloats the operational DB with a different concern. (b) S3-only — no visibility without querying Athena; loses events if S3 is unreachable.
Why logging + S3: Logging to stdout is zero-config and integrates with existing container log collection for real-time visibility. S3 provides the durable, queryable archive (Athena) for compliance.
Key design rules:
- Audit failures never propagate —
write_audit_entry()catches all exceptions and logs them, but never raises. - S3 keys use
YYYY/MM/DD/HH/MM/prefix for Athena partitioning. ContentEncoding: gzipset so S3 API consumers get transparent decompression.
Field conventions: See Audit Log Field Conventions in the operational documentation.
DDR-29: Events processed in work-item threads, not consumer threads
Decision: The SQS consumer thread stores events in the
pending_events table. Existing per-work-item threads drain and
process them.
Context: The ProcessingManager already runs one thread per work item with per-item ownership (DDR-51). Processing events in the same thread means events and actions are naturally serialized per work item.
Alternatives: (a) Consumer threads handle events directly — concurrency conflicts with action threads. (b) FIFO queue + in-memory dispatch — receipt handle management, visibility timeout extensions, message loss on crash.
Rationale: Option (c). Events join the existing thread structure. The decider runs after each event, determining follow-up actions. The single-thread-per-item invariant is preserved.
DDR-30: pending_events table as durable event store
Decision: Use a PostgreSQL table (pending_events) instead of
FIFO queues or in-memory queues for event delivery to work-item threads.
Context: Since work-item threads do the processing, the SQS consumer just needs to deliver events to them durably.
Alternatives: (a) FIFO queue + in-memory dispatch — complex EventBroker, receipt handle management, message loss on crash. (b) FIFO queue + direct handling — concurrency conflicts.
Rationale: The DB provides durability (crash recovery), serialization
(single thread per work item), and ordering (last_event_at guard). No
FIFO queue, router thread, MessageGroupId computation, in-memory queues,
or visibility timeout management needed.
DDR-31: SQS delete after INSERT, not after full processing
Decision: Delete the SQS message after INSERT+COMMIT into
pending_events, not after full event processing.
Context: The hummingbird-agent deletes after handling (no DB intermediary). The workqueue-service can delete after INSERT+COMMIT because the event is crash-recoverable in PostgreSQL.
Alternatives: Delete after full processing — requires visibility timeout extensions, receipt handle passing, heartbeat management.
Rationale: After INSERT+COMMIT the event is durable in PostgreSQL.
Duplicate SQS delivery may INSERT the same event twice; the
last_event_at guard makes processing idempotent.
DDR-32: Resolver filters by phase, consumer stays read-only
Decision: The resolver filters by phase = 'active' on the
MRWorkItem query. SQLAlchemy’s joined table inheritance handles the
work_items table access implicitly. The consumer does SELECT +
INSERT only, no UPDATE, no row lock contention.
Context: Only active items accept event-driven transitions (see
DDR-37). Filtering at resolve time avoids unnecessary INSERTs into
pending_events, thread spawns, and no-op processing for terminal
items.
Alternatives: (a) Resolve without phase filter, guard only at processing time — wastes work for terminal items. (b) Consumer checks phase and applies transitions directly — contention with work-item threads.
Rationale: The phase filter adds negligible cost (PK lookup on
work_items, typically one row). The consumer remains read-only.
Business logic (phase transitions, decider, audit) is applied by the
generic handle_event layer in the work-item thread.
DDR-33: Raw events stored, interpretation separated from application
Decision: The pending_events table stores the raw event JSON
body and event type. Type-specific handlers interpret events (read-only
on the work item), the generic layer applies the result.
Context: Event processing has three concerns: interpretation (what does this event mean?), application (phase transition, decider, audit), and routing (which work items?). Mixing them makes handlers hard to test and inconsistent across types.
Alternatives: (a) Handlers apply all changes directly — mixes interpretation with mutation, each handler reimplements decider/audit boilerplate. (b) Pre-process events in the consumer — consumer needs type-specific knowledge.
Rationale: Handlers return an EventResult dataclass (phase,
event_ts, trigger metadata). The generic handle_event applies the
phase transition via update_work_item_from_event, runs the decider,
and emits audit entries. Handlers have read-only access to the work
item and session (e.g. to check existing dimensions or detail fields)
but EventResult is their only output — they never mutate the work
item directly. This ensures decider/audit logic is consistent across
all item types. See DDR-38 for the dimension delivery extension.
DDR-34: Per-work-item advisory lock for insert/drain coordination
Superseded by DDR-51. The insert/exit handshake was replaced by a
SELECT ... FOR UPDATErow lock onwork_itemsaround the exit decision, which serializes every enqueue writer. See DDR-51 for the design and its motivation.
DDR-35: Sweep for startup recovery
Decision: Extend the periodic sweep to also check for work items
with unprocessed pending_events rows.
Context: Events that arrive while the service is stopped, or that survive a mid-processing crash, must be processed on restart.
Alternatives: Separate startup-only scan.
Rationale: Unifying startup recovery with the periodic sweep keeps
a single code path. The sweep runs immediately on manager.start(),
catching events from downtime. It also serves as a safety net behind
the advisory lock.
DDR-36: Consumer lifecycle and shutdown ordering
Decision: The SQS consumer is a single daemon thread started from the FastAPI lifespan. Shutdown stops the consumer before work-item threads.
Context: If work-item threads stop while the consumer is still INSERTing, events could be orphaned. The consumer must stop first.
Alternatives: Non-daemon thread with explicit join.
Rationale: Single thread is sufficient for low event volume (MR
state changes). Daemon thread ensures process exit even if the thread
hangs. Shutdown ordering: (1) set shutdown_event, (2) join consumer
thread (timeout=30s), (3) call manager.stop(). No consumer when
WORKQUEUE_SERVICE_EVENTS_QUEUE_URL is unset.
DDR-37: Terminal phase semantics — only active items accept events
Decision: Only items in phase = 'active' accept event-driven
phase transitions. completed, failed, and cancelled are terminal
phases — recovery requires explicit action via the retry API.
Context: Initially only failed was guarded (phase != 'failed').
completed and cancelled items could be modified by events: a
closed event could transition a completed item to cancelled, and an
opened event could reactivate a cancelled item.
Alternatives: (a) Guard only failed — inconsistent, completed
items are accidentally mutable. (b) Growing exclusion list
(NOT IN ('failed', 'completed')) — fragile if new phases are added.
(c) Positive match phase = 'active'.
Rationale: Option (c). A single positive match is clearer and
future-proof. If a work item was cancelled (MR closed) or completed
(MR merged), only deliberate human action (retry API) should
reactivate it — not an automated event. The retry API accepts both
failed and cancelled items. The resolver also filters by
phase = 'active' at query time (DDR-32) to avoid unnecessary
INSERT/processing for terminal items.
DDR-38: Dimension updates via EventResult
Decision: Event handlers return dimension upserts as part of
EventResult. The generic handle_event layer applies them.
Context: DDR-33 established that handlers return structured results and the generic layer applies them. Dimensions (DDR-1, DDR-3) need the same pattern — handlers identify which dimensions changed, the generic layer upserts them and emits audit entries.
Alternatives: (a) Handlers call crud.upsert_dimension() directly —
breaks the read-only handler contract from DDR-33. (b) Separate
dimension-specific consumer — duplicates the resolver/handler
infrastructure.
Rationale: EventResult.dimensions is a list of DimensionUpdate
dataclasses. After phase transition logic, handle_event iterates over
dimensions and calls crud.upsert_dimension() for each, emitting an
AuditEntry with event_type="dimension_change" using the existing
dimensions field. Dimension upserts run even when the phase transition
is stale (the early return is refactored to skip only the phase/decider
block). This ensures dimension-only events (e.g. note events with no
phase change) are still applied.
DDR-39: Server-derived idempotency keys with phase-aware uniqueness
Decision: Replace the client-supplied idempotency_key field with a
server-derived key mr:{project}:{source_branch}:{generation}. Enforce
uniqueness only among non-terminal phases via a partial unique index.
Context: The original API required callers to supply a unique
idempotency_key per submission. Recurring callers (dist-git sync,
upstream sync) had to invent unique keys for each invocation and track
them across runs. The global unique constraint also blocked new
submissions for a branch that had a previous completed or cancelled item
with the same key.
Alternatives: (a) Client-supplied key with a prefix convention —
callers still need to generate and track keys. (b) Server-derived key
from the natural deduplication identity (project, source_branch) with
a generation counter for lifecycle restarts.
Rationale: Option (b). The idempotency key is a DB-level safety net,
not a caller concern. The natural identity is (project, source_branch);
the generation counter increments when a new item is created after the
previous one reached a terminal phase. The partial unique index
(WHERE phase NOT IN ('completed', 'cancelled')) ensures at most one
active/failed item per idempotency key while allowing completed items to
coexist with their successors.
DDR-40: Upsert semantics for recurring MR management
Decision: Replace one-shot insert_mr_work_item with
upsert_mr_work_item that looks up an existing active/failed item by
(project, source_branch) and determines the correct action from a
content hash comparison.
Context: Recurring callers submit MRs for the same branch repeatedly. Under the old insert model, each submission created a new work item — even when the previous one was still active, leading to orphaned branches and duplicate MRs.
Alternatives: (a) Client manages deduplication — check before submit, handle conflicts. (b) Server-side upsert — callers always submit current desired state, the service deduplicates transparently.
Rationale: Option (b). The upsert does a
(project, source_branch) lookup among non-terminal items and branches
on the result:
- No match → create a new item (
actionable = true) - Active, same content → no-op (return existing,
"unchanged") - Active, different content → update fields, set
actionable = true, resetneeds_mr = "true" - Failed → reset to active, update content, clear error (
"retried"), setactionable = true - Terminal (completed/cancelled) → create a new generation (DDR-39)
Concurrent insert races are handled via DDR-39’s partial unique index —
the loser catches the IntegrityError, rolls back, and retries.
DDR-41: ThreadPoolExecutor with bounded workers
Decision: Replace unbounded per-item threading.Thread spawning
with a fixed-size ThreadPoolExecutor (10 workers).
Context: The periodic tick mechanism (DDR-42) can wake many idle items simultaneously. Unbounded thread creation would cause resource spikes (~109 threads in the worst case).
Alternatives: (a) Keep per-item threads with a semaphore. (b) Bounded thread pool.
Rationale: Option (b). ThreadPoolExecutor provides built-in
queuing when all workers are busy, deduplication via _submitted set,
and clean shutdown via pool.shutdown(wait=True). The 10-worker limit
caps concurrent GitLab API calls.
DDR-42: Hash-bucketed tick events for periodic re-evaluation
Superseded by DDR-48. The synthetic
tick_30mandtick_5mevents were replaced by thepending_untilcolumn onwork_item_statuses. See DDR-48 for the design and its motivation.
DDR-43: Empty diff detection via repository_compare
Decision: Detect and handle empty MR diffs at two call sites:
ensure_mr (prevention on create and force-push) and check_empty_diff
(periodic via sweep).
Context: The detection-execution race causes empty MRs when the target branch moves between change detection and MR creation. Also, existing MRs can become empty after creation.
Alternatives: (a) Compare file content hashes before commit. (b)
GitLab repository_compare API after commit/push.
Rationale: Option (b). The repository_compare endpoint returns
the diff between two branches. Empty diffs list means the branches
are identical. This is authoritative and handles edge cases that hash
comparison might miss (e.g. whitespace normalization). Cost: one API
call per check (~4 calls/minute across all items for periodic checks).
DDR-44: Per-item processing deadline
Decision: Exit the work item loop after 60 seconds of wall-clock time.
Context: The tick mechanism (DDR-42) and check_stale_mr action
could cause extended processing loops for items that repeatedly generate
new actions. A deadline prevents any single item from holding a pool
worker indefinitely.
Alternatives: (a) Limit loop iterations. (b) Wall-clock deadline.
Rationale: Option (b). Wall-clock time is predictable regardless of action duration. The deadline is between loop iterations, not mid-executor, so executors have their own HTTP timeouts. The item is re-picked by the next sweep if work remains.
DDR-45: Dimension-driven decider with needs_* flags
Decision: The decider scans needs_* dimensions in priority order
instead of encoding per-concern conditionals.
Context: The original decider checked item.mr_iid is None for
create, dashboard_link dimension absence for dashboard notes, and
would need more checks for auto-merge, stale detection, metadata sync.
Each new concern added another conditional.
Alternatives: (a) Per-concern conditionals in decider. (b)
Dimension-driven: events/executors set needs_* flags, decider scans
them.
Rationale: Option (b). The decider becomes a simple priority scan.
Executors own their trigger-to-next-step logic by setting the
appropriate needs_* dimension. Events set needs_reconcile to
request re-evaluation. Adding a new concern = new dimension + handler
in reconciler, no decider changes needed.
Update (DDR-48): The original three-category model (has_*,
needs_*, observed state) was unified into a single needs_* trigger
model with "true"/"false" status. See DDR-48.
DDR-46: reconcile_mr pattern
Updated by DDR-52. The “merge latency to seconds” trade-off below did not hold on busy target branches: the branch churn (largely the workqueue’s own fleet) resets mergeability to
uncheckedfaster than the 30-minute cadence re-samples it, starving managed MRs for days. DDR-52 adds the deadline-driven sweep and an adaptive recheck backoff that remove this failure mode; the direct-merge decision itself stands.
Decision: Single reconcile_mr executor handles all MR state
convergence, with cheap (always) and expensive (flagged) tiers.
Uses direct mr.merge() instead of GitLab auto-merge.
Context: Separate executors per concern (check_stale_mr,
enable_auto_merge, sync_metadata) each needed their own
registration, decider rule, and dimension tracking. GitLab auto-merge
(auto_merge=True) introduced state-management complexity:
merge_when_pipeline_succeeds tracking, stale SHA detection after
force-push, and recurring 422 noise for non-mergeable MRs.
Rationale: One mergerequests.get() call serves all cheap checks.
The reconciler checks detailed_merge_status == "mergeable" and calls
mr.merge() directly — no GitLab auto-merge state to track, no
skip-lists to maintain, no stale SHA detection. Pipeline completion
and approval webhooks trigger reconcile via the needs_reconcile
dimension, keeping merge latency to seconds. Expensive operations
(empty diff via repository_compare) only run when needs_empty_check
is actionable (periodic via pending_until, DDR-48). The reconciler
is idempotent — all corrections are conditional on actual vs desired
state.
DDR-47: CI job retry herder
Updated by DDR-48. The
needs_ci_retry="pending_tick_5m"status andtick_5mpromotion mechanism have been replaced bypending_untilon the dimension row.
Problem: Hummingbird CI pipelines frequently fail due to transient infrastructure errors (runner system failures, git clone timeouts, OOM kills). Manual retries waste developer time and delay merges.
Options considered:
- A. Executor action in workqueue-service (selected): Shares existing GitLab tokens, project config, DB-persisted state, and audit trail.
- B. Standalone herder service: Maximum CKI fidelity but requires a new deployment, duplicates token management, and has no MR lifecycle integration.
- C. Reconciliation polling: No new webhooks, but 30-minute detection latency and wastes API calls polling healthy pipelines.
Decision: Option A. The matcher engine is ported from the CKI
pipeline-herder with a reduced field set. Build events are received via
the existing SNS→SQS pipeline and resolved to MR work items by matching
pipeline_id against head_pipeline_id on mr_details.
DDR-48: Uniform needs_* dimension model with pending_until
Decision: Unify all dimensions under a single needs_* trigger
model with "true"/"false" status and an optional pending_until
column for deferred re-evaluation. Drop synthetic tick events.
Context: The dimension system had grown three distinct categories
(DDR-45): has_* positive completion markers, needs_* action
triggers, and observed state dimensions (approval, pipeline,
merge_status, has_conflicts). The has_* and needs_* split created
asymmetric decider logic. Observed state dimensions were write-only —
the reconciler set them but no code read them. The tick_5m/tick_30m
synthetic event mechanism (DDR-42) required hash bucketing, NOT EXISTS
guards, and special "pending_tick_5m" status values.
Alternatives: (a) Keep the three-category model and add
pending_until alongside tick events — increases complexity with two
deferral mechanisms. (b) Unify to needs_* with pending_until.
Rationale: Option (b).
- Uniform semantics: every dimension is a
needs_*trigger. Status is"true"(action needed) or"false"(handled). The decider uses a single check pattern:!= "false"means actionable. - Absent = actionable: missing dimension rows are treated as needing action, so new items don’t require explicit initialization of every dimension.
pending_untilreplaces ticks: executors setpending_untilto schedule future re-evaluation. The sweep finds expiredpending_untiland notifies — no synthetic events, no hash bucketing, no"pending_tick_5m"status values.- Dropped dimensions:
approval,merge_status,has_conflictswere write-only. Thepipelinedimension carried a single integer and is better modeled as a typed column onmr_details. - Deferred dimensions: the decider pre-filters dimensions where
pending_untilis in the future, treating them as"false".
DDR-49: Replace pending_action TEXT with actionable BOOLEAN
Decision: Drop the pending_action TEXT column. Replace it with
actionable BOOLEAN DEFAULT FALSE. The decider is called in exactly one
place: the processing loop’s derive-and-dispatch step.
Context: pending_action served three roles: (1) sweep
discoverability (WHERE pending_action IS NOT NULL), (2) processing loop
continue/exit signal, (3) executor dispatch key. Roles 2 and 3 made the
column authoritative for which action to run, creating a sync burden
with the dimension-driven decider.
Alternatives:
- Remove entirely — rejected: the decider treats absent dimensions as actionable, so a SQL query over dimension rows cannot find items whose next action derives from a missing row. A wake-up signal persisted to the DB is required.
next_action TEXT(renamed cache) — rejected: requiresdecider.decide()calls in the CRUD layer. The authoritative-looking name invites future dispatch reads.actionable BOOLEAN— selected.
Rationale: A boolean is sufficient to bridge the absent-dimension
discoverability gap. Stale True is harmless (false-positive sweep →
loop starts, decider returns None, exits immediately). actionable is
written in exactly two places: True when work is enqueued (creation,
update, retry, event insertion), False only on the loop’s exit branch
(committed atomically when both decider.decide() returns None and no
pending events exist). This invariant ensures the item stays
sweep-visible throughout processing — a crash at any point leaves
actionable = True, preventing items from becoming permanently stuck.
DDR-50: Event drain tolerates concurrent deletion
Decision: The event drain (_process_pending_events) reads the
(id, event_type, event_body) of all listed pending events before
processing the first one, and deletes rows by id. A row deleted by
another pod mid-drain makes the delete a no-op instead of an error.
Context: The lease is acquired once per loop run and renewed only
while executor actions run; it is neither renewed nor re-checked during
the drain, and the per-item deadline is only checked between drains
(DDR-44). A slow drain can therefore outlive the lease — most notably
during rolling deployments, where the terminating pod’s in-flight drain
overlaps the new pod’s startup sweep. Committing per event expires the
remaining ORM instances, so the next iteration’s attribute access lazily
refreshes against the database; when the other pod already deleted that
row, SQLAlchemy raises ObjectDeletedError and the whole work-item loop
dies (observed as Sentry issue 7829339838, 35 seconds after a rollout).
Alternatives:
- Hold the per-item advisory lock across the drain — rejected: the lock is transaction-scoped (DDR-34) and the drain commits per event for crash isolation; spanning all events would require a single transaction and lose per-event progress.
- Renew or re-check the lease inside the drain — rejected: shrinks but does not close the takeover window (the lease can still expire mid-iteration), and does not fix the stale-instance crash itself.
- Re-query each event before processing it — rejected: adds a query per event, and the row can still vanish between the check and the delete.
Rationale: Reading the attributes while the instances are fresh —
before the first commit — means no lazy refresh can fire mid-drain, and
delete-by-id is naturally idempotent. An event may then be applied by
both pods, but application is already at-least-once for crash recovery
(DDR-31) and idempotent (dimension upserts, last_event_at guard). The
drain becomes robust no matter which actor deletes a row.
DDR-51: Per-item processing ownership and exit safety
Decision: Loop ownership is a loop-lifetime session advisory lock —
pg_try_advisory_lock(0x5751, work_item_id.int & 0x7FFFFFFF) taken on a
pinned connection at loop entry — and the loop’s exit decision runs under
a SELECT ... FOR UPDATE row lock on work_items. The leased_until
lease with its heartbeat (DDR-6) and the insert/exit advisory-lock
handshake (DDR-34) are removed; the column is dropped by the contract
migration after the redesigned version has fully rolled out.
Context: DDR-29/DDR-30 require at most one processing loop per work
item across pods, but no mechanism delivered that invariant. The lease
was acquired once per loop run, renewed only while an action ran, and had
no owner, so it excluded only the next action, not the loop. The exit
wrote actionable = false blindly, so an API enqueue committing between
the loop’s state read and that write was overwritten — permanently
stalling items whose only wake-up signal was a PROCESS_IMMEDIATELY
dimension (no pending_events row, invisible to every sweep leg). The
transaction-scoped advisory lock covered only the consumer’s INSERT and
the exit, not the API writers. The 2026-11-04 ObjectDeletedError
(Sentry 7829339838) was one symptom: a new pod’s startup sweep took an
item whose lease the old loop had cleared between actions. !1494
(DDR-50) fixed the crash, not the ownership model.
The two-32-bit-key form (classid=0x5751, objsubid=2 in pg_locks)
deliberately does not overlap the legacy single-64-bit-key space
(objsubid=1), so during a mixed-version rolling update an old pod’s
handshake and a new pod’s session lock cannot block each other.
The key is the item UUID’s low 31 bits — the positive range of the
int4 second argument. Two unrelated items whose UUIDs share those
bits collide: the loser returns immediately (counted in
hummingbird_workqueue_processing_lock_skipped_total) and the sweep
re-spawns it once the winner’s loop finishes. A collision delays one
item; it never allows two loops for the same item and never loses work.
Every enqueue path already updates the work_items row in the same
transaction as its durable write (dimension upsert, pending_events
insert, retry mark). The row lock serializes the exit against all of
them without any new protocol; this contract is now explicit and must
hold for future enqueue paths.
Alternatives:
- Compare-and-set on a new
revisioncolumn — correct and non-blocking, but adds a migration and a permanent column, andonupdateis ORM-level, so old-pod writers would not bump it during the rollout. - CAS on
updated_atorxmin— no migration, butupdated_atis transaction-start time (uniqueness only by microsecond) andxminis an obscure system column with freeze/wrap edge cases. - Durable enqueue token (a wake-up row per enqueue, consumed only on a clean exit, enumerated by the sweep) — the structural fix that removes the shared-boolean race by construction; deferred as a larger change (a table, an insert on every enqueue path, a drain lifecycle, and a sweep leg).
- Extend the advisory lock to API writers — couples synchronous HTTP handlers to the processing lock for a whole loop and reintroduces the bug whenever a writer forgets.
- Fix the timestamp lease in place (owner token, renewal for the whole loop, re-assert before each action) — time-based expiry still cannot exclude a paused or partitioned worker holding the lock past its TTL, and keeps the heartbeat machinery.
SERIALIZABLEexit transaction — PostgreSQL SSI only guarantees serialization among serializable transactions; mixing isolation levels and retrying under contention is more machinery than the row lock.- Leader election / single-writer pod — removes rollout overlap but defeats replica throughput and distinct-item parallelism.
Rationale: The session lock makes “one loop per item” true by construction for the whole loop lifetime, not per action: a loser returns immediately without holding a worker slot, crashes recover through connection death plus the sweep, and no TTL or heartbeat is needed. The row lock makes the exit decision atomic with respect to every writer without a migration, and old-pod writers block on it too. A spawn request that arrives while the loop is still exiting is captured by a per-item rerun flag, and after a clean exit the loop unlocks, re-checks for work, and re-acquires if a writer slipped in.
Each running loop pins one pooled connection (pool sized to the worker
count with overflow headroom, pool_pre_ping, libpq keepalives and
tcp_user_timeout), and every loop session binds to it. A dead
connection therefore fails the work — the exception propagates and the
loop aborts — instead of leaving a loop running under a lock the server
already released; the still-actionable item is recovered by the sweep.
The API request path can block for the duration of the exit decision (a
few DB-only statements), so the exit transaction must stay DB-only.
During a mixed-version rolling update, an old pod’s leased loop and a
new pod’s locked loop can briefly both run the same executor action for
one item: the two-key lock cannot conflict with the old lease path.
Read-and-upsert actions (reconcile_mr, ensure_plr_status,
ensure_expected_plrs) converge on the same state. Creation actions and
retry_ci_jobs check their result field (e.g. mr_iid) before acting,
so a duplicate requires both loops to pass that check before either
commits — possible in the worst case, producing one duplicate external
mutation (a second MR or note, or a second CI retry) for an item
mid-action during the pod replacement. New pods deliberately do not skip
rows with an unexpired leased_until: a lease written by a pod that died
mid-rollout must not hide the item. The window is bounded by the rollout
and ends when the contract migration drops the column.
DDR-52: Deadline-driven sweep and adaptive merge-recheck
Decision: The sweep wakes at deadlines instead of polling: after
each pass it sleeps until the earliest armed pending_until (plus ~1 s
epsilon), capped at the former 5-minute interval as a backstop — or for
just the epsilon when a deferred timer is already due, letting the next
pass’s expired scan pick it up. unchecked/checking/unknown merge
statuses re-arm needs_reconcile at now + RetrySchedule.delay(step)
on an exponential schedule (15 s, doubling, 5-minute cap) with the step
persisted in the dimension detail, and a partial index on
work_item_statuses (pending_until) covers the sweep’s hot path.
Context: The direct-merge design (DDR-46 lineage; !1327) merges when
a reconcile GET observes detailed_merge_status == "mergeable", with a
30-minute retry cadence and the sweep (fixed 5 minutes) as the only
timer. On busy target branches this starves: every push to the target
branch resets stored mergeability to unchecked, GitLab recomputes it
only lazily on a single-MR GET (seconds), the service samples once per
30 minutes, and GitLab emits no webhook when a recheck completes.
Evidence: containers!22697 unmerged 6 days while green; 2026-11-10 logs
show 15/16 reconcile GETs observing unchecked and the two mergeable
observations losing the merge PUT race to concurrent fleet merges (422
“Branch cannot be merged”). The branch churn is largely the workqueue’s
own lockfile fleet (59 merges in two hours on 2026-11-10), so the
service invalidates its own merge candidates. The branch churn also
means any fixed short interval is a compromise: real blockers need
minutes, rechecks need seconds.
now is passed to the deadline query as a bound parameter rather than
SQL now() because pending_until values are written with the Python
clock; comparing in one clock domain avoids app-vs-DB skew shifting
wakeups. The MIN query mirrors find_expired_pending’s item filters
(active phase, accepted types) so completed items — which keep their
last dimension rows — cannot pin the sweep to a stale deadline. The
deadline lookup computes the earliest future deadline and a past-due
flag in one filtered-aggregate query — one snapshot — so a timer
expiring between the sweep’s expired scan and the lookup cannot be
missed by both; the past-due flag excludes actionable items, mirroring
the expired scan, so a running loop’s timer cannot spin the sweep. A
wake event interrupts the sleep when an earlier deadline is armed
in-process (set from spawn_loop()’s enqueue paths and after executor
action commits; sweep-initiated submissions deliberately do not poke —
they arm no timer, and poking would wake the sweep into a pass that
re-submits, spinning cross-pod while another pod’s loop holds the item
lock during a rollout); spurious wakes cost one indexed query and a
no-op sweep. Multi-pod: each pod computes its own MIN; the arming pod
wakes exactly, others see the work within the backstop cap, and
duplicate spawns are absorbed by the per-item advisory lock (DDR-51).
Alternatives:
- Shorter fixed sweep interval — wasted cycles and still granular; a +30 s timer would still round up to the interval.
- Short dimension timers without a sweep change — the 5-minute sweep granularity silently becomes the effective interval; the timer is a lie without deadline-driven wakeups.
- Sleeping and re-GETting inside the executor — holds a worker slot (10 max) and the pinned DB connection, brushes the 60-second item deadline (DDR-44), and violates the “reconcile_mr: cheap operations only” contract (DDR-46).
- Fixed fast interval without backoff — a permanently-
uncheckedMR would poll ~2880×/day. - Backoff step derived from row timestamps (
updated_atdeltas) — fragile, implicit, breaks on any other write to the dimension row. RetryAfterstate merged into the crud state union with a reserved detail key — more machinery and magic than needed for one consumer; revisit on a second non-MR consumer.- Re-enabling GitLab server-side auto-merge — removes service ownership of the merge decision (DDR-46); the adaptive recheck achieves convergence without it.
Rationale: A mergeable MR converges within one or two recheck probes
(~15–45 s) instead of winning a 30-minute lottery, and concurrent-merge
race losers recover within seconds, so the fleet race self-heals — the
422 → immediate retry → unchecked → +15 s → mergeable → merged
sequence replaces the 30-minute re-sample. unknown shares the fast
family (GitLab documents it alongside unchecked); a persistent unknown
settles at the 5-minute cap, and the mr.state != "opened" guard keeps
closed/merged MRs from being polled. Permanently-unchecked MRs poll at
the 5-minute cap (the same cadence as error backoff), visible via the
hummingbird_workqueue_merge_rechecks_total counter labeled by merge
status. Real blockers keep the 30-minute cadence and reset the backoff
step (detail=None, the documented clear path); webhook re-arms
preserve the step (the _PRESERVE sentinel) and consume one — each
immediate reconcile observing unchecked advances the backoff,
deliberately throttling poll storms on chatty MRs. The GitLab-error path
(the 5-minute error re-arm without a detail write) also preserves a
matured step, so the first unchecked probe after an outage waits at the
cap — deliberate. The sweep runs as often as the earliest deadline
(potentially every ~15 s under recheck storms); each pass is one indexed
MIN query, one expired-pending query, and the actionable-or-pending-events
union query, and only items with due work spawn loops — hence the
partial index. The direct-merge decision stands; this record removes its
starvation failure mode on busy branches. Rollout: the deadline-driven
sweep and its supporting partial index land first, then the reusable
RetrySchedule type, then the merge-recheck application (schedule
constant, detail step, outcome enum, metric, and docs); the migration
adds the partial index only.
4.5.34 - Service metrics
Long-running exporter for Hummingbird AWS Cost Explorer spend, Kubernetes Metrics API CPU/memory gauges, OpenShift cluster resource quota usage, PVC filesystem usage, GitLab CI/CD schedule status, and GitLab webhook alert status.
Features
- Explicit collectors - nothing runs unless listed in
METRICS_CONFIG - Kubernetes Metrics API - per-container CPU and memory gauges
- Cluster resource quotas - AppliedClusterResourceQuota usage and limits
- AWS Cost Explorer - yesterday’s
NetAmortizedCostby team and service - Volume filesystem usage - PVC disk usage via
shutil.disk_usage() - GitLab CI/CD schedules - pipeline schedule health via GraphQL
- GitLab webhooks - webhook alert status via the REST API
- Prometheus - HTTP metrics on port 9090
Configuration
Collectors are off unless listed in METRICS_CONFIG:
enabled: [kubernetes]
Unknown names fail startup.
| Variable | Purpose |
|---|---|
METRICS_CONFIG |
YAML with enabled collector list (required) |
KUBERNETES_CONFIG |
YAML with namespaces for the kubernetes collector |
VOLUMES_CONFIG |
YAML with volumes list for the volume collector |
GITLAB_URL |
GitLab instance URL (default https://gitlab.com) |
GITLAB_TOKEN |
GitLab API token with read_api scope (required for gitlab collectors) |
GITLAB_CONFIG |
YAML with namespaces (schedules) and webhooks (webhooks) lists |
METRICS_PORT |
HTTP port (default 9090) |
SENTRY_DSN |
Optional Sentry DSN |
AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY |
Cost Explorer IAM user (hub) |
AWS_DEFAULT_REGION |
Must be us-east-1 for Cost Explorer |
Kubernetes Metrics API
hummingbird_k8s_resource_usage{namespace,pod,container,resource,app} with
resource=cpu|memory and app sourced from the pod’s
app.kubernetes.io/name label (empty string if absent). Instant gauges from
/apis/metrics.k8s.io/v1beta1/namespaces/<ns>/pods. This is not cAdvisor:
managed OpenShift does not grant nodes/metrics or cluster-monitoring-view,
so kubelet /metrics/cadvisor and platform Prometheus federation are not
available.
If KUBERNETES_CONFIG.namespaces is unset, the collector uses the pod’s
ServiceAccount namespace.
Cluster resource quotas
The kubernetes collector also reads namespace-scoped
AppliedClusterResourceQuota objects from the quota.openshift.io/v1 API
and exposes two gauges that mirror the
openshift-state-metrics
schema with a hummingbird_ prefix:
hummingbird_clusterresourcequota_usage{name,resource,type}– cluster-wide totals fromstatus.total(type=hard|used)hummingbird_clusterresourcequota_namespace_usage{name,namespace,resource,type}– per-namespace breakdown fromstatus.namespaces(type=hard|used)
resource values match Kubernetes quantity names: cpu, memory,
requests.cpu, limits.memory, pods, etc. Quantities are converted to
base units (cores, bytes, count).
The same ACRQ may appear in multiple namespaces; the global metric is idempotent across duplicates. Stale series are removed when ACRQs disappear.
Requires list on appliedclusterresourcequotas in quota.openshift.io
(namespace-scoped Role – no ClusterRole needed).
Volume filesystem usage
The volume collector reports disk usage for PVC-backed volumes. It is
designed to run as a sidecar on pods that mount PVCs, since
kubelet_volume_stats_* metrics are inaccessible on managed clusters
(no ClusterRole for nodes/metrics) and RWO volumes cannot be mounted
cross-namespace.
Configure via VOLUMES_CONFIG:
VOLUMES_CONFIG: |
volumes:
- {name: my-postgres, path: /data}
Metrics (collected every 5 minutes):
hummingbird_volume_usage_bytes{name}– used byteshummingbird_volume_total_bytes{name}– total capacityhummingbird_volume_available_bytes{name}– available bytes
Deploy with enabled: [volume] and a non-default METRICS_PORT (e.g.
9091) to avoid port conflicts when running as a sidecar alongside other
containers. See the hummingbird-status StatefulSet in the
infrastructure repo for an example.
GitLab CI/CD schedules
The gitlab collector monitors the health of pipeline schedules across all projects in configured GitLab namespaces. It queries the GitLab GraphQL API once per hour, paginating through projects to fetch each schedule’s last pipeline status.
Configure via GITLAB_CONFIG:
GITLAB_CONFIG: |
namespaces:
- redhat/hummingbird
The GITLAB_TOKEN environment variable must contain a token with read_api
scope that has access to the monitored namespaces. Keep it in an OCP Secret
(not the ConfigMap).
Metric (collected every hour):
hummingbird_gitlab_schedule_status{instance,project,active,description,cron}– Enum reporting the last pipeline status of each CI/CD schedule
States: created, waiting_for_resource, preparing, pending, running,
success, failed, canceled, skipped, manual, scheduled, unknown.
Schedules with no last pipeline report unknown.
Stale series are removed automatically when schedules disappear.
GitLab webhooks
The gitlab_webhooks collector monitors the alert status of the webhooks
configured on specific GitLab projects. GitLab disables webhooks that fail
repeatedly (for example after a signing-token rotation), which silently stops
all event-driven automation behind them; this collector makes that state
visible. It queries the GitLab REST API once every 2 minutes.
Configure via GITLAB_CONFIG:
GITLAB_CONFIG: |
webhooks:
- redhat/hummingbird/tools
The GITLAB_TOKEN environment variable must contain a token with read_api
scope that has access to the monitored projects. Keep it in an OCP Secret
(not the ConfigMap).
Metric (collected every 2 minutes):
hummingbird_gitlab_webhook_alert_status{instance,project,url}– Enum reporting the alert status of each project webhook
States: executable, disabled, temporarily_disabled, unknown. An
unrecognized status reported by GitLab maps to unknown. Stale series are
removed automatically when hooks disappear from the configured projects.
The infrastructure repo consumes this metric with the
GitLabWebhookDisabled Mimir alert (severity critical) and a Grafana
webhook-status dashboard.
AWS costs
hummingbird_aws_cost{group,type,key} is yesterday’s NetAmortizedCost.
Queries run at process start and every 24 hours.
Why NetAmortizedCost
AWS Cost Explorer offers several cost metrics. This collector uses
NetAmortizedCost — the amortized cost after enterprise discounts.
This account is part of a consolidated billing organization with Compute Savings Plans shared across accounts. The cost metrics differ due to two independent effects:
- Enterprise discounts (affects all services): This account receives EDP (Enterprise Discount Program) and Private Rate Card discounts. In UnblendedCost and AmortizedCost these appear as separate negative line items; in the Net variants (NetAmortizedCost, NetUnblendedCost) they are folded into each line item directly.
- Savings Plans amortization (affects SP-covered services):
In UnblendedCost, SP costs appear in two record types:
SavingsPlanRecurringFeecarries the commitment cost, whileSavingsPlanCoveredUsageshows the on-demand equivalent with an equalSavingsPlanNegationthat cancels it — net zero. In AmortizedCost, this flips: the recurring fee goes to zero and the covered usage is valued at the SP effective cost instead. If only this account’s own SPs covered its usage, the amortized covered usage would equal the unblended recurring fee. The gap between them reveals how much org-wide SPs from other accounts contribute. To isolate the SP effect from enterprise discounts, compare Unblended against Amortized (SP amortization only) or Unblended against NetUnblended (discounts only).
NetAmortizedCost shows the true cost of this account’s resource
consumption to the organization, including compute covered by
Savings Plans owned by other accounts. For non-compute services,
amortized and unblended costs are identical.
Run service-metrics/scripts/cost_metric_comparison.py [YYYY-MM]
to see how these effects compose. It queries Cost Explorer for a
single month, groups by record type, and shows all five metrics
side by side — making visible how enterprise discounts fold into
Net variants, how SavingsPlanNegation cancels covered usage in
unblended, and how amortization redistributes SP costs. A summary
section breaks down what this account actually pays versus what
other accounts’ Savings Plans cover.
Groupings
group=all is the shared account by app-code and SERVICE.
group=hummingbird filters app-code=RPRM-001 and groups by SERVICE,
OPERATION, USAGE_TYPE, Name tag, and SERVICE+OPERATION.
Untagged Cost Explorer keys such as app-code$ become key=untagged.
Well-known SERVICE names are shortened (S3, EC2). Dual grouping uses
S3|PutObject.
The SAM template creates an IAM user with ce:GetCostAndUsage only. Access
keys are created after deploy and stored in Vault. Deployment lives in the
infrastructure repo.
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.
4.5.35 - VEX Checker
A CLI tool that works with the Red Hat CSAF VEX feed and the Hummingbird catalog API to check CVE statuses and track coverage across Jira, VEX advisories, and container image scan results.
Subcommands
| Subcommand | Purpose |
|---|---|
check |
Look up a single CVE and show Hummingbird product statuses |
reconcile |
Cross-reference Jira CVE tickets with the VEX feed |
catalog |
Cross-reference catalog API CVEs with Jira tickets |
Prerequisites
- Python 3.11+
- Internet access to
security.access.redhat.com(no VPN required) forcheck/reconcile; toapi-hummingbird.hummingbird-project.io(no VPN required) forcatalog JIRA_EMAILAtlassian account email — required forreconcileandcatalogJIRA_TOKENJira API token — required forreconcileandcatalog
check — Single CVE lookup
Fetches the CSAF VEX document for a given CVE and prints the status of all Hummingbird products, filtering out all other RHEL/OpenShift/etc. entries.
check: usage
./check_vex.py check CVE-YYYY-NNNNN [--json]
check: options
| Option | Description |
|---|---|
CVE-ID |
CVE to look up |
--json, -j |
Output results as JSON |
check: examples
$ ./check_vex.py check CVE-2014-8090
CVE-2014-8090 — CVE-2014-8090 ruby: REXML billion laughs attack (Moderate)
hummingbird-1:ruby.src known_affected none_available
hummingbird-1:ruby3.3.src known_affected none_available
hummingbird-1:ruby3.4.src known_affected none_available
hummingbird-1:ruby4.0.src known_affected none_available
$ ./check_vex.py check CVE-2014-8090 --json
{
"cve": "CVE-2014-8090",
"title": "...",
"severity": "Moderate",
"hummingbird": [
{ "product_id": "hummingbird-1:ruby.src", "status": "known_affected", "remediation": "none_available" },
...
]
}
check: exit codes
| Code | Meaning |
|---|---|
0 |
No VEX document found, no Hummingbird products listed, or no known_affected status |
1 |
One or more Hummingbird products are known_affected |
2 |
Invalid CVE ID format |
reconcile — Jira/VEX sync check
Queries the Jira HUM project (Security component) for all CVE tickets, then checks each CVE in the VEX feed. Reports two types of mismatches:
- Open in Jira but not
known_affectedin VEX — VEX says the issue is resolved but the Jira ticket is still open. - Closed in Jira but
known_affectedin VEX — the Jira ticket was closed but Red Hat’s advisory still marks Hummingbird as affected.
Continuous closed-ticket VEX reconciliation (Done-Errata → fixed,
Not a Bug → known_not_affected or package_not_listed, scoped to the
ticket’s package, with awaiting-vex work queue and dashboard persistence)
lives in hummingbird-cve-analysis / the dashboard Closed tab (HUM-5843).
reconcile remains the one-shot CLI audit.
Jira tickets are considered closed when their status is one of:
Done, Closed, Won't Fix, Not a Bug.
VEX documents are fetched in parallel (--workers, default 20) so the
command is fast even with hundreds of CVE tickets.
reconcile: usage
export JIRA_EMAIL=you@redhat.com
export JIRA_TOKEN=<your-api-token>
./check_vex.py reconcile [--workers N] [--json]
Credentials are read exclusively from environment variables to avoid exposing them in process listings.
reconcile: options
| Option | Description |
|---|---|
--jira-url |
Jira base URL (default: https://redhat.atlassian.net) |
--workers |
Parallel VEX fetch workers (default: 20) |
--json, -j |
Output results as JSON |
reconcile: example
$ ./check_vex.py reconcile
Fetching Jira tickets... 142 ticket(s)
Fetching VEX statuses for 89 unique CVE(s)...
89/89
2 mismatch(es) found:
HUM-1234 CVE-2025-1234 closed → known_affected (VEX still open)
Jira: https://redhat.atlassian.net/browse/HUM-1234
VEX: https://security.access.redhat.com/data/csaf/v2/vex-feed/2025/cve-2025-1234.json
HUM-5678 CVE-2024-5678 open → fixed (VEX not affected)
Jira: https://redhat.atlassian.net/browse/HUM-5678
VEX: https://security.access.redhat.com/data/csaf/v2/vex-feed/2024/cve-2024-5678.json
reconcile: exit codes
| Code | Meaning |
|---|---|
0 |
No mismatches found |
1 |
One or more mismatches detected |
2 |
Missing JIRA_EMAIL or JIRA_TOKEN |
catalog — Catalog API / Jira coverage check
Fetches all CVEs currently detected in Hummingbird container images (via Grype scan results stored in the catalog API) and cross-references them with Jira HUM Security tickets. Reports CVEs that have no corresponding Jira ticket.
CVEs with a closed Jira ticket are reported separately as expected propagation delay — the typical flow is CVE fix → RPM → image rebuild → Grype DB update, and Jira tickets are closed early while the catalog API reflects the latest Grype scan (updated ~once daily).
Non-CVE vulnerability identifiers (e.g. GHSA-*) from Grype are filtered out
since they are not tracked in Jira.
catalog: usage
export JIRA_EMAIL=you@redhat.com
export JIRA_TOKEN=<your-api-token>
./check_vex.py catalog [--api-url URL] [--workers N] [--json]
Each CVE line shows severity, age since first detection, affected images, and for tracked CVEs the Jira ticket key. Components with known fix versions are shown in brackets when available.
Closed Jira tickets are cross-referenced with the VEX feed to distinguish between different root causes:
- Done-Errata, VEX still affected – fix shipped in RPM but not yet propagated to container images; needs rebuild or lockfile refresh.
- Done-Errata, VEX resolved – fix fully propagated, Grype DB just needs its daily update to stop flagging.
- Won’t Do / Not a Bug, VEX resolved – VEX already marks CVE as not affected; Grype DB delay, will self-resolve.
- Won’t Do / Not a Bug, VEX still affected – genuine known issue that will persist in scans (accepted risk, upstream won’t fix, etc.).
catalog: options
| Option | Description |
|---|---|
--api-url |
Catalog API base URL (default: https://api-hummingbird.hummingbird-project.io/v1) |
--jira-url |
Jira base URL (default: https://redhat.atlassian.net) |
--workers |
Parallel VEX fetch workers (default: 20) |
--json, -j |
Output results as JSON |
catalog: example
$ ./check_vex.py catalog
Catalog: 72 CVEs across 29 images (scanned 2026-04-28T10:45:38Z)
Jira: 748 tickets (437 unique CVEs)
11 untracked CVE(s) (no Jira ticket):
CVE-2008-2662 High 8d ruby (12 tags) [ruby3.3@3.3.10, +36 more]
CVE-2026-2950 Medium 8d aspnet-runtime, ... (30 tags) [lodash@4.17.21]
...
45 tracked CVE(s) open in Jira:
CVE-2025-68114 High 8d php (3 tags) [capstone@5.0.6] HUM-925
...
8 closed in Jira (Done-Errata) but VEX still affected -- verify fix propagated:
CVE-2026-27143 Critical 8d caddy, go-fdo-client, ... (10 tags) HUM-969
4 closed in Jira (Done-Errata), VEX resolved -- Grype DB delay:
CVE-2025-61732 High 8d go, xcaddy (6 tags) HUM-1119
3 closed in Jira (Won't Do / Not a Bug), VEX resolved -- Grype DB delay:
CVE-2026-27140 High 8d caddy, go, ... (10 tags) HUM-967
1 closed in Jira (Won't Do / Not a Bug), VEX still affected -- genuine:
CVE-2025-12781 Medium 8d postgresql, python, ... (17 tags) HUM-1387
catalog: exit codes
| Code | Meaning |
|---|---|
0 |
All catalog CVEs are tracked (have a Jira ticket) |
1 |
One or more untracked CVEs found |
2 |
Missing JIRA_EMAIL or JIRA_TOKEN |
Development
# Run tests
cd vex-checker && python3 -m unittest discover tests -v
# Run linter (from repo root)
ruff check vex-checker
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.
4.5.36 - Hummingbird CVE agent
CLI and SQS worker that investigate open HUM Security tickets labeled
cve-needs-attention or upstream-fix-available, using AI where deterministic
analysis could not resolve the CVE. Fixes are handed to workqueue-service as
mr work items.
System overview
flowchart TB
TICKET["run HUM-XXXX"] --> COL
SNS["Successful cve_analysis --resolve<br/>SNS → SQS wake-up"] --> SWEEP["Worker: fresh Jira sweep"]
CLI["sweep [--limit N] [--workers N]"] --> SEARCH
SWEEP --> SEARCH["Open HUM Security tickets<br/>in-scope labels · no cve-next-release<br/>not embargoed · no prev analysis"]
SEARCH --> PREFLIGHT["Start and preflight all worker backends<br/>before ticket collection · skip if no tickets"]
PREFLIGHT --> THREAD
subgraph WORKER["Trusted worker · network and credentials"]
direction TB
THREAD["Sweep worker thread"] --> COL
COL["Collect facts<br/>Jira · Pulp · SBOM"] --> CAND{"Deterministic candidate?<br/>affected + backport + fix label/reference<br/>not stale"}
CAND -->|yes| ID{"Exact current version<br/>and SRPM NVR identified?"}
ID -->|no| HUMAN["needs_human · no actions"]
CAND -->|"no / stale"| AIINV["AI investigation<br/>bounded read-only tools"]
ID -->|yes| FETCH
AIINV -->|"source tools / needs_change + fix"| FETCH["Fetch pinned inputs<br/>hardened no-checkout Git transport<br/>checksummed sources · upstream patches"]
AIINV -->|verdict| RESULT["InvestigationResult"]
HUMAN --> RESULT
EVIDENCE["Bounded evidence + verified file_actions"] --> RESULT
RESULT --> OUT["Write result JSON<br/>stdout / --output before remote writes"]
OUT --> SUBMIT{"--submit-mr<br/>and prepared actions?"}
SUBMIT -->|yes| MR["Submit draft mr work item<br/>to workqueue-service"]
SUBMIT -->|no| POST
MR --> POST{"--post-to-jira?"}
POST -->|yes| COMMENT["Post Jira comment → verdict label<br/>→ analysis attachment"]
end
subgraph SANDBOX["Offline execution · no network or credentials"]
direction TB
BACKEND["WorkerSandbox owns backend + package tools<br/>one per worker thread"] --> FILES["per-ticket WORK workspace<br/>reset state/files between tickets"]
FILES --> PREP["RPM %prep<br/>existing downstream patches"]
PREP --> INSPECT["Source inspection<br/>+ disposable candidate checks"]
INSPECT -->|"backport path"| STAGE["Patch staging + spec edits"]
STAGE --> REPLAY["Final-spec replay from clean inputs"]
end
THREAD -->|owns| BACKEND
FETCH -->|"transfer each input once"| FILES
INSPECT -->|"AI tools: bounded evidence"| AIINV
REPLAY --> EVIDENCE
MR --> REVIEW["Konflux build + human review"]
classDef agent fill:#dbeafe,stroke:#2563eb,color:#111827
classDef sandbox fill:#dcfce7,stroke:#16a34a,color:#111827
class THREAD,COL,CAND,ID,HUMAN,AIINV,FETCH,EVIDENCE,RESULT,OUT,SUBMIT,POST,COMMENT agent
class BACKEND,FILES,PREP,INSPECT,STAGE,REPLAY sandbox
The agent runs as a CLI and is stateless: run handles one ticket, while
sweep pages through all eligible tickets and investigates them concurrently.
The deployed worker consumes sweep requests from SQS and invokes that same
bounded sweep; the event is only a wake-up signal, not a ticket snapshot.
CLI writes are opt-in; the deployed worker enables both Jira posting and MR
submission (see Writes).
Critical evidence network, download, or parse failures abort the investigation;
their local diagnostic report leaves the ticket eligible for a later retry.
Unsupported or ambiguous evidence becomes needs_human. The ticket-level state machine lives in the
CVE Lifecycle & State Vocabulary.
Tl;Dr
- Scoped to CVE Security tickets - investigates open HUM Security tickets
with either
cve-needs-attentionorupstream-fix-available; these are the deterministic-analysis residual and upstream-fix populations respectively. Sweeps skipcve-next-release, embargoed tickets, and tickets already carrying their<ticket>-cve-agent-analysis.jsonattachment. - Backporting - verifies upstream patches against pinned package source and prepares RPM changes; see Backporting.
- AI in one place - every LLM call goes through
ai.AgentAiClient, which uses the sharedhummingbird_ai.modelslibrary directly, so its token/cost metrics feed the shared GCP AI-costs dashboard. AI judges the fuzzy residual the deterministic path does not own; see What the model sees for the exact context. The model can use bounded read-only tools to search the SBOM, inspect package files, inspect checksum-pinned package sources, and look up exact npm release metadata. Before the first model turn, the agent prefetches predictable evidence through those same tools: ticket CVE advisories, matching SBOM components, and the package file listing and spec. Evidence is reused only within one sweep, keyed by its exact source identity and bounded counts: package artifacts, advisories, identical package-file reads and searches, and deterministic patch checks. Transient failures, corrupt artifacts, and final backport verification are never cached. Transient HTTP and transport failures are retried by one sweep-scoped policy: at most three attempts and 30 seconds of cumulative wait, honoringRetry-After. Redirects are still refused. A retryable external read failure reuses the worker sandbox only when it resets cleanly; a failed reset or any local corruption discards it. Each ticket has at most five tool-enabled turns plus one final-answer turn, whose request carries no tool definitions and explicitly asks for a JSON verdict using the available evidence. Invalid final replies receive a corrective request containing the result schema within the existing bounded retry budget. Exhausted retries fail the ticket without posting a result; no verdict is inferred from malformed output. Diagnostics log response length, finish reason, each collection step’s duration, and the model call duration, not response contents. Retries can make multiple model requests within a turn; six turns is not a six-request cap. SBOM searches return at most 20 components with binary-presence evidence (present,source_only, orunknown) from the shared SBOM classifier; a name/version match or source-only hit does not prove the component ships. When a matching SBOM result is unknown, the agent can scan published RPMs for that component. SBOM files are limited to 256 MiB. SBOM JSON is parsed incrementally; the sweep-scoped LRU retains at most eight full documents, so its entry limit is not a byte or process memory limit. - Structured results - every investigation produces a validated result
contract (there is no schema-version field). The result
statusis aVerdictfrom the shared lifecycle vocabulary (hummingbird_cve_analysis/lib/states.py), so the agent andcve_analysisspeak the same state language. See the CVE Lifecycle & State Vocabulary for the full state machine. Every result also carries the ticket’s collection records: each evidence step’s outcome (success,empty,partial,failure, orlimitation), its retryability, and its provenance. Initial collection records can appear in the model’s input facts; the caller owns the final collection records, including later tool outcomes.
Prerequisites
- Python 3.11+
- Podman with a running Linux VM for local source preparation, or Kubernetes sandbox access in deployment.
- A sandbox image with RPM build tools, Python RPM bindings,
python3-specfile,autoconf, Git, GNUpatch, and the RPM macros/helpers required by the package. GITHUB_TOKENfor GitHub backport lookup when unauthenticated API requests hit GitHub’s rate limit.- Jira API credentials (
JIRA_TOKEN, andJIRA_USERwhere basic auth applies) - Vertex AI access for the AI path:
GOOGLE_CLOUD_PROJECT, orGOOGLE_API_KEYfor the direct Gemini API. The deterministic fast path needs neither. - For
--submit-mronly:WORKQUEUE_SERVICE_URLand eitherWORKQUEUE_SERVICE_TOKEN_FILEorWORKQUEUE_SERVICE_TOKEN, withsubmitpermission on themrwork type. The deployed worker uses a projected token file; local development can use the environment token described in Writes: Jira comment and MR submission.
Installation
make hummingbird-cve-agent/setup
The target installs hummingbird-agent, hummingbird-ai, and
hummingbird-cve-analysis editable alongside this package. Equivalent by hand:
cd hummingbird-cve-agent
pip install -e ../hummingbird-ai -e ../hummingbird-agent -e ../hummingbird-cve-analysis -e ".[dev]"
For the environment, copy hummingbird-cve-agent/.env.example to .env and
source it for a quick way to set up all necessary variables. For local CA setup,
activate the Python environment used to run the CLI before sourcing .env, so
python -m certifi resolves that environment’s public CA bundle.
If the workqueue endpoint uses an internal CA, export REQUESTS_CA_BUNDLE
in the same shell as --submit-mr. Its PEM file must contain both the public
roots (for Google OAuth) and the internal root (for the workqueue endpoint).
Configuration
Operational settings come from the YAML config file; secrets and model credentials come from the environment.
| Variable | Purpose |
|---|---|
HUMMINGBIRD_CVE_AGENT_CONFIG |
Path to a YAML config file (optional) |
HUMMINGBIRD_CVE_AGENT_WORKERS |
Worker concurrency default (2; must be positive; --workers overrides) |
JIRA_TOKEN |
Jira API token |
JIRA_USER |
Jira basic-auth user (optional) |
SENTRY_DSN |
Sentry error reporting for the deployed worker |
GITHUB_TOKEN |
Read-only GitHub token for PR commit and advisory lookups (optional) |
GITLAB_TOKEN |
Read-only GitLab.com token for MR commit lookups (optional) |
GOOGLE_API_KEY |
Gemini API key (direct API mode) |
GOOGLE_CLOUD_PROJECT |
GCP project for Vertex AI mode |
WORKQUEUE_SERVICE_URL |
Base URL of a running workqueue-service instance (for --submit-mr) |
WORKQUEUE_SERVICE_TOKEN_FILE |
Path to projected bearer token (preferred; re-read on each submission) |
WORKQUEUE_SERVICE_TOKEN |
Bearer token fallback for local --submit-mr use |
SQS_QUEUE_URL |
Dedicated queue consumed by the deployed worker command |
YAML settings:
| Key | Default | Purpose |
|---|---|---|
jira_url |
Red Hat Jira | Jira base URL |
sandbox_type |
podman |
Sandbox backend; deployments use k8s |
sandbox |
Published, digest-pinned CI image | Override image locally to use the CVE RPM-preparation image |
model |
claude-sonnet-4-6 |
Model name passed to build_model |
model_regions |
Gemini and Claude region prefixes | Model-name prefix to Vertex region |
Neither forge token grants write access: they only authenticate the read-only
API calls that resolve a pull/merge request to its commits, and are needed only
for private or rate-limited repositories. Resolving one pull request costs two
API requests and unauthenticated GitHub access allows 60 per hour, so set
GITHUB_TOKEN before looking up many backports. The sandbox never holds a push
token.
Usage
hummingbird-cve-agent run HUM-1234
hummingbird-cve-agent run HUM-1234 --output results/HUM-1234.json
hummingbird-cve-agent run HUM-1234 --post-to-jira
hummingbird-cve-agent run HUM-1234 --submit-mr
hummingbird-cve-agent sweep
hummingbird-cve-agent sweep --limit 25 --workers 6 --output sweep.json
hummingbird-cve-agent sweep --post-to-jira --submit-mr
This gathers facts for the ticket, runs the investigation, and prints the
result as JSON or persists it to --output. See Backporting
for how upstream fixes become package changes and what happens when preparation
fails.
CLI logs include ticket and package context, e.g. - INFO (HUM-1234 openssl): collected facts for HUM-1234; the package is - until known, and unscoped logs
show - -.
sweep defaults to all eligible tickets and two concurrent workers. --limit
caps attempted investigations, including failed attempts; analyzed, embargoed,
and out-of-scope search results do not consume the limit. A sweep searches once
and then investigates for a long time, so a ticket can also lose scope after it
was selected, by human re-triage to cve-next-release or by an overlapping
sweep closing it. That is reported as out of scope, not as a failure: it does
not fail the run or change the exit code. Standard output shows
counts, verdicts, token usage, estimated cost, and compact per-ticket summaries.
--output FILE also saves the complete JSON report. The command exits nonzero if
any ticket fails, while continuing after handled ticket-level failures.
A source-initialization TypeError propagates out of investigate, so that
ticket produces no investigation-result JSON and makes no Jira/MR writes.
The sweep includes TypeError in its existing ticket-error handling, records
the failed ticket in its report, and continues other tickets normally; this is
not a catch-all for unexpected exceptions. Without
--post-to-jira, the sweep does not add the attachment used to mark a ticket
as investigated, so a later sweep can retry it.
Critical collection failures never post a recommendation or analysis attachment,
even when the write flags are enabled. The CLI and sweep retain bounded failure
records locally; transient failures are marked retryable and are not cached.
After finding eligible tickets, the sweep starts and preflights all
min(workers, eligible tickets) execution backends before any ticket collection,
investigation, or model calls. No eligible tickets means no backends. A thread
pool runs that many worker loops consuming a ticket queue, each with its own
backend; it does not create a backend per ticket or package. Preflight failure
aborts the sweep before any ticket starts or remote writes occur and cleans up
started backends. Backends are cleaned up when the sweep exits; a handled ticket
failure affects only that ticket, not other workers. Transport/collection failures
discard its backend so timed-out child processes cannot reach the next ticket;
the next ticket starts a fresh backend. Other handled failures reset ticket
state and workspace for reuse.
Backporting
The agent separates deciding that a package needs a fix from preparing the RPM changes. AI may judge affectedness or identify an upstream commit; the backporting code fetches and verifies the upstream patch, then prepares RPM changes. It does not rewrite or generate patch content (yet, see Planned work).
flowchart TD
C{"Affected + backport + fix reference<br/>+ upstream-fix-available?"} -->|Yes| S{"Analysis stale?"}
C -->|No| I["AI investigates"]
S -->|Yes| I
S -->|No| ID{"Exact current version<br/>and SRPM NVR identified?"}
ID -->|No| H["needs_human · no file_actions"]
ID -->|Yes| U["Resolve fix and select release patches"]
I -->|Needs change + fix| U
I -->|Otherwise| R["Return investigation result"]
U --> V["Per-ticket workspace on worker-owned backend<br/>RPM prep → disposable candidate checks"]
V --> W["Stage patches · edit spec/metadata<br/>replay final spec from clean inputs"]
V -->|Unsupported / rejected| F["Deterministic: needs_human<br/>AI: retain finding/confidence<br/>both: no file_actions"]
W -->|Unsupported / rejected| F
V -->|Critical collection failure| A["Abort · local diagnostics<br/>no remote writes"]
W -->|Verified| R
F --> R
H --> R
R --> J["Agent's result processing"]
1. Identify and select the upstream fix
The fast path skips AI only when the existing CVE analysis says the package is
affected, names a backport as the next step, the ticket has the
upstream-fix-available label, and the analysis still matches the package
version collected now and has not already reached a recorded fixed version on
its release line. Stale analysis goes to AI. A candidate that is not stale but
does not explicitly identify both the current version and exact SRPM NVR returns
needs_human, not an AI fallback.
A needs_change result with a usable fix reference enters the same preparation
path; otherwise, the agent returns the investigation result only.
Commit URLs are fetched directly. A merged PR URL is first resolved to its
commits in application order. A commit is accepted only when it is reachable
from the package’s declared upstream repository’s default branch or from its
exact release branch named in metadata/<package>.json. Commits from another
repository, including forks, are refused even if reachable. Open and closed-but-unmerged
PRs are rejected before their commits are fetched. An unknown release branch stays
unverified. A failed fetch is retried instead: it says nothing about the commit,
so it is never reported as unmerged. A squash-merged change has no
landed SHA in common with the branch commit, so such a URL stays unverified
rather than being assumed merged. Every patch fetched from a commit URL has
its full SHA checked against the cited commit before it can reach package
applicability checks or be saved for backport. The merged-backport PR diff
found by title is instead matched on its merged state and release-line base.
Branch-tagged patches must target the release
line of the shipped package; a mismatch requires human review. Mixed tagged and
untagged commits are rejected as ambiguous because untagged commits might be
prerequisites rather than alternative backports. After selecting the line, only
identical diff bodies are removed, retaining hunk positions, context, and file
metadata in the comparison so distinct fixes are not silently discarded. The
series has no commit-count limit: discovery work is bounded by an aggregate
10 MiB patch budget (and 2 MiB per patch), because patch count alone is not a
correctness limit.
2. Prepare and verify against package source
Each sweep worker thread owns one WorkerSandbox, which owns the execution
backend and package tools: a Podman container locally or an ephemeral Kubernetes
pod in deployment. Its .ticket(package) scope resets per-ticket state and
files in the WORK workspace. Sequential tickets reuse the physical container/pod,
but each gets fresh Preparation state, checksum pins, and candidates; none
carry over to the next ticket, except the sweep-scoped evidence cache. This avoids startup per investigation
without sharing ticket state. .open() lazily imports the package into the
existing worker backend. Standalone run uses its own WorkerSandbox and may
avoid backend startup entirely for an SBOM-only investigation; any backend it
starts is cleaned up when the run ends.
Within a ticket, the offline package workspace is shared by file/source
inspection, disposable candidate checks, patch staging, spec edits, and final
replay. Preparation performs RPM operations on that backend; it does not own
the backend or its lifecycle. Verified base trees and the disposable candidate
copy are released before the edited-spec replay, so a ticket never holds both
tree sets at once.
RPM macro expansion and
rpmbuild -bp --nodeps run only inside that sandbox. RPM owns architecture
conditions, source selection, macros, patch ordering, archive extraction,
and shell commands.
The worker has no package checkout. A temporary, hardened no-checkout Git object
transport pins one repository commit and exports only rpms/<package>/,
metadata/<package>.json and mock/dist-git-client.ini from that
same commit. The worker brokers source/lookaside downloads, validates checksums,
and fetches upstream patches; the selected inputs are validated and then
transferred once as a single archive stream, preserving executable package
scripts. Every RPM preparation uses fresh
sandbox-local copies of those inputs, so a script changing its sources cannot contaminate
the next architecture or final replay. Downloads are limited to 128 MiB each, with a shared 256 MiB
input budget. The execution sandbox has no network or worker credentials, so
missing inputs, macros, signatures, or preparation helpers block preparation
instead of triggering dependency installation or guessed behavior.
The worker retains immutable metadata, checksum pins, and lookaside routing facts from that Git snapshot. It never uses sandbox-modified copies to choose download destinations or checksums; these acquisition facts are not a second working tree.
Inspection reads and searches the real prepared tree, cached per architecture. Reads reject escaping paths and symlinks and are limited to 8 MiB per file; searches are limited to 48 MiB and 50,000 filesystem entries. Commands have timeouts, bounded diagnostics, and a 256 MiB individual-file limit. Podman keeps its original shared startup behavior and writable filesystem. CPU, memory, storage, process-count, and container-lifetime policy belong to deployment infrastructure; the CVE agent does not impose them on shared Podman users. Infrastructure enforcement is deferred, so these application-level checks do not provide a whole-sandbox resource bound for local runs.
Candidates are checked on disposable copies using GNU patch with fuzz zero.
Before each candidate is applied, a reverse dry-run on the clean copy separates
source that already carries the fix from source that needs a rebased patch: the
first blocks backporting as CVE already applied, the second as a patch that
does not apply. Mismatched context is rejected without retrying with greater
fuzz. Exact-context hunks that land at a line offset still pass, but they are
recorded and reported to the MR reviewer.
Original source paths take precedence. When they are missing, deterministic
adaptation can remove a nested monorepo prefix, add a downstream prefix such as
src/ or vendor/<package>/, or replace one shared directory prefix with another.
It preserves relative paths and requires a unique mapping supported by every
initially existing file touched by the ordered series. Added files must not
overwrite existing destinations. A bounded search ignores symlink directories;
ambiguous or incomplete searches are rejected, and addition-only patches do not
guess a target directory. Only diff headers change, never source hunks or commit
messages; individual file renames and source-level rewrites are not inferred.
Automatic backporting must use the same adapted patch on every applicable
supported target (x86_64, aarch64, and s390x). Source tools accept an explicit
arch; when multiple targets exist, they require it rather than guessing which
published binary the model is investigating. A selected architecture remains in
effect for later source-tool calls on the same ticket unless explicitly changed.
The installed specfile library performs structural edits. The edited spec is
then prepared again from clean inputs, using RPM’s own patch backend and options.
RPM’s default patch fuzz is explicitly set to zero for every preparation.
Every candidate-touched file must match its verified final content, including
deletions; a later source overlay cannot silently erase the fix. If necessary,
explicit architecture-guarded patch applications are appended and verified again.
The CVE-agent sandbox image supplies RPM configuration, macro packages, and prep helpers. Preparation does not establish buildroot equivalence, native execution on another architecture, compilation, runtime behavior, or that a fix resolves the CVE. Konflux builds and human review remain required.
3. Review the generated changes
Only verified patch text is staged. One file is created for each selected upstream patch; the spec and package metadata are updated as follows:
| Output | Change |
|---|---|
rpms/<package>/<NNNNN>-<cve>-<slug>.patch |
New numbered patch file; the slug uses up to five words from the ticket summary, or the short commit ID if no usable summary exists. |
Package .spec |
Adds patch declarations and arranges their application. Bumps Release by a trailing micro version (4.1 → 4.2), including %{gcc_release}.1 → %{gcc_release}.2 and values from a %global macro, so local changes stay out of Fedora’s integer namespace; date-shaped releases require human review. |
metadata/<package>.json |
Marks imported packages modified and adds the ticket/CVE to modification_reason. Independent packages remain independent; unrelated metadata is preserved. |
| Why preparation stopped | Result |
|---|---|
Supported %prep, path, or patch checks reject the fast-path fix |
needs_human, no file_actions. |
AI investigation says needs_change, but automatic preparation fails |
Keep the finding and confidence; add the preparation failure to the summary; no file_actions. |
| Critical source fetch, checksum, or other evidence collection fails | Abort with a local diagnostic report; retry behavior follows the failure classification. |
4. Hand off the prepared changes
Use --submit-mr to hand off prepared actions; see
Writes for submission and review
details.
Planned work
HUM-8512 tracks one bounded AI attempt to adapt a patch after a hunk conflict. The planned flow replays the existing package patch stack, then verifies the adapted candidate from a clean tree with zero fuzz. It also proposes regenerating the candidate as a clean downstream diff and verifying that diff before staging. Source-level AI adaptation and patch regeneration are not implemented today; deterministic directory-prefix mapping is tried automatically during source checks and backport verification.
Writes: Jira comment and MR submission
Both writes are off by default in CLI use; each is enabled solely by its own flag, so a
bare run never touches Jira or GitLab. The two flags are independent — a
comment can be posted without submitting an MR and vice versa — and a failed
write exits non-zero. The result writer runs first, before either remote write.
When both are enabled, the MR is submitted before the Jira
comment and analysis attachment, so a failed MR submission leaves the ticket
eligible for a later sweep.
-
--post-to-jiraposts a deterministicRECOMMENDATION: ...comment rendered from the validatedInvestigationResult— no AI call at comment time — throughcve_analysis’sjira_client, which handles ADF formatting and duplicate suppression, so re-running a command is idempotent. It also synchronizes onecve-agent-*Jira label to the current verdict, replacing any prior CVE Agent verdict label and preserving unrelated labels. A label update failure stops the write before the analysis attachment is uploaded. The comment carries the verdict, summary, cited evidence, and the next review step when human review is needed, including when a necessary package fix has no automated backport. The full result JSON is attached as<TICKET>-cve-agent-analysis.jsonwith the supporting evidence, file actions, commit message, and atoken_costblock (input_tokens,output_tokens,cache_read_tokens,cache_creation_tokens,unpriced_calls,estimated_cost_usd) measured from every model response of the investigation. The cost is a subtotal of the calls with known pricing;unpriced_callscounts the rest, so a non-zero count makesestimated_cost_usda lower bound. A duplicate comment does not prevent the agent from synchronizing the label or uploading a missing analysis attachment. Existing tickets are not backfilled; a later--post-to-jirarun synchronizes their verdict label.Verdict Jira label needs_changecve-agent-needs-changenot_affectedcve-agent-not-affectedalready_fixedcve-agent-already-fixedno_upstream_fixcve-agent-no-upstream-fixneeds_humancve-agent-needs-humanerrorcve-agent-errorAs tickets are currently evaluated only once,
no_upstream_fixtherefore would need a TTL (for example 24 hours) after which the ticket is investigated again, since a fix may have landed since the verdict. -
--submit-mrsubmits a draft MR work item to a runningworkqueue-serviceinstance (work items default toauto_merge=false). It is a no-op unless the result carries a prepared backport (file_actions+commit_message). Submission POSTs toWORKQUEUE_SERVICE_URLwith the current token fromWORKQUEUE_SERVICE_TOKEN_FILEwhen configured, otherwiseWORKQUEUE_SERVICE_TOKEN; the file is re-read for each submission so projected token rotation is honored. The agent is headless, so there is no interactive login fallback. The description follows the rpms CVE workflow’s Scope/Fix/Verification structure: the Jira tracker, package, CVEs, and built NVR; the summary, selected upstream fixes, and investigation steps; and the checks performed versus those left to Konflux and human review. Currently,workqueue-serviceonly applies this description when updating an existing MR.
Automated sweep deployment
After a successful cve_analysis --resolve cycle, the analysis CronJob publishes
a small SNS event requesting a sweep. A dedicated SQS queue filters for that
event, and one long-running agent Deployment runs the existing Jira search and
investigates eligible tickets. The message is a wake-up signal; it contains no
ticket data. Every request starts a fresh Jira search. The deployed child sweep
uses two workers by default; HUMMINGBIRD_CVE_AGENT_WORKERS sets that default,
and the worker’s --workers option overrides it. The value must be positive.
The sweep attempts at most forty tickets and enables both --post-to-jira and
--submit-mr.
The consumer renews message visibility while the sweep runs, enforces a one-hour deadline, and acknowledges only a completed sweep report. A completed pass is acknowledged even if some individual tickets failed; those tickets stay eligible for a later sweep. Failed or interrupted sweeps are retried by SQS. Repeated requests are handled sequentially by the single worker replica.
Before rollout, infrastructure must budget for the configured number of concurrent sandbox pods plus the controller pod, enforce per-pod resource limits, and set sandbox deadlines to cover the full one-hour sweep (the shared 1,800-second default is insufficient). Reap orphan sandbox pods before deployment. These are infrastructure requirements; this change adds no resource-policy flags or shared backend changes.
The analysis publisher has permission to publish to its environment’s SNS topic;
the agent has permission to consume only its dedicated SQS queue. Neither needs
cluster-wide Kubernetes API access; the agent needs pod management only in the
sandbox namespace. The worker uses a projected short-lived token for the
workqueue-service audience, where its identity can only submit mr work items.
Both environment switches start disabled in
infrastructure/vars.d/hummingbird-cve-agent.yml. Deploy the SAM stack and
provision each environment’s publisher and worker IAM keys in Vault before
enabling its switch:
| Variable | Purpose |
|---|---|
HUMMINGBIRD_CVE_AGENT_SWEEP_ENABLED_STAGING |
Enable staging event publishing and the worker replica. |
HUMMINGBIRD_CVE_AGENT_SWEEP_ENABLED_PRODUCTION |
Enable production event publishing and the worker replica. |
Eval harness
eval scores investigations against completed human /cve sessions. The eval
set, eval-tickets.txt, holds Jira ticket keys and nothing else — ticket
content is internal data and is never committed — so facts are fetched live and
running the eval needs Jira access.
hummingbird-cve-agent eval eval-tickets.txt --output eval-report.json
Read-only: every Jira request is a GET (issue, comments, changelog);
test_evaluate asserts it.
A resolved ticket gives the answer away — fixed_in_build set, labels moved to
awaiting-vex, status Closed, newest analysis comment post-fix — so each ticket
is rewound through its changelog, dropping later comments.
The rewind cutoff is the last scope-label handoff plus five minutes, allowing
the assessment comment from the same triage burst to remain visible. Resolution
time is the fallback when no handoff was recorded or it occurred after resolution;
the fix-build timestamp is not the cutoff. fixed_in_build is also blanked so
the recorded answer cannot leak through that field.
What is scored
Two axes, because one is not enough:
- Disposition — where each side says the ticket ends up. The verdict is
mapped to its eventual disposition, not through
states.VERDICT_OUTCOME: that shared table answers “what does the agent do next”, soneeds_changeis In Progress — but a ticket the human backported still endsDone-Errata, and scoring against the immediate state would mark a correctneeds_changeas a miss and reward a wrongalready_fixed. - Route — which branch the orchestrator would take, since
already_fixed(Fixed in Build) andneeds_change(MR creation) share theDone-Errataending and are indistinguishable on the first axis. The human’s branch is derived from Pulp upload times: a fix build published after the handoff means the package had to change (needs_change), an earlier one means the shipped package already carried the fix (already_fixed); aNot a Bugclosure isnot_affected. A build Pulp no longer lists is reported asn/aand excluded from the route denominator rather than guessed.
Verdicts that reach no conclusion (needs_human, error) and conclusions no
verdict can produce (unknown resolutions) never match. Package EOL Won't Do
closures are deterministic cve_analysis outcomes and excluded before the AI
eval runs. A ticket that cannot be rewound or investigated scores error
without aborting the run.
HUM-7554 agree verdict=already_fixed expected=Closed / Done-Errata actual=Closed / Done-Errata route=!=needs_change
HUM-7537 DISAGREE verdict=needs_human expected=Closed / Won't Do actual= route=n/a
disposition agreement: 1/2 (50%)
route agreement: 0/1 (0%), 1 undetermined
note: package facts (SRPM, SBOM, rpms repo) are current state, not the state at decision time, so agreement is retrospective
Only the Jira side is rewound. Package facts — the latest SRPM, the latest SBOM,
and the pinned rpms repository inputs — are read live, so an old ticket is
re-investigated against today’s package state rather than the state that existed
when the human decided. Where the fix has since shipped, already_fixed is
partly handed to the agent, so the score is retrospective current-state
agreement, not a historical replay; every report prints that caveat. True
replay needs as-of-cutoff SRPM and SBOM selection plus a date-pinned rpms
revision, and is deliberately left to a follow-up.
What the model sees
Only non-candidates reach the model. The prompt combines static /cve
instructions with validated TicketFacts: ticket, package and CVE details,
Jira content, analysis/discussion, built NVR, and SBOM identity.
The full SBOM is not included. Evidence is collected through bounded tools:
| Evidence | Tool | What it establishes |
|---|---|---|
| SBOM contents | search_sbom |
Searches SPDX metadata, including npm PURLs. Only present confirms shipping; ambiguous versions or unknown require investigation. |
| Packaging files | search_package_files, read_package_file |
Finds build settings, patches and packaging evidence. Searches can be scoped to a relative file or directory; truncated results are incomplete, and empty results or scans stopped by file, byte or member limits are inconclusive. |
| Other packages | lookup_shipped_package |
Returns the latest source-package SBOM, NVR, component version and binary presence. source_only or missing data is inconclusive. |
| Prepared source | read_package_source and source searches |
Inspects sandboxed, spec-pinned source and records provenance. built_nvr_match: unverified means it is not proven to match the ticket build. Preparation failures are not bypassed. |
| npm metadata | Registry lookup | Provides publication time, gitHead, repository and integrity. Metadata alone proves neither shipping nor fixedness. |
| CVE advisories | lookup_cve_advisory |
Returns affected packages, vulnerable ranges and patched versions from GitHub advisories. Missing, truncated or mismatched results are inconclusive. |
| Known upstream fix | check_upstream_fix |
Checks a commit from the package’s upstream repository against prepared source. It must be merged into the upstream default branch or its declared release branch. Clean application means that exact diff is absent; already_applied means it reverse-applies cleanly. Other failures are inconclusive. |
| Upstream history | Upstream source probe | Searches an official repository once per ticket within a 20-candidate and 10 MiB discovery budget. incomplete: true means the search was truncated, so a missing candidate is inconclusive. Ancestry and applicability require a resolved exact base commit; unresolved candidates are not evidence. No candidate found means needs_human, not no_upstream_fix. |
Evidence rules
- Keep conclusions narrow. Patch applicability,
%filesentries, missing disable flags and failed searches alone do not prove affectedness, shipping or fix inclusion. Unresolved conclusions requireneeds_human. - Respect bounds. Tools enforce request, response and shared read budgets. Reads support line windows; excerpts may be truncated. Upstream access uses approved HTTPS hosts without redirect fallback.
- Do not repeat permanent failures.
retryable: falsemeans move on and record the gap. Source tools permit retrying witharchwhen architecture selection is required. - Preserve partial evidence. Upstream collection failures abort the run;
unresolved refs remain partial evidence. A bounded upstream search that hits
its candidate or byte limit returns
incompleteand is recorded aspartial.
Model output
The model returns only status, confidence, summary, evidence and
vex_justification. vex_justification is accepted only with not_affected
and only as Component not Present or Vulnerable Code not Present.
The caller supplies file_actions, commit_message, ai_generated, ticket,
package and cves, and replaces collection records with complete evidence.
Security
The agent processes untrusted input, including executable RPM macros and prep commands. The credential-bearing worker never executes or imports RPM specs.
- Isolated source execution. Spec parsing, structural edits, preparation,
and candidate application run in an ephemeral, uncredentialed sandbox. There
are no host mounts or worker secrets, and command environments are cleared.
Sandbox backends come from the shared Hummingbird Sandbox library.
Podman disables networking; Kubernetes requires deny-all egress policy in the
dedicated sandbox namespace. Privileged
mock, full builds, and runtime dependency installation are not used. Reusing a container with a reset workspace is not a fresh security boundary: use controlled images and packaging, not mutually hostile tenants. - Deployment isolation. Repository Git operations use private, temporary no-checkout object transport; package files exist only in the offline sandbox. The worker may manage execution pods only in the sandbox namespace once explicit API credentials are available. Those pods run non-root, drop capabilities, disable privilege escalation and token mounts, and use infrastructure-managed pod deadlines and resource limits sufficient for the sweep. Normal context-manager exit deletes sandbox pods; forced termination can leave pods until their active deadline, so operators must remove leftovers. Standalone investigations clean up at run end. Sandbox-produced source trees are never extracted into the worker.
- Secret-free repository subprocesses. Worker Git transport operations use a minimal, allowlisted environment and do not inherit the Jira or model tokens used by the trusted worker and sweep CLI.
- Uncredentialed, hardened git. Repository transport is public and read-only with
no push token. Credential helpers and interactive prompts are disabled,
system/global git config is neutralized (an attacker-controlled
GIT_CONFIG_GLOBALcannot change behaviour), and submodule recursion is off. - Input validation. Package names are restricted to a safe charset (no path
traversal or absolute paths), file reads are confined to the package
directory, saved patch names must be
*.patchand cannot overwrite existing files, and patch URLs must use HTTPS on a recognized upstream forge.
Development
See the main README for development workflows.
Build the local CVE sandbox image for explicit local configuration overrides:
podman build -f hummingbird-sandbox/images/cve/Containerfile \
-t localhost/hb-cve-sandbox:latest hummingbird-sandbox/
make hummingbird-cve-agent/setup
cd hummingbird-cve-agent
PYTHONPATH=.. python3 -m unittest discover -s tests -t .
# Opt into real offline RPM tests using an image with the required macros:
PYTHONPATH=.. HUMMINGBIRD_RPM_TEST_IMAGE=registry.example.com/rpm-prep:dev \
python3 -m unittest tests.test_rpm_integration
The integration suite uses Podman without network access. Only source downloads are mocked; RPM parsing, Go preparation, architecture conditions, source overlays, patch checks, and final-spec replay execute for real. Standard unit discovery skips these tests unless the image variable is set.
The Containerfile pins the RPM repository commit supplying hum in HUM_COMMIT
and records it in the io.hummingbird-project.hum-revision image label. Keep that
pin aligned with the CVE analysis Containerfile and .gitlab-ci.yml, so test and
coverage jobs exercise the same library as both images. The CI revision test in
ci/internal/tests/test_hum_revision.py checks that the pins match. The agent’s
fetch layer comes before Python dependencies, so dependency-only changes reuse
the pinned checkout.
Rebuild and deploy both CVE images to refresh their bundled library. Updating a
runtime RPMs checkout does not replace an already-imported hum.meta.
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.
4.5.37 - Upstream Monitor
Upstream release discovery and catalog API for Hummingbird. Application code
lives in upstream-monitor/. Project definitions belong to the separate
upstream-monitor-projects repository.
Features
- Release discovery from Git, GitHub, GitLab, Anitya, and HTTP sources
- Version filtering, release streams, lifecycle dates, and Git provenance
- Catalog API and web interface
- File, PostgreSQL, or S3 storage
- Git catalog reconciliation and optional NATS JetStream events
Prerequisites
- Go 1.27 or newer and a C compiler for local builds and tests
- Podman for container builds
- A project catalog and writable file storage, PostgreSQL, or S3 for results
- Deployment managed credentials for authenticated upstreams and API operations
Build and Deployment
From the tools repository root:
podman build -t localhost/upstream-monitor:dev upstream-monitor/
Konflux builds ARM64 and AMD64 images as upstream-monitor--tools and releases
them to quay.io/hummingbird-ci/upstream-monitor. The image runs as UID 65532
and contains these commands:
| Command | Purpose |
|---|---|
upstream-monitor |
Poll upstreams and persist results, the default entry point |
upstream-monitor-api |
Serve the catalog API and web interface |
upstream-monitor-enricher |
Resolve optional Git provenance |
upstream-monitor-project-reconciler |
Reconcile Git definitions into PostgreSQL |
The infrastructure repository owns deployments, storage, messaging, credentials, and image pins. Register the Konflux component before building, and pin the released image digest in infrastructure and the catalog validator. Tools does not deploy the service or bundle a project catalog.
Usage
Validate a catalog
From the tools repository root:
podman run --rm --pull=never --network=none --read-only \
-e PROJECTS_DIR=/projects \
-v "$PWD/upstream-monitor/examples/file-layout/config/projects:/projects:ro" \
localhost/upstream-monitor:dev validate
Replace the mounted directory with a checkout of the deployment catalog to validate it. Validation does not contact upstreams or write to a database. An empty or missing directory validates zero projects, so check the reported count before treating validation as successful.
Run locally
From the application directory, start the monitor:
cd upstream-monitor
PROJECTS_DIR=examples/file-layout/config/projects \
go run ./cmd/upstream-monitor
In another terminal, from the same directory, start the API:
PROJECTS_DIR=examples/file-layout/config/projects \
go run ./cmd/upstream-monitor-api
Both processes must use the same project and results storage. File results
default to observations/, with provider cache data under cache/.
Without project definitions, the monitor has nothing to check.
For an upstream check without persisting results:
go run ./cmd/upstream-monitor dry-run \
--projects-dir /path/to/projects --project example-widget
dry-run contacts the upstream and can consume API quota. It resolves credentials
through source.credential_ref, not the monitor’s GitHub App configuration.
Configuration
| Variable | Default | Purpose |
|---|---|---|
PROJECT_STORAGE_BACKEND |
file |
Definitions: file, postgres, or s3 |
RESULTS_STORAGE_BACKEND |
file |
Results: file, postgres, or s3 |
PROJECT_MUTATIONS |
auto |
Definition writes: auto, enabled, or disabled |
PROJECTS_DIR |
config/projects |
External project JSON directory |
POSTGRES_DSN |
empty | Required when either backend is PostgreSQL |
S3_BUCKET |
empty | Required when either backend is S3 |
S3_PREFIX |
empty | Unique prefix for each installation sharing a bucket |
S3_REGION |
us-east-1 |
AWS region |
S3_ENDPOINT |
empty | Optional S3 compatible endpoint |
S3_USE_PATH_STYLE |
false |
Enable path style S3 addressing |
OBSERVATIONS_DIR |
observations |
File results directory |
CACHE_DIR |
cache |
File backend provider cache |
LISTEN_ADDR |
:8080 |
API listen address |
POLL_INTERVAL |
5m |
Scheduler interval and default project check interval |
UPSTREAM_TIMEOUT |
5m |
Maximum duration of an upstream check |
MAX_CONCURRENT_CHECKS |
4 |
Concurrent checks per worker |
MAX_CONCURRENT_GIT_FETCHES |
1 |
Concurrent uncached Git fetches per worker |
ENRICHMENT_ENABLED |
false |
Include provenance in API responses |
File results support a single writer. Use PostgreSQL or S3 for shared worker coordination. S3 must support conditional writes. Do not expire observation or coordination objects through bucket lifecycle policies.
Use GITHUB_TOKEN for token based GitHub access. For a GitHub App, set
GITHUB_APP_ID, GITHUB_APP_INSTALLATION_ID, and
GITHUB_APP_PRIVATE_KEY_FILE together. A project’s source.credential_ref
names an environment variable containing a credential, never the credential
itself. Keep secrets outside Git and container images.
See configuration defaults and environment overrides for remaining cache, lease, retry, and concurrency settings.
Project Definitions
Each JSON filename must match its id. IDs start with a lowercase letter,
contain only lowercase letters, digits, and hyphens, and have at most 128
characters. Do not supply uid, which the service assigns.
{
"id": "example-widget",
"name": "Example widget",
"homepage": "https://example.org/widget",
"source": {
"type": "github-releases",
"location": "example/widget"
},
"version_policy": {
"scheme": "semantic",
"include_prereleases": false
},
"stream_policy": {
"granularity": "minor"
},
"check_policy": {
"mode": "fixed",
"interval": "15m"
}
}
| Source type | Location |
|---|---|
git-tags |
Git remote URL |
github-releases |
GitHub owner/repository |
gitlab-releases, gitlab-tags |
GitLab numeric project ID or namespace path |
anitya |
Numeric Anitya project ID |
http-regex |
HTTP(S) URL with a source.extract regular expression |
Set source.base_url for a separate GitLab or Anitya instance, and
source.tag_pattern to select a tag family. Upstream destinations are not host
allowlisted, so infrastructure must enforce the required network egress policy.
The default version scheme is semantic. Other choices are calendar, rpm,
pep440 (python), and lexical (generic). Use transforms to normalize
versions, and version_filters and prerelease_filters to control release
selection. Streams group releases by major or minor, with optional semantic
numeric component buckets.
Optional lifecycle modes are latest-release, latest-n-lines, date-based,
endoflife-date, and manual, with per stream overrides. Fixed check policies
use interval. Adaptive policies back off between min_interval and
max_interval, resetting after a new release. The maximum interval is also the
worst polling detection delay.
Catalog Reconciliation
The project reconciler validates an atomic git-sync checkout and replaces the
PostgreSQL definition set. Supply POSTGRES_DSN through the environment:
PROJECT_STORAGE_BACKEND=postgres \
PROJECT_MUTATIONS=disabled \
PROJECT_CHECKOUT_LINK=/git-sync/current \
PROJECT_CHECKOUT_SUBPATH=projects \
go run ./cmd/upstream-monitor-project-reconciler
Run from upstream-monitor/. The projects subpath matches the separate
catalog repository. Set PROJECT_MUTATIONS=disabled on every process using
that catalog. Workers and API replicas read its database copy.
Invalid revisions leave the last valid catalog active. Reconciliation removes definitions absent from Git. Verify completeness and back up the database before enabling it. Git cannot restore deleted observation history.
The monitor initializes database schemas. The API does not. Schema and image rollouts belong to infrastructure, not API startup.
API and Authentication
Open / for the catalog browser. The generated API schema is available at
/openapi.json and /openapi.yaml. /docs redirects to the JSON schema.
| Endpoint | Purpose |
|---|---|
/healthz |
Process health |
/readyz |
Project store readability, not proof of successful polling |
/v1/status |
Aggregate and per project check status |
/v1/project-reconciliation |
Attempted and active Git revisions and reconciliation errors |
/v1/projects |
Project search and pagination |
Use last_success_at and project errors to assess freshness. Follow
X-Next-Cursor for paginated reads. Definition updates and deletes require the
current ETag in If-Match. A queued scan retains observations, while
rescan clears that project’s snapshot and release history before checking.
A registered project may have no stored snapshot before its first successful
scan. When project status has no last_success_at and does not report
succeeded, GET /v1/projects/{project_id} returns 200 OK with the project
identity, empty releases and streams arrays, and observed_at set to the zero
timestamp 0001-01-01T00:00:00Z. The /streams and project /releases endpoints
also return empty arrays. These reads do not persist a snapshot or mark a scan
successful. Read /v1/projects/{project_id}/status for scan state and failure
details.
Unknown projects and streams still return 404 Not Found. A missing snapshot
after a successful scan also remains an error. Unexpected storage failures,
including an inability to read project status, remain server errors rather than
empty results. Without a configured status reader, a missing snapshot remains
404 Not Found because the API cannot verify that no scan has succeeded.
Catalog reads are public. Definition reads, writes, inline checks, and scan
requests require authentication and Cedar authorization. PROJECT_MUTATIONS=auto
disables file definition writes but permits supported PostgreSQL and S3 writes.
API_AUTH_MODE |
Required configuration |
|---|---|
disabled |
Default. Authenticated operations are unavailable. |
static-token |
API_AUTH_TOKEN, API_AUTH_PRINCIPAL, API_AUTH_ROLES |
kubernetes-tokenreview |
API_KUBERNETES_AUDIENCE and a role mapping file |
oidc |
API_OIDC_ISSUER, API_OIDC_AUDIENCE, and a role mapping file |
trusted-proxy |
API_TRUSTED_PROXY_CIDRS and a role mapping file |
insecure |
Local development only. Never use in production. |
API_AUTH_ROLE_MAPPINGS_FILE maps subjects and groups to Cedar roles:
{
"subjects": {
"system:serviceaccount:ci:release-bot": ["ProjectChecker"]
},
"groups": {
"editors": ["ProjectEditor"]
}
}
The default policy grants definition reads to ProjectReader, inline checks
to ProjectChecker, editing and scan operations to ProjectEditor, and all
actions to ProjectAdmin. Use API_CEDAR_POLICY_FILE to replace that policy.
TokenReview requires a mounted reviewer token and permission to create
tokenreviews.authentication.k8s.io. Trust proxy identity headers only from an
isolated authenticating proxy that callers cannot bypass. Remove inbound
identity headers at the proxy, and prefer loopback CIDRs for a sidecar rather
than trusting a whole cluster network.
Project status
Each project fetch runs independently. A fetch failure or timeout records a project error and schedules a retry with backoff. Other project checks continue.
Repository lease errors do not stop the monitor. The scheduler reports the error and defers unfinished projects until the next poll interval. Completed checks retain their results. Deferred checks do not count as upstream failures. A batch lease claim error defers the due projects for the same interval. Cancellation still stops the monitor.
Workers report progress and results through the scheduler’s event channel. Only the scheduler changes the in-memory project schedules and statuses.
Successful checks return to scheduled. waiting and succeeded are accepted
for compatibility with older stored records:
stateDiagram-v2
[*] --> queued : definition due now
[*] --> scheduled : definition with future check
[*] --> waiting : load legacy record
[*] --> succeeded : load older successful record
waiting --> queued : restart with due check
waiting --> scheduled : restart with future check
succeeded --> queued : next check claimed
scheduled --> queued : check due, scan, or definition changed
queued --> processing : worker starts
queued --> scheduled : repository lease unavailable
processing --> scheduled : snapshot persisted
processing --> failed : check or persistence error
failed --> queued : retry or definition changed
processing --> queued : recover interrupted file worker
Records include the project identity, definition fingerprint, pending scan
request, queue/start/finish/next check times, last_success_at, failure count,
last error, and release/stream counts.
Events and Monitoring
Set NATS_URL to enable JetStream publishing and NATS_CREDS_FILE for mounted
credentials. Default subjects are hummingbird.upstream.version.v1 and
hummingbird.upstream.stream.v1. Set NATS_MANAGE_STREAM=false if
infrastructure provisions the stream. Delivery is at least once, so consumers
must deduplicate using event_key. Publication failures retain events for retry.
Logs are JSON on stdout by default, configured through LOG_LEVEL and
LOG_FORMAT. Set OTEL_METRICS_EXPORTER=otlp with standard OpenTelemetry
endpoint settings to export metrics. Unset exporter configuration does not
connect to a local collector.
Monitor time since the last successful check, check failures, GitHub quota, and outbox depth and oldest pending age. Quiet projects remain healthy when checks succeed and adaptive intervals grow.
Development
Run from upstream-monitor/:
go vet ./...
go test ./...
go build ./cmd/...
go run ./cmd/upstream-monitor validate --projects-dir examples/file-layout/config/projects
The GitLab job also rejects empty fixture directories. Container smoke tests mount the examples and check API health through the K8s test runner.
Integration tests require disposable services. Set TEST_POSTGRES_DSN for
PostgreSQL storage, TEST_POSTGRES_RECONCILE_DSN for a separate catalog
replacement database, TEST_S3_ENDPOINT and TEST_S3_BUCKET for S3, or
NATS_CLUSTER_TEST_URL for a JetStream cluster with three stream replicas.
Never point these tests at production services.
License
See the repository LICENSE.
4.5.38 - Hummingbird OSV Generator
A CLI tool that converts Red Hat vulnerability data into OSV records for Hummingbird RPMs, so that vulnerability scanners can match them against the packages installed in an image.
For how a run works end to end, see the workflow.
Features
- Two sources: shipped fixes from the advisories repository and all known affects, including unfixed CVEs, from OSIDB
- Binary RPM names: scanners match installed binary packages, so source RPMs are expanded to their binary RPMs
- Fixed and unfixed states: records carry
fixed,affected,will_not_fixorunder_investigationindatabase_specific.status - RPM epochs: fixed versions are written as
epoch:version-release - Incremental sync: content hashing means only changed inputs are parsed and only changed records are written
- Embargo safe: embargoed or non-public OSIDB data is never published
Output format
The output follows OSV schema 1.9.0 with one record per source RPM per CVE and
one affected entry per binary RPM.
| Field | Content |
|---|---|
id |
Opaque, stable identifier: x_HBA prefix, CVE year and a hash |
upstream |
The CVE the record covers |
affected[].package.ecosystem |
Ecosystem scoped to one product stream by its CPE |
affected[].package.purl |
Versionless package URL |
affected[].ranges |
introduced: 0 and, once shipped, fixed: <epoch:version-release> |
affected[].ecosystem_specific.advisory_id |
Advisory that supplied the fixed version, absent on unfixed entries |
database_specific.status |
fixed, affected, under_investigation or will_not_fix |
modified |
Time of the last content change |
The x_ in the identifier prefix marks a local database. It is dropped once
HBA is registered with the OSV schema project. The ecosystem name is also a
placeholder until it is registered.
Prerequisites
- Go: the version in
hummingbird-osv-generator/go.mod - Advisories checkout: a clone of the advisories repository, passed as its
data/directory - OSIDB access: the OSIDB base URL, reachable from where the tool runs
- Git: only when using
-gitor-git-remote
Installation
git clone https://gitlab.com/redhat/hummingbird/tools.git
cd tools/hummingbird-osv-generator
go build -o rhcve-osv-sync ./cmd/rhcve-osv-sync/
Usage
Full sync for one product stream, committing the output to git:
./rhcve-osv-sync \
-advisories /path/to/advisories/data \
-osidb-url https://osidb.example.com \
-streams hummingbird-1 \
-products hummingbird \
-output /path/to/osv-output \
-state /path/to/state.db \
-full -git
Later runs without -full use an OSIDB delta query. Run a full
reconciliation daily so that missed changes are corrected.
Binary RPM expansion
OSIDB tracks CVEs against source RPM names, while scanners need binary RPM
names. -expansion selects how a source RPM is expanded:
| Mode | Source |
|---|---|
osidb |
OSIDB subpackage_purls |
advisory |
Binary RPM names derived from advisory purls |
chain |
OSIDB first, then the advisory mapping |
The default is chain.
Incremental sync
State lives in a SQLite database. Each run goes through these phases: advisory diff, OSIDB diff, convert, write output and commit state. The workflow describes each phase.
- Advisory files are content-hashed and only changed files are parsed
- OSIDB uses a delta query, plus a full reconciliation on demand
- Output files are compared with a manifest and only changed files are written
- Output is written before state is committed, so a crash is safe
- If the manifest is lost, the next full run rebuilds it from the output and withdraws stale records
- If OSIDB fails and the convert phase does not need it, advisory output is still published and the process exits non-zero so a scheduler can retry or alert
Unchanged input produces unchanged output, so duplicate scheduling wastes work but does not corrupt the output.
Configuration
| Flag | Required | Description |
|---|---|---|
-advisories |
Yes | Path to the advisories repository data/ directory |
-osidb-url |
Yes | OSIDB base URL |
-streams |
Yes | Comma-separated ps_update_stream values, also filters advisories by product_stream |
-output |
Yes | Output directory for the OSV JSON files |
-state |
Yes | Path to the SQLite state database |
-products |
No | Filter advisory conversion by CPE product name |
-full |
No | Force a full OSIDB reconciliation |
-expansion |
No | Binary RPM expansion mode: osidb, advisory or chain |
-git |
No | Run git init, add and commit on the output after the sync |
-git-remote |
No | Git remote to push to after the commit (implies -git) |
-log-json |
No | Write structured JSON logs |
Development
cd hummingbird-osv-generator
# Vet and test, including the OSV schema validation test
go vet ./...
go test ./...
The hummingbird-osv-generator CI job runs the same commands on merge
requests.
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.
4.5.38.1 - Hummingbird OSV Generator Workflow
How a run of the Hummingbird OSV Generator turns its inputs into OSV records: what it reads, the phases of a sync, and what it writes.
Overview
flowchart LR
adv[Advisories data directory] --> p1[Advisory diff]
osidb[OSIDB] --> p2[OSIDB diff]
db[(State database)] --> p1
db --> p2
p1 --> p3[Convert]
p2 --> p3
osidb --> p3
p3 --> p4[Write output]
p4 --> out[OSV JSON files]
p4 --> p5[Commit state]
p5 --> db
out --> git[Optional git commit and push]
One run reads the advisories and OSIDB, converts what changed into OSV records, writes only the records whose content changed, and records what it did in the state database. Each run is a complete pass: nothing is kept in memory between runs.
Inputs
| Input | Flag | What the generator reads |
|---|---|---|
| Advisories directory | -advisories |
Advisory YAML files: shipped fixes with per-artifact package URLs and CVE fix lists |
| OSIDB | -osidb-url |
Affects for the given update streams, and the flaws for their CVEs |
| Update streams | -streams |
Which OSIDB update streams to read. Also filters advisories by product_stream |
| Products | -products |
Optional filter on the CPE product name of an advisory |
| State database | -state |
SQLite file from earlier runs, created on the first run |
| Output directory | -output |
Read only when the record ledger was lost and has to be rebuilt from the published files |
Advisories
- Only security advisories (RHSA) are read. Bug fix and enhancement advisories are skipped, even if they mention CVEs.
- Each advisory gives the product CPE, and each artifact gives a package URL and the CVEs it fixes. The package URL carries the binary RPM name, the version and release, and the RPM epoch.
- From the advisories the generator also builds two lookups: the update stream to CPE map, and the source RPM to binary RPM mapping.
OSIDB
An affect is ignored when it has no CVE ID, when it is embargoed or not public, or when it is marked not affected (including a delegated not affected). An affect whose tracker is resolved as shipped (Done-Errata or Fixed) is also ignored, because the advisory supplies that fix.
Flaws supply the CVE title and description, severity, CWE and Red Hat statement. A flaw that is embargoed or not public is ignored, and any text cached from it is dropped. A rejected flaw marks its records as withdrawn.
State database
| Table | Content |
|---|---|
input_hashes |
Content hashes of advisory files and of OSIDB affects, used to detect changes |
sync_meta |
Cursors: the last sync, the last full reconciliation, and the last flaw change window |
cve_meta |
Cached CVE text and severity, refreshed from the OSIDB flaw change feed |
output_manifest |
One row per published record: path, content hash, time written and time last modified |
The input hashes and the cursors are a cache that can be rebuilt. The manifest also holds each record’s last modification time, so it is durable state.
Phases
- Advisory diff. Hash every advisory file and compare with the stored hashes. The result is the list of added, changed and removed files.
- OSIDB diff. Decide between a delta and a full reconciliation. A delta
asks OSIDB for affects updated since the last sync, minus a clock skew
allowance, and compares their hashes. A full reconciliation fetches every
affect and also detects deletions. By default a full reconciliation runs on
the first run, when
-fullis given, and again once a day. If OSIDB fails here, the run goes on with advisory data when nothing needs OSIDB, and ends with an error (see Failure handling). - Convert. Build records from both sources:
- Fixed records come from the advisories.
- Records for every other status come from OSIDB affects. Each affect is
expanded from its source RPM to binary RPM names with the mode chosen by
-expansion. - CVE text and severity come from the flaw cache, refreshed from the OSIDB flaw change feed. If the refresh fails, cached values are used.
- When an advisory changed and output already exists, the complete OSIDB data is fetched, so that a record is never published from partial input.
- Write output. Compare each record’s content hash with the manifest and write only the records that changed. Remove records that are no longer produced (see Failure handling for when this is safe).
- Commit state. Store the new input hashes, the manifest and the cursors. The state is committed after the output is written, so a crash between the two repeats work and does not lose changes.
If -git or -git-remote is given, the output directory is then committed, and
pushed when a remote is given.
Records
A record covers one source RPM in one update stream for one CVE. It has one
affected entry for each binary RPM, and each entry keeps the fixed version of
its own binary because subpackages can have different epochs.
When one source RPM has entries in different states, the record takes the status with the highest precedence:
| Precedence | Status | Meaning |
|---|---|---|
| 1 | fixed |
An advisory shipped the fix |
| 2 | affected |
OSIDB says the product is affected and no fix has shipped |
| 3 | under_investigation |
OSIDB has a new affect with no resolution yet |
| 4 | will_not_fix |
OSIDB resolution is won’t fix or defer |
A shipped fix always wins over an unfixed affect for the same record.
published is the earliest of the dates known for the record: when the flaw
became public, when the affect was created, and when an advisory shipped.
modified changes only when the content of the record changes: a new record
takes its source update time, an unchanged record keeps its value, and a
changed record gets the time of the write.
Status over time
The status is recomputed from the inputs on every run and is not stored. A typical record moves as follows.
stateDiagram-v2
[*] --> UnderInvestigation: OSIDB affect created, no resolution
[*] --> Affected: OSIDB affect with a resolution
[*] --> WillNotFix: OSIDB resolution won't fix or defer
UnderInvestigation --> Affected: triaged as affected
UnderInvestigation --> WillNotFix: resolution won't fix or defer
WillNotFix --> Affected: resolution changed back
Affected --> Fixed: advisory ships the fix
UnderInvestigation --> Fixed: advisory ships the fix
WillNotFix --> Fixed: advisory ships the fix
Affected --> UnderInvestigation: OSIDB affect reset to new
UnderInvestigation --> Removed: marked not affected, embargoed or not public
Affected --> Removed: marked not affected, embargoed or not public
WillNotFix --> Removed: marked not affected, embargoed or not public
Fixed --> Removed: advisory no longer lists the CVE and OSIDB has no affect
Removed --> [*]
state "under_investigation" as UnderInvestigation
state "affected" as Affected
state "will_not_fix" as WillNotFix
state "fixed" as Fixed
state "file removed from the output" as Removed
A removed record is deleted from the output directory and from the manifest. A
record whose flaw was rejected is different: it stays in the output with a
withdrawn timestamp.
Outputs
| Output | Location | Content |
|---|---|---|
| OSV records | <output>/<product path>/<ID>.json |
One JSON file per record. The product path comes from the CPE, for example redhat/hummingbird/1 |
| State database | The -state file |
Updated hashes, manifest, cursors and CVE cache (see State database) |
| Git commit (optional) | The -output directory |
All changes committed, with a message that summarizes the run |
| Run summary | standard error | Counts of records created, updated, withdrawn and unchanged, and files written and removed |
The git step initializes the repository if needed, skips the commit when nothing
changed, and pushes the current branch when -git-remote is given. It runs only
after a sync that ended without an error.
The process exits with 0 on success, 1 when the sync or the git step fails, and 2 for invalid flags.
Failure handling
- OSIDB failure during the diff. If the convert phase does not need OSIDB data, advisory output is still written and the state is committed, and the process exits with 1 so a scheduler can retry or alert. The git step does not run in that case, so the written files are committed by the next successful run. Records that depend on OSIDB are not removed, because the record set is incomplete.
- OSIDB unavailable when convert needs it. This happens when an advisory changed and output already exists, because the complete OSIDB data is needed to avoid a partial record. The run fails before writing anything.
- No OSIDB changes in a delta run. The advisory-only result is not compared with existing records, and no record is removed, because OSIDB data is missing from it.
- Lost state database. The next run behaves like a first run.
- Lost manifest, output kept. A full run rebuilds the manifest from the published files and rewrites each record once, so stale records can still be removed.
- A corrupt published file fails the run when the manifest is rebuilt, so it cannot hide stale state.
- Repeated or duplicate runs. The same input produces the same output, so a duplicate run wastes work and does not corrupt anything.
4.5.39 - S3 Lookaside Cache
Infrastructure for caching dist-git lookaside content, providing access to source artifacts for the Hummingbird build pipeline.
Overview
This stack provides S3-based infrastructure for dist-git artifacts with CloudFront CDN. It caches content from upstream sources like Git tarballs.
A companion stack (s3-lookaside-cache-upload-role) creates a GitLab OIDC
identity provider and an IAM role that GitLab CI jobs can assume using
short-lived JWT tokens, eliminating the need for long-lived access keys.
Architecture
Cache Infrastructure
flowchart LR
clients["Build Clients"]
cf["CloudFront\nDistribution"]
s3["S3 Bucket\n(dist-git-cache)"]
logs["S3 Logs\nBucket"]
backup["AWS Backup\nVault"]
headers["Security\nHeaders"]
clients --> cf --> s3
cf --> headers
s3 --> logs --> backup
GitLab CI Upload Role
flowchart LR
gitlab["GitLab CI\n(.gitlab-ci.yml)"]
sts["AWS STS\n(validates JWT\nvia OIDC/JWKS)"]
role["IAM Role\n(scoped to\nS3 PutObject)"]
s3["S3 Bucket\n(dist-git cache)"]
gitlab -- "id_token" --> sts
sts -- "AssumeRoleWithWebIdentity" --> role
role -- "temporary credentials" --> sts
sts -- "credentials" --> gitlab
gitlab -- "s3:PutObject" --> s3
Components
Cache Stack (s3-lookaside-cache)
| Resource | Type | Description |
|---|---|---|
DistGitCacheBucket |
S3 Bucket | Main cache storage with versioning and object lock |
DistGitLogBucket |
S3 Bucket | Access logs with tiered storage lifecycle |
DistGitCacheDistribution |
CloudFront | CDN with HTTP/2+3, IPv6, TLS 1.2+ |
DistGitCacheOAC |
Origin Access Ctrl | Secure S3 access from CloudFront |
DistGitCacheCachePolicy |
Cache Policy | 1-day default TTL, 1-year max, Gzip/Brotli |
DistGitCacheResponseHeadersPolicy |
Response Headers | HSTS, X-Frame-Options, XSS protection |
DistGitBackupVault |
Backup Vault | AWS Backup vault for data protection |
DistGitBackupPlan |
Backup Plan | Daily (35-day) and weekly (365-day) backups |
DistGitUploadPolicy |
IAM Managed Policy | Grants s3:PutObject to the cache bucket |
Upload Role Stack (s3-lookaside-cache-upload-role)
| Resource | Type | Description |
|---|---|---|
GitLabOIDCProvider |
IAM OIDC Provider | Registers gitlab.com as a trusted identity provider |
DistGitUploadRole |
IAM Role | Web identity role assumable by the configured GitLab project |
Parameters
Cache Stack Parameters
| Parameter | Description |
|---|---|
ResourcePrefix |
Prefix for resource names (e.g., arr-hummingbird-prod-dist-git-cache) |
BucketName |
Globally unique name for the cache bucket (logs bucket appends -logs) |
TrustedKeyGroupId |
CloudFront key group ID whose keys may validate signed URLs — created and managed by the CKI credential manager; empty = the distribution serves unsigned/public |
Upload Role Parameters
| Parameter | Description |
|---|---|
ResourcePrefix |
Prefix for resource names (e.g., arr-hummingbird-prod-dist-git-upload) |
CacheStackName |
Name of the deployed s3-lookaside-cache CloudFormation stack |
AdditionalCacheStackName |
Optional second cache stack whose upload policy is attached to the role |
GitLabProjectPath |
GitLab project path allowed to assume the role (e.g., redhat/hummingbird/rpms) |
S3 Key Structure
Files are stored using the dist-git lookaside path convention:
{namespace}/{package}/{filename}/{hashType}/{hash}/{filename}
Example:
rpms/tar/tar-1.35.tar.xz/sha512/abc123.../tar-1.35.tar.xz
Trust Policy
The upload role’s trust policy restricts access using three conditions (all
StringEquals):
- Audience (
gitlab.com:aud): Must behttps://gitlab.com - Subject (
gitlab.com:sub): Must matchproject_path:<GitLabProjectPath>:ref_type:branch:ref:main - Protected ref (
gitlab.com:ref_protected): Must be"true"
Only the main branch of the configured project can assume the role.
GitLab CI Usage
Configure your .gitlab-ci.yml to assume the role using
id_tokens. The AWS CLI
automatically calls AssumeRoleWithWebIdentity when
AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN are set:
upload to cache:
image:
name: amazon/aws-cli:latest
entrypoint: [""]
id_tokens:
GITLAB_OIDC_TOKEN:
aud: https://gitlab.com
script:
- set +x
- printenv GITLAB_OIDC_TOKEN > /tmp/oidc-token
- export AWS_WEB_IDENTITY_TOKEN_FILE=/tmp/oidc-token
- export AWS_ROLE_ARN="arn:aws:iam::${AWS_ACCOUNT_ID}:role/${ROLE_NAME}"
- export AWS_ROLE_SESSION_NAME="gitlab-ci-${CI_JOB_ID}"
- aws s3 cp "$FILE" "s3://${BUCKET}/${S3_KEY}"
- rm -f /tmp/oidc-token
The aud value in id_tokens must match the audience configured in the OIDC
identity provider (https://gitlab.com). The token is written to a file
because the AWS SDK reads it from AWS_WEB_IDENTITY_TOKEN_FILE rather than
accepting it inline.
Security Features
- Encryption: AES256 server-side encryption with S3 bucket keys
- Public Access: All public access blocked on both buckets
- Transport Security: HTTPS enforced via bucket policy
- Object Lock: GOVERNANCE mode with 1-day default retention (will be increased in the future)
- CloudFront OAC: Modern Origin Access Control (not legacy OAI)
- Security Headers: HSTS, X-Frame-Options (DENY), X-Content-Type-Options, X-XSS-Protection, strict referrer policy
- TLS 1.2 Minimum: For all CloudFront connections
- OIDC Federation: Short-lived tokens instead of long-lived access keys
Signing Key Rotation
CloudFront public-key material is immutable — a rotation always creates a
new key object. The signing key is a managed credential — one token group
per key pair with token_type: cloudfront_signing_key. The
CKI credentials documentation
is the reference for the type: fields, selectors, CloudFront resource
naming, and the key-group ownership (the manager creates the key group,
registers every public key in it, and maintains the membership).
This stack references the key group by ID: the TrustedKeyGroupId parameter
carries the CloudFront key group ID, and the distribution trusts exactly the
keys that group holds. An empty parameter (the default) leaves the
distribution unsigned/public — the public-cache sibling stack deploys that
way. Key-group membership changes take effect without redeploying this
stack: the distribution follows the live group.
Rotate from an infrastructure repository worktree with direnv set up (Vault
tooling), following the standard prepare → switch → clean cycle; the
managed-credentials runbook (documentation/operating/secrets.md in the
infrastructure repository) covers the merge, bake and rollback
pattern:
python3 -m cki_tools.credentials.manager prepare --token-name <KEY>
python3 -m cki_tools.credentials.manager switch --token-name <KEY>
python3 -m cki_tools.credentials.manager clean --token-name <KEY>
Merge the secrets.yml change between steps. On switch, the signer
(kubernetes/setup-konflux/48-…, reading the [deployed] private key and
[deployed]#key_pair_id) moves to the new key. Clean merges after a bake
period; the key-group drop is the revocation.
Fresh deployments
manager create bootstraps the credential and the key group — the founding
version deploys immediately. Bring up a signed deployment in this order:
- Add the signing-key entry to
secrets.ymlwithtoken_type: cloudfront_signing_keyand theprofile_nameandaws_name_prefixmeta. - Run
python3 -m cki_tools.credentials.manager create --token-name <KEY>. - Set the stack’s
TrustedKeyGroupIdparameter to the recordedkey_group_id. - Deploy the stack.
The deployment bot holds the CloudFront public-key and key-group read
actions (GetPublicKey, GetPublicKeyConfig, GetKeyGroup,
GetKeyGroupConfig) on Resource: "*": these actions authorize without a
resource ARN, so only Resource: "*" grants match — resource-scoped grants
silently never apply (verified live during the 2026-11 signing-key
rotation). The CKI credential manager performs all key and key-group
changes with operator credentials. The deployment-bot template reflects
this.
Backup Strategy
| Schedule | Retention | Cold Storage |
|---|---|---|
| Daily (5:00 AM UTC) | 35 days | - |
| Weekly (Sunday 5:00 AM UTC) | 365 days | After 30 days |
Continuous backup is enabled for point-in-time recovery.
Log Lifecycle
Access logs transition through storage tiers:
| Age | Storage Class |
|---|---|
| 0-30 days | Standard |
| 30-60 days | Standard-IA |
| 60-90 days | Glacier IR |
| 90+ days | Expired |
Outputs
Cache Stack Outputs
| Output | Description |
|---|---|
BucketName |
Name of the cache S3 bucket |
BucketArn |
ARN of the cache S3 bucket |
LogBucketName |
Name of the logs S3 bucket |
LogBucketArn |
ARN of the logs S3 bucket |
DistributionDomainName |
CloudFront domain name |
DistributionId |
CloudFront distribution ID |
DistributionArn |
CloudFront distribution ARN |
BackupVaultName |
AWS Backup vault name |
BackupVaultArn |
AWS Backup vault ARN |
BackupPlanId |
AWS Backup plan ID |
UploadPolicyArn |
ARN of the managed policy granting upload access |
Upload Role Outputs
| Output | Description |
|---|---|
RoleArn |
ARN of the upload role (pass to GitLab CI as ROLE_ARN) |
RoleName |
Name of the upload role |
OIDCProviderArn |
ARN of the GitLab OIDC identity provider |
References
4.5.40 - Docs Content Negotiation
CloudFront distribution with HTTP content negotiation for the Hummingbird
public documentation site (hummingbird-project.io). When a client sends
Accept: text/markdown, a CloudFront Function rewrites the request URI to the
pre-built .md equivalent so GitLab Pages serves Hugo’s markdown output
instead of HTML.
Overview
Hugo generates .md files alongside .html for every page. GitLab Pages is a
static file server — it cannot inspect Accept headers and select a response
type. This stack interposes a CloudFront distribution between DNS and GitLab
Pages, adding the header-based routing that GitLab Pages cannot provide.
The stack is deployed by the infrastructure repository. After each deploy,
Route53 A and AAAA alias records for hummingbird-project.io are updated to
point at the CloudFront distribution.
Architecture
flowchart LR
client["Client\n(browser / agent)"]
dns["Route53\nhummingbird-project.io"]
cf["CloudFront\nDistribution"]
fn_req["CloudFront Function\n(viewer-request)"]
fn_resp["CloudFront Function\n(viewer-response)"]
pages["GitLab Pages\nredhat.gitlab.io"]
client -->|"HTTPS"| dns
dns -->|"A/AAAA alias"| cf
cf --> fn_req
fn_req -->|"rewrite URI if\nAccept: text/markdown"| cf
cf -->|"forwards viewer Host header"| pages
pages --> fn_resp
fn_resp -->|"auth redirects → 404"| client
Content Negotiation Logic
The cloudfront-js-2.0 viewer-request function runs at every edge location
before the cache is consulted:
- If the URI starts with
/l/→ normalize and pass through (regardless of headers). These are stable alias paths that GitLab Pages handles via a single_redirectssplat rule per alias (/l/foo/*). The function normalizes variant request forms so the splat rule matches:/l/foogets a trailing slash,/l/foo.mdbecomes/l/foo/index.md. Requests already in/l/foo/or/l/foo/index.mdform pass through unchanged. - If
Acceptdoes not includetext/markdown→ pass through unchanged (HTML response served as before). - Rewrite only known HTML-like patterns:
/docs/(trailing slash) →/docs/index.md/docs/page.html→/docs/page.md/docs/page(no extension) →/docs/page/index.md
- Everything else (
.css,.js,.png,.md,.txt, etc.) → pass through unchanged.
Because the rewrite happens before the cache key is computed (viewer-request
event), HTML and markdown responses land in separate cache entries. No custom
cache policy is needed — the standard CachingOptimized managed policy is
used.
Auth-Redirect Guard
A second CloudFront Function (AuthRedirectGuardFunction) runs on
viewer-response. It detects 302 responses whose Location header points to
projects.gitlab.io/auth — the GitLab Pages access-control gate — and
converts them into a 404 Not Found before they reach the viewer.
GitLab Pages access control causes the origin to return a 302 auth redirect
for nonexistent paths instead of a proper 404. CloudFront caches origin
responses per Accept-Encoding variant (gzip vs brotli), so these redirects
can persist for up to 24 hours. Without the guard, clients follow a redirect
chain to gitlab.com/users/sign_in and land on a Cloudflare challenge page.
Origin
The origin is redhat.gitlab.io (the GitLab Pages default domain). GitLab
Pages routes requests based on the Host header. The AllViewer origin
request policy forwards the viewer’s Host: hummingbird-project.io header
to the origin, so the correct project is served regardless of what DNS
hummingbird-project.io resolves to.
Components
| Resource | Type | Description |
|---|---|---|
ContentNegotiationFunction |
CloudFront Function | viewer-request; rewrites URI for Accept: text/markdown |
AuthRedirectGuardFunction |
CloudFront Function | viewer-response; converts auth redirects to 404 |
DocsCertificate |
ACM Certificate | TLS for the custom domain; DNS-validated via Route53 |
DocsDistribution |
CloudFront Distribution | CDN proxy in front of GitLab Pages |
Parameters
| Parameter | Description |
|---|---|
ResourcePrefix |
Prefix for resource names (e.g., arr-hummingbird-prod-docs-cdn) |
DocsDomainName |
Custom domain for the documentation site (e.g., hummingbird-project.io) |
HostedZoneId |
Route53 hosted zone ID — used for ACM DNS validation CNAME auto-creation |
Outputs
| Output | Description |
|---|---|
DistributionDomainName |
CloudFront domain name |
DistributionId |
CloudFront distribution ID |
Deployment
Deployed by the infrastructure repository. Stack name and region are
configured in samconfig.toml.j2 in the infrastructure repo.
On the first deploy, CloudFormation requests an ACM certificate and
auto-creates the DNS validation CNAME in Route53 (via HostedZoneId). Once
validated (~2–5 minutes), the distribution is created and Route53 alias records
are updated to point at it.
Verification
# Content negotiation: should return text/markdown
curl -H "Accept: text/markdown" https://hummingbird-project.io/docs/ -sI
# Normal browsing: should return text/html (unchanged)
curl https://hummingbird-project.io/docs/ -sI
# /l/ alias paths are not rewritten — GitLab Pages issues HTTP 301 instead
curl -H "Accept: text/markdown" https://hummingbird-project.io/l/adding-images -sI
# Nonexistent paths return 404 (auth-redirect guard converts the origin's 302)
curl https://hummingbird-project.io/this-does-not-exist -sI
# Static assets are excluded from rewriting
curl -H "Accept: text/markdown" https://hummingbird-project.io/favicon.ico -sI
# Alternate domains still redirect to primary (unchanged)
curl https://project-hummingbird.io/ -sI # expect 308
curl https://hummingbird-project.org/ -sI # expect 308
# DNS resolves to CloudFront, not the GitLab Pages IP
dig hummingbird-project.io A +short
Scope
Only the primary domain (hummingbird-project.io) is proxied through
CloudFront. The alternate domains (project-hummingbird.io,
hummingbird-project.org) continue pointing directly to GitLab Pages, which
issues 308 redirects to the primary domain.
4.5.41 - Hummingbird MR Collaboration
How to open merge requests and help on someone else’s MR in the
redhat/hummingbird/tools monorepo. Applies to all components (including
redhat-catalog/).
Features
- Same-repo branches — contributors work in the shared GitLab project, not personal forks for day-to-day changes
- One MR per change — seniors push fixes to the author’s source branch instead of opening a nested MR
- Automatic CI — each push to the MR branch re-runs the pipeline, GitLab Pages preview, and dashboard status links
Prerequisites
- GitLab account with Developer (or higher) access on
redhat/hummingbird/tools - Git clone of the monorepo
- Optional: glab CLI authenticated to GitLab
Git remotes
Many developers use two remotes:
| Remote | Typical URL | Use for |
|---|---|---|
upstream |
git@gitlab.com:redhat/hummingbird/tools.git |
Fetch MRs, push to shared branches |
origin |
Personal fork (if configured) | Optional; not used for team MRs |
Add upstream if missing:
git remote add upstream git@gitlab.com:redhat/hummingbird/tools.git
git fetch upstream
Do not open fork MRs for routine team work. mr-auto-approver unconditionally rejects MRs where
source_project_id != target_project_id.
Open an MR (author)
git clone git@gitlab.com:redhat/hummingbird/tools.git
cd tools
git checkout -b hum-XXXX-short-description
# edit, commit
git push -u upstream hum-XXXX-short-description
Open an MR targeting main in the GitLab UI.
Branch naming: prefer hum-XXXX-* or a descriptive feat/*, fix/*, or
experiment/* prefix (see Environment Promotion
for experiment/* deploy behavior).
Help on someone else’s MR (reviewer)
Use the MR source branch name from the GitLab MR page (not necessarily your local checkout name).
1. Check out the MR locally
Option A — fetch by MR number (works without glab):
git fetch upstream merge-requests/<IID>/head:mr-<IID>
git checkout mr-<IID>
Option B — checkout the source branch:
git fetch upstream <source-branch>
git checkout -B <source-branch> upstream/<source-branch>
Option C — glab:
glab mr checkout <IID> --repo redhat/hummingbird/tools
2. Edit and verify locally
Component-specific checks (example for Red Hat Catalog):
cd redhat-catalog
npm run ci-checks
npm run start:dev # optional UI smoke test
3. Push to the author’s branch
If your local branch name matches the MR source branch:
git push upstream <source-branch>
If you checked out via mr-<IID> (local name differs from remote), push
explicitly:
git push upstream HEAD:<source-branch>
Example: local branch mr-748, remote source branch feat/image-request-form:
git push upstream HEAD:feat/image-request-form
Optional — rename locally so future pushes are simpler:
git branch -m mr-<IID> <source-branch>
git push -u upstream <source-branch>
The existing MR updates; CI re-runs; no second MR is needed.
What runs on each MR push
For redhat-catalog/** changes, see
Red Hat Catalog Environment Promotion — Pipeline Flow.
Summary:
| Stage | Result |
|---|---|
| Test | Build, lint, type-check, unit tests, audit, SAST |
| Visual | Mock visual diff (MR head vs merge-base) |
| Preview | GitLab Pages at /mr-{IID} |
| Status | Internal MR note with Hummingbird dashboard link |
| Agent | Code review on open/update; /hummingbird analyze-failures on CI fail |
After merge to main, staging deploys automatically; production requires manual
gate and UAT sign-off.
Troubleshooting
error: src refspec <branch> does not match any
Git has no local branch with that name. You are probably on mr-<IID> while
pushing git push upstream feat/....
Fix: push the current branch to the remote source branch:
git push upstream HEAD:<source-branch>
Push rejected / permission denied
- Confirm Developer+ access on
redhat/hummingbird/tools - Confirm you are pushing to
upstream, not a personal fork - Check whether the MR author enabled “Prevent pushing to source branch” (uncommon)
fatal: couldn't find remote ref merge-requests/.../head
Fetch from upstream, not origin:
git fetch upstream merge-requests/<IID>/head:mr-<IID>
Team tips
| Tip | Why |
|---|---|
| Assign yourself when taking over pushes | Clear ownership on the MR |
| Use MR Pages preview | Review UI without local dev |
| Use dashboard link in MR internal note | Per-commit Konflux/build status |
| Use GitLab suggestions | Small fixes without a local checkout |
Use experiment/<name> for long exploration |
Live host before opening an MR |
Related documentation
- Red Hat Catalog Environment Promotion
- MR Auto-Approver — fork rejection and approval rules
- Hummingbird Dashboard — Merge Requests
- Hummingbird Agent —
/hummingbirdMR commands
4.5.42 - K8s Integration Tests
Smoke tests that verify tools container images work in a real Kubernetes cluster. Each test creates short-lived pods, checks expected behavior, and cleans up. Tests run automatically on every MR via the Konflux K8s test pipeline.
Test Runner
Tests are executed by ci/run_tests_k8s.sh. For each component it:
- Reads
{component}/tests-k8s.yml - Parses each test entry (YAML -> JSON via Python)
- Runs the
commandblock withkubectlwired to the target cluster - Cleans up labeled resources (
hum-k8s-test=<run-id>) after each test
Running Locally
Test against a local cluster (kind, minikube, or remote):
# Single component with a published image
IMAGE_URL=quay.io/hummingbird-ci/gitlab-ci:latest \
IMAGE_NAME=gitlab-ci--tools \
ci/run_tests_k8s.sh --context kind-kind gitlab-ci--tools
# Single component with a locally-built image
podman build -t localhost/hummingbird-dashboard:dev hummingbird-dashboard/
kind load docker-image localhost/hummingbird-dashboard:dev
IMAGE_URL=localhost/hummingbird-dashboard:dev \
IMAGE_NAME=hummingbird-dashboard--tools \
ci/run_tests_k8s.sh --context kind-kind hummingbird-dashboard--tools
CI Integration
In Konflux, the tools-k8s-test
IntegrationTestScenario
triggers on every MR. The scenario is config-driven: it is declared in
konflux-templates/scenarios.yml and rendered into
konflux-templates/rendered.yml. The tools-k8s-test and
tools-conforma IntegrationTestScenario CRs are deployed to the Konflux
cluster by the infrastructure tools-releases sync, which fetches
konflux-templates/rendered.yml from main and runs on every merge; they
are intentionally not deployed from MR pipelines, because ITS is static
repo-level config that must not change between MRs (concurrent MR
pipelines would otherwise reset the live CR to stale versions). The pipeline:
- Checks whether
tests-k8s.ymlexists for the changed component - Provisions an ephemeral EaaS namespace
- Runs
ci/run_tests_k8s.shwith the built image
Components without tests-k8s.yml skip the test (the pipeline exits early
after the check step).
When an MR touches multiple components of the same application, Konflux
runs per-component PipelineRuns (one per component snapshot) and a
group PipelineRun (one for the PR-group snapshot, created after all
per-component builds in the application complete). The group run invokes
the script once with --group-component-name (repeated) and per-component
IMAGE_URL_<COMPONENT> env vars; per-component runs use the single-component
IMAGE_URL/IMAGE_NAME pair. --group-component-name is handled
identically to --component-name. An MR touching fewer than two components
of the same application gets no group run (Konflux creates a group snapshot
only when ≥2 components of the same application are on the same source branch).
Adding Tests for a New Component
-
Create
{component}/tests-k8s.ymlwith one or more named tests:--- smoke-test: command: | name="${TEST_GROUP}-smoke-${TEST_RUN_ID}" kubectl run "${name}" --image="${TEST_IMAGE:?}" --restart=Never \ --labels="${TEST_RUN_LABEL}" \ --command -- echo "hello" kubectl wait --for=jsonpath='{.status.phase}'=Succeeded \ "pod/${name}" --timeout=120s || test_fail "Pod did not succeed" kubectl logs "${name}" -
Use
${TEST_IMAGE}for the image under test,${TEST_RUN_ID}and${TEST_RUN_LABEL}for unique naming and cleanup. -
Call
test_fail "message"to fail a test with a clear message. -
Keep tests fast (under 2 minutes) — these are smoke tests, not integration suites.
Common pitfalls
- All lines in
command: |block scalars must be indented relative to thecommandkey. Unindented lines break YAML parsing. - Use
command -vinstead ofwhichto check for commands —whichis not available in all container images. - For multi-statement Python one-liners, use semicolons on a single line:
python3 -c 'import foo; import bar; print("ok")'
Debugging Failures
See the EaaS and Debugging guide for accessing ephemeral namespaces and using Kubearchive for historical PipelineRun data.
4.5.43 - Trigger Manifests
Trigger manifests are declarative JSON files committed alongside each Hummingbird source repo. They map changed file paths to the Konflux build PipelineRuns expected to fire, and list the IntegrationTestScenario CRs expected to run after a successful build. The workqueue service reads them to compute build and test expectations for open MRs; any other tooling that needs expectations can read the same files.
The same manifest concept is generated by the containers, tools, and rpms repos.
Location
Each repo carries two manifests:
.tekton/trigger-manifest-pull-request.json— PR (pre-merge) event.tekton/trigger-manifest-push.json— push (post-merge) event
Both are kept committed and regenerated by the repo’s own generator:
- tools —
ci/internal/render_template.py(make all) - containers —
ci/internal/generate_pac_resources.sh(make global-resources, which is also a prerequisite ofmake all-host) - rpms —
ci/generate_resources.py(make generate-host, ormake generateto run inside the build container)
The tree_status CI job enforces that the committed files match the
generator output on every pipeline, so the files are always fresh.
Schema
Both manifests share the same top-level structure:
{
"type": "push",
"excludes": [],
"applications": { ... },
"integration_test_scenarios": { ... }
}
type
String. Either "push" or "pull_request". Identifies which event class
the manifest covers.
excludes
Array of path regex strings. Any changed path matching one of these
patterns suppresses triggering entirely, regardless of the applications
section. Tools and rpms carry an empty array; containers carries exclusion
patterns for image-documentation files (e.g. README.md.j2, report.yml,
Containerfile.j2, properties.yml) that do not affect builds.
applications
Object keyed by Konflux application name. Each value is an object keyed by component name, whose value is an array of path regex strings:
"applications": {
"tools": {
"workqueue-service--tools": ["^workqueue-service/"],
"hummingbird-agent--tools": [
"^hummingbird-agent/",
"^hummingbird-ai/",
"^hummingbird-sandbox/"
]
}
}
A component is considered affected by a commit when at least one of its trigger regexes matches at least one changed path. An application with no affected components produces no build expectations for that commit.
integration_test_scenarios
Object keyed by application name. Present in a manifest only when the repo has integration-test scenarios applicable to that event type; an absent key is treated as “no test expectations” by consumers (backward-compatible with old SHAs before scenarios were added).
Concretely: the push manifest carries integration_test_scenarios only
when the repo has component-context scenarios (they fire on individual
component snapshots post-merge). Tools defines one such scenario
(conforma), so its push manifest includes the section. Rpms defines
scenarios but all are pull_request-context, so its push manifest has no
integration_test_scenarios key. Containers currently defines no
scenarios, so neither manifest includes the section.
The shape differs between the two manifests:
PR manifest (.tekton/trigger-manifest-pull-request.json):
"integration_test_scenarios": {
"tools": [
{ "name": "tools-conforma", "group": false },
{ "name": "tools-k8s-test", "group": true }
]
}
Push manifest (.tekton/trigger-manifest-push.json):
"integration_test_scenarios": {
"tools": [{ "name": "tools-conforma" }]
}
Each entry:
| Field | Type | Present in | Meaning |
|---|---|---|---|
name |
string | both manifests | The IntegrationTestScenario CR name. Matches the test.appstudio.openshift.io/scenario label on test PipelineRuns. |
group |
boolean | PR only | true for pull_request-context scenarios, which also trigger a group-snapshot run when ≥2 components are affected. false for component-context scenarios. |
Scenarios marked optional: true in konflux-templates/scenarios.yml are
excluded from both manifests. component-context scenarios appear in both
manifests (they fire on individual component snapshots). pull_request-context
scenarios appear only in the PR manifest (no group snapshots exist
post-merge).
Consumer semantics
For each application with ≥1 affected component, every listed scenario must
have ≥1 observed test PipelineRun and all observed runs must be green. For
group: true entries, an additional group-snapshot run is required when
the application has ≥2 affected components. Scenario names are matched
against the test.appstudio.openshift.io/scenario labels on test
PipelineRuns.
See the workqueue service documentation for the full consumer-side implementation.
Generation
scenarios.yml schema
Each repo that owns IntegrationTestScenario CRs carries a
konflux-templates/scenarios.yml matrix file. This is the canonical schema
documentation — the file itself carries no inline comments (it is
machine-managed via PyYAML round-trip; this page is the schema reference).
applications:
<application>:
- template: <type>
suffix: <name>
triggers:
- pull_request
- component
optional: true
vars:
KEY: value
Entry fields:
| Field | Required | Description |
|---|---|---|
template |
yes | Scenario type. Maps to scenarios/<template>.yml.j2 and becomes the CR name suffix: <application>-<template>. |
suffix |
no | CR-name override. When set, CR name = <application>-<suffix> instead of <application>-<template>. |
triggers |
yes* | Konflux trigger context names (pull_request, component). Unknown or empty values on non-optional entries fail the generator. *Optional entries may omit it. |
optional |
no | If true, the scenario is excluded from both trigger manifests. |
vars |
no | Per-instance param overrides passed to the template, rendered after the template’s default params. |
All CR content — params, resolverRef, Renovate markers, timeout
annotations — lives in the per-type template file
(konflux-templates/scenarios/<template>.yml.j2). The matrix holds only
the instantiation relationships: application × template × triggers ×
overrides. Unknown or empty triggers on non-optional entries cause the
generator to fail hard, so the manifests are always consistent with the
deployed IntegrationTestScenario CRs.
Component-to-path mappings
The component-to-path mappings in the applications section are derived
differently per repo:
- tools — each component’s
Containerfilelocation, with an optional.build-contextoverride file for extra trigger paths. - containers —
images/*/properties.ymlfiles aggregated with shared configuration fromimages/variables.yml(distros, variants, hermetic settings). - rpms —
rpms/<package>/directories, with PR-time additions fromtest/rpms/<package>.ymlper-package test specs.
To regenerate both manifests after editing scenarios.yml or adding a
component:
make all
5 - Agentic SDLC
Project Hummingbird’s core mission is velocity: shipping RPM updates, CVE fixes, and new
container images as fast as possible. The pipeline already delivers at a scale unprecedented
within Red Hat — consistently over 1,000 automated commits per week across the
containers and rpms repositories, a throughput that would
be impossible to sustain manually — work that in most Linux distributions is still performed
manually by dedicated teams. Managing 400+ packages and 1,750+ image builds with a
24-hour security SLO, that pipeline is not a nice-to-have — it is what makes the project
viable with a small team.
The pipeline solves throughput. What it does not solve is the full software development lifecycle: assessing what to build, implementing it, documenting it, testing it, reviewing UX, coordinating releases. For a small team maintaining hundreds of containers and building an entire OS, staffing all of those functions manually does not scale — and agentic AI is what changes that equation. Where rule-based automation handles the deterministic, agents take on everything that has historically required a human to pick up, think through, and act on. Hummingbird is building toward becoming an open source reference implementation for responsible agentic AI in software engineering: demonstrating through production use, not promises, that this kind of automation can be adopted incrementally and grounded in engineering discipline.
The Agentic SDLC Vision
The target state is a development pipeline driven by agents at every stage — with humans engaged where their judgment is needed, not where their time is consumed by routine work.
Every change flows through the same lifecycle:
1. Request. A customer, product team, or automated trigger submits a change: a new container image, an RPM update, a feature request, a bug fix.
2. Analysis. A product management agent assesses the request — evaluating feasibility, priority, and fit against existing components and conventions. For straightforward requests it proceeds automatically; for complex or ambiguous ones it surfaces the relevant context and flags the case for human input before continuing. This is the work that normally sits in a product manager’s queue.
3. Implementation. The request splits based on its nature:
- Deterministic changes — dependency updates, lockfile syncs, rebuilds triggered by upstream changes — are implemented by rule-based automation and merged automatically once CI passes. No ticket, no approval, no delay.
- Everything else is handled by an engineering agent: a developer co-programming with an agent, or an agent working autonomously to write the code or configuration, run validation, and open a merge request. Agentic AI is the default answer for everything that has historically required a human to pick it up, think through it, and act on it.
- Specialist agents contribute alongside: a tech writer agent drafts or updates the documentation and long-form Red Hat product content; a testing agent generates or extends tests; and a user design agent applies UX patterns and practices to any interface-related changes.
4. Review. Automated agents check for policy compliance and code quality, including the production code review agent. The same agent capabilities that contribute during implementation also apply here: a tech writer agent reviews documentation changes for clarity and correctness; a testing agent reviews test coverage and flags gaps; and a user design agent reviews UX-related changes against established patterns. A human engineer makes the final call on all of it. No agent-produced work is merged without human approval — the human is not eliminated, but elevated to the role where judgment actually matters.
5. Merge. The change ships.
The goal is maximum automation, pursued pragmatically rather than dogmatically — the right tool for each kind of work. Where agentic AI excels is in the space just before human judgment is required: gathering context, surfacing options, resolving the routine, so that when a human does engage, the decision in front of them is sharp, informed, and fast.
Responsible AI
Agent-produced code and decisions require human approval before merge. This is not a temporary constraint — it is a design principle. Humans may stay in the loop to steer or closely supervise complex work when that helps, but the standing requirement is approval — not constant supervision of every agent run. The feedback between human judgment and agent behavior is what keeps the system trustworthy over time.
The high-velocity automated pipeline — over 1,000 commits per week — operates through rule-based automation and does not involve agents; no human review is required there. The failure analysis agent does run at high volume — triggering on every pipeline, build, or test failure — but it produces analysis and comments, not code changes requiring approval. That alone saves hundreds of minutes per week of manual log triage. Agent work that requires human approval is scoped to SDLC tasks: feature requests, documentation, code review — a much lower volume where the requirement is entirely manageable.
Security
Agents run in containers — locally via Podman, in production on OpenShift. This is not a novel security model: it is the same approach the industry has long used for untrusted code, applied consistently. The Hummingbird agent platform is architecturally designed to ensure agents have no access to secrets or tokens. That single constraint effectively neutralizes prompt injection as a threat: an agent that cannot access credentials or act on external systems is, in practice, just another contributor — and the review process treats it as such. The same gatekeeping that applies to any third-party contribution applies to agent-produced work. The worst-case outcome is a low-quality merge request — and that is precisely what the review process, including the review agents themselves, is designed to catch.
Learning from Feedback
Agents learn from feedback — and this is already working in production. When a reviewer comments on an agent-produced merge request, the agent reads those conversations, incorporates the feedback, and updates its work. It avoids re-raising issues already addressed and respects when a developer has explained an intentional design choice. That feedback is also preserved — stored so that future agent runs can draw on it, improving with each iteration rather than repeating the same mistakes — an approach grounded in the frameworks established in Pascal Bornet’s Agentic Artificial Intelligence.
From Pair Programmer to Autonomous Agent
Deploying production agents requires a foundation. The approach that works is not to automate first and document later — it is the reverse.
Phase 1 — Agent as pair programmer. Agents assist individual developers: drafting code, explaining unfamiliar systems, proposing changes for review. This phase builds understanding of what agents need: context, conventions, constraints.
Phase 2 — Documentation as fuel. Agent output quality is directly proportional to
documentation quality. Every repository carries an AGENTS.md — a file with instructions
targeting agents specifically, but one that mostly points to other documentation rather than
duplicating it. That underlying documentation is written for humans and agents alike: clear,
specific, and rich with rationale. The AGENTS.md is the entry point; the broader docs are
the substance. An agent with good documentation reasons from the same foundation as the team;
an agent without it guesses.
Phase 3 — Agents in production. With the foundation in place, agents move into the critical path: first as reviewers proposing concrete changes, then as autonomous contributors on well-defined tasks. The next section describes what that looks like in practice for Hummingbird today.
An agent running autonomously in production is not fundamentally different from a developer running an agent during pair programming. The same agent, the same context, the same documentation — the difference is who initiates the run and how much human supervision follows. If an agent consistently does good work when a developer is in the loop, that is a strong signal the project is ready to let it operate more independently. The quality of pair programming performance is the leading indicator for autonomous deployment readiness. Trust is built incrementally, one well-handled task at a time.
See From Pair Programmer to Autonomous Agent by Valentin Rothberg for the full treatment.
Documentation as Infrastructure
Writing Documentation with Agents
In an agentic system, documentation is infrastructure — the primary interface between the team and its agents, and the quality of what an agent produces is bounded by the quality of what it can read. That makes writing documentation with agent assistance not a separate task but a natural extension of the work: the same agent that helps an engineer write and review code can help write the documentation for it — capturing decisions, explaining rationale, drafting contribution guides. This is not a shortcut; it is the same pair-programming model applied to a different kind of output. This is how Hummingbird built its documentation from the start: engineers worked alongside agents to draft and improve it in the same way they wrote code — iteratively, collaboratively, with humans in the lead. The result is a virtuous cycle: better docs make agents more capable, and more capable agents help produce better docs.
Tech writers are not a prerequisite — engineers writing clear, intentional documentation with agent assistance is sufficient to start. But their skills become more valuable in this model, not less: precision over generality, rationale alongside rules, explicit anti-patterns. A tech writer contributing to an agentic codebase is directly shaping the behavior of the agents that work within it.
In Hummingbird, this is not theoretical. The containers repository is
developed in active collaboration with Red Hat tech writers, whose customer-facing product
documentation lives in the same repository as the code — keeping internal conventions, agent
context, and customer-facing content in one place, shared across engineers, tech writers, and
agents alike.
Knowledge Transfer Across Teams
The same principle — knowledge encoded in markdown, readable by humans and agents alike — also answers a broader question: how does one team’s expertise become available to agents maintained by another? The answer is the same as everything else in this system — markdown. A UX team encodes their patterns and design practices in a skills file; the user design agent picks it up. The documentation team maintains their style guide, standards, and best practices — including Red Hat’s documentation guidelines — encoded once and available to every agent that produces or reviews written content. No code changes, no integrations, no handoff meetings. Any team can contribute their domain knowledge to the agentic system simply by writing it down in a form that both humans and agents can read. The markdown file is the interface between teams.
Agents in Production Today
Two agents are running in production today, built on the
Hummingbird Agent platform. They operate on every merge request
across the containers and rpms repositories, which
together see over 1,000 automated commits per week — a throughput that would be impossible to
review and maintain manually without agent assistance.
More agents are under active development in parallel, with production deployment planned after Red Hat Summit 2026 — likely by end of Q2. One example is the merge conflict agent — which will automatically resolve merge conflicts that arise during automated RPM updates. The full roadmap is tracked in the HUM-687 epic.
Failure Analysis Agent
When a merge request pipeline fails, this agent automatically investigates and posts its findings directly on the merge request. It pulls data from multiple sources — GitLab CI job logs, Konflux build and test pipeline runs, Testing Farm test results — and correlates them to identify root causes. Where the same underlying issue causes multiple failures across different images or architectures, it groups them so engineers see patterns rather than an overwhelming list of individual failures. Every finding includes links to the specific logs and artifacts so engineers can dig deeper without having to hunt.
What previously required clicking through multiple levels of web UI, downloading logs, and working through thousands of lines of output to find a single root cause now happens automatically. It is not rocket science — but it is some of the most time-consuming and draining work an engineer can be asked to do repeatedly.
Workflow definition: analyze-failures.md
Code Review Agent
When a merge request is opened, this agent performs a technical code review with the priorities of a senior engineer: security first, then correctness, then performance, then maintainability. It provides specific line references and concrete code examples for every issue it raises — not vague suggestions, but actual fixes. It reads prior review discussions before posting, avoids re-raising issues already addressed, and respects when a developer has explained an intentional design choice.
The result is consistent, high-quality review coverage on every merge request, with human reviewers free to focus on the issues that genuinely need their judgment.
The team is already pleased with the quality of the reviews — and this is only the beginning. A false positive costs nothing: a human reviewer reads it, disagrees, and moves on. The design ensures that bad agent output has no blast radius. And as human reviewers comment on agent-produced work, the agent learns and incorporates that feedback — improving with every iteration.
Workflow definition: code-review.md
The near-term impact is measured in cycle time. A feature request that previously took weeks to move from intake through product assessment, implementation, documentation, and review can be compressed to days, sometimes hours. That changes what a small team can realistically take on. The longer-term picture is equally straightforward: the system already delivers value today. If agents continue to improve, Hummingbird benefits more. If they plateau, the team continues to operate at a level that would otherwise require far more people. The infrastructure built now is not a bet on a technology that might not materialize — it is an investment that is already paying off, with optionality for more.
6 - Architecture Decision Records
Architecture Decision Records (ADRs) capture significant technical decisions made across the Hummingbird project, along with their context and consequences.
Records
Implemented decisions are also accepted. An acceptance date is listed only when recorded in the ADR.
| ID | Title | Status | Proposed | Accepted |
|---|---|---|---|---|
| ADR-0001 | Unified Observability Stack | Implemented | 2026-07-19 | Not recorded |
| ADR-0002 | AI-Assisted SDLC Workqueue | Proposed | 2026-08-24 | Pending |
| ADR-0003 | RPM metadata and versioning | Accepted | 2026-08-31 | 2026-11-17 |
| ADR-0004 | Move private_product into package metadata | Accepted | 2026-11-05 | 2026-11-18 |
6.1 - ADR-0001: Unified Observability Stack
- Status: Implemented
- Date: 2026-07-19 (updated 2026-08-19)
- Author: Robert Sturla
- Jira: HUM-4790 (Epic), HUM-4791 (Design task)
Context
Hummingbird had no single place to view the health of builds, infrastructure,
and services. Multiple disconnected monitoring and alerting implementations
existed — CloudWatch dashboards, a custom cloudwatch-log-forwarder Lambda,
an SNS AlertsTopic for consumer CloudWatch alarms, and per-service custom
dashboards. Workloads span multiple clusters with no cross-cluster visibility.
This created blind spots, duplicated effort, and inconsistent alerting
behaviour.
One standardised observability stack is needed. No duplication. One toolchain, used everywhere, federated across clusters from day one.
Operational runbooks live in documentation/monitoring.md in the
infrastructure repository. This ADR records the
architecture decision.
Cluster Constraints
Hummingbird has no cluster-admin access on any cluster. All resources must be deployed within tenant-provisioned namespaces. Operator CRDs and RBAC grants are controlled by the cluster platform team — what is available differs between clusters.
This constraint drives a hub-spoke split architecture: the hub (MPP) runs the backends and Grafana, while spoke clusters run lightweight collectors that forward all telemetry to the hub.
Decision
Stack
Hub (MPP, hummingbird--monitoring-hub):
| Signal | Backend | Storage |
|---|---|---|
| Metrics | Mimir (monolithic) | S3 via ObjectBucketClaim |
| Logs | Loki (monolithic) | S3 via ObjectBucketClaim |
| Collection | Alloy Hub (YACE + textfile) | N/A |
| Visualisation | Grafana | N/A |
| Alerting | Mimir ruler + Alertmanager | N/A |
Spoke (every cluster with hummingbird--monitoring):
| Signal | Backend | Storage |
|---|---|---|
| Metrics | Alloy prometheus.scrape → hub Mimir remote-write |
N/A |
| Logs | Alloy loki.source.kubernetes → hub Loki |
N/A |
| Collection | Alloy + kube-state-metrics | N/A |
| Visualisation | None (use hub Grafana) | N/A |
Spoke clusters have no local backends. All storage, alerting, and visualisation is centralised on the hub. Each spoke forwards to both the staging and production hubs.
Distributed tracing (Tempo) is not deployed. See Future Work.
Alerting Architecture
Mimir evaluates PromQL alerting rules (ruler) and delivers notifications through its built-in Alertmanager. Grafana is not in the evaluation path.
Mimir ruler (PromQL) → Mimir Alertmanager → Slack / email
- Rules are Git-provisioned in the infrastructure repository and mounted into Mimir as a ConfigMap.
- All alerting runs on the hub. Spoke clusters do not run Alertmanager. Spoke alerting depends on network connectivity to the hub — if a spoke loses connectivity, alerts for that cluster are delayed until the connection is restored. This is an accepted tradeoff given the operator constraints on spoke clusters (see Cluster Constraints).
- Staging Alertmanager receivers are empty (a blackhole) so staging does not page Slack or email.
Alert Routing
| Severity | Channel |
|---|---|
critical |
Slack + email |
info |
Slack |
| everything else | Slack + email |
Production only; staging is a blackhole.
Deployment Model
The hub runs the backends, Grafana, and Alloy Hub. Spoke clusters run Alloy and kube-state-metrics.
Hub (MPP):
- Namespace: dedicated
hummingbird--monitoring-hubnamespace (tenant-provisioned) - Tenant-scoped: all resources deployed in namespaces owned by Hummingbird, no cluster-admin dependency
- Mimir (monolithic): accepts Prometheus remote-write from spoke Alloy and Alloy Hub, stores TSDB blocks in S3, serves PromQL, and runs the ruler and Alertmanager. Grafana queries Mimir in-cluster (no nginx basic auth on the Service port).
- Loki (monolithic): receives logs from spoke Alloy via authenticated Routes. Grafana queries Loki in-cluster.
- Alloy Hub: a dedicated Alloy instance in the hub namespace.
CloudWatch metrics are ingested with
prometheus.exporter.cloudwatch(YACE), filtered to Hummingbird resources (app-code=RPRM-001). Credential-expiry metrics are ingested with the textfile collector from a CI-generated ConfigMap. Alloy Hub remote-writes to in-cluster Mimir (Service DNS; no Route or CA bundle). - Grafana: exposed via an internal Route. Authentication is Generic
OAuth against gitlab.com (
redhat/hummingbirdgroup). GitLab Owner and Maintainer map to Grafana Admin, Developer to Editor, everyone else in the allowed group to Viewer. Datasources and dashboards live ingrafana_data/and are synced withgrafana_data/deploy.sh. Grafana also has a CloudWatch datasource for ad-hoc browsing of AWS logs and metrics; that path is query-only and is not used for alerting or pod-log forwarding. - External access: Loki and Mimir Routes sit behind an nginx sidecar with HTTP basic auth. In-cluster Service access (Grafana, Alloy Hub) does not require auth.
Spoke (every cluster):
- Namespace:
hummingbird--monitoring - Alloy: collects container logs via
loki.source.kubernetesand scrapesprometheus.io/scrapeServices, including kube-state-metrics. Forwards logs to hub Loki and metrics to hub Mimir. Dual-writes to staging and production hubs. - kube-state-metrics: Kubernetes object metrics (deployments, pods, jobs, PVCs, resource quotas) with a curated allowlist.
- No local Mimir, Loki, Grafana, or Alertmanager.
AWS Lambda Services
Some Hummingbird services (e.g. container-catalog) run as AWS Lambda functions. Observability for those services uses the existing AWS signals rather than an OpenTelemetry Lambda extension:
- Metrics: Alloy Hub YACE scrapes curated
AWS/Lambdametrics (Invocations, Errors, Duration, Throttles) into Mimir. Mimir alerting rules cover error rate and throttling. - Logs: Grafana queries CloudWatch Logs directly via the CloudWatch datasource. Pod logs are not involved; Lambda logs stay in CloudWatch.
- Low-traffic utility Lambdas are covered by the same YACE allowlist
when tagged
app-code=RPRM-001.
Log Collection
Container stdout/stderr is tailed by spoke Alloy using
loki.source.kubernetes (Kubernetes API; no DaemonSet, no sidecar, no
ClusterLogForwarder). Logs are labeled with cluster, namespace, pod,
container, and app, then dual-written to hub Loki.
This requires no application code changes. ClusterLogForwarder is available on some clusters but is not used.
High Availability
Hub:
| Component | Replicas | Rationale |
|---|---|---|
| Mimir | 1 | Monolithic; brief downtime acceptable |
| Loki | 1 | Monolithic; brief downtime acceptable |
| Grafana | 1 | View-only; brief downtime acceptable |
| Alloy Hub | 1 | Stateless scrape; gap until next scrape is OK |
Spoke:
| Component | Replicas | Rationale |
|---|---|---|
| Alloy | 1 | Stateless forwarder; brief gap OK |
| kube-state-metrics | 1 | Stateless; brief gap OK |
Spoke clusters have no stateful observability components. HA concerns are concentrated on the hub. Single-replica monolithic Mimir and Loki are an accepted tradeoff against the proposed Prometheus + Thanos HA pair: there is no existing Prometheus fleet to retrofit, and Alloy already remote-writes.
Retention
| Environment | Mimir | Loki |
|---|---|---|
| Production | 90d | 90d |
| Staging | 14d | 14d |
S3 is provisioned with ObjectBucketClaim. There is no separate warm/cold tier or multi-year archive.
Operator Availability
CRD availability was audited on 2026-08-09. Hummingbird has no cluster-admin access — only operator CRDs explicitly granted to tenant namespaces are usable. That audit is why this stack does not use Cluster Observability Operator (MonitoringStack / Thanos), LokiStack, TempoStack, ClusterLogForwarder, or an OpenTelemetry Collector as the live path:
- LokiStack and TempoStack are not available to tenant namespaces on any cluster. Loki is a standalone Deployment. Tempo is deferred.
- COO MonitoringStack is available on MPP but not on spoke, and would have required Prometheus + Thanos for a fleet we do not have. Mimir accepts remote-write directly from Alloy.
- ClusterLogForwarder is available, but Alloy
loki.source.kubernetescollects the same pod logs without a second operator. - OpenTelemetry Collector was deployed earlier and removed once Alloy wrote directly to Loki and Mimir.
Relationship to Existing Dashboard
The existing hummingbird-dashboard app serves two distinct roles:
- Data visualisation — CVE status charts, failed release counts, build metrics
- Operational workflows — click to rerun a failed release, interactive triage, actionable controls
This observability stack takes over data collection, storage, and visualisation (role 1). Grafana is the primary place for charts, time-series dashboards, and alert-driven visualisations. The dashboard app retains operational workflow features (role 2) — interactive actions that Grafana cannot provide.
The two systems complement each other:
- The dashboard app can embed Grafana panels for richer visualisation without reimplementing charting
- The dashboard app can query the same backends (Mimir, Loki) directly for data it needs to drive workflows
- Over time, the dashboard app consumes data from the unified backends instead of maintaining parallel collection
Alerting (e.g. failed release Slack notifications) moves to Alertmanager. The dashboard app no longer needs its own alerting path.
What This Replaces
| Previous | Replaced by |
|---|---|
cloudwatch-log-forwarder Lambda |
Spoke Alloy → hub Loki |
SNS AlertsTopic + consumer CloudWatch alarms |
Mimir ruler + Alertmanager |
| Per-service custom dashboards | Grafana dashboards |
| Dashboard app data collection | Unified backends (Mimir, Loki) |
| Dashboard app alerting (Slack) | Alertmanager |
| Dashboard app visualisations | Grafana (embeddable in the dashboard app) |
The dashboard app itself is not decommissioned — its operational workflow features (release reruns, interactive triage) remain. Only its data collection and visualisation responsibilities shift to the unified stack.
hummingbird-events-topic is the live event bus. It was not an
alerting channel and is not replaced by this stack.
Consumer CloudWatch alarms and the log-forwarder were removed after the Mimir/Loki path was in place. Remaining CloudWatch use is intentional:
- Grafana CloudWatch datasource for ad-hoc AWS log and metric browsing
- CVE error-budget SLO dashboards and
hummingbird-slo-alerts(no Grafana port yet)
Operations
Day-to-day endpoints, datasources, query examples, and onboarding a new
cluster are documented in documentation/monitoring.md in the
infrastructure repository.
Future Work
- Distributed tracing: Tempo (or equivalent) for request and pipeline traces. Not deployed; traces are not a current signal.
- CVE lifecycle tracing: instrument CVE analysis, RPM build, and container image pipelines with spans. Trace a CVE from detection → RPM fix → image rebuild → publish. Requires a trace backend and a schema (separate story).
- Spoke local alerting: if operator access on spoke clusters changes, a local evaluation path could remove the hub dependency for spoke alerts. Not required today.
Alternatives Considered
Alerting: Grafana-only vs Alertmanager-only vs hybrid
- Grafana-only: less mature routing, dedup, and silencing.
- Hybrid Grafana + Alertmanager (original proposal): Grafana would evaluate rules across metrics, logs, and traces, then forward to Alertmanager. That is not what shipped: there is no Tempo, and log alerting is not required yet.
- Mimir ruler + Alertmanager (chosen): PromQL rules and delivery in one component. Matches the metrics-first rollout. Applications can still POST to the Alertmanager API later if needed.
Metrics backend: Prometheus + Thanos vs Mimir
Prometheus + Thanos (MonitoringStack) was the original hub design. It fits retrofitting an existing Prometheus fleet. Hummingbird had no such fleet — Alloy already remote-writes. Mimir is one component for receive, storage, PromQL, ruler, and Alertmanager, instead of Prometheus + Thanos Sidecar + Store Gateway + Querier + a separate Alertmanager.
Logs: Sumo Logic vs Loki
Red Hat has a company-wide Sumo Logic license. However, the Sumo Logic Grafana plugin requires Grafana Enterprise or Grafana Cloud — it is not available in open-source Grafana. Loki integrates natively with open-source Grafana and keeps logs on-cluster.
Log collection: ClusterLogForwarder vs Alloy
ClusterLogForwarder is available on hub and spoke. Alloy
loki.source.kubernetes collects the same pod logs and already runs on
every spoke for metrics, so a second operator was not justified.
Spoke architecture: full stack vs collector-only
- Full stack per cluster: each cluster runs its own backends and Grafana. Local alerting survives network partitions. Requires operator CRDs Hummingbird does not have on spoke.
- Collector-only on spoke (chosen): Alloy + kube-state-metrics forwarding to the hub. Simpler spoke footprint. Tradeoff: spoke alerting depends on hub connectivity.
Lambda: OTel extension vs CloudWatch ingest
An OpenTelemetry Collector Lambda extension would export OTLP to the hub. That needs a network path from Lambda to MPP and a collector that no longer exists. YACE metrics into Mimir plus Grafana’s CloudWatch datasource for logs reuse AWS signals already produced by the functions.
HA: all components vs selective vs single replica
Full HA doubles cost with little benefit for view-only tools. The original proposal kept HA only on Prometheus and Alertmanager. The as-built stack uses single-replica monolithic Mimir and Loki; Alertmanager is in-process with Mimir. Brief downtime is accepted.
Consequences
- Team must learn Grafana dashboarding and PromQL/LogQL basics
- New hub services should expose Prometheus metrics (scrape annotation)
and may add a Grafana dashboard under
grafana_data/ - New spoke services must be scraped by Alloy (scrape annotation) and emit logs to stdout/stderr
- Spoke alerting depends on hub connectivity — no local alerting on spoke
- Deploying a new hub cluster requires Mimir, Loki, Grafana, and Alloy Hub, plus ObjectBucketClaims
- Deploying a new spoke cluster requires a
hummingbird--monitoringnamespace, Alloy, and kube-state-metrics pointing at both hubs - CloudWatch remains for Lambda logs (Grafana datasource) and CVE SLO error-budget dashboards; it is not used for pod-log forwarding or consumer alarms
6.2 - ADR-0002: AI-Assisted SDLC Workqueue
- Status: Proposed
- Date: 2026-08-24
- Author: Michael Hofmann
- Jira: HUM-6342 (spike)
Context
The containers and rpms repositories
produce over 1,000 automated commits per week. The automation that drives
this throughput is spread across multiple independent mechanisms: GitLab
CI scheduled jobs running shell scripts, Renovate CronJobs in the
infrastructure repo, AWS Lambda functions for MR approval, and manual
developer workflows run from dev machines. Each mechanism creates and
manages MRs with its own lifecycle, credentials, and error handling.
There is no unified system that tracks an MR from creation through approval to merge. The Hummingbird Agent already runs AI-powered failure analysis and code review on every MR, but cannot create or modify MRs. The security foundation for agent-authored MRs is designed in HUM-851 but not yet implemented.
This ADR documents the current state of MR automation across all repositories and entrypoints as discovery for a spike to design a unified workqueue-based system.
Scope
This document inventories what exists today. It does not prescribe the target architecture — that is the output of the spike. Design questions and patterns are included to inform the spike investigation.
Current State
Repositories
rpms (redhat/hummingbird/rpms): RPM package definitions. Each
package lives in its own directory with a .spec file and
metadata/<package>.json for upstream repo URL, CVE product mapping, and
modification status. Fedora dist-git imports and upstream version bumps
are the primary automated flows.
containers (redhat/hummingbird/containers): Container image
definitions. Each image lives in images/<name>/ with Containerfiles and
RPM lockfiles (rpms.lock.yaml). Lockfile refresh and Renovate
dependency updates are the primary automated flows.
tools (redhat/hummingbird/tools): Lambdas, CLI tools, and the
Hummingbird Agent framework. Contains mr-auto-approver,
gitlab-event-forwarder, hummingbird-agent, and the CVE analysis
pipeline.
infrastructure (redhat/hummingbird/infrastructure): Kubernetes
manifests, AWS SAM templates, and deployment configuration. Hosts
Renovate CronJob definitions and agent deployment manifests.
MR Creation Entrypoints
| Entrypoint | Repo | Trigger | What it creates |
|---|---|---|---|
dist_git_update_multi_mr.sh |
rpms | GitLab schedule (every 4h) | One MR per package needing Fedora dist-git sync |
upstream_update_multi_mr.sh |
rpms | GitLab schedule | One MR per package with a newer upstream release |
rebuild_multi_mr.sh |
rpms | Manual (operator-triggered) | One MR per package needing rebuild |
create_lockfile_update_mrs.sh |
containers | GitLab schedule | One MR per image with changed RPM lockfiles |
ci/create_mr.sh (metrics) |
containers | GitLab schedule (daily) | Single chore MR for metrics report |
| Renovate (rpms) | rpms | K8s CronJob (hourly) | Tekton pipeline digest and Konflux release-catalog updates only |
| Renovate (containers) | containers | K8s CronJob (hourly) | Dependency updates, lockfile maintenance, pipeline migration |
| Renovate (tools/infra/k8s-test) | tools/infra/k8s-test | K8s CronJob (hourly) | Dependency updates |
CVE analysis (advisory_handler) |
CEE advisories | K8s CronJob (30 min) | Advisory MRs; handles rebase on conflicts |
/cve skill |
rpms | Manual (Cursor/Claude) | Fix MRs from staged CVE analysis data in Jira |
gitlab_sync |
tools | Various | File sync MRs with poll, rebase, and best-effort approve |
| Manual dev workflows | rpms/containers | Human on dev machine | Package updates, CVE fixes, spec changes |
MR Creation Mechanism: Git Push Options
All shell-script-based MR creation uses git push options, not the GitLab Merge Requests REST API. The push options control MR creation, title, description, labels, draft status, and merge-when-pipeline-succeeds (MWPS):
git push \
--push-option merge_request.create \
--push-option "merge_request.title=chore(rpms): Update foo to 1.2.3" \
--push-option merge_request.remove_source_branch \
--push-option merge_request.merge_when_pipeline_succeeds \
origin HEAD:refs/heads/chore/upstream-update-foo
Approval is always a separate step — never part of the creation scripts.
MR Lifecycle and Approval Flows
GitLab Project Settings (containers and rpms)
| Setting | Value |
|---|---|
| Approvals required | 1 (any_approver); author cannot self-approve |
| Pipeline must succeed | Yes |
| Discussions resolved | Required |
| Jira required | Yes (prevent_merge_without_jira_issue) |
| Merge trains / merge pipelines | Off (Konflux external statuses must stay blocking) |
| Merge method | Merge commit |
| Auto-merge | MWPS via merge_request.merge_when_pipeline_succeeds push option |
Bot Identities
| Role | Identity | Token type |
|---|---|---|
| Create chore MRs | chore-mr |
Project access token |
| Approve chore MRs (CI) | chore-mr-approval |
Separate project access token |
| Create lockfile MRs | lockfile-update |
Project access token |
| Approve lockfile MRs (CI) | lockfile-update-approval |
Separate project access token |
| Renovate + Konflux PAC | pipelines-as-code |
MintMaker token |
| Lambda approvals | Per-project tokens | Separate project access tokens |
Approval Paths
Three parallel approval mechanisms exist, with overlapping scope:
1. CI approval jobs — delayed jobs in each repo’s .gitlab-ci.yml:
-
rpms
chore_mr_approval: gates oncheck_konflux_statuses.py(polls every 60s up to 2 hours for Konflux statuses to appear and succeed), then approves withCHORE_MR_APPROVAL_GITLAB_TOKEN. Triggered for authorchore-mrorpipelines-as-codeonchore/*branches. -
containers
chore_mr_approval: same author/branch gate, approves withCHORE_MR_APPROVAL_GITLAB_TOKEN. No Konflux polling.allow_failure: true. -
containers
lockfile_update_approval: triggered forpipelines-as-codeonlock-file-maintenance(-vulnerability)?branches. Realwhen: delayed/start_in: 10 minutes. No Konflux polling.
2. mr-auto-approver Lambda — event-driven via GitLab webhooks:
GitLab webhook → gitlab-event-forwarder → SNS → mr-auto-approver Lambda
Approves only (does not set MWPS). Rejects forks. Checks Konflux statuses where configured.
| Project | Allowed authors | Branch patterns | Konflux check |
|---|---|---|---|
| containers | PAC + chore-mr | renovate/.*, chore/.* |
Yes |
| rpms | PAC + chore-mr | chore/.* only |
Yes |
| tools | PAC | renovate/.* (deny renovate/redhat-catalog/.*) |
No |
| infrastructure | renovate token | renovate/.* |
No |
The Lambda is intended to replace CI approval jobs; both currently run in parallel for chore/lockfile paths.
3. Renovate automerge — Renovate sets automerge: true in its MR
configuration, but GitLab still requires a non-author approval before
merge. For rpms Renovate MRs, neither CI approval jobs nor the Lambda
cover renovate/.* branches — these MRs are stuck until a human
approves.
Konflux Status Gate
External commit statuses prefixed Konflux kflux-prd-rh03 / are set by
Konflux PipelineRuns triggered via Pipelines-as-Code. The
check_konflux_statuses.py script (rpms) polls these statuses:
- Missing → wait (poll continues)
- Failed → do not approve
- Invoked/running/pending/success → approve (MWPS then waits for final green, including Testing Farm via IntegrationTestScenarios)
Lifecycle by MR Type
RPMs dist-git clean (chore/dist-git-update-<pkg>): schedule creates
→ MWPS at creation → GitLab CI + Konflux builds + Testing Farm → CI
approval job (poll Konflux, up to 2h) + Lambda → merge. Typical: tens of
minutes to hours.
RPMs dist-git conflict: same branch, but MR is created as draft with
no-test label and no MWPS. Requires human conflict resolution (or
claude_resolve_conflict.sh), then remove draft/no-test and
re-trigger.
RPMs upstream (chore/upstream-update-<pkg>): same as dist-git
clean. Uses OIDC AWS lookaside upload for tarballs.
RPMs rebuild (chore/rebuild-<pkg>): same approval path. Typically
operator-triggered, not scheduled.
RPMs Renovate (renovate/…): Renovate sets automerge, but neither CI
approval job nor Lambda covers renovate/.* branches on rpms → stuck
until human approves. The renovate-babysit agent workflow posts a
triage note but does not approve or merge.
Containers Renovate lockfile (lock-file-maintenance):
lockfile_update_approval (10 min delay) + Lambda (renovate/.* with
Konflux check) → Renovate automerge / MWPS. Typical: 30–90+ minutes.
Containers other Renovate: Lambda covers renovate/.*. No
lockfile_update_approval unless branch matches lockfile pattern.
Containers metrics report (chore/metrics-report): daily schedule →
MWPS → chore_mr_approval + Lambda.
Containers CI lockfile (rpm-lockfile-updates/<image>): schedule is
currently active: false. Approval path is incomplete — CI rule matches
lock-file-maintenance, Lambda does not match
rpm-lockfile-updates/.*.
Shell Script Automation Detail
rpms create_mr.sh (shared helper)
Used by all rpms multi-MR scripts. Expects commits already on HEAD.
Inputs: --branch NAME (required), --title TEXT (required), optional
--description, --auto-merge, --draft, --label NAME (repeatable).
Algorithm:
- Count commits ahead of target branch; if 0 → exit 0.
- Dedup:
git ls-remote --headsfor branch name. If present → exit 2 (“MR already exists”). - Create branch, push with MR push options.
- Parse push stdout for MR URL; if missing → exit 1.
Exit code contract: 0 = MR created, 2 = remote branch exists (skip), 1 = failure.
Limitation: stale remote branch from a closed MR still triggers exit 2.
Does not update existing MR/branch content. No --force-with-lease.
rpms dist_git_update_multi_mr.sh
Creates one MR per package needing Fedora dist-git sync.
Modes: production (cwd = repo, always create MRs) vs clone dry-run vs
clone + --create-mrs.
Algorithm:
- Select
metadata/*.jsonfiles; apply--clean-only/--modified-onlyfilter; skip native packages. - Record
START_COMMITon target branch. - For each package: run
./ci/dist_git.py update PACKAGE. Exit 0 + new commit = clean update. Exit 2 + new commit = conflict update. - Collect all new commits, hard-reset target to
START_COMMIT. - Per commit: parse subject
(
^(Update|Sync) ([^ ]+) from ... to ...), create branchchore/dist-git-update-<pkg>, cherry-pick. Conflict → draft +no-test, no auto-merge. Clean →--auto-merge. - Call
create_mr.sh.
Branch convention: chore/dist-git-update-<pkg>. Title:
chore(rpms): <commit subject> or CONFLICT: chore(rpms): <subject>.
rpms upstream_update_multi_mr.sh
Same multi-MR / cherry-pick pattern as dist-git, for upstream version bumps.
Algorithm: runs
./ci/check_upstream_versions.py check --update --sign-off,
collects commits, splits into per-package MRs. Always
--auto-merge. Branch: chore/upstream-update-<pkg>.
rpms rebuild_multi_mr.sh
Does not generate rebuilds. Expects local commits from
dist_git.py rebuild or rebuild-rev-deps, then splits them into
per-package MRs.
Unique feature: --force flag deletes existing remote branch before
create, working around stale branches from closed MRs. Branch:
chore/rebuild-<pkg>.
containers create_lockfile_update_mrs.sh
Creates or updates one MR per image group when RPM lockfiles changed.
Prerequisite: lockfiles already refreshed (make all-host FORCE_REFRESH=true).
Algorithm (process_single_image):
- Branch:
rpm-lockfile-updates/<image>. - For each distro/variant lockfile: compare to HEAD after stripping
.arches[].packages[].urland.arches[].source[].urlviayq(URL churn ignored). - If no meaningful changes → skip.
- If
push-mode≠forceandorigin/<branch>exists: compare to remote; if identical → skip. git switch --force-createbranch, commit lockfiles.- Push
--force-with-lease+ MR push options.
Key difference from rpms: uses --force-with-lease to update existing
branches in place. Dedup compares lockfile content, not just branch
existence.
containers create_mr.sh (simpler helper)
Used by metrics report. Creates from working tree (not existing
commits). git add --all, commit, --force-with-lease push. No dedup —
overwrites existing branch.
Deduplication Summary
| Script | Existing branch | Content refresh |
|---|---|---|
rpms create_mr.sh |
Skip (exit 2) | Never |
| rpms multi-MR scripts | Via create_mr.sh |
Never (unless rebuild --force) |
| containers lockfile | Skip if remote content matches | Yes (--force-with-lease) |
containers create_mr.sh |
Overwrites | Yes |
Hummingbird Agent Framework
The Hummingbird Agent is an event-driven LLM agent deployed on OpenShift that processes GitLab events via SQS.
Architecture
- Event-driven: consumes SQS events (pipeline failures, MR events, slash commands). No REST API — triggered only by SQS or CLI.
- Workflow-driven: investigation logic defined in markdown files as LLM system prompts. New behaviors do not require code changes.
- Sandboxed execution: all LLM commands run in isolated containers (Podman/K8s/KubeVirt) with no network access and no credentials.
- Multi-model: supports Gemini and Claude via Vertex AI.
- Token separation: model tokens (read-only, Reporter-level) are distinct from orchestrator tokens (write-capable, used only by deterministic code outside the sandbox).
- Single output action:
post_gitlab_noteis the only way the agent affects the outside world. It cannot create, modify, or merge MRs.
Current Workflows
analyze-failures: triggered on pipeline failure. Bulk-fetches
GitLab CI job logs, Konflux PipelineRun/TaskRun metadata and logs, and
Testing Farm results. Groups failures by root cause. Posts a structured
MR note with collapsible forensic evidence and links.
- Tools:
gitlab_get_mr_details,gitlab_get_mr_diff,gitlab_get_commit_statuses,konflux_list_pipelineruns,konflux_get_pod_log,tf_get_results,tf_get_test_log,sandbox_exec - Pattern: bulk metadata fetch → selective deep log pulls → grouped analysis → structured note
- Flexibility: medium-low (tightly coupled to Konflux/Tekton/TF pipelines)
code-review: triggered on MR open/update. Reads the unified diff,
prior review discussions, and per-repo rules. Posts review findings with
severity, category, and concrete code fix examples.
- Tools:
gitlab_get_mr_details,gitlab_get_mr_unified_diff,gitlab_get_mr_discussions,gitlab_get_file_at_ref,sandbox_exec - Pattern: diff-first analysis, discussion-aware follow-up (avoids re-raising resolved issues), per-project rules overlay
- Per-repo rules:
workflows/repo-rules/<group>-<repo>.mdfiles provide project-specific review guidance (e.g.,hummingbird-rpms.mddocuments package metadata conventions) - Flexibility: high (generic for any GitLab MR; domain knowledge plugs
in via
repo-rules/)
renovate-babysit: triggered on successful Renovate MR pipeline.
Classifies the update as safe or risky based on diff scope, upstream
changelogs (via web search), and local API usage. Posts a triage verdict.
- Tools:
gitlab_get_mr_details,gitlab_get_mr_unified_diff,web_search,gitlab_get_repo_archive,sandbox_exec - Pattern: diff scope classification, embedded changelog first then web search, archive + grep for call-site impact
- Flexibility: medium-high (ecosystem-agnostic risk matrix; does not approve/merge — triage note only)
What the Agent Has That Maps to a Workqueue
- SQS FIFO queue with message grouping (serializes per session)
- Session persistence (S3
context.json) - Config hot-reload in serve mode
- Per-project token separation (model / action / orchestrator)
- Rate limiting (per-workflow per-MR)
- Data source abstraction (GitLab, Konflux, Testing Farm)
- Auto-resolve on push/success
What the Agent Lacks for MR Lifecycle Management
- No MR creation capability (no git write operations, no push access)
- Read-only GitLab tokens for the model
- No workqueue for MR lifecycle state (events are fire-and-forget)
- Single output action (
post_gitlab_noteonly) - No safe-outputs pattern (no validation of proposed writes)
- No deterministic fallback paths for well-understood operations
- No request/response API (SQS consumption and CLI only; external services cannot invoke a workflow and retrieve structured results)
Event Bus
The existing event pipeline could serve as the backbone for a workqueue:
GitLab MR / note / pipeline webhooks
→ gitlab-event-forwarder (API Gateway → Lambda)
→ SNS hummingbird-events-topic
→ mr-auto-approver Lambda (approve safe bot MRs)
→ hummingbird-agent SQS (analyze / review / babysit)
→ hummingbird-status SQS (pipeline tracking)
mr-auto-approver is a pure rule-based approver — no LLM involvement.
hummingbird-mr-human-tracker detects human fixes on Renovate/bot MRs
and tracks them in Jira epics.
Security Foundation: HUM-851
HUM-851 (“Enable Secure Agent MR Creation”) is an existing epic (status: New) that designs the security posture for agents creating MRs.
Core Principle
Instead of protecting CI secrets from agent-authored MR pipelines,
eliminate all secrets from containers/rpms/tools CI entirely. Move all
secret-dependent jobs to the infrastructure repo’s CI. With zero secrets,
a compromised agent MR pipeline can only access CI_JOB_TOKEN
(single-project-scoped for bot users) and the runner’s network.
What Moves to Infrastructure Repo CI
| Job | Current project | Current trigger |
|---|---|---|
deploy_hugo_tag |
containers, rpms, tools | On merge to main |
lockfile_update |
containers | Scheduled |
| Lockfile/chore approval | containers, rpms | After MR CI passes |
| Chore MR creation | containers, rpms | Scheduled |
dist_git_update |
rpms | Scheduled |
upstream_version_update |
rpms | Scheduled |
metrics_report |
containers | Scheduled |
update_quay_description |
containers | Scheduled/on merge |
This migration directly enables centralized MR creation.
Target Security Posture
- Zero CI secrets in target projects
- Restricted runners with egress filtered to package registries only
- Agent never holds write tokens — writes happen in deterministic orchestrator code
- Diff safety validation — blocklist prevents changes to CI config,
.tekton/, dependency files - Human review required for every agent MR, auto-labeled
agent-generated - Rate limiting per-project and global; auto-close stale MRs
- Monitoring: every agent MR logged and Slack-notified
Task Breakdown
| Task | Status | What it does |
|---|---|---|
| HUM-852 | New | Threat model and security design |
| HUM-853 | New | Move ops jobs to infra CI |
| HUM-854 | Closed | K8s CronJob/Job manifests (done) |
| HUM-855 | New | Validate and remove CI variables |
| HUM-856 | New | Restrict runner egress |
| HUM-857 | New | MR creation orchestrator action |
| HUM-858 | New | Rate limiting and abuse prevention |
| HUM-859 | New | Monitoring and incident response |
Renovate Deployment
Deployed as Kubernetes CronJobs in the infrastructure repo
(kubernetes/renovate/20-cronjobs.yml.j2):
- 5 repos: containers, infrastructure, k8s-test-pipeline, rpms, tools
- Each gets its own CronJob running hourly with
concurrencyPolicy: Replace - Uses MintMaker image (
MINTMAKER_IMAGE) - containers/rpms use bash wrappers and custom config files
- Custom env variables for RPM lockfile cache, Rust/Cargo paths
CVE Analysis Pipeline
The CVE analysis pipeline is a complex, mature system. The relevant interface points for a workqueue are:
hummingbird-cve-analysisCronJob (K8s, every 30 min prod) — runscve_analysis --resolvewhich triages Jira Security tickets, transitions statuses, creates advisory MRs, manages labels.collect_cve_dashboard— runs after analysis, posts lifecycle data to the dashboard API./cveskill — reads staged CVE analysis data from Jira tickets and creates fix MRs in the rpms repo.
The workqueue integration point: when /cve or CVE analysis determines
an MR should be created, that request becomes a work item.
RPM-to-Container Pipeline Flow
A single RPM spec change triggers a cascade that spans both repos and multiple build systems. The full flow is documented in the RPM Pipeline and Image Pipeline docs.
RPM spec change (rpms repo)
→ MR validation (GitLab CI + Konflux build + Testing Farm)
→ Merge to main
→ Konflux RPM build (Tekton PipelineRun per package, mock hermetic)
→ RPM signing (Kerberos-based)
→ Publish to Pulp (packages.redhat.com)
→ Lockfile update detects new RPM versions (containers repo)
→ Lockfile MR created (Renovate or create_lockfile_update_mrs.sh)
→ Container build (Konflux, multi-arch)
→ Container testing (Testing Farm + K8s tests)
→ Enterprise Contract validation (Conforma)
→ Release to Quay.io
Key observations for the workqueue:
- The cascade is implicit — there is no explicit trigger from RPM
publish to container lockfile update. Lockfile updates discover new
RPMs by polling Pulp (via
make all-host FORCE_REFRESH=trueor Renovate’s RPM lockfile maintenance). - A single RPM change can fan out to many container images (any image that includes that package in its lockfile).
- The RPM pipeline has 5 stages (spec change, MR validation, build, signing, publishing). The container pipeline has 6 stages (source templates, generation, build, testing, Enterprise Contract, release).
- Testing Farm runs integration tests on merge requests only, not on main branch builds. Container tests include reverse dependency testing (rebuilding dependent images locally in the test environment).
- Enterprise Contract (Conforma) validates supply chain security, hermetic builds, and policy compliance before release.
Decision
To be determined by the spike investigation. This section will be updated with the chosen architecture.
The spike should investigate a unified workqueue-based system where:
- Queue feeders replace or wrap the current scheduled scripts and Renovate outputs
- Queue processors handle MR creation, approval, rebasing, and merge through deterministic code
- Agentic capabilities (review, patch generation, triage) are invoked as composable services with structured output
- The security model follows HUM-851’s design: agents reason read-only, orchestrators execute writes
Design Patterns and Prior Art
GitHub Agentic Workflows (gh-aw)
gh-aw provides a reference architecture for AI-assisted repository automation with security guardrails:
- Read-only agent execution: agents run with no write permissions, no secrets
- Safe Outputs: agents buffer intended write operations as structured JSON artifacts; separate permission-controlled jobs validate and execute them
- Threat detection: a separate AI agent scans agent outputs before write execution
- Plan-level trust: trust is embedded in the workflow definition (what the agent is allowed to do), not in the agent’s runtime decisions
Key pattern: separate reasoning (agentic, sandboxed, read-only) from execution (deterministic, permission-controlled, auditable).
Existing Hummingbird Patterns
- SQS FIFO queue with message grouping: already serializes work per session; can be extended to serialize per-MR
- Workflow-as-prompt: markdown files define behavior; new behaviors do not need code changes
- Data source abstraction: GitLab API access is modular
- Token tier separation: model read tokens, action write tokens, orchestrator tokens — maps directly to the safe-outputs pattern
- Sandbox isolation: reusable for agentic review/analysis tasks
Agent as Composable AI Service
One approach to bridge the agent framework and a workqueue is to evolve the agent from a self-contained event processor into a building block that non-AI services can call:
- Request/response API: HTTP endpoint accepts workflow requests, returns a UUID; caller polls for structured JSON results. The caller (e.g., a workqueue herder) applies deterministic logic to decide what to do with the output.
- Structured output: agent produces typed JSON (not just a GitLab note), enabling callers to process results programmatically.
- Safe-output actions: new action types beyond
post_gitlab_note(e.g.,create_mr,update_mr) with per-workflow path allowlists, diff size limits, and branch naming constraints. The orchestrator validates each action against the workflow’s configured constraints before execution.
Open Questions
- Queue technology: extend SQS FIFO (already in use by the agent) or use a different queue (K8s-native, Redis, PostgreSQL)?
- Processor deployment: K8s Deployments polling SQS, GitLab CI jobs triggered by webhooks, Lambda functions, or a combination?
- Renovate strategy: keep Renovate as a separate MR creator and add a herder, or replace Renovate’s MR creation with the workqueue?
- Write credentials: per-repo project access tokens (current model) or a single bot account? How does HUM-851’s “eliminate CI secrets” model interact with workqueue processors that need write access?
- Safe outputs validation: simple schema validation, deterministic allowlist checks, or full threat detection (as in gh-aw)?
- Review scope: which MRs need AI review before merging vs which can be auto-merged with deterministic checks only?
- State tracking: queue, GitLab labels, a database, or the existing
hummingbird-statusPostgreSQL? - Agent evolution: request/response API, safe-output actions, or hybrid? How does this relate to HUM-857?
- HUM-851 relationship: is the workqueue spike a superset of HUM-851, or a separate epic that depends on HUM-851 Phase 1?
Consequences
To be updated after the spike investigation determines the architecture.
Known consequences of any workqueue approach:
- Centralizing MR creation provides a single point for audit, rate limiting, and monitoring
- Moving secret-dependent jobs to the infrastructure repo (per HUM-851) is a prerequisite regardless of workqueue design
- The existing CI approval jobs, Lambda approver, and Renovate automerge have overlapping scope that must be reconciled
- RPMs Renovate MRs currently have no automated approval path — any design must address this gap
- The agent’s current workflows (failure analysis, code review, Renovate babysit) complement the workqueue and should not be disrupted
File and System Inventory
Infrastructure repo
| Path | Contents |
|---|---|
kubernetes/renovate/ |
Renovate CronJob definitions (5 repos) |
kubernetes/hummingbird-agent/ |
Agent deployment manifests (8 files) |
kubernetes/hummingbird-cve-analysis/ |
CVE analysis CronJob |
aws/hummingbird-agent/ |
Agent SQS/SNS SAM templates |
Tools repo
| Path | Contents |
|---|---|
hummingbird-agent/ |
Agent framework (loop, sandbox, models, data sources, workflows) |
hummingbird-agent/workflows/ |
Workflow definitions (analyze-failures, code-review, renovate-babysit) |
hummingbird-agent/workflows/repo-rules/ |
Per-project review rules (hummingbird-rpms.md) |
hummingbird-cve-analysis/ |
CVE analysis pipeline (preserve) |
hummingbird-dashboard/ |
Dashboard web app (preserve) |
mr-auto-approver/ |
Lambda: rule-based MR approval |
hummingbird-mr-human-tracker/ |
Lambda: tracks human fixes on bot MRs |
gitlab-ci/gitlab_sync/ |
File sync MR creation with rebase |
gitlab-event-forwarder/ |
Webhook → SNS bridge |
hummingbird-events-topic/ |
SNS topic infrastructure |
RPMs repo
| Path | Contents |
|---|---|
ci/create_mr.sh |
Shared MR creation helper (push options, dedup by branch) |
ci/dist_git_update_multi_mr.sh |
Dist-git sync: commit-all → reset → cherry-pick per package |
ci/upstream_update_multi_mr.sh |
Upstream version bumps: same cherry-pick pattern |
ci/rebuild_multi_mr.sh |
Rebuild MR splitter (with --force for stale branches) |
ci/dist_git.py |
CLI for import/update/sync/rebuild operations |
ci/check_konflux_statuses.py |
Polls Konflux external statuses before approval |
ci/claude_resolve_conflict.sh |
AI-assisted conflict resolution for dist-git MRs |
metadata/<package>.json |
Per-package metadata (upstream URL, CVE mapping, status) |
Containers repo
| Path | Contents |
|---|---|
ci/create_lockfile_update_mrs.sh |
Per-image lockfile MR creation (force-with-lease, content dedup) |
ci/create_mr.sh |
Generic chore MR helper (force-with-lease, no dedup) |
ci/internal/shared_lib.sh |
Shared utilities (get_distro_variants, build helpers) |
images/<name>/ |
Per-image Containerfiles and RPM lockfiles |
6.3 - ADR-0003: RPM metadata and versioning
- Status: Accepted
- Date: 2026-08-31
- Accepted: 2026-11-17
- Author: Brent Baude
- Jira: HUM-6212
Context
Each RPM package has build inputs in rpms/<package>/ and a machine-readable record in
metadata/<package>.json. Automation uses metadata to decide whether Fedora updates may
overwrite local build inputs, which upstream project to track, and how CVE analysis maps a
package to its upstream identity.
Package metadata needs consistent definitions for the relationship to Fedora, required fields,
and versioning behavior. In particular, storing a release base that can disagree with the spec’s
current Release: value caused incorrect rebuild increments in HUM-5182.
This ADR originally contained the full metadata specification. Subsequent proposals to change the metadata contract need a single current specification, rather than definitions spread across the original ADR and later decisions.
Decision
Adopt the RPM metadata and versioning specification as the authoritative contract
for metadata/<package>.json and its relationship to RPM versioning. The specification was
extracted from this ADR without changing the metadata contract.
The adopted model records a package’s relationship to Fedora through modification_status and
Fedora provenance. Metadata version matches the spec’s resolved Version:. The spec’s Release:
line is the sole authoritative release value for what ships. fedora.release, when present,
records only the Fedora base from which that value is built.
No metadata field records a release base for packages ahead of Fedora or packages without a Fedora origin. Rebuilds increment the spec’s current release directly in those cases, removing the stored value that could disagree with the spec and cause the bug class behind HUM-5182.
Specification changes
Changes to the metadata contract or its semantics require an ADR and a corresponding specification update. The ADR records the proposal, rationale, and compatibility implications. The specification defines the resulting contract and remains the source of truth.
Review the ADR and specification update together, using linked merge requests when necessary. Mark the ADR accepted when the specification change is approved, and record the change in the specification’s change history with a reference to that ADR. Editorial corrections and example clarifications that do not change the contract do not require an ADR.
Consequences
metadata/*.jsonacross the rpms repository must be brought in line with the specification. That migration remains separate, tracked work. Accepted status records the decision, not completion of the migration.dist_git.py,check_upstream_versions.py, and CVE analysis tooling can implement the specification’s validation rules directly.- The spec’s
Release:line remains the sole authoritative value for what ships; no metadata field may be treated as a substitute for it, includingfedora.release. - Future contract changes update the standalone specification, without rewriting this ADR’s original metadata decision or requiring readers to assemble the contract from multiple ADRs.
- RPM operating documentation explains workflows and links to the specification for definitions. It does not define a competing metadata contract.
Specification sections
The following headings preserve links to sections originally embedded in this ADR.
Package states
See Package states in the specification.
Field reference
See Field reference in the specification.
Computing the built NVR
See Computing the built NVR in the specification.
Validation rules
See Validation rules in the specification.
Examples
See Examples in the specification.
6.4 - ADR-0004: Move private_product into package metadata
- Status: Accepted
- Authors: Prarit Bhargava
- Jira: HUM-11758
- Date: 2026-11-05
- Accepted: 2026-11-18
- Last updated: 2026-11-19
This record relocates the design approved in infrastructure!1979. The decision is unchanged.
Summary
This ADR moves the per-package private_product setting out of
ci/package-overrides.yaml in the rpms repository and into the package
metadata JSON (metadata/<package>.json). It also requires that every read,
write, and validation of private_product is implemented in the core library,
hum.meta, and that the CI scripts that use the value go through that library
instead of parsing YAML or JSON themselves.
The RPM metadata and versioning specification defines the authoritative metadata contract. This ADR records the decision and rationale for extending it.
Motivation
private_product routes a package to a private, per-product Pulp repository
(for example nodejs20) instead of the public one. A private product groups
packages for publication to access-restricted repositories. Today the setting
is stored in ci/package-overrides.yaml, which is a build-configuration file,
not package metadata. This causes the following problems:
- Implicit classification. Because the setting is not part of the package
metadata, other tooling tends to infer private or long-term-support (LTS)
status from the package name, for example a
-ltssuffix. A name is not a reliable signal: an upstream project can legitimately have a name that ends in-ltswithout being an LTS package. Routing and classification must be driven by explicit data, never by naming conventions. - Split source of truth. Everything else about a package (origin, version,
modification status, upstream tracking) lives in
metadata/<package>.jsonand is handled byhum.meta.private_productis the exception, so a reader has to consult two files to understand one package. - Duplicated file handling.
ci/generate_resources.py,ci/check_nevr_conflicts.py, andci/dist_git.pyeach read or editpackage-overrides.yamldirectly.dist_git.add_private_product()edits the file line by line to preserve formatting. This is the kind of direct file manipulation thathum.metaexists to replace, and it is not covered by the library’s validation.
Depends on
- ADR-0003: RPM metadata and versioning adopts the metadata specification that this ADR extends.
- HUM-6232 ported the metadata consumers onto
hum.meta.
Impacts
User facing
None for external users. Packages continue to be built and published to the
same repositories. Maintainers set private_product in the package’s metadata
JSON instead of package-overrides.yaml.
Repositories
- documentation/
- gorget/
- rpms/
- containers/
- infrastructure/
- skills/
- tools/ (CI, Website)
rpms/ holds the metadata, the library, and all affected scripts.
documentation/ holds this ADR and the metadata specification.
infrastructure/ removes the original design document in favor of this ADR.
Decision
1. Metadata field
Add an optional private_product field to metadata/<package>.json:
- Type: string. When absent, the package is public.
- The value is the product name and must match the
private_productof an entry inci/konflux_rpa_config.yml. - The field is valid in every metadata state; it is orthogonal to the clean, modified, and independent states.
- The
private_productkey inci/konflux_rpa_config.ymlis a property of the RPA entry, not of a package. It is unchanged by this ADR.
2. Core library changes (hum.meta)
All handling of the field is implemented in hum.meta. No script may read or
write private_product by parsing a metadata or overrides file directly.
hum.meta.model: addprivate_product: str | NonetoPackageMetadataand the concrete subclasses, with a validated setter and JSON round-tripping.hum.meta.validation: validate the value (non-empty string, allowed character set). Check that it matches an RPA entry inci/konflux_rpa_config.yml; this check moves out ofgenerate_resources.py.hum.meta.repository: provide the read and write paths. Setting the field is aload, set,saveoperation onMetadataRepository, which replaces the line-based YAML edit.- Queries: provide library helpers to enumerate packages by product
(
{package: private_product}and the inverse) so consumers do not scan files. - Tests in
hum/meta/tests/cover the field, its validation, and the repository round trip.
3. Consumers
Each consumer switches to the library:
| File | Change |
|---|---|
ci/generate_resources.py |
Use the library for per-package product lookup and the RPA cross-check. Public RPAs still exclude packages that have a product. |
ci/check_nevr_conflicts.py |
Replace load_private_products() with a library query. Behavior is unchanged. |
ci/dist_git.py |
add_private_product() sets the field through MetadataRepository instead of editing package-overrides.yaml. It still adds the RPA entry in konflux_rpa_config.yml. |
The private_product documentation block in ci/package-overrides.yaml is
removed once no entries remain.
4. Migration
Existing entries (nodejs20 and the example entry) are moved into the
matching metadata/<package>.json files in the same MR that ports the
consumers, so the data and the code change together. This follows the pattern
used for HUM-6232. A one-time script is acceptable, but it is not kept in the
repository afterwards. During the migration, no package may have the field in
both places.
Risks and open questions
- Schema change process. Metadata changes require an ADR and a corresponding specification update. Review and approve this ADR with the specification update before merging any implementation MRs.
- Two-file consistency. A package’s product and the RPA entry live in different files. The library validation in section 2 is the guard against drift. The open question is whether the RPA entry should also move into metadata in a later ADR.
- Allowed characters. The allowed character set for product names still needs to be defined.
- Other readers of the overrides file. Scripts outside
rpms/ciand documentation may refer toprivate_productinpackage-overrides.yaml. The implementation MR must search for these before the key is removed.
Implementation notes
- Implementation is tracked in HUM-13038.
- Land the library change (
hum.meta, with tests) first, then the consumer and migration MR, then the operating documentation updates. - Documentation to update in
rpms/:documentation/operating/private-rpm-repositories.md,documentation/operating/package-metadata-fields.md, anddocumentation/background/rpm-pipeline.md. - Update the tests in
test/test_generate_resources.py,test/test_check_nevr_conflicts.py, andtest/test_dist_git.pyto build their fixtures from metadata JSON instead ofpackage-overrides.yaml. - When the implementation is complete, a follow-up MR changes this ADR’s status to Implemented.
7 - Specifications
Specifications define current technical contracts. Architecture decision records capture the decisions and rationale behind those contracts.
7.1 - RPM metadata and versioning specification
This is the authoritative specification for RPM package metadata and versioning. ADR-0003 records the adopting decision and rationale. Contract changes require an ADR and a corresponding update to this specification; editorial corrections do not require an ADR.
Scope
The metadata file at metadata/<package>.json and the package directory at rpms/<package>/
form a pair, one per package. The package directory holds the build input: the spec file,
patches, and the sources checksum manifest that mock consumes to produce RPMs. This
specification does not change anything about that directory.
The metadata file is the machine-readable record of a package’s relationship to Fedora and
upstream. Automation reads it to decide what it may do: whether dist_git.py update may
overwrite the package directory, which upstream project check_upstream_versions.py tracks, and
how CVE analysis maps the package to its upstream identity. The optional private_product field
records a product assignment for Hummingbird’s internal use.
The metadata file does not duplicate build inputs, and it does not record the release currently
shipped: the spec’s Release: line is the sole authoritative value for what ships, and is what
NVR computation uses. See Computing the built NVR. For packages on
Fedora’s version, fedora.release also appears in metadata, but only as the base value the
spec’s Release: is built from, never as a substitute for it.
The two locations are updated independently; neither is derived from the other. They must stay
consistent by convention, not by construction. modification_status is correct only if it
accurately reflects whether the package directory actually diverges from Fedora’s import.
This document normatively defines the target structure of metadata/<package>.json and how
release and versioning behave as a function of modification_status. It describes the target
metadata shape, not what is present on disk in metadata/*.json today. Bringing existing
metadata in line with this specification is separate, tracked work.
Package states
Every package occupies exactly one of three states, describing its relationship to Fedora at a
given moment. A state is not itself a metadata field. It is what modification_status means in
combination with which version the package currently ships. The rest of this specification refers
to these states by name, so they are defined here first.
A package’s state follows from its modification_status, which is itself defined by the
package’s relationship to a Fedora origin. modification_status is one of three values:
clean: the package has a Fedora origin, and currently matches that Fedora import.modified: the package has a Fedora origin, but currently diverges from it, either through local changes while still on Fedora’s version, or by having moved ahead to a newer upstream version.independent: the package has no Fedora origin at all.
For a modified package, the state further depends on whether it currently ships Fedora’s exact
version or has moved ahead to a newer upstream version. The fedora object’s complete or partial
shape records which of those is the case; it does not itself define the state.
The table below names all three states by the version they currently ship, not just by
modification_status, since modified alone is ambiguous between the two Fedora-origin cases:
| Version currently shipped | modification_status |
|---|---|
| Fedora’s version | clean or modified |
| An upstream version ahead of Fedora | modified only |
| No Fedora origin (independent) | independent |
Field reference
This is the complete field-by-field reference for metadata/<package>.json, covering every field
consumed by tooling in the rpms repository, not only the fields that determine modification_status
or the Fedora relationship. Fields are listed in four groups, in order: package identity and state
(version through modification_reason), the Fedora-origin group (fedora and its sub-fields),
fields that configure upstream-version tracking, and fields unrelated to either Fedora or upstream
tracking (private_product, cve_product, version_transform, fix_status). The “Presence”
column states the condition under which a field appears in the JSON; fields are omitted entirely
rather than set to null or an empty value when their condition does not hold.
| Field | Type | Presence | Description |
|---|---|---|---|
version |
string | Always | The version currently shipped. |
modification_status |
enum: clean, modified, independent |
Always | Whether the package matches its Fedora import unmodified (clean), carries local changes (modified), or has no Fedora origin at all (independent). |
modification_reason |
string | Only in the modified state |
Why the package carries local changes. Absent for clean and independent packages. |
fedora |
object | In the clean state or either sub-case of the modified state |
Everything about the Fedora build this package tracks or last tracked. Absent entirely in the independent state. |
fedora.git_url |
string | Whenever fedora is present |
The Fedora dist-git repository URL. |
fedora.branch |
string | Whenever fedora is present |
The Fedora dist-git branch (e.g. rawhide). |
fedora.sha |
string | Whenever fedora is present |
The exact Fedora commit imported or last synced. |
fedora.release |
string | Only in the on-Fedora-version state (clean, or modified while still on Fedora’s version) |
The release number of the Fedora build tracked, dist tag stripped. Absent once the package moves to an upstream version ahead of Fedora. |
upstream_repo |
string (git URL) | Always | The canonical upstream project repository. Falls back to the Fedora dist-git URL when no independent upstream repository exists. |
upstream_branch |
string | Optional; independent of state | Pins one package’s upstream line when several packages share a single upstream_repo (e.g. an nodejs2x-style family). |
version_from_ref |
object: {"type": "commit-date"} |
Optional; independent of state; requires upstream_repo |
Configures dist_git.py update’s fixed-ref pin-refresh logic to compute a gorget source-pipeline’s --version string automatically from a newly-pinned commit, instead of requiring manual resolution. commit-date is currently the only supported type. |
track_upstream |
string: "latest" or a version prefix (e.g. "1.26") |
Optional; independent of state | Enables check_upstream_versions.py tracking of this package; a version prefix constrains which upstream releases are accepted. |
release_monitoring_project_id |
integer or string | Optional; independent of state; only meaningful paired with track_upstream |
The release-monitoring.org (Anitya) project ID or name check_upstream_versions.py queries. |
version_source |
string: currently only "gitlab_tags" |
Optional; independent of state | Selects an alternate upstream-version source for check_upstream_versions.py instead of Anitya. |
tag_strip_prefix |
string, default "v" |
Optional; independent of state | Used with version_source: "gitlab_tags" to strip a tag-name prefix before comparing versions. |
version_suffix_strip |
string | Optional; independent of state | A suffix check_upstream_versions.py strips from an Anitya-reported version before comparison and update (e.g. "-RELEASE"). |
upstream_version_transform |
string | Optional; independent of state | A named transform check_upstream_versions.py applies to normalize an Anitya-reported version into RPM version syntax (e.g. openjdk_to_rpm). A distinct field from version_transform below, with a different consumer. Do not conflate the two. |
source_availability_check |
string | Optional; independent of state | A named checker (registered in SOURCE_AVAILABILITY_CHECKERS) that HEAD-probes a candidate source URL before check_upstream_versions.py accepts that version. |
private_product |
string | Optional; independent of state | An internal product identifier used for private package distribution. Absent for public packages. |
cve_product |
string, or list of strings | Optional; independent of state | A CVE vendor/product override, independent of Fedora or upstream tracking. |
version_transform |
string | Optional; independent of state | A version-mapping rule consumed by CVE analysis tooling in the tools repository (e.g. dotnet_sdk_to_runtime). Not consumed by anything in the rpms repository’s own ci/ scripts. |
fix_status |
integer | Optional; independent of state | Not consumed by any tooling in the rpms repository today. Carried over from history and likely stale. Not a field to use for new work. |
Private product assignment
private_product is an optional, non-empty string identifying a private product, such as
nodejs20, for Hummingbird’s internal use. When absent, the package is public. The field is valid
in every package state and does not change the package’s relationship to Fedora or its versioning
behavior.
ADR-0004 describes the tooling behavior and implementation requirements.
Computing the built NVR
The built RPM is identified by its NVR: Name, Version, and Release. Name is the spec/package name
and is unaffected by any of this. Only Version: and Release: vary with state.
Metadata version and the spec Version: line always match. Metadata stores it separately not
because the two can diverge, but because reading Version: reliably from every spec (which may
build it from macros or indirection) isn’t practical, so metadata is the field automation trusts
instead. Release: works the other way: the spec Release: line is the sole authoritative
value for what ships, in every state. fedora.release, when present, is only the base value
Release: is built from, never a substitute for it.
The table below shows, for each state, where the spec’s Version: and Release: lines come
from. The prose beneath it gives the precise rebuild-increment rules that don’t compress into a
table cell.
| Version currently shipped | Spec Version: |
Spec Release: |
|---|---|---|
Fedora’s version (clean or modified) |
Matches metadata version |
fedora.release plus %{?dist}, with an optional trailing .N for local rebuilds |
An upstream version ahead of Fedora (modified only) |
Matches metadata version |
Starts at 0.1%{?dist} the first time the package moves ahead of Fedora; local rebuilds increment the trailing .N directly from the spec’s own current value |
| No Fedora origin (independent) | Matches metadata version |
A one-time human-chosen base (typically 1 or 0.1) plus %{?dist}; local rebuilds increment the trailing .N directly |
On Fedora version (clean or modified): fedora.release holds Fedora’s confirmed release
number, dist tag stripped. The spec Release: is that value plus %{?dist}, with an optional
trailing .N if the package has been locally rebuilt without a source change since the last
import or sync. dist_git.py rebuild computes the next .N by comparing the spec’s current
Release: against fedora.release: if they match exactly, this is the first local rebuild since
import or sync, and .1 is appended; if the spec already carries a trailing .N ahead of
fedora.release, that trailing number is incremented.
On upstream version (always modified): No fedora.release, and no other metadata field
records a base value. The spec uses Release: 0.1%{?dist} the first time a package moves ahead
of Fedora. Every subsequent local rebuild increments the trailing dot-number directly from the
spec’s own current Release: value (0.1 → 0.2 → 0.3, and so on), never by re-deriving it
from a stored metadata value, because none exists. This is what makes the bug class behind
HUM-5182 structurally impossible: that bug occurred because a stored placeholder value could
disagree with the spec’s actual current Release:; with nothing stored to disagree with, the
mis-increment cannot happen.
Independent: The same rule as the on-upstream-version case applies. No metadata field
records a base value, and the spec Release: is authoritative. Rebuilds increment its trailing
.N directly. The only difference is the very first value chosen when the package is added
(typically 1 or 0.1) is a one-time human decision, not derived from anything.
Validation rules
Every rule below is a single, unconditional statement: given the field values present, exactly one outcome is correct. None of them carry a qualifier like “should,” “generally,” or “in most cases,” and none fold an exception into the wording. A rule that needs an exception is really two rules, stated separately. The rationale for why each rule holds lives elsewhere in this document (Package states, Field reference, Computing the built NVR); this section exists so a validator can implement each line as a direct check with nothing left to interpret.
versionis present.modification_statusis present.upstream_repois present.modification_statusisclean,modified, orindependent.version_source, when present, isgitlab_tags.source_availability_check, when present, is a name registered inSOURCE_AVAILABILITY_CHECKERS.- The top-level field
releasedoes not appear. - If
modification_statusisclean,fedorais present. - If
modification_statusismodified,fedorais present. - If
modification_statusisindependent,fedorais absent. - If
fedorais present, it includesgit_url,branch, andsha. - If
modification_statusisclean,fedora.releaseis present. - If
modification_statusismodified,fedora.releaseis present if and only ifversionequals the version Fedora ships atfedora.sha. fedora.release, when present, does not include a dist-tag suffix (e.g..fc42,.el9).- If
modification_statusismodified,modification_reasonis present. - If
modification_statusiscleanorindependent,modification_reasonis absent. - If
track_upstreamis present andversion_sourceis absent,release_monitoring_project_idis present. - If
release_monitoring_project_idis present,track_upstreamis present. - If
version_sourceisgitlab_tags,release_monitoring_project_idis absent. - If
tag_strip_prefixis present,version_sourceisgitlab_tags. - The spec’s
Version:line, resolved through macro expansion, equals metadataversion. - If
version_from_refis present,upstream_repois present. version_from_ref.type, when present, iscommit-date.private_product, when present, is a non-empty string.
Examples
Metadata for a package on Fedora version, clean:
{
"version": "5.4.3.0",
"modification_status": "clean",
"fedora": {
"release": "2",
"git_url": "https://src.fedoraproject.org/rpms/dnf5.git",
"branch": "rawhide",
"sha": "f174d0fcedc78a34d74b800697862b93683ad5e0"
},
"upstream_repo": "https://github.com/rpm-software-management/dnf5"
}
Metadata for a package on Fedora version, modified:
{
"version": "5.3.15",
"modification_status": "modified",
"modification_reason": "Added gorget source-pipeline.yaml (HUM-5841; pattern from HUM-4622): a fresh fetch of Source0's bash-5.3.tar.gz no longer matches the recorded checksum -- upstream re-published the base tarball (translation file + build stamp only, GPG signature re-verified against the already-trusted key) -- so Fedora's cache no longer serves bytes matching the corrected checksum",
"fedora": {
"release": "2",
"git_url": "https://src.fedoraproject.org/rpms/bash.git",
"branch": "rawhide",
"sha": "2768211b5135c7169f513965d79e9f89e0ca6124"
},
"upstream_repo": "https://git.savannah.gnu.org/git/bash.git"
}
Metadata for a package on an upstream version ahead of Fedora, modified:
{
"version": "0.9~rc4",
"modification_status": "modified",
"modification_reason": "update to 0.9rc4",
"fedora": {
"git_url": "https://src.fedoraproject.org/rpms/avahi.git",
"branch": "rawhide",
"sha": "195919c9f9a8dff0e996921afd8736df8f12715b"
},
"upstream_repo": "https://github.com/avahi/avahi"
}
Note the absence of fedora.release and of any top-level release field. This package is ahead
of whatever Fedora currently ships, so there is no confirmed Fedora release to record.
Metadata for an independent package:
{
"version": "1.3.3",
"modification_status": "independent",
"upstream_repo": "https://github.com/oras-project/oras",
"track_upstream": "latest",
"release_monitoring_project_id": 205787
}
No fedora object and no release-related field. The release currently shipped lives only in
that package’s spec Release: line.
For a package assigned to the private nodejs20 product, its metadata includes this field:
{
"private_product": "nodejs20"
}
This is a partial record; all fields required by the package’s state remain required.
Change history
Record contract changes here with a reference to the ADR that approved each change.
| Date | Change | Decision |
|---|---|---|
| 2026-11-17 | Extracted the specification from ADR-0003 without changing the metadata contract. | ADR-0003 |
| 2026-11-18 | Added optional private_product metadata for internal product classification. |
ADR-0004 |