# Project Hummingbird > A collection of minimal, hardened, and secure container images with a significantly reduced attack surface. Compact index: [llms.txt](/llms.txt) -------------------------------------------------------------------------------- # Using a Project Hummingbird container image url: https://hummingbird-project.io/docs/using/ -------------------------------------------------------------------------------- # Using a Project Hummingbird container image url: https://hummingbird-project.io/docs/using/overview/ Project Hummingbird builds a collection of minimal, hardened, and secure container images with a significantly reduced attack surface. This strong focus on security combined with a highly automated update workflow aims to minimize CVE counts, targeting near-zero vulnerabilities. All images support amd64 and arm64 architectures. ## Quick Start All images are available from the [Red Hat Hardened Images registry](https://images.redhat.com) and work directly with Podman, Docker, or Kubernetes: ```bash # Run a command directly with the curl image podman run registry.access.redhat.com/hi/curl:latest -v https://example.com # Start a PostgreSQL database podman run -e POSTGRES_PASSWORD=mysecret -p 5432:5432 registry.access.redhat.com/hi/postgresql:latest ``` Example container build using the Hummingbird Python image: ```Dockerfile FROM registry.access.redhat.com/hi/python:latest COPY myapp.py /app/ WORKDIR /app CMD ["python3", "myapp.py"] ``` Documentation for each image is available on the [Red Hat Hardened Images Catalog](https://images.redhat.com). ## Contents - [Available Images](#available-images) - [Hardened for Security](#hardened-for-security) - [Distroless Containers](#distroless-containers) - [Understanding Image Variants](#understanding-image-variants) - [Image Verification](#image-verification) - [Vulnerability Scanning](#vulnerability-scanning) - [Sharing Host Data](#sharing-host-data) - [Custom CA Certificates](#custom-ca-certificates) - [Compatibility](#compatibility) - [Reproducible Builds](#reproducible-builds) - [Content-Based Layers](#content-based-layers) - [Source Containers](#source-containers) - [Roadmap](#roadmap) - [Relationship to Fedora](#relationship-to-fedora) - [Contributing](#contributing) ## Available Images Available images span language runtimes, databases, web servers, CLI tools, and base runtime images. Browse the complete catalog at the [Red Hat Hardened Images Catalog](https://images.redhat.com). ## Hardened for Security Hummingbird applies several measures to harden container images: - **Minimal Software Footprint**: Images include only essential packages required for the workload, significantly reducing the attack surface and CVE count. - **Rapid Update Deployment**: Package updates ship as quickly as possible, ensuring fixes are consumed early. - **Non-Root User Default**: Containers default to a non-root user (UID 65532) where technically possible, reducing privileges within the container. - **Hermetic Build Environment**: All containers are built in a hermetic environment without network access, preventing unintended package drift and maintaining full control over software versions. - **Distroless Security**: Shipping only what is strictly necessary for the workload reduces the attack surface and makes certain types of attacks impossible. - **FIPS-Validated Cryptography**: FIPS variants provide NIST-validated cryptographic modules for compliance-sensitive workloads. ## Distroless Containers Hummingbird builds distroless containers — images that do not ship with a package manager and most do not even provide a shell. The distroless design makes the bundled application the container's entrypoint, offering a streamlined experience. For example, with the [curl image](https://images.redhat.com/?name=curl), arguments can be passed directly: `podman run registry.access.redhat.com/hi/curl:latest -v https://www.redhat.com/en`. Purpose-built containers reduce the burden on users. Instead of building custom container images and managing their vulnerabilities, a Hummingbird image with the needed application avoids CVE management overhead entirely. ## Understanding Image Variants Hummingbird provides different variants to support various use cases while maintaining security by default. ### Default Variant (`:latest`) - Distroless: no package manager, no shell - Minimal attack surface - Recommended for production - Example: `registry.access.redhat.com/hi/python:latest` ### Builder Variant (`:latest-builder`) - Includes `dnf` package manager and `bash` - For installing additional dependencies - Intended for multi-stage builds and development - Example: `registry.access.redhat.com/hi/python:latest-builder` Some images provide additional variants such as [PHP's FPM variant](https://images.redhat.com/?name=php). ### FIPS Variants (`:latest-fips`, `:latest-fips-builder`) FIPS variants ship [FIPS 140-3 validated][cmvp] cryptographic modules for workloads that require certified cryptography. Validation applies **only when running on RHEL systems installed in FIPS mode**. Two cryptographic stacks are available depending on the image: | Stack | Images | FIPS packages | |---------|-------------------------------------------|--------------------------------------------------------| | OpenSSL | Go, Nginx, Node.js, Python, Ruby, curl | `openssl-config-fips`, `crypto-policies-config-fips` | | NSS | OpenJDK | `nss-softokn-fips`, `nss-softokn-freebl-fips` | FIPS is available in both distroless (`:latest-fips`) and builder (`:latest-fips-builder`) variants. FIPS variants are available for the Hummingbird distro only (not Rawhide). For implementation details, see the [FIPS Variant Guide][fips-guide]. [cmvp]: https://csrc.nist.gov/projects/cryptographic-module-validation-program ### Support Levels Images are published under two support levels: - **Red Hat supported** — the majority of images. Built, tested, and maintained by the Hummingbird team with full CI/CD coverage, security updates, and vulnerability tracking. Available at `registry.access.redhat.com/hi/` with official Red Hat signing keys — this is the recommended registry for all users. - **Community** — images where the upstream project does not follow standard open-source release practices (e.g., MinIO) or where the image is experimental (e.g., bootc-os). Built and tested in the same infrastructure but without the same support commitments. Community also serves as a staging ground for images being evaluated for promotion to Red Hat supported status. Published to `quay.io/hummingbird-community/`. ### Registry Organizations | Registry | Description | |------------------------------------------|-----------------------------------------------------------------| | `registry.access.redhat.com/hi/` | Red Hat Hardened Container Images (recommended) | | `quay.io/hummingbird/` | Mirror of Red Hat supported images without official signing | | `quay.io/hummingbird-community/` | Community-supported images (e.g., MinIO, bootc-os) | | `quay.io/hummingbird-rawhide/` | Upstream Fedora Rawhide packages directly | | `quay.io/hummingbird-ci/` | Build infrastructure images | Tool images used by pipelines (e.g., `quay.io/hummingbird-ci/gitlab-ci`) are published from the [tools](https://gitlab.com/redhat/hummingbird/tools) repository. ### Tagging Strategy Images follow a version-based tagging scheme: - `:latest` — most recent version (may change) - `:` — specific version (e.g., `:3.11`, `:16`) - `:-builder` — builder variant of a specific version - `:-fips` — FIPS variant of a specific version - `:-fips-builder` — FIPS builder variant of a specific version Use versioned tags in production for reproducible builds. The `:latest` tag is convenient for development but may introduce unexpected changes. For best practices on tag selection and digest pinning, see [How to name, version, and reference container images](https://developers.redhat.com/articles/2025/01/28/how-name-version-and-reference-container-images). **Multi-version images:** For languages with multiple supported version families (Python, Node.js, .NET, etc.), separate images exist for each major version: - `registry.access.redhat.com/hi/python:latest` — latest Python (currently 3.14) - `registry.access.redhat.com/hi/python:3.11` — Python 3.11.x (stays on 3.11) - `registry.access.redhat.com/hi/python:3.13` — Python 3.13.x (stays on 3.13) Version constraints prevent these images from accidentally upgrading to incompatible versions. See [version constraints](documentation/operating/version-constraints.md) for details. ### Multi-Stage Build Pattern The recommended pattern for compiled languages: ```Dockerfile # Build stage: use builder variant to install dependencies and compile FROM registry.access.redhat.com/hi/go:latest-builder AS builder RUN dnf install -y COPY . /src WORKDIR /src RUN go build -o /app . # Runtime stage: use minimal base image for the compiled binary FROM registry.access.redhat.com/hi/core-runtime:latest COPY --from=builder /app /app ENTRYPOINT ["/app"] ``` This approach provides build-time flexibility while maintaining a minimal production image. ## Image Verification Hummingbird images are signed and can be verified using [cosign](https://docs.sigstore.dev/cosign/system_config/installation/): ```bash cosign verify \ --key "https://security.access.redhat.com/data/63405576.txt" \ --insecure-ignore-tlog \ registry.access.redhat.com/hi/: ``` ## Vulnerability Scanning Use [Syft](https://github.com/anchore/syft) and [Grype](https://github.com/anchore/grype) to locally inspect and scan images. For details on how production SBOMs are generated, see [Software Bill of Materials](https://hummingbird-project.io/docs/background/containers/security-labels-and-metadata/#software-bill-of-materials-sbom). To scan an image for vulnerabilities: ```bash grype registry.access.redhat.com/hi/: ``` For current vulnerability information across all variants and versions, see the [Red Hat Hardened Images Catalog](https://images.redhat.com). ## Sharing Host Data By default, containers do not have access to host filesystem content. Volume mounts must be added explicitly: ```bash podman run -v /path/on/host:/path/in/container registry.access.redhat.com/hi/curl:latest ... ``` ### SELinux Relabeling On systems with SELinux enabled (such as Fedora and RHEL), mounted volumes must be relabeled to allow container access. Use the `:z` or `:Z` option: ```bash podman run -v /path/on/host:/path/in/container:z registry.access.redhat.com/hi/curl:latest ... ``` The `:z` option relabels content to be accessible by any container sharing the mount. The `:Z` option relabels content to be uniquely accessible to one container. ### File Permissions Most Hummingbird images default to a non-root user. When mounting host directories that the container needs to write to, either make the directory world-writable (within a private directory): ```bash mkdir -m 700 /path/on/host mkdir -m 777 /path/on/host/mnt podman run -v /path/on/host/mnt:/path/in/container:z ... ``` Or run as the root user (recommended only with rootless Podman, where the root user in the container maps to the calling user on the host): ```bash podman run --user root -v /path/on/host:/path/in/container:z ... ``` For read-only access, no permission changes are needed as long as files are world-readable. ## Custom CA Certificates Hummingbird images can be configured to trust custom Certificate Authority (CA) certificates for TLS connections. The approach depends on the image's TLS stack. ### OpenSSL-Based Images (curl, Nginx, Python, etc.) Mount a CA bundle to replace or extend the system trust store: ```bash podman run --rm \ -v /path/to/ca.crt:/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:ro,Z \ registry.access.redhat.com/hi/curl:latest https://internal-server/ ``` For details on building derived images with merged trust stores, see the [Custom CA (OpenSSL) guide][custom-ca-openssl]. ### Java-Based Images (OpenJDK, Tomcat) Java uses its own PKCS12 truststore format. A custom truststore can be created using `keytool` and mounted at runtime: ```bash podman run --rm \ -v /path/to/cacerts:/etc/pki/ca-trust/extracted/java/cacerts:ro,Z \ registry.access.redhat.com/hi/openjdk:latest java -jar app.jar ``` For step-by-step truststore creation, see the [Custom CA (Java) guide][custom-ca-java]. ## Compatibility Hummingbird images are designed for compatibility with popular images from Docker Hub, Red Hat UBI, and other registries, enabling straightforward migration of existing workloads. **Key difference**: Hummingbird images default to a non-root user (UID 65532) where technically possible, while most traditional images run as root. This may require adjusting file permissions on mounted volumes: ```bash # Ensure correct ownership for mounted data chown -R 65532:65532 /path/to/data ``` For detailed comparisons (environment variables, ports, sizes, default users), see the [compatibility report][report.md], also available in [machine-readable form][report.json]. ## Reproducible Builds [Reproducible builds](https://reproducible-builds.org/) ensure that the same inputs and build environment always produce bit-for-bit identical output. This allows independent verification that a published image corresponds to its claimed source materials, making it straightforward to detect if malware was injected during the build process. Hummingbird images are fully reproducible. Given the signed SLSA provenance attestation that accompanies each image, anyone can rebuild the image from its inputs (this git repo and the RPMs it describes) to verify it matches the published version exactly. For further background, see [Reproducible builds: Project Hummingbird](https://developers.redhat.com/articles/2026/03/26/reproducible-builds-project-hummingbird). ### Verifying Reproducibility The `cosign` and `podman` tools are required. First, download and verify the SLSA provenance attestation using `cosign` (see [Image Verification](#image-verification) for key details): ```bash IMAGE=registry.access.redhat.com/hi/curl:latest cosign verify-attestation --key "https://gitlab.com/redhat/hummingbird/containers/-/raw/main/ci/key.pub?ref_type=heads" --insecure-ignore-tlog \ --type slsaprovenance $IMAGE > attestation.json ``` Then, feed the attestation into the rebuild tool (capture the image ID for comparison): ```bash iid=$(podman run -i --rm --privileged -v /mnt \ quay.io/hummingbird-ci/hummingbird-builder rebuild < attestation.json) ``` > [!NOTE] > The `--privileged` flag is required because the build process uses nested containerization. However, this command is expected to be run rootless. A rootless container cannot gain more privileges than the calling user. To verify reproducibility, pull the published image and compare image IDs: ```bash iid2=$(podman pull $IMAGE) [ $iid = $iid2 ] && echo "Identical" ``` > [!NOTE] > The containers-storage image ID (a hash over the manifest and uncompressed content) is used here, not the repo digest (a hash over compressed content). Comparing uncompressed content avoids depending on the exact compression algorithm or registry format. To keep the rebuilt image for further inspection, use the `DUMP_OCIARCHIVE` environment variable: ```bash podman run -i --rm --privileged -e DUMP_OCIARCHIVE=1 -v /mnt \ quay.io/hummingbird-ci/hummingbird-builder rebuild < attestation.json | podman load ``` ## Content-Based Layers Most container images have layers that mirror the Containerfile structure: each `RUN` or `COPY` instruction creates a layer. A single package update can invalidate a layer much larger than the package itself, requiring clients to re-pull content that has not changed. Hummingbird images use [chunkah] to split images into content-based layers. Instead of layers reflecting the build process, files are grouped by the packages they belong to. This benefits both the network level (only new layers need downloading) and the storage level (common layers are stored once on disk). Inspect which packages live in which layers using `podman history` (or `docker history`): ```bash podman pull registry.access.redhat.com/hi/jq podman history registry.access.redhat.com/hi/jq ``` [chunkah]: https://github.com/coreos/chunkah ## Source Containers The source code for all Hummingbird containers is available as source containers. A source container includes RPM and non-RPM content shipped in an image, pushed alongside each available tag with the `-source` suffix. Inspect source container contents with [skopeo](https://github.com/containers/skopeo): ```bash IMAGE=registry.access.redhat.com/hi/curl:8.19.0-source cd $(mktemp -d) mkdir source skopeo copy --override-os=linux docker://$IMAGE dir:source cd source mv version manifest.json .. for f in $(ls); do tar xvf $f; done ``` ## Roadmap - **Image catalog expansion**: The catalog of available images will continue to expand significantly. - **Image size optimization**: Container-specific package configurations and dependency management to further reduce image sizes. - **Enhanced CVE and SBOM tooling**: Improved vulnerability metadata and scanning integration, including methods like cargo-auditable for Rust supply chain transparency. - **Bootable containers**: Expanding support for image-based OS deployments and updates. ## Relationship to Fedora Hummingbird builds and maintains its own RPM packages independently. Spec files are derived from Fedora but built in Hummingbird's own infrastructure, allowing project-specific modifications and optimizations. Key differences from Fedora: - Use of monorepos — one monorepo for packages and one for containers, simplifying management and automation. - Heavy reliance on CI/CD — dependency management, testing, and releasing are automated. - Secure supply chain requirements — container images and packages are built using Konflux, providing SLSA level 3 compliance. - Greater emphasis on upstream tracking — software closely tracks upstream projects' lifecycles, including parallel stable versions where offered. There is a natural affinity between the two projects, and opportunities exist to contribute work from Hummingbird back into Fedora. ## Contributing Interested in contributing? The following resources cover the development workflow: - **[Quickstart Guide][contributing-quickstart]** — building and testing images locally, project structure, CI/CD pipeline - **[Adding Images](documentation/contributing/adding-images.md)** — step-by-step guide for new images and versioned variants - **[FIPS Variant Guide][fips-guide]** — adding FIPS-validated cryptography to an image - **[Testing Images](documentation/contributing/testing-images.md)** — running tests locally and understanding automatic infrastructure retries - **[Development Workflow](documentation/contributing/development-workflow.md)** — local setup and workflow ## License This project is licensed under the MIT License - see the [LICENSE.txt](LICENSE.txt) file for details. [contributing-quickstart]: https://hummingbird-project.io/l/contributing-quickstart [custom-ca-java]: https://hummingbird-project.io/l/custom-ca-java [custom-ca-openssl]: https://hummingbird-project.io/l/custom-ca-openssl [fips-guide]: https://hummingbird-project.io/l/fips-variant-guide [report.md]: https://gitlab.com/redhat/hummingbird/containers/blob/main/report.md [report.json]: https://gitlab.com/redhat/hummingbird/containers/blob/main/report.json -------------------------------------------------------------------------------- # Custom CA Certificates (OpenSSL) url: https://hummingbird-project.io/docs/using/custom-ca-openssl/ ## Overview You can configure OpenSSL-based container images (curl, nginx, etc.) to trust custom Certificate Authority (CA) certificates for TLS connections. There are two approaches for custom CAs depending on your needs: 1. **Custom bundle file** - Replace system CAs entirely with your own bundle 2. **Derived image** - Build a new image with merged trust store ## Approach 1: Custom Bundle File Use this when you want to trust **only** your custom CAs and block all default CAs. This is a common case with [OpenShift's custom PKI bundle][os-custom-pki]. Mount your CA bundle to `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem`: ### Custom bundle with Podman ```bash podman run --rm \ -v /path/to/ca.crt:/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:ro,Z \ quay.io/hummingbird/curl https://your-server/ ``` ### Custom bundle with Kubernetes ```yaml apiVersion: v1 kind: ConfigMap metadata: name: custom-ca-bundle data: tls-ca-bundle.pem: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ... more certificates if needed ... --- apiVersion: v1 kind: Pod spec: containers: - name: app image: quay.io/hummingbird/curl volumeMounts: - name: custom-ca mountPath: /etc/pki/ca-trust/extracted/pem readOnly: true volumes: - name: custom-ca configMap: name: custom-ca-bundle ``` ### Custom bundle on OpenShift with CA injection On OpenShift, the cluster admin may have already added your organization's CA certificates to the cluster-wide trust store. You can use OpenShift's automatic CA injection feature to make these certificates available to your pods. Create a ConfigMap with the `config.openshift.io/inject-trusted-cabundle=true` label. OpenShift will automatically populate it with the cluster CA bundle as a key named `ca-bundle.crt`: ```yaml # OpenShift will inject a "ca-bundle.crt" into this automatically apiVersion: v1 kind: ConfigMap metadata: name: trusted-ca labels: config.openshift.io/inject-trusted-cabundle: "true" --- apiVersion: v1 kind: Pod spec: containers: - name: app image: quay.io/hummingbird/curl volumeMounts: - name: trusted-ca mountPath: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem subPath: ca-bundle.crt readOnly: true volumes: - name: trusted-ca configMap: name: trusted-ca ``` The key detail is using `subPath: ca-bundle.crt` to mount OpenShift's injected file to the correct location (`tls-ca-bundle.pem`) that OpenSSL expects. See the [Configuring a custom PKI][os-custom-pki] OpenShift documentation for details on cluster-wide CA configuration. [os-custom-pki]: https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html/configuring_network_settings/configuring-a-custom-pki ## Approach 2: Derived Image Use this approach when you need to trust both your custom CA and the image's builtin default CAs. This creates a new image with a merged trust store with the `trust anchor` command. Create a multi-stage Containerfile with your `ca.crt` in the build context. This example derives from the `curl` image: ```dockerfile FROM quay.io/hummingbird/curl:latest-builder AS builder USER root COPY ca.crt /tmp/ RUN trust anchor /tmp/ca.crt FROM quay.io/hummingbird/curl:latest COPY --from=builder /etc/pki /etc/pki ``` Build and use your custom image: ```bash podman build -t my-curl-with-ca . podman run --rm my-curl-with-ca https://your-server/ ``` -------------------------------------------------------------------------------- # Custom CA Certificates (Java) url: https://hummingbird-project.io/docs/using/custom-ca-java/ ## Overview You can configure Java-based container images (openjdk, etc.) to trust custom Certificate Authority (CA) certificates for TLS connections. Java uses its own truststore format (PKCS12) rather than PEM files. Java keystores require a password for integrity checking. The system truststore uses the standard password `changeit`, which is a well-known default used for tamper detection rather than confidentiality. ## Volume Mount Approach Create a custom truststore using the `keytool` command. You can either trust only your specific CAs (for internal-only applications) or merge with the default public CAs (typical for applications connecting to both internal and external services). ```bash CERTDIR=$(mktemp -d) # or use a persistent directory for repeated runs chmod 777 "$CERTDIR" # to work with non-privileged containers # OPTIONAL: Copy the default Java truststore to merge with public CAs # Skip this step to trust ONLY your custom CAs podman run --rm -v "${CERTDIR}:/work:Z" \ quay.io/hummingbird/openjdk:latest \ cp /etc/pki/ca-trust/extracted/java/cacerts /work/cacerts # Import your custom CA podman run --rm -v "${CERTDIR}:/work:Z" \ quay.io/hummingbird/openjdk:latest \ keytool -import -noprompt -trustcacerts -alias my-custom-ca \ -file /work/my-ca.crt \ -keystore /work/cacerts \ -storepass changeit -storetype PKCS12 \ -J-Dkeystore.pkcs12.certProtectionAlgorithm=NONE ``` The `-storetype PKCS12` and `-J-Dkeystore.pkcs12.certProtectionAlgorithm=NONE` flags are required for compatibility with FIPS images. The system truststore is PKCS12 format with unencrypted certificate entries (generated by `p11-kit`). Without these flags, `keytool` defaults to PBE-encrypted certificate bags that the FIPS security provider cannot decrypt. ### Podman Example Mount the custom truststore to the system location: ```bash podman run --rm \ -v "${CERTDIR}/cacerts:/etc/pki/ca-trust/extracted/java/cacerts:ro,Z" \ quay.io/hummingbird/openjdk:latest \ java -jar myapp.jar ``` ### Kubernetes Example In Kubernetes, use an init container to create the truststore, then share it via an emptyDir volume: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: custom-ca data: ca.crt: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- --- apiVersion: v1 kind: Pod spec: initContainers: - name: prepare-truststore image: quay.io/hummingbird/openjdk:latest command: ["sh", "-c"] args: - | # OPTIONAL: Copy the default truststore to merge with public CAs # Skip this line to trust ONLY your custom CAs cp /etc/pki/ca-trust/extracted/java/cacerts /truststore/cacerts chmod 666 /truststore/cacerts keytool -import -noprompt -trustcacerts -alias custom-ca \ -file /ca/ca.crt -keystore /truststore/cacerts \ -storepass changeit -storetype PKCS12 \ -J-Dkeystore.pkcs12.certProtectionAlgorithm=NONE volumeMounts: - name: custom-ca mountPath: /ca - name: truststore mountPath: /truststore containers: - name: app image: quay.io/hummingbird/openjdk:latest volumeMounts: - name: truststore mountPath: /etc/pki/ca-trust/extracted/java readOnly: true volumes: - name: custom-ca configMap: name: custom-ca - name: truststore emptyDir: {} ``` For the trust-only-specific-CAs approach, omit the `cp` command and create the keystore directly with `keytool -import`. ## Derived Image Approach Use this approach when it's more practical to bake the certificates into the image instead of generating the truststore on each container startup. This creates a new image with a merged trust store with the `trust anchor` command. Create a multi-stage Containerfile with your `ca.crt` in the build context. This example derives from the `openjdk` image: ```dockerfile FROM quay.io/hummingbird/openjdk:latest-builder AS builder USER root COPY ca.crt /tmp/ RUN trust anchor /tmp/ca.crt FROM quay.io/hummingbird/openjdk:latest COPY --from=builder /etc/pki /etc/pki ``` Build and use your custom image: ```bash podman build -t my-openjdk-with-ca . podman run --rm my-openjdk-with-ca java -jar myapp.jar ``` -------------------------------------------------------------------------------- # Custom CA Certificates (Python) url: https://hummingbird-project.io/docs/using/custom-ca-python/ ## Overview You can configure Python container images to trust custom Certificate Authority (CA) certificates for TLS connections. Python libraries have different trust store behaviors compared to other [OpenSSL-based images](custom-ca-openssl.md): - **[urllib](https://docs.python.org/3/library/urllib.request.html)** (built-in) reads CAs from *both* of these locations: 1. `/etc/pki/tls/cert.pem` - the main CA bundle file 2. `/etc/pki/tls/certs/` - directory with hashed certificate symlinks - **[requests](https://pypi.org/project/requests/)** (third-party) ignores `/etc/pki` entirely and requires the `REQUESTS_CA_BUNDLE` environment variable to use system CAs **Note**: The `requests` library is not included in the standard Python image. To use it, you need to build a derived image: ```dockerfile FROM quay.io/hummingbird/python:latest RUN ["pip3", "install", "requests"] ``` ```bash podman build -t localhost/my-python-with-requests . ``` There are two approaches for custom CAs depending on your needs: 1. **Custom bundle file** - Replace system CAs entirely with your own bundle 2. **Add custom CAs** - Merge your custom CAs with the system defaults ## Approach 1: Custom Bundle File Use this when you want to trust **only** your custom CAs and block all default CAs. This is a common case with [OpenShift's custom PKI bundle][os-custom-pki]. Mount your CA bundle to `/etc/pki/tls/cert.pem`, set `$REQUESTS_CA_BUNDLE` to that, and block the hashed directory with an empty directory: ### Custom bundle with Podman ```bash podman run --rm \ -v /path/to/ca.crt:/etc/pki/tls/cert.pem:ro,Z \ --tmpfs /etc/pki/tls/certs:notmpcopyup \ -e REQUESTS_CA_BUNDLE=/etc/pki/tls/cert.pem \ localhost/my-python-with-requests python3 -c " import urllib.request print(urllib.request.urlopen('https://your-server/').status) import requests print(requests.get('https://your-server/').status_code) " ``` **Note:** Podman requires the `:notmpcopyup` option as mounting tmpfs behaves like an overlay. Docker's `tmpfs` creates an empty directory by default, and does not know that option, so drop that option with Docker. ### Custom bundle with Kubernetes ```yaml apiVersion: v1 kind: ConfigMap metadata: name: custom-ca-bundle data: cert.pem: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ... more certificates if needed ... --- apiVersion: v1 kind: Pod spec: containers: - name: app # Use quay.io/hummingbird/python for urllib only # Use your derived image with requests installed for both libraries image: registry.example.com/my-python-with-requests:latest env: - name: REQUESTS_CA_BUNDLE value: /etc/pki/tls/cert.pem volumeMounts: - name: custom-ca mountPath: /etc/pki/tls/cert.pem subPath: cert.pem readOnly: true - name: empty-certs mountPath: /etc/pki/tls/certs readOnly: true volumes: - name: custom-ca configMap: name: custom-ca-bundle - name: empty-certs emptyDir: {} ``` ### Custom bundle on OpenShift with CA injection On OpenShift, the cluster admin may have already added your organization's CA certificates to the cluster-wide trust store. You can use OpenShift's automatic CA injection feature to make these certificates available to your pods. Create a ConfigMap with the `config.openshift.io/inject-trusted-cabundle=true` label. OpenShift will automatically populate it with the cluster CA bundle as a key named `ca-bundle.crt`: ```yaml # OpenShift will inject a "ca-bundle.crt" into this automatically apiVersion: v1 kind: ConfigMap metadata: name: trusted-ca labels: config.openshift.io/inject-trusted-cabundle: "true" --- apiVersion: v1 kind: Pod spec: containers: - name: app # Use quay.io/hummingbird/python for urllib only # Use your derived image with requests installed for both libraries image: registry.example.com/my-python-with-requests:latest env: - name: REQUESTS_CA_BUNDLE value: /etc/pki/tls/cert.pem volumeMounts: - name: trusted-ca mountPath: /etc/pki/tls/cert.pem subPath: ca-bundle.crt readOnly: true - name: empty-certs mountPath: /etc/pki/tls/certs readOnly: true volumes: - name: trusted-ca configMap: name: trusted-ca - name: empty-certs emptyDir: {} ``` The key detail is using `subPath: ca-bundle.crt` to mount OpenShift's injected file to the correct location (`/etc/pki/tls/cert.pem`) that Python expects. See the [Configuring a custom PKI][os-custom-pki] OpenShift documentation for details on cluster-wide CA configuration. [os-custom-pki]: https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html/configuring_network_settings/configuring-a-custom-pki ## Approach 2: Add Custom CAs Use this when you want to **merge** your custom CAs with the system defaults. Mount your CA certificate as an individual file in `/etc/pki/tls/certs/` with a properly hashed filename. The filename must be the output of `openssl x509 -noout -subject_hash -in ca.crt` followed by `.0`. Also set `$REQUESTS_CA_BUNDLE` to that directory: ### Custom CA file with Podman ```bash # Generate the hashed filename CA_HASH=$(openssl x509 -noout -subject_hash -in ca.crt) echo "Hashed filename: ${CA_HASH}.0" # Mount the CA certificate as an individual file in the certs directory podman run --rm \ -v ./ca.crt:/etc/pki/tls/certs/${CA_HASH}.0:ro,Z \ -e REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ \ localhost/my-python-with-requests python3 -c " import urllib.request print(urllib.request.urlopen('https://your-server/').status) print(urllib.request.urlopen('https://google.com/').status) # Default CAs still work import requests print(requests.get('https://your-server/').status_code) print(requests.get('https://google.com/').status_code) # Default CAs still work " ``` ### Custom CA file with Kubernetes ```yaml apiVersion: v1 kind: ConfigMap metadata: name: custom-ca data: ca.crt: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- --- apiVersion: v1 kind: Pod spec: containers: - name: app # Use quay.io/hummingbird/python for urllib only # Use your derived image with requests installed for both libraries image: registry.example.com/my-python-with-requests:latest env: - name: REQUESTS_CA_BUNDLE value: /etc/pki/tls/certs/ volumeMounts: - name: custom-ca # REQUIRED: replace example hash with: `openssl x509 -noout -subject_hash -in ca.crt` and append .0 mountPath: /etc/pki/tls/certs/12abc456.0 subPath: ca.crt readOnly: true volumes: - name: custom-ca configMap: name: custom-ca ``` -------------------------------------------------------------------------------- # Contributing to a Project Hummingbird container image url: https://hummingbird-project.io/docs/contributing/ Welcome to the Project Hummingbird contributor documentation. This section provides guides and references for contributing to container images. ## Getting Started New to contributing? Start here: - **[Quickstart Guide](quickstart.md)** - Get your first contribution done in 5 minutes - **[Development Workflow](development-workflow.md)** - Detailed development environment and workflow ## Guides Step-by-step guides for common tasks: - **[Adding Images](adding-images.md)** - How to add a new container image - **[Testing](testing-images.md)** - How to run and write tests locally ## Reference Detailed reference documentation: - **[Image Configuration Reference](image-configuration-reference.md)** - Complete `properties.yml` reference - **[Test Configuration Reference](test-configuration-reference.md)** - Complete test definition reference -------------------------------------------------------------------------------- # Quickstart Guide url: https://hummingbird-project.io/docs/contributing/quickstart/ description: Get your first contribution done in 5 minutes Welcome! This guide will help you make your first contribution to Project Hummingbird container images. For more detailed information, see the full [contributor documentation](https://hummingbird-project.io/l/contributing-images/). ## Prerequisites - **Container tools**: [Podman](https://podman.io/) or [Docker](https://www.docker.com/) - **Build tools**: `buildah`, `make`, `git` - **Python**: Python 3.11+ with dependencies from `requirements.txt` Install on Fedora/RHEL: ```bash sudo dnf install podman buildah make git python3 python3-pip pip install -r requirements.txt ``` ## Quick Start Workflow ### 1. Fork and Clone 1. Fork the repository on [GitLab](https://gitlab.com/redhat/hummingbird/containers/-/forks/new) 2. Clone your fork: ```bash git clone --recurse-submodules https://gitlab.com//containers.git cd containers git remote add upstream https://gitlab.com/redhat/hummingbird/containers.git ``` ### 2. Make Your Changes Edit files in the `images//` directory: - `properties.yml` - Image configuration - `Containerfile.j2` - Container build template - `tests-container.yml` - Integration tests ### 3. Generate and Build ```bash # Update dependent files make # Build the image ci/build_images.sh ``` ### 4. Test Your Changes ```bash # Run integration tests ci/run_tests_container.sh # Run linters and checks make check ``` ### 5. Submit a Merge Request 1. Create a branch: `git checkout -b fix-nginx-config` 2. Commit your changes: `git commit -am "Fix nginx configuration"` 3. Push to your fork: `git push origin fix-nginx-config` 4. Open a merge request from your fork to the upstream repository on [GitLab](https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/new) **Note for external contributors**: If you're not a project member, the CI pipeline won't run automatically. Please ping one of the [project maintainers](https://gitlab.com/redhat/hummingbird/containers/-/project_members?max_role=static-40) in your merge request to trigger the build and test pipeline. ## License By contributing to this project, you agree that your contributions will be licensed under the MIT License. See [LICENSE.txt](LICENSE.txt) for the full license text. ## Common Tasks ### Adding a New Image ```bash # Create image directory mkdir images/myapp # Copy templates cp images/Containerfile.j2 images/myapp/Containerfile.j2 cp images/properties.yml images/myapp/properties.yml # Edit the template, configuration, and tests vim images/myapp/properties.yml # Configure packages and variants vim images/myapp/Containerfile.j2 # Customize the build template vim images/myapp/tests-container.yml # Add integration tests # Update dependent files make # Build and test ci/build_images.sh myapp ci/run_tests_container.sh myapp ``` ### Testing with Docker ```bash # Automatic Docker-in-Docker setup ci/build_images.sh --engine docker --setup ci/run_tests_container.sh --engine docker --setup ``` ### Building Specific Variants ```bash # Build only the rawhide builder variant ci/build_images.sh nginx/rawhide/builder # Test only the rawhide default variant ci/run_tests_container.sh nginx/rawhide/default # Build/test all variants for a specific distro ci/build_images.sh nginx/rawhide ci/run_tests_container.sh nginx/rawhide ``` ## Next Steps - [Development Workflow](https://hummingbird-project.io/l/development-workflow) - Detailed development environment and workflow - [Adding Images](https://hummingbird-project.io/l/adding-images) - Step-by-step guide for adding new container images - [Testing](https://hummingbird-project.io/l/testing-images) - How to run and write tests locally ## Getting Help - **Documentation**: [hummingbird-project.io/l/contributing-images/](https://hummingbird-project.io/l/contributing-images/) - **Code**: [gitlab.com/redhat/hummingbird](https://gitlab.com/redhat/hummingbird) -------------------------------------------------------------------------------- # Development Workflow url: https://hummingbird-project.io/docs/contributing/development-workflow/ description: Detailed development environment setup and contribution workflow ## Prerequisites Ensure the required tools are installed: - **Container tools**: [Podman](https://podman.io/) or [Docker](https://www.docker.com/) - **Build tools**: `buildah`, `make`, `git` - **Python**: Python 3 with `PyYAML` package Install on Fedora/RHEL: ```bash sudo dnf install podman buildah make git python3-pyyaml ``` ### macOS Setup macOS requires bash 5+ and GNU command-line tools. Install via [Homebrew](https://brew.sh/): ```bash brew install podman bash grep coreutils podman machine init && podman machine start # Add to ~/.zshrc for persistence export PATH="/opt/homebrew/bin:/opt/homebrew/opt/grep/libexec/gnubin:/opt/homebrew/opt/coreutils/libexec/gnubin:$PATH" ``` > **Note:** On Intel Macs, use `/usr/local/` instead of `/opt/homebrew/`. ## Repository Setup ### Fork and Clone 1. Fork the [containers repository](https://gitlab.com/redhat/hummingbird/containers/-/forks/new) on GitLab 2. Clone your fork with submodules: ```bash git clone --recurse-submodules https://gitlab.com//containers.git cd containers ``` 1. Add the upstream remote: ```bash git remote add upstream https://gitlab.com/redhat/hummingbird/containers.git ``` ## Development Workflow ### 1. Create a Branch Create a descriptive branch starting from upstream main: ```bash git fetch upstream git checkout -b feature-add-redis-image upstream/main ``` ### 2. Make Changes Edit the relevant files in `images//`: - `properties.yml` - Image configuration (packages, variants, tags) - `Containerfile.j2` - Container build template - `tests-container.yml` - Container tests - `tests-k8s.yml` - K8s tests - `README.md.j2` - Image documentation template ### 3. Generate Files Regenerate derived files: ```bash make ``` ### 4. Build Locally Build the image to verify changes: ```bash ci/build_images.sh # Build specific variant ci/build_images.sh /rawhide/builder # Build with verbose output ci/build_images.sh --verbose ``` ### 5. Test Locally **Container tests:** ```bash ci/run_tests_container.sh # Test specific variant ci/run_tests_container.sh /rawhide/default # Test with verbose output ci/run_tests_container.sh --verbose ``` By default, tests use Podman. To test with Docker: ```bash # Automatic Docker-in-Docker setup ci/run_tests_container.sh --engine docker --setup ``` **K8s tests:** ```bash ci/run_tests_k8s.sh --context # Test specific variant ci/run_tests_k8s.sh --context /rawhide/default # Test with verbose output ci/run_tests_k8s.sh --verbose --context ``` See the [Testing Guide](testing-images.md) for K8s test prerequisites and local development workflow. **Testing base images:** When modifying base images (like `core-runtime`), test dependent images: ```bash ci/build_images.sh core-runtime ci/build_images.sh --build-deps core-runtime ci/run_tests_container.sh --include-reverse-deps core-runtime ci/run_tests_k8s.sh --include-reverse-deps --context core-runtime ``` **Troubleshooting test failures:** Use `--pause` flag to inspect resources before cleanup: ```bash ci/run_tests_container.sh --pause ci/run_tests_k8s.sh --pause --context ``` ### 6. Run Linters Ensure code quality: ```bash make check ``` ### 7. Commit and Push Commit changes with a descriptive message and push to the fork: ```bash git add . git commit -m "feat: add Redis container image - Add Redis 7.x image with default and builder variants - Include basic health check tests - Add compatibility notes for official Redis image" git push origin feature-add-redis-image ``` ### 8. Open a Merge Request Open a merge request on GitLab: 1. Go to the [containers repository](https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/new) 2. Select your fork and branch as the source, `main` as the target 3. Fill in the description with what changed, why, and what tests were added ### 9. CI Pipeline The CI pipeline automatically: - Builds images for multiple architectures - Runs integration tests - Checks for linting issues **Note for external contributors**: If not a project member, the CI pipeline won't run automatically. Ping one of the [project maintainers](https://gitlab.com/redhat/hummingbird/containers/-/project_members?max_role=static-40) in the merge request to trigger the pipeline. ## Next Steps - [Adding Images](adding-images.md) - Step-by-step guide for adding new images - [Testing Guide](testing-images.md) - Comprehensive testing documentation - [Image Pipeline](https://hummingbird-project.io/l/image-pipeline) - How the complete pipeline works -------------------------------------------------------------------------------- # Resolving Merge Conflicts in Generated Files url: https://hummingbird-project.io/docs/contributing/resolve-merge-conflicts-generated/ description: How to resolve merge conflicts in generated files during rebases ## Overview During rebases of branches in this repository, you can commonly trigger merge conflicts in files that are generated by the various toolings and Makefile targets. ## Rules * This documentation is only for generated files. * If there is not a section below for the conflicted file, you may need to check `make help` or search the Makefile for how to regenerate it (e.g., `grep -r "target_file" Makefile`). ## Procedures for generated files The following sections describe how to rebase given merge conflicts in specific paths. ### Regenerating rpms.lock.yaml When rebasing a branch, you may encounter conflicts in `rpms.lock.yaml` files. You should first delete the conflicting file, then run `make container`. Once in the container, rerun `make` and pass the relative path of the target. The file will be regenerated. Below shows an interactive example. If attempting to do this non-interactively, remove the file as described and then run something like: ```bash podman run --pull=newer --rm -v "$PWD:$PWD:z" -w "$PWD" \ quay.io/hummingbird-ci/gitlab-ci:latest \ make ``` ```bash # 1. Delete the conflicted lockfile (replace with your actual conflicting path) rm # 2. Enter the CI container make container # 3. Inside container, regenerate the lockfile make # 4. Exit container exit # 5. Stage the regenerated file and continue the rebase git add git rebase --continue # 6. After rebase completes, validate the result make -j$(nproc) check ``` ### Regenerating Containerfile If the merge conflict is in one or more of the generated Containerfiles, then always accept the upstream (rebasing-onto) branch as the source of truth (i.e., `git checkout --theirs -- ` and `git add `). When you have committed the upstream versions and are done with the rebase, you must then regenerate the Containerfiles so they pick up your changes. Containerfiles can be generated in two ways: To rebuild Containerfiles for all images that are changed by your PR: ```bash make -j$(nproc) ``` To rebuild a specific Containerfile, you can also use `make` and pass the path of the generated Containerfile as an argument. ```bash make images/nginx/rawhide/default/Containerfile ``` After regenerating Containerfiles: ```bash # Stage regenerated Containerfiles git add # If still in rebase, continue git rebase --continue # After rebase completes, validate the result make -j$(nproc) check ``` -------------------------------------------------------------------------------- # Adding New Images url: https://hummingbird-project.io/docs/contributing/adding-images/ description: Step-by-step guide for adding new container images to the project This guide covers two scenarios: - **Adding a new image** - Creating an entirely new image (e.g., adding `nginx` for the first time) - **Adding a versioned image** - Adding a new version of an existing image (e.g., `go-1-25` when `go` already exists) Choose the appropriate section below based on your use case. ## Adding Completely New Images Follow these steps to add an entirely new container image to the project. ### 1. Create Image Directory Create a directory for the new image: ```bash mkdir images/your-service ``` ### 2. Copy Base Templates Copy the template files: ```bash cp images/properties.yml images/your-service/properties.yml cp images/Containerfile.j2 images/your-service/Containerfile.j2 ``` ### 3. Configure Image Properties Edit `images/your-service/properties.yml` to configure: - **Variants**: Define variants (default: `[default, builder]`) - **Packages**: List RPM packages needed in `rpm_packages.all` - **Main package**: Set `main_package` for version labeling - **Repository and stream**: Set `repository` and `stream` (both required). Add a YAML comment above `stream` documenting the reasoning (see [Choosing a Stream Value](#choosing-a-stream-value)) - **Tags**: Configure image tags - **Release metadata**: Set `application_category`, `summary`, `description`, and `url` - **Support level**: New images start with `support_level: community` Example minimal configuration: ```yaml --- rpm_packages: all: - coreutils-single - your-main-package default: - ca-certificates-bundle # smaller than ca-certificates; sufficient for non-builder variants builder: - ca-certificates # full package needed for builder toolchains main_package: your-main-package repository: your-service # -- see "Choosing a Stream Value" section stream: "latest" support_level: community application_category: "Other" summary: "One-line description of the image" description: >- Description of the software, its purpose, and its main capabilities. url: "https://upstream-project.example.com/" tags: - value: latest ``` Prefer `ca-certificates-bundle` over `ca-certificates` in non-builder variants — it is smaller and sufficient for TLS trust roots. Builder variants need the full `ca-certificates` package for toolchain compatibility. See the [Image Configuration Reference](image-configuration-reference.md) for complete `properties.yml` options. ### 4. Customize Containerfile Template Edit `images/your-service/Containerfile.j2`: - Add service-specific configuration (entrypoint, exposed ports, volumes, environment variables) - The template automatically includes packages from `properties.yml` via the `{{ main_packages_arg() }}` macro The base template macros handle the standard container setup automatically: | Macro | What it provides | |-----------------------|-------------------------------------------------------------------| | `{{ final_stage() }}` | `ENV HOME=/tmp`, `WORKDIR /tmp`, labels, OCI archive extraction | | `{{ set_user() }}` | `USER 65532` (when `user: default` in `properties.yml`) | The `user` field in `properties.yml` defaults to `default` (UID 65532) if not specified. No passwd entry is needed — Linux allows running as any UID. See the [Container Configuration](image-configuration-reference.md#user) reference for per-variant and custom user options. #### When to override the defaults The macros provide sensible defaults, but the image should match what the upstream image does. If the upstream image uses a custom user, `WORKDIR`, or `HOME`, the Hummingbird image should match for compatibility. Check the upstream image's `Dockerfile` or inspect it with `podman inspect` before deciding. When the defaults are sufficient, avoid duplicating what the macros already provide: ```dockerfile {# The macros already handle HOME=/tmp, WORKDIR=/tmp, USER 65532. #} {# Only add useradd/groupadd, WORKDIR, HOME, or USER when the #} {# upstream image does something different that users depend on. #} ``` When the upstream image requires a specific user or working directory, set `user:` in `properties.yml` and add `WORKDIR` in `Containerfile.j2` as needed. Use `{{ default_user }}` instead of hardcoding `65532`. #### Docker/registry auth in README templates For images that interact with container registries, the auth config must be mounted to `$HOME/.docker/config.json` inside the container. Check what `HOME` is set to — for default-user images it is `/tmp`: ```bash -v ${XDG_RUNTIME_DIR}/containers/auth.json:/tmp/.docker/config.json:ro,Z ``` #### Multi-Architecture Support When Containerfiles or build scripts need to reference the target architecture (e.g., downloading architecture-specific binaries), use the correct naming convention for each context. | | Docker | Linux | |---------------|----------------------------|------------------| | Intel | `amd64` | `x86_64` | | ARM | `arm64` | `aarch64` | | Containerfile | `$TARGETARCH` ARG variable | `$(arch)` output | - **In Containerfiles or Go builds**: Use the `$TARGETARCH` ARG variable provided by [buildah][targetarch] (values: `amd64`, `arm64`) - **In shell scripts or RPM builds**: Use the [arch] or [uname -m][uname] commands (values: `x86_64`, `aarch64`) [targetarch]: https://github.com/containers/common/blob/a5ccdae846b629b5ceaefa6ffd5c6511409c3487/docs/Containerfile.5.md#L628 [arch]: https://www.gnu.org/software/coreutils/arch [uname]: https://www.gnu.org/software/coreutils/uname ### 5. Add Comparison Report Create the required `images/your-service/report.yml`. Its first comparison supplies the upstream image and documentation URL used by compatibility reports and generated README sections: ```yaml --- compare: - pull: docker.io/upstream/your-service:latest url: https://upstream-project.example.com/docs/ ``` Add `ignore` entries only for intentional differences from the upstream image. See existing images with similar entrypoints and runtime behavior for examples. ### 6. Add Integration Tests Create test files in `images/your-service/`: - `tests-container.yml` - Required container tests (Podman/Docker) - `tests-k8s.yml` - Optional Kubernetes tests for workload or cluster integration behavior **Container test example:** ```yaml --- version-check: command: | test_engine_run --rm "${TEST_IMAGE}" your-service --version ``` **K8s test example:** ```yaml --- run-as-pod: command: | name="test-pod-${TEST_RUN_ID}" kubectl run "${name}" --image="${TEST_IMAGE}" --restart=Never --labels="${TEST_RUN_LABEL}" kubectl wait --for=jsonpath='{.status.phase}'=Succeeded "pod/${name}" --timeout=60s ``` See the [Testing Guide](testing-images.md) for how to write and run tests. ### 7. Add Image Documentation Create `images/your-service/README.md.j2` with a high-level description, usage instructions, and compatibility notes. The template generates multiple README files for different target audiences (defined in `images/variables.yml` under `readme_targets`). **Important notes:** - The `readme_heading()` macro automatically generates the heading with the correct product name - The `readme_description()` macro renders the image description from `description` in `properties.yml`; use it after the heading instead of hardcoding introductory text - Use `{{ readme_targets[target].registry }}` for registry references (not hardcoded `quay.io/hummingbird`) - The `readme_available_tags()` macro renders the variants and tags table - The `readme_trailing_sections()` macro appends compatibility, verification, vulnerability, and about sections **Example template:** ```jinja2 {{ readme_heading() }} {{ readme_description() }} {{ readme_available_tags(variants, tags) }} ## Usage podman run -d -p 8080:8080 {{ readme_targets[target].registry }}/your-service:latest {{ readme_trailing_sections() }} ``` For more details, see [README Generation](../background/image-pipeline.md#readme-generation). ### 8. Generate, Validate, and Submit Follow the standard contribution workflow from the [Development Workflow guide](development-workflow.md): - Generate files (`make`); commit all generated changes - Build locally (`ci/build_images.sh your-service`) - Run container tests (`ci/run_tests_container.sh your-service`) - Run Kubernetes tests when `tests-k8s.yml` exists (`ci/run_tests_k8s.sh your-service`) - Run linters (`make check`) - Commit and push changes - Open a merge request - Wait for CI pipeline ### 9. Add Konflux Components and Trigger Testing When generated `konflux-templates/rendered.yml` changes, the merge request pipeline runs `add_components_to_konflux` automatically. This adds only the new components. After the job completes, add a `/retest` comment to the merge request so Konflux starts the new image pipelines. The manual `deploy_components_to_konflux` job redeploys all components and is not needed for normal image onboarding. ## Adding Versioned Images When adding a new version of an existing image (e.g., `go-1-25`, `helm-4`, or `rabbitmq-4-3`), follow this streamlined workflow. ### 1. Check Package Availability Versioned packages may be available in different repositories: - **Hummingbird repository** - Custom packages built by the Hummingbird project (e.g., `golang1.25`, `nodejs24`, `dotnet-sdk-10.0`) - **Fedora repositories** - Standard Fedora packages (e.g., `golang`, `python3`) The project uses two distros. Their current repository files are defined in `images/variables.yml`: - `rawhide` - Uses only Fedora Rawhide packages (`fedora-44.repo`) - `hummingbird` - Uses Fedora 43 + Hummingbird packages (`fedora-43.repo` + `hummingbird.repo`) **To check if a package is available in Hummingbird:** ```bash # Check the Hummingbird RPMs repository ls ../rpms/rpms/ | grep ``` **Important:** If the package is **only** available in the Hummingbird repository, add `distros: [hummingbird]` to `properties.yml` (see step 3). ### 2. Create Image Directory and Copy Files Follow the naming pattern `-` with dashes (e.g., `go-1-25`, `nodejs-24`, `dotnet-sdk-10-0`): ```bash # Create directory mkdir images/go-1-25 # Copy templates from the closest existing version, not the generic templates cp images/go-1-26/Containerfile.j2 images/go-1-25/ cp -P images/go-1-26/tests-container.yml images/go-1-25/ cp images/go-1-26/report.yml images/go-1-25/ ``` Tests shared by a version family may instead live under `ci/shared-tests/`, with each image's `tests-container.yml` and `tests-k8s.yml` symlinked to the shared files. Preserve the sibling image's arrangement. Only the version that publishes `latest` owns `README.md.j2` and the generated README files for the shared repository. When a new version becomes the latest, move or copy the template to that version and remove it from the previous version. Older versions without `latest` do not need a README template. README files are generated from a template in the same image directory; they are not shared automatically. ### 3. Configure properties.yml Create `images/go-1-25/properties.yml` with the versioned package: ```yaml --- main_package: golang1.25 repository: go # endoflife.date: parallel at major.minor (Docker Hub + Wolfi) stream: "1.25" application_category: "Programming Languages & Runtimes" summary: "Statically-compiled language with concurrency and a large standard library" description: >- Statically typed, compiled programming language with garbage collection, memory safety, and a large standard library. It includes the complete Go toolchain. url: "https://go.dev" distros: - hummingbird # Only if package is unavailable in Fedora user: root version_constraints: '1.25.*' tags: # NOTE: Do NOT include 'latest' or major-version tags for older versions # Only the newest version (e.g., go-1-26) should have those tags - value: '{{ package_major_minor_version("golang1.25") }}' label: io.hummingbird-project.major-minor-version - value: '{{ package_version("golang1.25") }}' label: org.opencontainers.image.version rpm_packages: all: - coreutils-single - git-core - glibc-devel - golang1.25 - make ``` **Key differences from a new image:** - Set `main_package` to the versioned package name - Set `repository` to match the base image family (for registry organization) - Set `stream` to the version this image tracks (e.g., `"1.25"` for go-1-25), with a YAML comment above it documenting the reasoning (copy from an existing sibling image) - Set `version_constraints` so lockfile updates cannot cross the image's version boundary - Copy `application_category`, `summary`, `description`, and `url` from the sibling image, then update any version-specific text - Add `distros: [hummingbird]` if the package is only in the Hummingbird repository - Preserve sibling settings such as variants, FIPS packages, user, and support level unless the new upstream version requires a deliberate change - **Tag strategy for multiple versions:** - Only the newest version includes `latest` - Include a major-version tag on each stream when it is unique (e.g., Helm 3 and Helm 4) - If several streams share a major version (e.g., Go 1.25 and 1.26), only the newest includes that shared major-version tag - Every version includes its non-conflicting stream tag and full version tag ### 4. Update Containerfile Template Edit `images/go-1-25/Containerfile.j2` to reference the versioned package in any environment variables or version-specific commands: ```diff ENV GOPATH=/go \ GOTOOLCHAIN=local \ - GOLANG_VERSION={{ package_version('golang') }} \ + GOLANG_VERSION={{ package_version('golang1.25') }} \ PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ``` ### 5. Update Comparison Images Edit `images/go-1-25/report.yml` to compare against version-specific upstream images: ```yaml compare: - pull: docker.io/golang:1.25 url: https://hub.docker.com/_/golang - pull: cgr.dev/chainguard/go:latest-dev url: https://images.chainguard.dev/directory/image/go/overview ``` ### 6. Tag Strategy for Multiple Versions When multiple versions share a repository, coordinate tags to avoid conflicts. The following Go versions both have major version 1, so only the newest can publish the `1` tag. **Newest version** (`go-1-26`): ```yaml tags: - value: latest # Points to newest - value: '{{ package_major_version("golang1.26") }}' # e.g., "1" label: io.hummingbird-project.major-version - value: '{{ package_major_minor_version("golang1.26") }}' # e.g., "1.26" label: io.hummingbird-project.major-minor-version - value: '{{ package_version("golang1.26") }}' # Full version label: org.opencontainers.image.version ``` **Older versions** (`go-1-25`): ```yaml tags: # NO 'latest' tag # NO major-version tag (would conflict with go-1-26's "1" tag) - value: '{{ package_major_minor_version("golang1.25") }}' # e.g., "1.25" label: io.hummingbird-project.major-minor-version - value: '{{ package_version("golang1.25") }}' # Full version label: org.opencontainers.image.version ``` **Why:** This ensures users can reference: - `quay.io/hummingbird/go:latest` → go-1-26 - `quay.io/hummingbird/go:1` → go-1-26 (latest 1.x) - `quay.io/hummingbird/go:1.25` → go-1-25 (specific minor version) - `quay.io/hummingbird/go:1.26` → go-1-26 (specific minor version) **When to update:** If you add `go-1-27`, update `go-1-26` to remove its `latest` and `major-version` tags, and add them to `go-1-27`. ### 7. Generate Files and Test ```bash # Generate distro directories, lockfiles, VERSION, TAGS, etc. make # Build the image ci/build_images.sh go-1-25/hummingbird/default # Run tests ci/run_tests_container.sh go-1-25/hummingbird/default # Run linters make check ``` ### Common Issues **Error: `No match for argument: golang1.25`** The package is not available in the configured distro repositories. If it is available only in the Hummingbird repository, add `distros: [hummingbird]` to `properties.yml`, then regenerate files: ```bash make ``` Generation removes stale distro directories. Do not edit or remove generated files individually. ## Choosing a Stream Value Every image must have a `stream` field in `properties.yml`. The stream is a version series identity (e.g., `"24"`, `"3.11"`, `"10.1"`, `"latest"`) that is emitted as the `io.hummingbird-project.stream` label on the container image. It enables the image catalog to provide consistent structured data about multi-stream repositories. ### Why Stream Matters The image catalog must provide consistent structured data across time, including for images published before a second stream exists. Labels on published images cannot be changed retroactively. If a repository gains a new stream, all previously published images must already carry the correct version-based stream value — otherwise the catalog has inconsistent data that requires fragile repository-specific fixups. ### Decision Process ```mermaid flowchart TD Start["New image: choose stream value"] --> Q1{"Does endoflife.date show\nmultiple concurrently\nsupported branches?"} Q1 -->|Yes| Assess["Assess ecosystem granularity:\nupstream project, Docker Hub,\nDebian/Ubuntu, Wolfi/Chainguard,\nFedora, endoflife.date"] Assess --> Version["Use version-based stream\n(match ecosystem granularity)"] Q1 -->|No| Q2{"Does the software have a\nmeaningful version series\nidentity?"} Q2 -->|Yes| Current["Use version-based stream\n(current major version)"] Q2 -->|No| Latest["stream: latest"] Version --> Doc["Document reasoning\nas YAML comment"] Current --> Doc Latest --> Doc ``` Document the reasoning for the stream choice as a YAML comment above the `stream` field in `properties.yml`. This ensures future maintainers understand why a particular granularity was chosen: ```yaml # endoflife.date: parallel at major.minor (2.8, 3.0, 3.2) stream: "3.0" ``` ```yaml # rolling release with date-based versions, no version branches stream: "latest" ``` #### Step 1: Does endoflife.date show multiple concurrently supported branches? Check [endoflife.date](https://endoflife.date) for the upstream project. It provides structured data on upstream release branches, security support timelines, and versioning schemes for most projects. If it shows multiple concurrently supported branches, use a version-based stream at the granularity of those branches. If the product is not on endoflife.date, that is itself evidence that parallel version lifecycle management is not a concern for that software — proceed to step 2. The stream granularity must match the level at which upstream maintains parallel security-supported branches. Check: - **Upstream project**: At what version granularity are parallel branches maintained? - **Docker Hub official images**: Are there parallel version tags? At what granularity? - **Debian/Ubuntu**: Are there parallel versioned packages? - **Wolfi/Chainguard**: Are there versioned package variants? - **Fedora**: Are there parallel versioned packages? Single-version availability is **not** evidence against versioning, but multi-version availability **is** strong evidence for it. Important: the image's directory name is **not** evidence for granularity. The directory may use a coarser version or no version at all. The stream must reflect upstream's actual branch structure. Similarly, the purpose of upstream's branches (LTS vs stable, stable vs mainline, STS vs LTS) is irrelevant to granularity. If they are parallel branches receiving security updates, the stream granularity must match. Match the upstream project's version naming convention. If the project and ecosystem consistently identify branches using a format that includes a minor component (e.g., .NET uses `8.0`, `9.0` — never just `8` or `9`), use that format for the stream value. This applies even when endoflife.date abbreviates to just the major number. #### Step 2: Does the software have a meaningful version series identity? If the software uses numbered versioning where the major (or major.minor) number identifies a version series, use a version-based stream at the current major version. The risk is asymmetric: using a version-based stream when a new major version never ships is harmless (the label is still correct), while using `"latest"` when a new version does ship breaks catalog consistency. Note: if endoflife.date lists minor versions (e.g., memcached 1.4, 1.5, 1.6) but each immediately supersedes the previous with no overlap in support periods, the minor number is a sequential release counter, not a branch identity. Use the major version as the stream. Reserve `"latest"` only for images where no meaningful version series identity exists. This is a narrow criterion with two cases: - **Rolling/date-based releases** with no version series (e.g., minio uses `RELEASE.2024-01-18...`) - **Base images that track the distro** rather than independent software (e.g., core-runtime) #### Step 3: For unversioned images, track the current version For images without a version in their directory name or version constraints, the stream value reflects the version currently shipped and must be updated when the upstream version changes. This is analogous to `version_constraints` — an explicit declaration that is updated periodically. ### Field Reference See the [Image Configuration Reference](image-configuration-reference.md#stream) for the `stream` field definition and examples. ## Next Steps - [Testing Guide](testing-images.md) - How to run and write tests - [Image Configuration Reference](image-configuration-reference.md) - Complete `properties.yml` reference - [FIPS Variant Guide](fips-variant-guide.md) - How to add FIPS variants to images - [Image Pipeline](https://hummingbird-project.io/l/image-pipeline) - How the complete pipeline works -------------------------------------------------------------------------------- # Adding FIPS Variants url: https://hummingbird-project.io/docs/contributing/fips-variant-guide/ description: How FIPS variants work and step-by-step guide for adding them ## Overview FIPS variants provide container images that use only [FIPS 140-3 validated][cmvp] cryptographic modules. FIPS is implemented as a cross-cutting modifier variant (like `builder`), not a separate image — the FIPS variant uses the same base image structure but layers FIPS configuration on top via additional RPM packages. FIPS variants are restricted to the **Hummingbird distro only**, because FIPS-validated packages are not available in Fedora Rawhide. FIPS images must not contain any non-validated cryptographic libraries. The presence of an unvalidated crypto library (such as `libgcrypt` or `gnutls`) would undermine the FIPS compliance guarantee, even if the application itself does not use that library. Global tests enforce this constraint. [cmvp]: https://csrc.nist.gov/projects/cryptographic-module-validation-program ## FIPS Validation Scope The cryptographic modules shipped in FIPS images are FIPS 140-3 validated through the NIST Cryptographic Module Validation Program (CMVP). This validation applies **only when running on RHEL systems installed in FIPS mode**. Running FIPS images outside this environment (e.g., on Fedora, on RHEL not in FIPS mode, or on other Linux distributions) is outside the validated configuration. As a best-effort goal, FIPS images aim to behave similarly on both FIPS and non-FIPS hosts — restricting cryptographic operations to FIPS-approved algorithms regardless of host configuration. This is not a guarantee; some images (particularly NSS-based ones like OpenJDK) may behave differently depending on host FIPS mode. ## Validated Cryptographic Modules All validated modules in Hummingbird are pre-built binaries from RHEL 9.2, submitted to NIST for FIPS 140-3 validation: | Module | Package | Verified by global test | |-----------------------|----------------------------|------------------------------------| | OpenSSL FIPS provider | `openssl-config-fips` | `fips-provider-matches-ubi9` | | NSS softokn | `nss-softokn-fips` | `fips-nss-modules-match-rhel92` | | NSS freebl | `nss-softokn-freebl-fips` | `fips-nss-modules-match-rhel92` | Global tests validate each module's checksum against the known-good RHEL 9.2 validated binaries. Checksums are architecture-specific because the binaries differ per architecture. All FIPS variants also install `crypto-policies-config-fips`, which sets the system-wide crypto policy to `FIPS`. ### Two FIPS Stacks Two FIPS stacks exist depending on which crypto library the image's software uses: **OpenSSL stack** — `crypto-policies-config-fips` + `openssl-config-fips`. The OpenSSL FIPS provider is enabled via a drop-in configuration file at `/etc/pki/tls/openssl.d/fips-provider-enable.cnf`. Enforcement is container-side: non-approved algorithms are rejected even on non-FIPS hosts. Used by: Nginx, Node.js, Python, Ruby. Go images also include these packages for system tools (e.g., `git-core`), but Go itself uses its own [native FIPS 140-3 module][go-fips-blog]. **NSS stack** — `crypto-policies-config-fips` + `nss-softokn-fips` + `nss-softokn-freebl-fips`. NSS checks the host kernel's FIPS mode (`/proc/sys/crypto/fips_enabled`), so enforcement behavior differs between FIPS and non-FIPS hosts. The primary Java security provider switches to `SunPKCS11-NSS-FIPS`. Used by: OpenJDK. Both stacks share `crypto-policies-config-fips` for the system crypto policy, but differ in which cryptographic library provides the validated implementation. ## Adding a FIPS Variant ### 1. Determine the FIPS Stack Identify which crypto library the image's software uses: - **OpenSSL** — most languages and servers (Python, Node.js, Ruby, Nginx, Go) - **NSS** — Java/OpenJDK (uses NSS via PKCS#11) This determines which FIPS packages to install. ### 2. Update properties.yml Add `fips` to `additional_variants` with a Hummingbird distro restriction, and add the FIPS packages to `rpm_packages.fips`. **OpenSSL-based image:** ```yaml additional_variants: - name: fips distros: [hummingbird] rpm_packages: fips: - crypto-policies-config-fips - openssl-config-fips ``` **NSS-based image (OpenJDK):** OpenJDK has existing `runtime` and `runtime-builder` base variants. FIPS is added as both `fips` (full JDK) and `runtime-fips` (headless JRE): ```yaml additional_variants: - name: fips distros: [hummingbird] - name: runtime-fips distros: [hummingbird] rpm_packages: fips: - java-21-openjdk-devel # variant-specific packages - crypto-policies-config-fips - nss-softokn-fips - nss-softokn-freebl-fips runtime-fips: - crypto-policies-config-fips - nss-softokn-fips - nss-softokn-freebl-fips ``` See the [Image Configuration Reference](image-configuration-reference.md) for complete `properties.yml` options, including [additional_variants](image-configuration-reference.md#additional_variants) and [rpm_packages](image-configuration-reference.md#rpm_packages). ### 3. Generate Files Run `make` to generate the `hummingbird/fips/` directory structure (Containerfile, RPM lockfiles, VERSION, TAGS): ```bash make ``` ### 4. Write FIPS Tests Add FIPS-specific tests to `tests-container.yml` using `filters.variants` to target FIPS variants. A standard set of FIPS tests validates: 1. **Non-approved algorithm rejected** — e.g., MD5 (for security use), Blowfish, RC2 2. **Approved algorithm works** — e.g., SHA-256 3. **Non-approved cipher rejected** — e.g., 3DES, CHACHA20-POLY1305 4. **Approved cipher works** — e.g., AES-256-GCM (encrypt/decrypt roundtrip) **Example (Python):** ```yaml fips-rejects-md5: filters: variants: ["*fips*"] command: | test_engine_run --rm "${TEST_IMAGE:?}" python3 -c " import hashlib try: hashlib.new('md5') print('MD5_ALLOWED') except ValueError: print('MD5_REJECTED') " | grep -q MD5_REJECTED \ || test_fail "FIPS mode should reject MD5" fips-allows-sha256: filters: variants: ["*fips*"] command: | test_engine_run --rm "${TEST_IMAGE:?}" python3 -c " import hashlib print(hashlib.sha256(b'test').hexdigest()) " | grep -q 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 \ || test_fail "FIPS mode should allow SHA-256" ``` **Host FIPS mode considerations:** Where possible, write tests **without** `filters.fips` so they validate behavior on both FIPS and non-FIPS hosts. This supports the best-effort goal of consistent behavior across host types. Use `filters.fips: true` only for tests that inherently require a FIPS-enabled host kernel — primarily NSS-based images where enforcement depends on the host: ```yaml # NSS-based: rejection only works on FIPS hosts fips-rejects-blowfish: filters: variants: ["*fips*"] fips: true # requires FIPS-enabled host kernel command: | ... ``` See the [Test Configuration Reference](test-configuration-reference.md) for [variant filters](test-configuration-reference.md#variant-and-distro-filtering) and [FIPS mode selection](test-configuration-reference.md#fips-mode-selection). ### 5. Build and Test ```bash # Build the FIPS variant ci/build_images.sh /hummingbird/fips # Run tests ci/run_tests_container.sh /hummingbird/fips # Run linters make check ``` ## Go-Specific FIPS Requirements Go 1.24+ includes a native FIPS 140-3 cryptographic module that has been [CMVP validated][go-fips-blog] ([CMVP Certificate #5247][cmvp-cert]). Unlike the previous `golang-fips` approach (which replaced the compiler with a fork using OpenSSL as the crypto backend), native FIPS uses the same `golang` package with build-time and runtime environment variables: - **`GOFIPS140=v1.0.0`** (build-time): Tells `go build` to include the CMVP-validated FIPS module (v1.0.0) in the resulting binary. Set in the image environment so all builds in the container use it. - **`GODEBUG=fips140=on`** (runtime): Enables FIPS 140-3 mode — uses NIST DRBG for randomness, negotiates only FIPS-approved TLS, and runs mandatory self-tests. Set in the image environment as the default for binaries run in the container. - **Same package**: The FIPS variant installs the standard `golang1.25` package (no `version_package` override needed). - **No cgo dependency**: Native FIPS works with both `CGO_ENABLED=0` and `CGO_ENABLED=1` binaries. ```yaml main_package: golang1.25 rpm_packages: fips: - crypto-policies-config-fips - golang1.25 - openssl-config-fips ``` The `openssl-config-fips` package is retained for system tools in the image (e.g., `git-core`) that use OpenSSL for TLS connections. Go-specific FIPS tests verify: - `crypto/fips140.Enabled()` returns `true` (native FIPS module is active) - `go version -m` output includes `build GOFIPS140=v1.0.0-c2097c7c` (CMVP-validated module, per the security policy section 11.1) - FIPS works with `CGO_ENABLED=0` static binaries (a key advantage over the old `golang-fips` approach) [go-fips-blog]: https://go.dev/blog/fips140 [cmvp-cert]: https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/5247 ## OpenJDK-Specific FIPS Requirements OpenJDK uses the **NSS stack** instead of OpenSSL, which has several implications: - **No `openssl-config-fips`** — uses `nss-softokn-fips` + `nss-softokn-freebl-fips` instead - **Host FIPS mode dependency** — NSS checks `/proc/sys/crypto/fips_enabled`, so FIPS enforcement behavior differs between FIPS and non-FIPS hosts. Crypto rejection tests must use `filters.fips: true`. - **Compound variants** — OpenJDK introduces `runtime-fips` alongside `fips`, combining the `runtime` base variant with the FIPS modifier. Both share the same NSS FIPS packages; `fips` additionally includes `java-*-openjdk-devel`. - **Security provider** — in FIPS mode, the primary Java security provider becomes `SunPKCS11-NSS-FIPS` instead of the default `SUN`/`SunJCE` providers. ## Global FIPS Tests The following tests run automatically for all FIPS images via `ci/global-tests/tests-container.yml`. Each test skips gracefully for images using the other FIPS stack. | Test | Validates | Applies to | |---------------------------------|----------------------------------------------------------|---------------| | `fips-provider-matches-ubi9` | OpenSSL FIPS module checksum matches RHEL 9.2 binaries | OpenSSL stack | | `fips-nss-modules-match-rhel92` | NSS module checksums match RHEL 9.2 validated binaries | NSS stack | | `fips-no-libgcrypt` | `libgcrypt` is not present (not a validated module) | All FIPS | | `openssl-fips-config-installed` | OpenSSL FIPS drop-in config exists | OpenSSL stack | | `crypto-policy-is-fips` | System crypto policy is set to `FIPS` | All FIPS | Image-specific FIPS tests (algorithm rejection/acceptance) are defined in each image's `tests-container.yml`. ## Testing on FIPS Hosts The test runner detects host FIPS mode from `/proc/sys/crypto/fips_enabled` and exports the `TEST_FIPS` environment variable (`true` or `false`). Tests can filter on this using `filters.fips`: ```yaml # Runs only on FIPS-enabled hosts fips-host-required: filters: fips: true command: | ... # Runs only on non-FIPS hosts non-fips-only: filters: fips: false command: | ... # Runs regardless of host FIPS mode (default when filters.fips is omitted) always-runs: command: | ... ``` **OpenSSL-based images**: Most FIPS tests run regardless of host FIPS mode, because the OpenSSL FIPS provider enforces algorithm restrictions at the container level. **NSS-based images**: Crypto rejection tests require `filters.fips: true`, because NSS enforcement depends on the host kernel's FIPS mode. ## Next Steps - [Image Configuration Reference](image-configuration-reference.md) — `additional_variants`, `rpm_packages`, `version_package` - [Test Configuration Reference](test-configuration-reference.md) — variant filters, FIPS mode selection - [Testing Guide](testing-images.md) — how to run and write tests -------------------------------------------------------------------------------- # Testing Guide url: https://hummingbird-project.io/docs/contributing/testing-images/ description: How to run and write integration tests locally and work with CI Container tests validate image functionality with Docker and Podman. K8s tests validate images in real Kubernetes environments. ## Running Container Tests Locally ### Prerequisites - **Container Engine**: [Podman](https://podman.io/) or [Docker](https://www.docker.com/) - **Python**: `PyYAML` package (`pip install PyYAML` or `dnf install python3-pyyaml`) ### With Podman (Recommended) Tests work directly with Podman: ```bash ci/run_tests_container.sh # Test specific distro ci/run_tests_container.sh /rawhide # Test specific distro/variant ci/run_tests_container.sh /rawhide/default # Verbose output (shows passing test output and bash trace) ci/run_tests_container.sh --verbose ``` ### With Docker Use `--setup` to automatically configure Docker-in-Docker: ```bash ci/run_tests_container.sh --engine docker --setup ``` The `--setup` flag: - Starts `hummingbird-docker-dind` container with `mirror.gcr.io/docker:dind` - Configures environment variables (`DOCKER_HOST`, `DOCKER_CERT_PATH`, etc.) - Waits for Docker daemon to be ready - Reuses existing container on subsequent runs **Prerequisites:** - Docker CLI (`dnf install docker-cli`) - Podman (to run the dind container) ### Building Images Build images before testing: ```bash # With Podman ci/build_images.sh # With Docker ci/build_images.sh --engine docker --setup ``` ### Testing Base Images When modifying base images, test dependent images: ```bash ci/build_images.sh core-runtime ci/build_images.sh --build-deps core-runtime ci/run_tests_container.sh --include-reverse-deps core-runtime ``` ### Reproducing CI Failures Reproduce Testing Farm failures locally: ```bash # Single distro/variant tests ci/run_tests_container.sh --include-reverse-deps --engine podman /rawhide/default ci/run_tests_container.sh --include-reverse-deps --engine docker --setup /rawhide/default # Group tests (all distros/variants) ci/run_tests_container.sh --engine podman /rawhide/group ci/run_tests_container.sh --engine docker --setup /rawhide/group ``` ### Troubleshooting Container Tests #### Inspecting Failed Tests Use `--pause` to inspect containers before cleanup: ```bash ci/run_tests_container.sh --pause ``` #### Viewing Test Output By default, only failing tests show output. Use `--verbose` to see passing test output: ```bash ci/run_tests_container.sh --verbose ``` This also enables bash trace mode (`set -x`), showing each command as it executes. ## Running K8s Tests Locally K8s tests require kubectl and access to a Kubernetes cluster (via kubeconfig or context). ### Basic Usage ```bash ci/run_tests_k8s.sh --context # Test specific distro/variant ci/run_tests_k8s.sh --context /rawhide/default # Verbose output ci/run_tests_k8s.sh --verbose --context ``` **Safety**: The script requires explicit `--context` or `--kubeconfig` to prevent accidental operations on production clusters. ### Local Development Workflow Build images locally and push to the internal registry: ```bash # Build the image podman build -t my-nginx:dev images/nginx/hummingbird/default/ # Push to internal registry and test ci/run_tests_k8s.sh --context mpp-preprod --push-image my-nginx:dev nginx/hummingbird/default ``` This requires `oc login` to the target cluster and `oc project ` to set the target namespace. ### Testing Published Images Test published images without building locally: ```bash IMAGE_URL=quay.io/hummingbird/nginx:latest \ IMAGE_NAME=nginx--hummingbird--default \ ci/run_tests_k8s.sh --context mpp-preprod nginx/hummingbird/default ``` ### Troubleshooting K8s Tests Use `--pause` to inspect resources before cleanup, or `--verbose` to see passing test output: ```bash ci/run_tests_k8s.sh --pause --context ci/run_tests_k8s.sh --verbose --context ``` Tests that encounter recognized transient infrastructure failures run up to three times. The runner cleans up test resources and waits 60 seconds before each retry. Application assertion failures are reported immediately. Recognized failures include transient registry and container daemon errors, package-download timeouts from the Hummingbird repository, and transient Go `crypto/rand` entropy-read failures on FIPS-enabled hosts. ## Writing Tests ### Test File Locations - **Container tests**: `images//tests-container.yml` - **K8s tests**: `images//tests-k8s.yml` Both use the same YAML format with different available environment variables. ### Basic Tests Create a test file with named tests: ```yaml --- version-check: command: | test_engine_run --rm "${TEST_IMAGE}" your-service --version basic-functionality: command: | test_engine_run --rm "${TEST_IMAGE}" your-service --help ``` Container tests must use `test_engine_run` to start containers. When a detached container has both a name and a network, the helper waits for Podman network DNS registration automatically before returning. Fresh lookup processes avoid negative DNS caching in a single client process: ```bash test_engine_run -d --network "${NETWORK_NAME}" --name "${CONTAINER_NAME}" "${TEST_IMAGE}" ``` For K8s tests: ```yaml --- cluster-access: command: | kubectl auth can-i get pods || test_fail "No pod access" run-as-pod: command: | name="test-pod-${TEST_RUN_ID}" kubectl run "${name}" --image="${TEST_IMAGE}" --restart=Never --labels="${TEST_RUN_LABEL}" kubectl wait --for=jsonpath='{.status.phase}'=Succeeded "pod/${name}" --timeout=60s ``` ### External Test Scripts For complex tests, use a separate shell script: ```yaml --- complex-test: command: ./test-complex-scenario.sh ``` Create `images//test-complex-scenario.sh`: ```bash #!/bin/bash set -euo pipefail # Load TEST_IMAGES array for cross-variant testing # shellcheck disable=SC1090 source "${TEST_IMAGES_PATH:?}" # Run test test_engine_run --rm "${TEST_IMAGE:?}" your-service test ``` ### Variant-Specific Tests Limit tests to specific variants: ```yaml build-tools-test: variants: [builder] command: | test_engine_run --rm "${TEST_IMAGE}" make --version ``` ### Cross-Variant Tests Test interactions between variants: ```yaml cross-variant-test: variants: [group] command: | builder="${TEST_IMAGES[nginx/builder]:?}" default="${TEST_IMAGES[nginx/default]:?}" test_engine_run --rm "${builder}" nginx -version test_engine_run --rm "${default}" nginx -version ``` ### Known Issues (Container Tests Only) Mark container tests with known failures: ```yaml test-with-known-issue: command: | result=$(test_engine_run --rm "${TEST_IMAGE}" some-command) [[ "$result" == "expected" ]] || test_fail "Custom error message" known_issues: - description: "Known configuration issue" issue: "https://issues.redhat.com/browse/PROJ-1234" pattern: "Custom error message" fails: "sometimes" ``` See the [Test Configuration Reference](test-configuration-reference.md) for complete configuration options. ## Working with CI Tests ### Container Tests (Testing Farm) Container tests run on Testing Farm with RHEL-9-Nightly systems for both `x86_64` and `aarch64` architectures. #### Viewing CI Test Results 1. Open the merge request in GitLab 2. Navigate to the Jobs page for the pipeline 3. Identify the Testing Farm job (e.g., `containers-rawhide-testing-farm-x86-64`) 4. Click on the job name to go to the Konflux PipelineRun page 5. If `wait-for-results` is already finished, select it in the pipeline details, and then switch to the Testing Farm job via the `ARTIFACTS_URL` link in the right side pane; otherwise, select the `scheduler` job, switch to the Testing Farm API details via the `tf-request` link in the right side pane, and then follow the `run.artifacts` link in the JSON data ### K8s Tests (Konflux Ephemeral Namespace) K8s tests run in Konflux ephemeral namespaces provisioned via EaaS (Environment as a Service). #### Viewing K8s Test Results 1. Open the merge request in GitLab 2. Navigate to the Jobs page for the pipeline 3. Identify the K8s test job (e.g., `containers-rawhide-k8s-test`) 4. Click on the job name to go to the Konflux PipelineRun page 5. View the `run-tests` task logs for test output ### Rerunning CI Tests Retrigger a specific pipeline using slash commands in merge request comments: ```text /retest nginx--hummingbird--default-on-pull-request ``` For automated retriggers of only failed checks, see [Retrying Konflux Checks](https://hummingbird-project.io/l/retrying-konflux-checks). ## Next Steps - [Test Configuration Reference](test-configuration-reference.md) - Complete test configuration reference - [Adding Images](adding-images.md) - How to add new images with tests -------------------------------------------------------------------------------- # Adding Kubernetes Tests url: https://hummingbird-project.io/docs/contributing/adding-k8s-tests/ description: How to add Kubernetes and OpenShift integration tests to container images This guide describes how to add Kubernetes tests to container images. ## Overview Kubernetes tests validate that container images work correctly in Kubernetes and OpenShift environments, particularly: - Directory permissions (important for OpenShift arbitrary UIDs) - Version checks - Application functionality (web servers, databases, etc.) - Multi-container scenarios (sidecar patterns) ## Confirm Test Design Before implementing: 1. Read `images/{image_name}/tests-container.yml` to understand existing tests 2. Identify which tests need K8s validation: - Directory permissions (important for OpenShift arbitrary UIDs) - Version check - Application functionality (web server, database, etc.) - Multi-container scenarios (sidecar patterns) 3. Confirm test approach before creating files ## Rules - Script naming: `test-k8s-*.sh` prefix (e.g., `test-k8s-version.sh`) - Script location: `images/{image_name}/test-scripts/` - YAML location: `images/{image_name}/tests-k8s.yml` - Keep YAML lean: complex logic goes in scripts, not inline YAML - No external downloads: avoid network flakes (exception: container images) - OpenShift compatibility: writable dirs must use gid=0 + g+w permissions ## Validation - If `images/{image_name}/` doesn't exist → error "Image not found" - If no `tests-container.yml` → error "Add container tests first" ## Create Test Scripts If directory doesn't exist → create `images/{image_name}/test-scripts/` ### Test Script Template ```bash #!/bin/bash set -euo pipefail # Test logic test_result=$(command) if [[ ! "${test_result}" =~ "expected" ]]; then echo "FAIL: Description" echo "Got: ${test_result}" exit 1 fi echo "SUCCESS: Test passed" exit 0 ``` ### Directory Permissions Test If testing OpenShift compatibility → create `test-k8s-directory-permissions.sh`: - Check writable directories exist and are writable - Verify running as non-root (UID != 0) - Test actual write with touch/rm ### Version Check Test If testing version → create `test-k8s-version.sh`: - Run version command - Check major version only (not major.minor.patch) - Avoids breaking on patch updates ### Application-Specific Tests - If web server → deployment test with sidecar (curl container) - If database → connection/persistence test - If tool → primary functionality test ## Create tests-k8s.yml Pattern for each test: ```yaml test-name: variants: [default] command: | # Create ConfigMap with script kubectl create configmap "${TEST_GROUP}"-scripts-"${TEST_RUN_ID}" \ --from-file=test-k8s-name.sh=test-scripts/test-k8s-name.sh \ --dry-run=client -o yaml | kubectl label -f - --local -o yaml hum-k8s-test="${TEST_RUN_ID}" | kubectl apply -f - # Create Pod kubectl apply -f - < --push-image localhost/{image_name}:test {image_name}/{distro}/default ``` **Note**: Push images to `localhost/{image_name}:test`, not to `quay.io`. Tests should never push to the official registry. 5. Monitor pods: `oc get pods -w` ## Common Issues - If tests fail OpenShift with "Permission denied" → check Containerfile uses gid=0 and g+w - If wrong image used in OpenShift → verify `ci/run_tests_k8s.sh` doesn't override TEST_IMAGE when `--push-image` set - If multi-container test can't see files → add emptyDir volume - If readiness probe times out in OpenShift → add `timeoutSeconds: 5` ## Automated Skill For AI-assisted implementation, use the `/add-k8s-tests ` command which follows these patterns automatically. -------------------------------------------------------------------------------- # Image Configuration Reference url: https://hummingbird-project.io/docs/contributing/image-configuration-reference/ description: Complete reference for properties.yml configuration and image settings Complete reference for configuring container images via `properties.yml`. ## properties.yml Overview Each image can have custom settings via a `properties.yml` file. This reference documents all available configuration options. **Minimal example:** ```yaml --- rpm_packages: all: - nginx main_package: nginx tags: - value: latest ``` **Complete structure:** ```yaml --- # Distro configuration (see Image Variants section) # distros: [hummingbird] # Override default distros (use when package unavailable in some distros) # Image variants (see Image Variants section) # Use additional_variants to extend defaults, or variants to override completely additional_variants: [fpm, fpm-builder] # Adds to default variants # variants: [default, builder] # Alternative: completely replaces defaults # Container runtime configuration (see Container Configuration section) user: default # Optional: 'default' for default_user (65532), or literal user ID/name # Package management (see Package Management section) rpm_packages: build-deps: [...] all: - package-name # All architectures - name: arch-specific-package # Arch-specific (see Package Management) arches: only: x86_64 : [...] : [...] additional_repos: [...] allow_fedora_repos: true # Optional: allow Fedora repos for hummingbird images (temporary workaround) # Build configuration (see Build Configuration section) hermetic: true # Exceptional legacy source builds only; do not use for new images: # build_from_source: true # prefetch_gomod_path: upstream-submodule # Versioning (see Versioning section) main_package: package-name # version_package: # Optional: override package for version lookup per variant/distro # fips: package-fips-name # Example: variant-specific package name # Testing configuration (see Testing Configuration section) reverse_dependency_tests: false # Optional: disable for images like curl # Release configuration (see Release Configuration section) repository: image-name # Required: Quay.io repository name # stream: "latest" # Required: version series identity for the image catalog application_category: "Storage" # Required: Pyxis application category # support_level: community # Optional: community or experimental (default: Red Hat supported) # Image metadata (see Image Metadata section) summary: "Short one-liner description" # Required when adding labels description: >- # Required when adding labels Short paragraph describing the image. 1-2 sentences about key features. url: "https://upstream-project.org" # Required when adding labels tags: - value: latest - value: '{{ package_major_version("package-name") }}' label: io.hummingbird-project.major-version # Compliance scanning (enabled by default, see Compliance Scanning section) # oscap: # enabled: false # Optional: set to false to opt out # profiles: # Optional: override which profiles run per variant # cis: true # stig: # variants: ["*fips*"] # exclude_rules: # Optional: exclude specific rules (concatenated with globals) # - id: xccdf_org.ssgproject.content_rule_example # reason: "Reason for exclusion" # variants: ["*fips*"] # Optional: glob patterns # profiles: [cis] # Optional: limit to specific profiles ``` ## Image Metadata Images can declare metadata fields for container image labels: ### summary - **Type**: String - **Required**: When adding image labels (currently caddy only, expanding to all) - **Length**: ~40-80 characters (one-liner) Short phrase describing what the image is. Used in table/list views and catalog cards. Do not start with the image name -- the name is always visible from context (UI headings, `name` label, inspect output). **Example**: `"Web server with automatic HTTPS"` ### description - **Type**: String (use `>-` YAML scalar for multi-line readability) - **Required**: When adding image labels - **Length**: ~100-250 characters (1-2 sentences) Short paragraph for card views, `podman inspect`, and Kubernetes UIs (`io.k8s.description`). Describes what the software is and its key distinguishing features. Use `>-` (folded scalar) in YAML to keep the value readable across multiple lines -- it folds to a single-line string. Do not use `|` (literal block) as embedded newlines break label syntax. Avoid embedded double quotes and backslashes -- values are emitted as-is in Containerfile LABEL instructions with no escaping. The same `description` value is used when generating the image README: the `readme_description()` macro in `README.md.j2` templates renders it as the introductory paragraph. Do not duplicate this text in the template; use `{{ readme_description() }}`. **Example**: ```yaml description: >- Extensible server platform with automatic HTTPS by default. Provides HTTP/3, reverse proxying, load balancing, and static file serving with minimal configuration. ``` ### url - **Type**: String (URL) - **Required**: When adding image labels URL for the upstream project homepage. Used in `org.opencontainers.image.url` and the Conforma `url` label. **Example**: `"https://caddyserver.com"` ## Image Variants Images can have multiple variants (e.g., `default` and `builder`): - **default**: Minimal runtime environment - **builder**: Includes development tools and build dependencies - **custom**: Define custom variants in `properties.yml` Each variant gets its own: - Containerfile in `images///Containerfile` - RPM lockfiles in `images///rpms` - Build pipeline and release tags (non-default variants get `-` suffix) ### variants - **Type**: Array of strings - **Default**: `[default, builder]` - **Description**: List of image variants to generate. Completely overrides the default variants. Each variant gets its own Containerfile, lockfiles, build pipeline, and release tags. Non-default variants get a `-` suffix in their image tags. - **Example**: `[default]` (CI images that only need the default variant) - **Note**: Prefer `additional_variants` when extending defaults; use `variants` only when you need to exclude default variants. ### additional_variants - **Type**: Array of strings or objects - **Default**: None - **Description**: Additional variants to add to the base variants. The base is `variants` if specified, otherwise the default variants. Use this when you want to extend without repeating. **Simple format (string array):** ```yaml additional_variants: [fpm, fpm-builder] # Adds FPM variants to base variants for all distros ``` **Object format with distro restrictions:** Each variant can be an object with `name` and optional `distros` fields to restrict which distros the variant is built for: ```yaml additional_variants: - name: fips distros: [hummingbird] # Only build fips variant for hummingbird distro - name: fpm # Simple string still works in array - name: fpm-builder distros: [hummingbird, rawhide] # Build for specific distros ``` The `distros` field supports glob patterns: ```yaml additional_variants: - name: fips distros: ["*bird"] # Matches hummingbird - name: special distros: ["raw*"] # Matches rawhide ``` **When to use distro restrictions:** Use distro restrictions when a variant requires packages or features only available in specific distros. For example, FIPS variants may only be available for Hummingbird where FIPS-validated packages are present ### variant_descriptions - **Type**: Object (string-to-string mapping) - **Required**: When image has `description` and defines `additional_variants` with non-default base specializations - **Description**: Maps base variant names to human-readable descriptions. These descriptions are emitted as the `io.hummingbird-project.variant.description` label on each container image. The global description for `default` is defined in `images/variables.yml`; image-specific entries add descriptions for bases like `fpm`, `runtime`, or `openssl`. Modifier display (builder, fips) is handled by catalog consumers using the boolean variant labels, not by these descriptions. **Writing guidelines:** - Describe the variant **holistically** — what the image variant *is*, not just what packages it adds on top of `default`. Someone reading the description with no surrounding context (e.g., via `podman inspect`) should understand the variant's identity. - Use a **noun phrase** (2-6 words), not a full sentence. - Do not start with the image name (the image context is already provided by the image name label). - Avoid the word "runtime" in the `default` description (the OpenJDK `runtime` variant is a separate base, and OpenJDK's `default` includes the full JDK). - Avoid expanding abbreviations redundantly (e.g., "JRE runtime" is redundant because JRE already means "Java Runtime Environment"). - Prefer functional descriptions (what capability the variant provides) over repeating the variant name as a product name. **Example** (PHP image with FPM base): ```yaml variant_descriptions: fpm: "PHP FastCGI process manager" ``` **Example** (OpenJDK image with runtime base): ```yaml variant_descriptions: runtime: "Headless Java runtime" ``` ### distros - **Type**: Array of strings - **Default**: `[rawhide, hummingbird]` (from `images/variables.yml`) - **Description**: List of distros to build for this image. Overrides the global `default_distros`. Use this to disable a distro for images where a required package is not available in that distro. - **Example**: `[hummingbird]` disables Rawhide builds for this image - **Note**: Each enabled distro creates a full set of variants, so the final combinations are distros × variants. ## Container Configuration ### user - **Type**: String, integer, or object (per-variant dict) - **Required**: No (defaults to `default`) - **Description**: Specifies which user the container runs as. The `USER` directive is rendered via the `{{ set_user() }}` macro at the end of Containerfile templates. The special (and default) value `default` maps to the `default_user` variable (UID 65532). For `user: default`, the `{{ final_stage() }}` macro sets `HOME=/tmp` and `WORKDIR /tmp` early in the image to ensure unprivileged users have a writable default environment. Images can override `WORKDIR` later in their Containerfile.j2. For builder variants with `user: default`, it additionally sets `ENV CONTAINER_DEFAULT_USER=65532`. **Same user for all variants:** ```yaml user: default ``` ```yaml user: root # Root user ``` ```yaml user: postgres # Literal username ``` **Per-variant configuration:** Use when different variants need different users: ```yaml user: default: root builder: root fpm: default fpm-builder: default ``` ## Package Management Packages are defined in `properties.yml` under `rpm_packages`: ```yaml rpm_packages: build-deps: # Build-time dependencies (installed in builder layer, not in final # image). Included in lockfiles for hermetic builds. - golang # Example: Go compiler for building from source - rpm-build # Example: Tools for building RPMs during image construction all: # Included in all variants (Containerfiles and lockfiles) - coreutils-single - your-main-package - name: grub2-efi-x64 # Arch-specific: only on x86_64 arches: only: x86_64 - name: grub2-efi-aa64 # Arch-specific: only on aarch64 arches: only: aarch64 hummingbird: # Included in all Hummingbird variants (distro-level key) - versioned-package-1.2 rawhide: # Included in all Rawhide variants (distro-level key) - unversioned-package builder: # Included only in "builder" variant - compiler-packages - debug-tools custom-variant: # Included only in "custom-variant" variant - variant-specific-packages ``` ### Package entry format Each entry in an `rpm_packages` list can be either a **string** (installed on all architectures) or an **object** with architecture constraints: **String entry** (all architectures): ```yaml - coreutils-single ``` **Object entry** (arch-specific): ```yaml - name: grub2-efi-x64 arches: only: x86_64 # Install only on x86_64 ``` ```yaml - name: some-package arches: not: aarch64 # Install on all architectures except aarch64 ``` The `arches.only` and `arches.not` fields accept either a single string or a list of strings. Supported architectures: `aarch64`, `x86_64`. **How arch-specific packages work:** 1. **Lockfiles**: Arch-specific entries are passed to rpm-lockfile-prototype using its native per-arch package format. The lockfile resolver only includes matching packages for each architecture. 2. **Containerfiles**: Arch-specific packages are installed via a `case` statement on `TARGETARCH`, so the correct packages are installed during multi-arch builds. ### rpm_packages.build-deps - **Type**: Array of strings or arch-specific objects - **Description**: Build-time dependencies needed during image construction. Automatically installed in the builder layer (not in the final image) and included in lockfiles for hermetic builds. Use for compilers, build tools, and other packages needed only during the build process. ### rpm_packages.all - **Type**: Array of strings or arch-specific objects - **Description**: Packages included in all variants, in both Containerfiles and lockfiles ### rpm_packages.\ - **Type**: Array of strings or arch-specific objects - **Description**: Packages included in all variants for the specified distro. Useful when different distros ship different package names (e.g. versioned vs unversioned) and the packages should apply to every variant within that distro. ### rpm_packages.\ - **Type**: Array of strings or arch-specific objects - **Description**: Packages included only in the specified variant, regardless of distro ### additional_repos - **Type**: Array of strings (repository filenames) - **Default**: None - **Description**: Additional yum repository files from `yum-repos/` to include in the RPM lockfile generation. These repos are added to the variant-specific repos (see [Global Variables Reference - default_variant_repos][default-variant-repos]). Specify only the filename, not the full path. - **Example**: `[konflux-ci-rpm-lockfile-prototype-main-fedora-rawhide.repo]` - **Usage**: Only needed for packages not available in standard Fedora repos or variant-specific repos ### allow_fedora_repos - **Type**: Boolean - **Default**: `false` - **Description**: Allow Fedora repositories during lockfile generation for Hummingbird production images. By default, production Hummingbird images exclude Fedora repos to ensure packages come only from the Hummingbird repository. Set to `true` to temporarily allow Fedora repos when required packages are not yet available in the Hummingbird repository. - **Example**: `true` - **Usage**: Typically used temporarily for packages not yet built for Hummingbird (e.g., .NET runtime on x86_64, OpenJDK packages). Should be removed once packages are available in the Hummingbird repository. - **Note**: Toolchain and similar images under `images/` (such as `hummingbird-builder`) use Fedora repositories for signature validation regardless of this setting. ## Hermetic Builds For hermetic builds (enabled by default), all packages from: - `rpm_packages.build-deps` - included in lockfiles and Containerfiles (builder layer only) - `rpm_packages.all` - included in lockfiles and Containerfiles (final image) - `rpm_packages.` - included in lockfiles and Containerfiles for all variants of a distro - `rpm_packages.` - included in lockfiles and Containerfiles for matching variants - `default_rpm_packages.builder` - automatically added for builder variants (see [Global Variables Reference - default_rpm_packages][default-rpm-packages]) are added to `rpms.lock.yaml` and prefetched using cachi2 for offline builds. **Build-time vs Runtime Packages:** - **`rpm_packages.build-deps`**: Installed in the builder layer only (not in final image) - **`rpm_packages.all`**: Installed in `${NEWROOT}` (the final image) - **`rpm_packages.`**: Installed in `${NEWROOT}` for all variants of the specified distro - **`rpm_packages.`**: Installed in `${NEWROOT}` for the specified variant ## Build Configuration ### hermetic - **Type**: Boolean - **Default**: `true` - **Description**: Controls whether the Konflux build pipeline passes `HERMETIC=true` to the buildah task, enabling hermetic (offline, network-isolated) builds with prefetched dependencies. - **Important**: Images with `hermetic: false` are **blocked from release** by the [Conforma hermetic_task policy](https://conforma.dev/docs/policy/packages/release_hermetic_task.html#hermetic_task__hermetic). `hermetic: false` should not be used for any image in the containers repository. ### build_from_source - **Type**: Boolean - **Default**: `false` - **Description**: Controls version extraction in the `package_version()` macro. When `true`, extracts version from `.gitmodules` for images built from git submodules. When `false`, extracts version from RPM package metadata. Does not control whether source code is compiled in the Containerfile. - **Warning**: Do not use for new images. Software built directly from source is not represented by RPM package metadata and cannot be scanned for CVEs by the project's package-based scanner. This setting exists only for exceptional legacy images. ### prefetch_gomod_path - **Type**: String (path relative to image directory) - **Default**: None - **Description**: Path to a Golang project directory (typically a git submodule) for prefetching Go modules via cachi2. Enables hermetic builds for Go projects without a vendor tree. The path is relative to the image directory. - **Usage**: Existing exceptional source-built images only. Do not use for new images because source-built software cannot be scanned for CVEs by the project's package-based scanner. ## Versioning ### main_package - **Type**: String - **Description**: Specifies the main package for version labeling **For RPM-based images:** Set to the package name. Version is extracted from RPM package metadata. ```yaml main_package: nginx ``` **For source-built images:** Existing exceptional source-built images set this to the submodule path. Version is extracted from the `branch` field in `.gitmodules`. Do not use this pattern for new images: source-built software cannot be scanned for CVEs by the project's package-based scanner. ```yaml main_package: images/minio/minio-upstream build_from_source: true ``` ### version_constraints - **Type**: String (glob pattern) - **Default**: None (no constraints) - **Description**: Defines a version constraint pattern to prevent unwanted major or minor version updates. The constraint applies to all distros for this image. Constraints are validated during `make check`, and builds fail if VERSION files violate the constraint. Primarily used for multi-version images (e.g., python-3-11, nodejs-20) to ensure they stay on their intended version family. **Constraint patterns:** Patterns use glob-style matching: - `3.11.*` - Match any 3.11.X version (recommended) - `20.*` - Match any 20.X version - `8.*` - Match any 8.X version - `1.25.*` - Match any 1.25.X version (for 3-part versioning like Go) The pattern matches against the content of VERSION files generated during the build process. **Example:** ```yaml main_package: python3.11 version_constraints: '3.11.*' # Ensure python-3-11 stays on 3.11.X across all distros ``` ```yaml main_package: dotnet-sdk-8.0 version_constraints: '8.*' # Applies to rawhide, hummingbird, and any other distros ``` **When to use:** - Multi-version images where each image targets a specific version family - Preventing automatic major/minor version bumps during Renovate updates - Ensuring version tags remain accurate (e.g., `python:3.11` doesn't become `python:3.12`) **Violation handling:** When a constraint is violated, `make check` fails with a detailed error message showing: - The VERSION file path - The actual version found - The constraint that was violated - The properties.yml file where the constraint is defined Options for resolution: 1. Create a new image for the new version (recommended for major/minor bumps) 2. Exclude the violating version via `yum-repos/*.repo` excludepkgs 3. Update the constraint to allow the new version See [Version Constraints](../operating/version-constraints.md) for detailed documentation. ### version_package - **Type**: Object (string-to-string mapping) - **Default**: None - **Description**: Override the package name used for version extraction on a per-distro, per-variant, or per-distro/variant basis. When a variant or distro installs a different package that provides the same software (e.g., a FIPS-enabled build), this field tells the version/tag macros which package name to look up in the lockfile. The lookup order is: `distro/variant`, then `variant`, then `distro`, then `main_package`. **Per-distro override:** Use when different distros ship the same software under different package names: ```yaml main_package: ruby version_package: hummingbird: ruby4.0 # hummingbird ships ruby4.0, rawhide ships ruby ``` **Per-variant override:** Use when a variant installs a differently-named package: ```yaml main_package: ruby version_package: minimal: ruby-minimal # minimal variant resolves version from ruby-minimal ``` **Per-distro/variant override:** Use when both distro and variant affect the package name: ```yaml main_package: ruby version_package: hummingbird/fips: ruby4.0-fips ``` **Usage in templates:** The resolved package name is available as `package_name_for_version` in Jinja2 templates: ```yaml tags: - value: '{{ package_version(package_name_for_version) }}' label: org.opencontainers.image.version ``` ```jinja2 ENV GOLANG_VERSION={{ package_version(package_name_for_version) }} ``` ## Testing Configuration ### reverse_dependency_tests - **Type**: Boolean - **Default**: `true` - **Description**: Controls whether this image participates in reverse dependency workflows (both building and testing). When `true`, changes to this image trigger rebuilds and tests of all images that depend on it. Dependencies are detected by searching for `TEST_IMAGES[...]` references in test scripts. Set to `false` for images that should not trigger reverse dependency workflows when changed. - **Example**: Set to `false` for `curl` to avoid triggering unnecessary reverse dependency tests **How it works in CI:** When an image is tested, the CI pipeline performs two phases: 1. **Build Phase**: Builds all dependencies needed for testing - Uses `ci/build_images.sh --build-deps` to build: - Forward dependencies (images the main image's tests depend on) - Reverse dependencies (images that depend on the main image, filtered by `reverse_dependency_tests: true`) - Forward dependencies of those reverse dependencies - Automatically deduplicates to ensure each image is built exactly once 2. **Test Phase**: Runs tests on the main image and its reverse dependencies - Uses `ci/run_tests_container.sh` and `ci/run_tests_k8s.sh` with `--include-reverse-deps` to test: - The main image - All reverse dependencies (filtered by `reverse_dependency_tests: true`) **Why set to `false`?** Set `reverse_dependency_tests: false` for images like `curl` that are: - Used pervasively across many images (too many reverse dependencies) - Have a small, stable API that can be fully tested in their own tests - Would cause excessive CI load if every dependent image were tested on each change ## Release Configuration The `repository` and `stream` fields together define the release identity for an image. Both are required and validated during CI. The `repository` specifies the Quay.io repository name, and the `stream` specifies the version series within that repository. ### repository - **Type**: String - **Required**: Yes - **Description**: The Quay.io repository name for this image. For images where the directory name matches the desired repository name (e.g., `images/nginx/` → `repository: nginx`), this is redundant but still required for explicitness. For versioned image directories, this groups multiple versions under one repository (e.g., `nodejs-20` and `nodejs-24` both use `repository: nodejs`). - **Example**: `nodejs` (used by `nodejs-20` and `nodejs-24` directories) ### stream - **Type**: String - **Required**: Yes - **Description**: The version series identity for this image within its repository. Emitted as the `io.hummingbird-project.stream` label on the container image, enabling the image catalog to provide consistent structured data about multi-stream repositories. The stream value must reflect upstream's actual branch structure at the granularity where parallel security-maintained branches exist. Use [endoflife.date](https://endoflife.date) to determine the correct granularity. **Key rules:** - The stream is **never derived from the image directory name**. The directory name is not evidence for stream granularity (e.g., a directory named `foo/` for an upstream with branches `2.8`, `3.0`, `3.2` still requires a major.minor stream like `"3.0"`). - The **purpose of upstream branches is irrelevant** to granularity. Whether upstream calls them "LTS", "stable", "mainline", or "STS" does not matter. If they are parallel branches receiving security updates, the stream granularity must match. - **Match the upstream project's version naming convention.** If the project identifies branches as `8.0`, `9.0` (not `8`, `9`), use that format even if endoflife.date abbreviates it. - **Sequential minor versions are not parallel branches.** If endoflife.date lists minor versions that each immediately supersede the previous with no support overlap, the minor is a release counter. Use the major version as the stream. - Use `"latest"` only for images where no meaningful version series identity exists: rolling-release tools with date-based versioning (e.g., minio) or base images that track the distro rather than independent software (e.g., core-runtime). - For **unversioned images** (no version in directory name, no version constraints), the stream value reflects the version currently shipped and must be updated when the upstream version changes. See the [stream assignment guidelines](adding-images.md#choosing-a-stream-value) for the full decision process and rationale. **Document the reasoning** for the stream choice as a YAML comment above the `stream` field. This ensures future maintainers understand why a particular granularity was chosen. **Examples:** ```yaml # endoflife.date: parallel at major (Docker Hub + Debian + Wolfi) stream: "24" # endoflife.date: parallel at major.minor (Docker Hub + Debian + Wolfi) stream: "3.11" # endoflife.date: branches are 10.1, 11.0, not 10, 11 stream: "10.1" # rolling release with date-based versions, no version branches stream: "latest" ``` ### application_category - **Type**: String - **Required**: Yes - **Description**: The Pyxis application category for the image's delivery repository. Emitted as the `io.hummingbird-project.application-category` label on the container image and used in the generated Pyxis repo config. All images sharing the same `repository` must use the same value. Validated during `make check` against the list in [`ci/application-categories.txt`](https://gitlab.com/redhat/hummingbird/containers/-/blob/main/ci/application-categories.txt). **Common values for Hummingbird images:** - `"Programming Languages & Runtimes"` -- language runtimes and SDKs (Go, Python, Node.js, .NET, Ruby, etc.) - `"Web Services"` -- web servers, reverse proxies, and load balancers (Nginx, httpd, Caddy, HAProxy, Tomcat) - `"Database & Data Management"` -- databases and caches (PostgreSQL, MariaDB, Valkey, Memcached) - `"Developer Tools"` -- CLI utilities and build tools (git, curl, jq) - `"Storage"` -- object storage (MinIO, MinIO Client) - `"Operating System"` -- base runtime images (core-runtime) **Example:** ```yaml application_category: "Web Services" ``` ### tags - **Type**: Array of objects - **Description**: Define image tags for Quay.io releases. Each variant gets these tags, with non-default variants receiving a `-` suffix. **Tag Object Fields:** - **`value`** (required): The tag value. Can be a literal string (e.g., `latest`) or a Jinja2 template using helper functions. - **`label`** (optional): If specified, the tag value is written as a `LABEL` in the Containerfile, and the actual rendered value from the label is used as the image tag. This ensures version tags match the actual package versions in the image. **Tag Helper Functions:** - `package_version("")` - Full version (e.g., `1.2.3-4.fc42`) - `package_major_version("")` - Major version only (e.g., `1`) - `package_major_minor_version("")` - Major.minor version (e.g., `1.2`) **Example:** ```yaml tags: - value: latest # Literal tag, no label - value: '{{ package_major_version("nginx") }}' # Template tag with label label: io.hummingbird-project.major-version - value: '{{ package_major_minor_version("nginx") }}' # Tag becomes actual version from label label: io.hummingbird-project.major-minor-version - value: '{{ package_version("nginx") }}' label: org.opencontainers.image.version ``` **Note**: For images with multiple versions, only set `latest` on the latest version. ### tag_suffix_aliases - **Type**: Mapping of variant names to arrays of suffixes - **Description**: Add aliases for every configured tag without creating another image variant. The suffix is appended directly to each base tag. Existing canonical tags remain unchanged. - **Example**: ```yaml tag_suffix_aliases: default: - jdk25 builder: - jdk25-builder ``` ### support_level - **Type**: String - **Default**: Omitted (image is Red Hat supported) - **Values**: `community`, `experimental` - **Description**: Controls the support level, Quay organization, and Konflux application assignment for hummingbird-distro images. The mapping from support level to Quay org is defined in `images/variables.yml` under `support_levels`; all other names (Konflux app, service account, RPA suffix) are derived from the Quay org name. When omitted, the image is Red Hat supported and assigned to `containers-hummingbird`. Rawhide images are always assigned to `containers-rawhide` regardless of this property. - **Example**: `support_level: community` - **Mapping**: | Value | Quay org | Konflux app | CPE | Pyxis/RPA | |----------------|-------------------------|-------------------------------------|-----|-----------| | (omitted) | `hummingbird` | `containers-hummingbird` | Yes | Included | | `community` | `hummingbird-community` | `containers-community-hummingbird` | No | Excluded | | `experimental` | `hummingbird-ci` | `containers-ci-hummingbird` | No | Excluded | - **Usage**: Use `community` for images maintained by the community (e.g., minio, bootc-os). Use `experimental` for internal build infrastructure (e.g., hummingbird-builder). These images are built and tested normally but published to their designated Quay organization, not the official Red Hat registry. **How Labels Work:** 1. Template is rendered and written to Containerfile: `LABEL org.opencontainers.image.version=1.27.3-1.fc42` 2. Build system extracts the actual value from the image label 3. Image is tagged with the extracted value: `quay.io/hummingbird/nginx:1.27.3-1.fc42` This ensures tags always match the actual package versions built into the image. ## Compliance Scanning OpenSCAP compliance scanning verifies CIS and STIG compliance during the build. Scanning runs against the container rootfs using `oscap-chroot` and fails the build if any enabled rules are violated. Scanning is enabled by default for all images. It only runs for the Hummingbird distro and is skipped for the `builder` base image. ### oscap.enabled - **Type**: Boolean - **Default**: `true` (from `images/variables.yml`) - **Description**: Controls whether OpenSCAP compliance scanning runs for this image. When `true`, a `verify-compliance` step is added to the generated Containerfile for all Hummingbird variants. Set to `false` to opt out. ```yaml oscap: enabled: false ``` ### oscap.profiles - **Type**: Object mapping profile names to enablement rules - **Default**: `{cis: true, stig: {variants: ["*fips*"]}}` (from `images/variables.yml`) - **Description**: Controls which compliance profiles run for each variant. Each profile can be set to `true` (all variants), `false` (disabled), or an object with a `variants` list of glob patterns. **Default behavior** (CIS for all variants, STIG only for FIPS variants): ```yaml oscap: profiles: cis: true stig: variants: - "*fips*" ``` **Enable STIG for all variants:** ```yaml oscap: profiles: stig: true ``` **Disable CIS for a specific image:** ```yaml oscap: profiles: cis: false ``` The `variants` field supports glob patterns using `fnmatch` syntax (e.g., `*fips*` matches `fips`, `fips-builder`, `openssl-fips`). ### oscap.exclude_rules - **Type**: Array of rule exclusion objects - **Default**: Global rule exclusions from `images/variables.yml` - **Description**: Rules to exclude from compliance scanning. Image-level exclusions are concatenated with global exclusions (not replaced). Each exclusion object has the following fields: | Field | Type | Required | Description | |------------|------------------|----------|-----------------------------------------------------------------------------------| | `id` | String | Yes | XCCDF rule ID (e.g., `xccdf_org.ssgproject.content_rule_configure_crypto_policy`) | | `reason` | String | Yes | Explanation for excluding this rule | | `variants` | Array of strings | No | Glob patterns for variants this exclusion applies to. Omit for all variants. | | `profiles` | Array of strings | No | Profile names (`cis`, `stig`) this exclusion applies to. Omit for all profiles. | **Exclude a rule for all variants and profiles:** ```yaml oscap: exclude_rules: - id: xccdf_org.ssgproject.content_rule_configure_crypto_policy reason: "Caddy uses Go's built-in TLS stack, not system crypto-policies" ``` **Exclude a rule only for specific variants:** ```yaml oscap: exclude_rules: - id: xccdf_org.ssgproject.content_rule_configure_crypto_policy reason: "crypto-policies state files not generated in installroot" variants: - "*fips*" ``` **Exclude a rule only from a specific profile:** ```yaml oscap: exclude_rules: - id: xccdf_org.ssgproject.content_rule_some_rule reason: "Not applicable to CIS for this image" profiles: - stig ``` **Combine variant and profile filters:** ```yaml oscap: exclude_rules: - id: xccdf_org.ssgproject.content_rule_configure_crypto_policy reason: "crypto-policies state files not generated in installroot for FIPS variants" variants: ["*fips*"] profiles: [cis] ``` ### How compliance scanning works 1. **Profile resolution**: `generate_jinja2.py` determines which profiles (CIS, STIG) are active for the current variant based on `oscap.profiles`. 2. **Rule filtering**: Exclusion rules are filtered by variant (glob matching) and profile, producing per-profile exclusion lists. 3. **Tailoring file**: If any active profile has exclusions, an XCCDF tailoring file (`oscap-tailoring.xml`) is generated in the variant directory with custom profiles extending the upstream CIS/STIG profiles. 4. **Containerfile**: The `verify-compliance` command is added with explicit `--cis` and/or `--stig` flags based on active profiles, plus a `--tailoring-file` argument if exclusions exist. 5. **Build-time scan**: `oscap-chroot` evaluates the rootfs against each active profile. The build fails if any non-excluded rule fails. ## Next Steps - [Adding Images][adding-images] - Step-by-step guide for adding new images - [Image Pipeline][image-pipeline] - How the complete pipeline works [default-variant-repos]: https://hummingbird-project.io/l/global-variables-reference#default_variant_repos [default-rpm-packages]: https://hummingbird-project.io/l/global-variables-reference#default_rpm_packages [adding-images]: adding-images.md [image-pipeline]: https://hummingbird-project.io/l/image-pipeline -------------------------------------------------------------------------------- # Test Configuration Reference url: https://hummingbird-project.io/docs/contributing/test-configuration-reference/ description: Complete reference for test definition format and configuration options Complete reference for configuring image tests via `tests-container.yml` and `tests-k8s.yml`. ## Test Definition Format Each image can have tests defined in: - `images//tests-container.yml` - Container tests (run with Podman/Docker) - `images//tests-k8s.yml` - K8s tests (run in Kubernetes cluster) Both files use the same YAML format: ```yaml --- script: command: ./test.sh inline: command: | image=${TEST_IMAGE:?} test_engine_run --rm "${image}" --version ``` ## Environment Variables The test runners provide environment variables to test commands. Some are common to both test types, others are specific to container or K8s tests. ### Common Variables | Variable | Description | |--------------------|--------------------------------------------------------| | `TEST_IMAGE` | The container image being tested | | `TEST_IMAGES` | Associative array with all `group/variant` image URLs | | `TEST_IMAGES_PATH` | Path to file containing serialized `TEST_IMAGES` array | | `TEST_VERBOSE` | Show test command output (`true` or `false`) | | `TEST_GROUP` | The image group being tested (`nginx`, `python`, etc.) | | `TEST_DISTRO` | The distro being tested (`hummingbird`, `rawhide`) | | `TEST_VARIANT` | The variant being tested (`default`, `builder`, etc.) | ### Container Test Variables | Variable | Description | |--------------------|--------------------------------------------------------| | `TEST_ENGINE` | Container engine to use (`podman` or `docker`) | | `TEST_USER_ID` | Current user ID (for permission handling) | | `TEST_FIPS` | Whether host is in FIPS mode (`true` or `false`) | ### K8s Test Variables | Variable | Description | |--------------------|--------------------------------------------------------| | `TEST_RUN_ID` | Unique ID for this test run (for resource naming) | | `TEST_RUN_LABEL` | Label selector for cleanup (`hum-k8s-test=`) | ### Helper Functions #### Common to All Tests Both test types provide the `test_fail` function: | Function | Description | |-------------|-------------------------------------------| | `test_fail` | Fail the test with a custom error message | In K8s tests, `kubectl` is pre-configured with the context/kubeconfig from CLI args. #### Container Tests Only Container tests provide helper functions that wrap the container engine with appropriate flags. Always use these functions instead of calling `${TEST_ENGINE}` directly for `create` and `run` commands. This ensures tests respect globally set options such as the hermetic mode setting (`--pull=never`). | Function | Description | |---------------------|---------------------------------------------------------| | `test_engine_create`| Run `${TEST_ENGINE} create` with appropriate pull flags | | `test_engine_run` | Run `${TEST_ENGINE} run` with appropriate pull flags | **Note**: For other container engine commands (`inspect`, `network`, `volume`, etc.), use `${TEST_ENGINE:?}` directly. ## Variant-Aware Testing The testing system automatically includes variant-specific default tests: - **Global tests** from `ci/{variant}-tests/tests-container.yml` - **Image-specific tests** from `images//tests-container.yml` ## Variant Selection Tests can specify which variants they apply to using the `filters.variants` field. The field supports both exact matches and glob patterns. ```yaml # Test that only runs for builder variants build-tools-test: filters: variants: [builder] command: | test_engine_run --rm "${TEST_IMAGE}" make --version # Test that runs for multiple specific variants multi-variant-test: filters: variants: [default, builder] command: | test_engine_run --rm "${TEST_IMAGE}" echo "Hello" # Test that runs for all variants (no filters.variants field) universal-test: command: | test_engine_run --rm "${TEST_IMAGE}" echo "Always runs" ``` ### Glob Patterns in Variant Filters Use glob patterns to match multiple variants dynamically: ```yaml # Test that runs for any FIPS variant (fips, fips-builder, etc.) fips-test: filters: variants: ["*fips*"] command: | test_engine_run --rm "${TEST_IMAGE}" check-fips-mode # Test that runs for all builder variants (builder, fpm-builder, etc.) builder-test: filters: variants: ["*-builder", "builder"] command: | test_engine_run --rm "${TEST_IMAGE}" dnf --version # Mix exact matches and patterns mixed-filter: filters: variants: [default, "*fips*"] command: | test_engine_run --rm "${TEST_IMAGE}" echo "Runs on default and all fips variants" ``` **Supported glob syntax:** - `*` - matches any sequence of characters - `?` - matches any single character ## Distro Selection Tests can specify which distros they apply to using the `filters.distros` field. Like variant filters, this field supports both exact matches and glob patterns. ```yaml # Test that only runs for hummingbird distro hummingbird-only-test: filters: distros: [hummingbird] command: | test_engine_run --rm "${TEST_IMAGE}" some-hummingbird-specific-check # Test that runs for multiple distros multi-distro-test: filters: distros: [hummingbird, rawhide] command: | test_engine_run --rm "${TEST_IMAGE}" echo "Runs on both" # Test that runs for all distros (no filters.distros field) universal-test: command: | test_engine_run --rm "${TEST_IMAGE}" echo "Always runs" ``` This is useful for tests that check distro-specific features. For example, CPE labels are only present on hummingbird images (Fedora/rawhide has no official CPE). ### Glob Patterns in Distro Filters Use glob patterns to match multiple distros dynamically: ```yaml # Test that runs for any distro ending in "bird" bird-distro-test: filters: distros: ["*bird"] command: | test_engine_run --rm "${TEST_IMAGE}" check-something # Test that runs for any distro starting with "raw" raw-distro-test: filters: distros: ["raw*"] command: | test_engine_run --rm "${TEST_IMAGE}" check-rawhide-feature ``` **Supported glob syntax:** - `*` - matches any sequence of characters - `?` - matches any single character ## FIPS Mode Selection Tests can specify whether they should run based on the host's FIPS mode using `filters.fips`: ```yaml # Test that only runs on FIPS-enabled hosts fips-required-test: filters: fips: true command: | test_engine_run --rm "${TEST_IMAGE}" openssl list -providers # Test that only runs on non-FIPS hosts non-fips-test: filters: fips: false command: | test_engine_run --rm "${TEST_IMAGE}" some-non-fips-check # Test that runs regardless of FIPS mode (no filters.fips field) universal-test: command: | test_engine_run --rm "${TEST_IMAGE}" echo "Always runs" ``` The `TEST_FIPS` environment variable is set to `true` when running on a FIPS-enabled host (detected from `/proc/sys/crypto/fips_enabled`), and `false` otherwise. ## Group Test Mode When running tests for all variants (without specifying a specific variant), the system supports a special "group" test mode for tests that need to work across multiple variants simultaneously: ```yaml # Test that runs only in group mode cross-variant-compatibility-test: filters: variants: [group] command: | # Access specific group/variant combinations using TEST_GROUP builder_image=${TEST_IMAGES[${TEST_GROUP}/builder]:?} default_image=${TEST_IMAGES[${TEST_GROUP}/default]:?} echo "Builder variant: ${builder_image}" echo "Default variant: ${default_image}" # Test that both variants have compatible APIs test_engine_run --rm "${builder_image}" nginx -version test_engine_run --rm "${default_image}" nginx -version ``` ## Using TEST_IMAGES in External Scripts External shell scripts that need to reference other image variants must source the TEST_IMAGES array file at the beginning: ```bash #!/bin/bash set -euo pipefail # Load TEST_IMAGES array # shellcheck disable=SC1090 source "${TEST_IMAGES_PATH:?}" # Now TEST_IMAGES is available - always use :? for proper error checking test_engine_run --rm "${TEST_IMAGES[curl/default]:?}" ... test_engine_run --rm "${TEST_IMAGES[httpd/default]:?}" ... ``` **Note:** Inline test commands in YAML files automatically have access to TEST_IMAGES and do not need this sourcing pattern. ## Using TEST_GROUP for Dynamic References The `TEST_GROUP` variable contains the current image group name (e.g., `nginx`, `python`, `aspnet-runtime-8-0`). Use it with `TEST_IMAGES` to dynamically reference the current image's variants without hardcoding the group name: ```yaml multi-stage-build: filters: variants: [builder] command: | # Build in builder variant, run in default variant "${TEST_ENGINE}" build -t localhost/myapp -f - . < **AI Agent Note:** When asked to exclude packages, always ask the user for a > ticket link or explanation first. This is required for the comment documenting > the exclusion. ## Overview When a faulty RPM build is published to Hummingbird repositories, it can be excluded from container images by adding an `excludepkgs` directive to the repository configuration. ## Workflow ### 1. Identify the source package NVR Identify the Name-Version-Release of the faulty source package. Example: `ncurses-6.5-8.20250614.hum1` ### 2. Generate excludepkgs value Query the Hummingbird repositories and generate the `excludepkgs` value: ```bash podman run --rm quay.io/hummingbird-ci/hummingbird-builder:latest sh -c ' dnf5 repoquery --queryformat="%{sourcerpm} %{name}-%{evr}.* " 2>/dev/null | sed -n "s/^ncurses-6.5-8.20250614.hum1.src.rpm //p" | tr "\n" " " ' ``` Replace `ncurses-6.5-8.20250614.hum1` with the source NVR. This outputs the ready-to-use `excludepkgs` value: ```text ncurses-6.5-8.20250614.hum1.* ncurses-base-6.5-8.20250614.hum1.* ... ``` ### 3. Add excludepkgs to repo file Edit `yum-repos/hummingbird.repo` and add the binary package NEVRs to `excludepkgs` in the `[hummingbird]` section (the binary repo, not `[hummingbird-source]`). Document the exclusion with a one-line comment containing the ticket link: ```ini [hummingbird] ... # HUM-1234: ncurses-6.5-8.20250614.hum1 causes segfault in terminfo parsing excludepkgs=ncurses-base-6.5-8.20250614.hum1.* ncurses-libs-6.5-8.20250614.hum1.* ``` Multiple packages are space-separated. Use `.*` suffix to match all architectures. ### 4. Merge the change Merge the repo file change to main. Renovate will automatically apply the exclusions when it next rebases existing lockfile update branches or creates new ones. ## Verification Check that the excluded package no longer appears in any `rpms/rpms.lock.yaml` files in subsequent Renovate-generated merge requests. ## Removing Exclusions When a newer version is available, exclusions can be removed. ### 1. Verify a newer version exists Check the latest available version (should differ from excluded `6.5-8.20250614.hum1`): ```bash podman run --rm quay.io/hummingbird-ci/hummingbird-builder:latest \ dnf5 repoquery --latest-limit=1 --queryformat="%{evr}" "ncurses-libs" 2>/dev/null ``` ### 2. Remove the exclusion Remove the `excludepkgs` line and its comment from `yum-repos/hummingbird.repo`. ### 3. Merge and verify Merge the change. Renovate will pick up the newer version in subsequent lockfile updates. -------------------------------------------------------------------------------- # Managing Konflux Comments url: https://hummingbird-project.io/docs/operating/managing-konflux-comments/ description: Clean up and manage excessive Konflux comments on merge requests using the comment deletion tool. When Konflux comments become excessive on merge requests, you can clean them up using the provided comment deletion tool. ## Prerequisites - **COM_GITLAB_TOKEN**: GitLab API token with `api` scope and Maintainer access ## Usage If Konflux comments become excessive on merge requests, you can clean them up using: ```bash # Show help for the comment deletion tool make delete-konflux-comments ARGS="--help" # Example: Delete comments from a specific merge request make delete-konflux-comments ARGS="https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/123" # Example: Delete only failed build/test comments (dry run) make delete-konflux-comments ARGS="--dry-run --comment-category failed -- https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/123" # Example: Delete all Konflux comments from a merge request make delete-konflux-comments ARGS="--all https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/123" ``` ## Comment Categories The tool can target specific types of Konflux comments: - **failed**: Only comments related to failed builds/tests - **all**: All Konflux-related comments (use with `--all` flag) ## Safety Features - **Dry Run Mode**: Use `--dry-run` to preview what would be deleted without actually removing comments - **Selective Deletion**: Target specific comment categories to avoid removing important information - **Confirmation**: The tool will show what it plans to delete before taking action ## Examples ```bash # Set your GitLab token export COM_GITLAB_TOKEN=your_token_here # Preview what failed comments would be deleted make delete-konflux-comments ARGS="--dry-run --comment-category failed -- https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/789" # Delete only failed build comments make delete-konflux-comments ARGS="--comment-category failed -- https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/789" # Delete all Konflux comments (use with caution) make delete-konflux-comments ARGS="--all https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/789" ``` ## When to Use Consider cleaning up Konflux comments when: - Merge requests have accumulated many outdated failure comments - Comments are making it difficult to find relevant discussion - You want to start fresh after resolving systematic issues - The comment thread has become cluttered with automated messages **Note**: Use this tool judiciously, as it permanently removes comments that might contain useful debugging information. ## Related Operations - [Retrying Konflux Checks](retrying-konflux-checks.md) - Retrigger failed CI pipeline runs -------------------------------------------------------------------------------- # Running Conforma Checks Locally url: https://hummingbird-project.io/docs/operating/running-conforma-checks-locally/ ## Overview [Conforma][conforma] (Enterprise Contract) policy checks run automatically in the Konflux pipeline before release (see [Image Pipeline — Stage 5][stage5]). These checks can also be run locally against any Konflux-built image to validate compliance, test policy changes, or investigate failures. [conforma]: https://conforma.dev/ [stage5]: https://hummingbird-project.io/l/image-pipeline#stage-5-enterprise-contract-validation ## Prerequisites Install the [Conforma CLI][ec-releases]: ```bash curl -sLO https://github.com/conforma/cli/releases/download/snapshot/ec_linux_amd64 chmod 755 ec_linux_amd64 mkdir -p ~/.local/bin mv ec_linux_amd64 ~/.local/bin/ec ``` [ec-releases]: https://github.com/conforma/cli/releases ## Running All Checks The full set of Conforma checks (signatures, attestations, SLSA provenance, labels, etc.) can be run against any Konflux-built image using `ec validate image` with the Konflux signing key. ### Public Key The Konflux signing public key is committed at [`ci/key.pub`][key-pub]. This is the same key stored in the Konflux member cluster at `k8s://openshift-pipelines/public-key`. [key-pub]: https://gitlab.com/redhat/hummingbird/containers/-/blob/main/ci/key.pub ### Obtaining the Image Reference Combine the `IMAGE_URL` and `IMAGE_DIGEST` results from a successful build PipelineRun: ```text @ ``` Merge request build images follow the pattern: ```text quay.io/redhat-user-workloads/hummingbird-tenant/:on-mr-- ``` ### Running the Validation Create a policy file that matches the [pipeline policy][policy-macro] but with any modifications needed for testing. For example, to run the pipeline policy with label checks enabled (removing the `labels.required_labels` and `labels.optional_labels` exclusions): ```bash cat > /tmp/policy.yaml << 'YAML' sources: - name: Release Policies config: exclude: - test.required_tests_passed:sast-snyk-check - test.no_skipped_tests:sast-snyk-check - test.required_tests_passed:sast-snyk-check-oci-ta - test.no_skipped_tests:sast-snyk-check-oci-ta - test.no_failed_tests:ecosystem-cert-preflight-checks - test.no_erred_tests:ecosystem-cert-preflight-checks - test.no_failed_informative_tests - test.no_test_warnings:deprecated-image-check - trusted_task.current - rpm_repos.ids_known - buildah_build_task.privileged_nested_param - schedule.weekday_restriction include: - '@redhat' data: - github.com/release-engineering/rhtap-ec-policy//data - oci::quay.io/konflux-ci/tekton-catalog/data-acceptable-bundles:latest - oci::quay.io/konflux-ci/konflux-vanguard/data-acceptable-bundles:latest - oci::quay.io/konflux-ci/integration-service-catalog/data-acceptable-bundles:latest policy: - oci::quay.io/enterprise-contract/ec-release-policy:konflux ruleData: allowed_registry_prefixes: - "quay.io/hummingbird-rawhide/" - "quay.io/hummingbird-ci/" - "quay.io/hummingbird-community/" - "quay.io/hummingbird/" - "quay.io/redhat-user-workloads/" YAML ``` Then validate: ```bash ec validate image \ --image "quay.io/redhat-user-workloads/hummingbird-tenant/@sha256:" \ --policy /tmp/policy.yaml \ --public-key ci/key.pub \ --ignore-rekor \ --strict=false \ --show-successes \ --output text ``` [policy-macro]: https://gitlab.com/redhat/hummingbird/containers/-/blob/main/konflux-templates/macros/policy.yml.j2 ### Options | Flag | Purpose | |--------------------|-----------------------------------------------------------------| | `--ignore-rekor` | Required — Konflux does not use a Rekor transparency log | | `--strict=false` | Return exit code 0 even on failures (useful for investigation) | | `--show-successes` | Include passing checks in output (off by default) | | `--output json` | Machine-readable output (also: `text`, `yaml`, `summary`) | ## Running Label Checks Only The label checks (`labels.required_labels`, `labels.optional_labels`) are the only Conforma rules that can be evaluated without signatures or attestations, because they only read OCI image metadata. This lighter-weight approach does not require the signing key and works against any registry image, including published (non-Konflux-built) images. ### Setup Fetch the policy rego files and upstream rule data (one-time): ```bash ec fetch policy \ --source oci::quay.io/enterprise-contract/ec-release-policy:konflux \ --dest /tmp/ec-policy git clone --depth=1 \ https://github.com/release-engineering/rhtap-ec-policy.git \ /tmp/rhtap-ec-policy ``` ### Running Checks ```bash ec opa eval \ --data /tmp/ec-policy/policy/*/policy \ --data /tmp/rhtap-ec-policy/data/rule_data.yml \ --input <(echo '{"image":{"ref":"quay.io/hummingbird-rawhide/caddy:latest"}}') \ '{"deny": data.labels.deny, "warn": data.labels.warn}' \ --format pretty ``` This works against any image accessible from a container registry — both Konflux-built and published images. It uses the `ec` CLI's embedded OPA engine with `ec.oci.*` built-in functions to read image config directly from the registry. > **Note:** The `--data` flag (not `--bundle`) must be used for the rego files. Loading the > policy as a bundle creates an OPA root conflict that prevents the rule data from being > resolved. The `ec.oci.*` built-in functions are only available in `ec opa`, not in > standalone `opa` or `conftest`. ## Interpreting Results For `ec validate image`: - **Successes** indicate passing policy rules (visible with `--show-successes`). - **Violations** are failures that block release. Each includes a `code` (rule name) and `msg`. - **Warnings** are non-blocking advisories. For `ec opa eval` (label checks only): - **`deny`** entries are failures. Each includes `code`, `msg`, and optionally `effective_on` (a date when the rule becomes enforced — future dates mean the rule is not yet active). - **`warn`** entries are non-blocking warnings for optional labels. - Empty arrays mean the image passes all label checks. ## Updating Cached Data The fetched policy and rule data are point-in-time snapshots. To pick up upstream changes: ```bash # Re-fetch the policy rego files (for ec opa eval) rm -rf /tmp/ec-policy ec fetch policy \ --source oci::quay.io/enterprise-contract/ec-release-policy:konflux \ --dest /tmp/ec-policy # Update the rule data (for ec opa eval) git -C /tmp/rhtap-ec-policy pull ``` The `ec validate image` command fetches policy and data bundles on each run based on the `policy.yaml` references, so no manual update is needed. -------------------------------------------------------------------------------- # Upgrading Docsy url: https://hummingbird-project.io/docs/operating/docsy-upgrades/ description: How to upgrade the Docsy theme and forward-port our layout overrides. ## Overview Both Hummingbird documentation sites use [Docsy][docsy] as their Hugo theme, installed as a Hugo module. Docsy releases roughly quarterly. This runbook covers how to upgrade to a new Docsy version. The **public docs** ([documentation][docs-repo]) own all layout overrides. The **internal docs** ([infrastructure][infra-repo]`/internal-docs/`) only have build scaffolding and pick up layouts from the public docs via Hugo module import. ## Layout override inventory We override four Docsy templates: three to inject the companion page banner and sidebar lock icons, and one to add `llms-full.txt` cross-linking. Each override is a copy of the upstream template with a small patch applied. ### `layouts/docs/_td-content.html` Overrides Docsy's shared `layouts/_td-content.html` for docs pages only. **Patch:** One line added after the description lead, before `
`: ```go-html-template {{ partial "companion.html" . }} ``` ### `layouts/docs/list.html` Overrides Docsy's `layouts/docs/list.html` for docs section pages. **Patch:** Same one line added after the description lead, before `
`: ```go-html-template {{ partial "companion.html" . }} ``` ### `layouts/partials/sidebar-tree.html` Overrides Docsy's `layouts/_partials/sidebar-tree.html` for the navigation sidebar. **Patch:** One snippet inserted in two places (foldable and non-foldable branches), after the `{{ $s.LinkTitle }}` `` and before ``: ```go-html-template {{- if isset $s.Params "companion" }}{{ if not $s.Params.companion }} {{ end }}{{ end -}} ``` ### `layouts/all.md` Overrides Docsy's `layouts/all.md` (the catch-all markdown output template). **Patch:** One line changed in the LLMS index section to add a link to `llms-full.txt` alongside the existing `llms.txt` link. ### Other overrides (no forward-porting needed) These files are fully custom and don't track an upstream template: | File | Purpose | | ---- | ------- | | `layouts/shortcodes/include.html` | Custom file-include shortcode | | `layouts/_default/_markup/render-link.html` | Link rewriting for hummingbird URLs and `/l/` aliases | | `layouts/_default/_markup/render-heading.html` | Delegates to Docsy's `td/render-heading.html` | | `layouts/partials/companion.html` | Environment-aware companion page banner | | `layouts/partials/breadcrumb.html` | Custom breadcrumb with stable-URL display | | `layouts/partials/footer.html` | Fully custom 3-column footer | | `layouts/home.html` | Custom home page with cover image and feature cards | | `layouts/index.llms.txt` | Custom recursive llms.txt with full doc tree | | `layouts/index.llms-full.txt` | Full content dump for LLM consumption | | `layouts/partials/llms-doc-tree.txt` | Recursive tree-walking partial for llms.txt | | `layouts/partials/llms-full-doc-tree.txt` | Recursive content partial for llms-full.txt | ## Upgrade procedure ### 1. Read the release notes Check the [Docsy changelog][docsy-changelog] and the release upgrade guide for the target version. Note any breaking changes that affect templates we override. ### 2. Update the Hugo module From the documentation repo worktree: ```bash # Update to new version (replace vX.Y.Z with the target version) podman run --pull=newer -it --rm -u 0 \ -v $(pwd):/src:z -w /src \ -e RUNNING_IN_CONTAINER=1 -e HUGO_CACHEDIR=/src/.hugo_cache \ quay.io/hummingbird-ci/gitlab-ci:latest sh -c \ "hugo mod get github.com/google/docsy/theme@vX.Y.Z && hugo mod tidy" ``` ### 3. Update npm dependencies Docsy v0.16.0+ sources Bootstrap and Font Awesome from npm: ```bash podman run --pull=never -it --rm -u 0 \ -v $(pwd):/src:z -w /src \ -e RUNNING_IN_CONTAINER=1 -e HUGO_CACHEDIR=/src/.hugo_cache \ quay.io/hummingbird-ci/gitlab-ci:latest sh -c \ "hugo mod npm pack && npm install --no-package-lock" ``` ### 4. Forward-port layout overrides For each of the four tracked overrides: 1. Find the new upstream template in the Hugo module cache: ```bash find .hugo_cache -path "*docsy/theme@vX.Y.Z*" \ -name "_td-content.html" -o \ -name "sidebar-tree.html" -o \ -name "all.md" -o \ \( -name "list.html" -path "*docs*" \) ``` 2. Compare the new upstream against our current override to see what changed upstream. 3. Start from the new upstream template and re-apply the patch documented in the inventory above. 4. For `docs/list.html`, keep all upstream hooks (especially `{{ .Render "_td-content-after-header" -}}`). ### 5. Verify dark mode and code highlighting The site uses Docsy's light/dark mode toggle and dark-aware code syntax highlighting. After upgrading, confirm these config and SCSS settings are still in place: - `config.yaml`: `params.ui.showLightDarkModeMenu: true` - `config.yaml`: `markup.highlight.noClasses: false` (required for CSS-based Chroma themes to work) - `assets/scss/_styles_project.scss`: `@import 'td/code-dark';` (loads light/dark Chroma themes: `friendly` for light, `native` for dark) If a Docsy upgrade changes the dark mode mechanism, check the [Look and Feel][docsy-lookandfeel] docs for updated instructions. ### 6. Build and validate ```bash make build # or: make build-host (if Hugo/Go are available locally) ``` The build should complete with no errors and no deprecation warnings. ## Validation checklist After upgrading, verify: - [ ] Build passes without errors or deprecation warnings - [ ] Companion banners render on docs pages with `companion: true` front matter - [ ] Lock icons appear in the internal site sidebar for internal-only pages - [ ] Home page cover image and search render correctly - [ ] Light/dark mode toggle works and all page elements adapt (navbar, sidebar, footer, code blocks) - [ ] Code blocks have dark background with readable syntax colors in dark mode - [ ] Agent-support outputs generate, if enabled (`.md` URL variants, `/llms.txt`, `/llms-full.txt`) - [ ] Stable URL aliases (`/l/...`) still resolve For the internal site, also build `infrastructure/internal-docs/` after updating its `config/internal/config.yaml` and `go.mod` to the same Docsy version. ## Updating the internal site The internal site only needs config and module changes (no layout work): 1. Update `internal-docs/config/internal/config.yaml`: change the `theme:` list to match the public site. 2. Update the Hugo module and rebuild: ```bash cd internal-docs make container # Inside container: hugo mod get github.com/google/docsy/theme@vX.Y.Z && hugo mod tidy exit make setup # npm dependencies make build ``` ## Rollback To revert to a previous Docsy version: ```bash podman run --pull=never -it --rm -u 0 \ -v $(pwd):/src:z -w /src \ -e RUNNING_IN_CONTAINER=1 -e HUGO_CACHEDIR=/src/.hugo_cache \ quay.io/hummingbird-ci/gitlab-ci:latest sh -c \ "hugo mod get github.com/google/docsy/theme@vPREVIOUS && hugo mod tidy" ``` Then restore the layout overrides from the previous commit (`git checkout HEAD~1 -- layouts/`). [docsy]: https://www.docsy.dev/ [docsy-changelog]: https://www.docsy.dev/project/about/changelog/ [docsy-lookandfeel]: https://www.docsy.dev/docs/content/lookandfeel/ [docs-repo]: https://gitlab.com/redhat/hummingbird/documentation [infra-repo]: https://gitlab.com/redhat/hummingbird/infrastructure -------------------------------------------------------------------------------- # Version Constraints for Multi-Version Images url: https://hummingbird-project.io/docs/operating/version-constraints/ ## Overview Version constraints prevent unwanted major or minor version updates for multi-version container images (e.g., python-3-11, nodejs-20). Constraints are defined in `properties.yml` and validated during `make check` by comparing VERSION files against defined patterns. ## Adding Version Constraints Edit the image's `properties.yml` to add a version constraint: ```yaml --- main_package: python3.11 version_constraints: '3.11.*' # Allow 3.11.X, block 3.12+ and 4.X ``` The constraint applies to all distros for the image: ```yaml --- main_package: dotnet-sdk-8.0 version_constraints: '8.*' # Applies to rawhide, hummingbird, and any other distros ``` ### Constraint Patterns Constraints use glob-style patterns: - `3.11.*` - Match any 3.11.X version (recommended for version families) - `20.*` - Match any 20.X version - `8.*` - Match any 8.X version - `1.25.*` - Match any 1.25.X version (for Go-style 3-part versions) The pattern matches against the version from the VERSION file. For example, if the VERSION file contains `3.11.14`, the constraint `3.11.*` will match. ## How Validation Works 1. **Lockfile Generation**: When `make all` runs, lockfiles are generated with resolved package versions 2. **VERSION File Creation**: `generate_jinja2.py` creates VERSION files from resolved package versions 3. **Constraint Checking**: When `make check` runs, `check_version_constraints.py` validates VERSION files against constraints 4. **Failure**: If a violation is detected, the build fails with a detailed error message 5. **Renovate Integration**: Renovate MRs that violate constraints fail CI and are blocked from automerge ## Handling Constraint Violations When a build fails due to a version constraint violation, you have several options: ### Option 1: Accept the new version (create new image) If the new major/minor version is acceptable: 1. Create a new image directory (e.g., `images/python-3-14/`) 2. Copy properties and configuration from the old version 3. Update the constraint in the new directory: ```yaml version_constraints: '3.14.*' ``` 4. Update `repository` field to group versions together 5. Consider adding `latest` tag to the new version if appropriate ### Option 2: Block the version update (exclude package) If the new version should be blocked entirely: 1. Add the package to `yum-repos/*.repo` excludepkgs 2. Follow the procedure in [Excluding Packages from Images](excluding-packages-from-images.md) 3. Renovate will skip the excluded version in future updates 4. When the issue is resolved, remove the exclusion ### Option 3: Update the constraint (allow version bump) If the version bump is acceptable for the existing image: 1. Update the constraint in `properties.yml`: ```yaml version_constraints: '3.12.*' # Allow 3.12.X now ``` 2. Consider if this changes the image's purpose (major version change) 3. Update image documentation and tags accordingly 4. Consider renaming the image directory to reflect the new version ## Examples ### Python Multi-Version Images ```yaml # images/python-3-11/properties.yml distros: - hummingbird main_package: python3.11 repository: python version_constraints: '3.11.*' ``` ```yaml # images/python-3-13/properties.yml distros: - hummingbird main_package: python3.13 repository: python version_constraints: '3.13.*' ``` ### Node.js Multi-Version Images ```yaml # images/nodejs-20/properties.yml main_package: nodejs20 repository: nodejs version_constraints: '20.*' ``` ```yaml # images/nodejs-24/properties.yml main_package: nodejs24 repository: nodejs version_constraints: '24.*' ``` ### .NET SDK Multi-Version Images ```yaml # images/dotnet-sdk-8-0/properties.yml main_package: dotnet-sdk-8.0 repository: dotnet-sdk version_constraints: '8.*' ``` ### .NET Runtime Multi-Version Images ```yaml # images/dotnet-runtime-8-0/properties.yml main_package: dotnet-runtime-8.0 repository: dotnet-runtime version_constraints: '8.*' ``` ### ASP.NET Runtime Multi-Version Images ```yaml # images/aspnet-runtime-8-0/properties.yml main_package: aspnetcore-runtime-8.0 repository: aspnet-runtime version_constraints: '8.*' ``` ### OpenJDK Multi-Version Images ```yaml # images/openjdk-21/properties.yml main_package: java-21-openjdk-headless repository: openjdk version_constraints: '21.*' ``` ### Go Multi-Version Images (3-part versions) ```yaml # images/go-1-25/properties.yml main_package: golang1.25 repository: go distros: - hummingbird version_constraints: '1.25.*' # Matches 1.25.0, 1.25.3, etc. ``` ## Troubleshooting ### Constraint not working - Verify package name matches exactly (case-sensitive) - Ensure pattern syntax is correct (use `*` wildcard) - Run `make all` to regenerate VERSION files before `make check` ### False positive violations - Check for typos in the constraint pattern - Verify the pattern matches the version format (e.g., `3.11.*` not `3.11*`) - Ensure you're using glob patterns, not regex ### Package renamed in repositories If a package is renamed (e.g., `python3.11` → `python311`): 1. Update `main_package` field 2. Update constraint with new package name (not needed - constraints check VERSION files) 3. Regenerate lockfiles and VERSION files with `make all` ## Implementation Details The version constraint system consists of: 1. **properties.yml**: Optional `version_constraints` field with a glob pattern string 2. **ci/check_version_constraints.py**: Python script that validates VERSION files 3. **Makefile integration**: Script runs as part of `make check` 4. **CI integration**: Failures block merge, including Renovate automerge The validation process: 1. Reads `properties.yml` for each image 2. If `version_constraints` defined, finds all VERSION files across all distros 3. Applies the same constraint pattern to all distros for the image 4. Compares VERSION file content against glob pattern using Python's `fnmatch` 5. Reports violations with file path, version, constraint, and properties file location 6. Exits with error code 1 if any violations found ## See Also - [Image Configuration Reference](../contributing/image-configuration-reference.md) - Complete `properties.yml` field reference - [Excluding Packages from Images](excluding-packages-from-images.md) - How to exclude problematic package versions - [Switching Fedora Streams](switching-fedora-streams.md) - How to pin distro versions -------------------------------------------------------------------------------- # Retrying Konflux Checks url: https://hummingbird-project.io/docs/operating/retrying-konflux-checks/ description: Automatically retrigger failed Konflux CI checks on merge requests using the retrigger script. When Konflux CI checks fail on merge requests, you can automatically retrigger them using the provided retrigger script. ## Prerequisites - **GITLAB_TOKEN**: GitLab API access token with `api` scope and Maintainer access ## Usage For automated retriggers of failed Konflux pipeline runs: ```bash # Retrigger only failed pipeline runs and wait for them to start make retrigger-failed-checks ARGS="https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/123" # Preview what would be retriggered without posting comments make retrigger-failed-checks ARGS="--dry-run https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/123" ``` ## How It Works The retrigger script will: 1. Extract failed pipeline runs from the commit statuses on the merge request 2. Generate `/retest pipeline-name` commands for each failed run 3. Post the retest commands as merge request comments (unless `--dry-run`) 4. Wait for the pipeline runs to start (unless `--dry-run`) ## Manual Retrigger Commands You can also manually retrigger specific checks using slash commands in merge request comments: ```text /retest gitlab-ci--default--main-on-pull-request ``` To retrigger a specific pipeline run as listed in [.tekton/images-on-pull-request.yaml](.tekton/images-on-pull-request.yaml). See the [Tekton Pipelines as Code documentation][pac-commands] for more details on available commands. [pac-commands]: https://pipelinesascode.com/docs/guide/gitops_commands/ ## Examples ```bash # Set your GitLab token export GITLAB_TOKEN=your_token_here # Retrigger failed checks for a specific merge request make retrigger-failed-checks ARGS="https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/456" # Dry run to see what would be retriggered make retrigger-failed-checks ARGS="--dry-run https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/456" ``` The script will output which pipeline runs are being retriggered and wait for them to start before completing. ## Related Operations - [Managing Konflux Comments](managing-konflux-comments.md) - Clean up excessive comments on merge requests -------------------------------------------------------------------------------- # Switching Fedora Streams url: https://hummingbird-project.io/docs/operating/switching-fedora-streams/ > **AI Agent Note:** When asked to switch Fedora streams, follow the workflow > below. The repo file naming convention determines the GPG key automatically. ## Overview The `rawhide` distro can be pinned to a specific Fedora release branch instead of following the rolling Rawhide stream. This provides stability during the period when Rawhide moves to N+1 but the current stable release is still N-1. For example, when Fedora 44 branches from Rawhide, the `rawhide` distro can be pinned to Fedora 44 packages while Rawhide itself starts tracking Fedora 45. ## Repo ID naming convention All Fedora repo files use a `fedora-` naming convention for their repo IDs, where `` matches the filename stem: - `fedora-43.repo` uses `[fedora-43]`, `[fedora-43-source]`, `[fedora-43-updates]`, etc. - `fedora-44.repo` uses `[fedora-44]`, `[fedora-44-source]`, etc. - `fedora-rawhide.repo` uses `[fedora-rawhide]`, `[fedora-rawhide-source]`, etc. This convention ensures: - No DNF repo ID conflicts when multiple repo files coexist in `yum-repos/` - Predictable, greppable repo IDs in lockfiles - The GPG key name can be derived automatically from the filename ## Workflow ### 1. Create or update the repo file Create a new file `yum-repos/fedora-.repo` (e.g., `fedora-45.repo`). The repo file **must** follow these conventions: - **Repo IDs match the filename stem**: Use `[fedora-45]`, `[fedora-45-debuginfo]`, and `[fedora-45-source]` as section names - **Version-specific URLs**: Point to the correct Fedora version's repository (e.g., `/development/45/` or `/releases/45/`) - **Version-specific GPG key**: Reference the correct signing key (e.g., `RPM-GPG-KEY-fedora-45-$basearch`) Use an existing repo file (e.g., [`yum-repos/fedora-44.repo`](../../yum-repos/fedora-44.repo)) as a template. Example for a development branch: ```ini [fedora-45] name=Fedora 45 - Developmental packages for the next Fedora release baseurl=https://koji-s3-cache.hummingbird-project.io/download-ib01.fedoraproject.org/pub/fedora/linux/development/45/Everything/$basearch/os/ enabled=1 countme=1 metadata_expire=6h repo_gpgcheck=0 type=rpm gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-45-$basearch skip_if_unavailable=False ``` ### 2. Update variables.yml Edit [`images/variables.yml`](../../images/variables.yml) and change the `default_variant_repos.rawhide` entry to reference the new repo file: ```yaml default_variant_repos: rawhide: - fedora-45.repo ``` ### 3. Regenerate all derived files Run regeneration to update rpms.in.yaml files, Containerfiles, and lockfiles: ```bash make -j 16 FORCE=true ``` ### 4. Verify GPG key derivation The Jinja templates in `macros/setup_newroot.yml.j2` automatically derive the GPG key name from the repo filename: - `fedora-45.repo` becomes `RPM-GPG-KEY-fedora-45-primary` - `fedora-44.repo` becomes `RPM-GPG-KEY-fedora-44-primary` - `fedora-rawhide.repo` becomes `RPM-GPG-KEY-fedora-rawhide-primary` No template changes are needed when switching versions. Verify by checking the generated `Containerfile` for the builder image contains the correct key: ```bash grep rpmkeys images/hummingbird-builder/rawhide/default/Containerfile ``` ## Switching back to rolling Rawhide To switch back to the rolling Rawhide stream, update `variables.yml` to reference `fedora-rawhide.repo`: ```yaml default_variant_repos: rawhide: - fedora-rawhide.repo ``` Then regenerate all derived files with `make -j 16 FORCE=true`. -------------------------------------------------------------------------------- # Disabling Rawhide for Images url: https://hummingbird-project.io/docs/operating/disabling-rawhide-for-images/ > **AI Agent Note:** When asked to disable Rawhide for images, follow the > workflow below. Run `make all` after editing properties to clean up stale > directories automatically. ## Overview When Rawhide is unstable (mass rebuild, broken critical packages, Fedora branching), Rawhide variants can be temporarily disabled for specific images or globally. This stops building and testing Rawhide variants while keeping Hummingbird variants unaffected. ## Disabling Rawhide for a single image ### 1. Edit properties.yml Add an explicit `distros` field to `images//properties.yml` that excludes `rawhide`: ```yaml distros: - hummingbird ``` This overrides the default distros (which include `rawhide`) for this image only. ### 2. Regenerate ```bash make all ``` The stale Rawhide directories are removed automatically. Generated resources (`.gitattributes`, Konflux templates, Tekton pipelines) are updated to exclude the Rawhide variants. ### 3. Commit and merge Commit the properties change together with the removed generated files. ## Disabling Rawhide for all images ### 1. Edit variables.yml Remove `rawhide` from `default_distros` in [`images/variables.yml`](../../images/variables.yml): ```yaml default_distros: - hummingbird ``` ### 2. Edit per-image overrides Images that explicitly list `distros` including `rawhide` in their own `properties.yml` are not affected by the `variables.yml` change. Edit those files to remove `rawhide` as well. ### 3. Regenerate ```bash make all ``` ### 4. Commit and merge Commit all properties changes together with the removed generated files. ## Re-enabling Rawhide Reverse the properties change (add `rawhide` back to `default_distros` or remove the per-image `distros` override), then regenerate: ```bash make all ``` The Rawhide directories no longer exist, so Make creates all targets from scratch. ## Verification After running `make all`: - `git status` shows Rawhide directories as deleted (when disabling) or new files (when re-enabling) - `.gitattributes` no longer lists Rawhide paths for affected images - `konflux-templates/rendered.yml` and `.tekton/` pipelines no longer reference Rawhide variants for affected images ## See Also - [Image Configuration Reference](../contributing/image-configuration-reference.md) - `distros` field documentation - [Switching Fedora Streams](switching-fedora-streams.md) - Pin Rawhide to a specific Fedora release branch -------------------------------------------------------------------------------- # Setting Up a Konflux Cluster url: https://hummingbird-project.io/docs/operating/setting-up-konflux-cluster/ description: How to set up a new Konflux cluster ## Setting Up the Cluster 1. Set up the new cluster in the [infrastructure] repository, mirroring the existing cluster configuration 2. In the Konflux UI, create a new Component for the Application with the same name as the git repository 3. Verify the Pipeline as Code `Repository` resource is created 4. Run two successive full pipelines in the [infrastructure] repository to deploy resources from [konflux/rendered.yml] and [fix up owner references][konflux-owner-references] 5. Delete the initial Component created from the UI 6. Disable Konflux build status comments (requires elevated permissions): ```bash kubectl patch \ --namespace hummingbird-tenant \ repository REPOSITORY_NAME \ --type merge \ --patch '{"spec":{"settings":{"gitlab":{"comment_strategy":"disable_all"}}}}' ``` [konflux-owner-references]: https://gitlab.com/redhat/hummingbird/infrastructure/-/tree/main/konflux-owner-references [infrastructure]: https://gitlab.com/redhat/hummingbird/infrastructure [konflux/rendered.yml]: https://gitlab.com/redhat/hummingbird/containers/-/blob/main/konflux-templates/rendered.yml ## Next Steps - [Retrying Konflux Checks](retrying-konflux-checks.md) - Retrigger failed CI pipeline runs - [Managing Konflux Comments](managing-konflux-comments.md) - Clean up excessive comments -------------------------------------------------------------------------------- # Deleting RPMs from Hummingbird Repos url: https://hummingbird-project.io/docs/operating/deleting-rpms-from-hummingbird-repos/ description: How to delete RPM packages from Hummingbird repositories ## Overview * This document should be used to remove RPMs from the Hummingbird repositories ## Setup * make sure you have followed the local development setup instructions in the [infrastructure repository](https://gitlab.com/redhat/hummingbird/infrastructure) * install pulp cli via dnf install pulp-cli * setup the local pulp environment via: ```bash cki_secret PULP_PUBLIC_RHEL_PRIMITIVES_CONFIG_FILE > ~/.config/pulp/cli.toml # or vault kv get -mount=apps -field cli.toml hummingbird/PULP_PUBLIC_RHEL_PRIMITIVES_CONFIG_FILE > ~/.config/pulp/cli.toml ``` ## Steps * Note the names of the rpms to delete. In this example, we'll use `systemd-stub` * Obtain the relevant metadata required to delete. * You need to execute this command once for each architecture. ```bash pulp --domain public-hummingbird rpm content -t package list --name systemd-stub | jq -r '.[] | "\(.pulp_href)"' ``` or for prettier content, ```bash pulp --domain public-hummingbird rpm content -t package list --name systemd-stub | jq -r '.[] | "\(.name) \(.version)-\(.release) \(.arch) \(.pulp_created) \(.pulp_href)"' ``` Example output from basic command: ```text /api/pulp/public-hummingbird/api/v3/content/rpm/packages/019b0e50-b89e-749b-a82b-405e00cdd1ed/ /api/pulp/public-hummingbird/api/v3/content/rpm/packages/019b0ded-4234-72b6-82e2-36803fc57a3b/ ``` * Using the `hrefs`, construct your delete commands: ```bash pulp --domain public-hummingbird rpm repository content remove --repository aarch64 --package-href /api/pulp/public-hummingbird/api/v3/content/rpm/packages/019b0e50-b89e-749b-a82b-405e00cdd1ed/ pulp --domain public-hummingbird rpm repository content remove --repository aarch64 --package-href /api/pulp/public-hummingbird/api/v3/content/rpm/packages/019b0ded-4234-72b6-82e2-36803fc57a3b/ ``` * Repeat for `x64_64` and `source` repositories * Verify deletion by navigating to * Deletion is a serial operation that can be impacted by other automated jobs updating the repositories. ## Related Operations * [Setting Up Pulp Repositories](setting-up-pulp-repositories.md) -------------------------------------------------------------------------------- # Updating Dist-git Packages url: https://hummingbird-project.io/docs/operating/updating-dist-git-packages/ description: How packages are automatically updated from Fedora dist-git Packages are automatically updated from Fedora dist-git. Each update creates a separate MR that is automatically approved and merged when CI passes. ## Testing Locally ```bash # Dry-run (check all packages, no MRs) ./ci/dist_git_update_multi_mr.sh --clone # Dry-run (check first 5 packages, no MRs) ./ci/dist_git_update_multi_mr.sh --clone --max-packages=5 # Check only a specific package (for testing/debugging) ./ci/dist_git_update_multi_mr.sh --clone --only-package=libgcrypt # Check only clean packages (skip modified/independent) ./ci/dist_git_update_multi_mr.sh --clone --clean-only # Check only modified packages (skip clean/independent) ./ci/dist_git_update_multi_mr.sh --clone --modified-only # Create up to 3 test MRs (checks all packages, stops after finding 3 updates) export CHORE_MR_GITLAB_TOKEN="glpat-xxxxxxxxxxxxxxxxxxxx" ./ci/dist_git_update_multi_mr.sh --clone --max-updates=3 --create-mrs # Check first 10 packages, create up to 3 MRs export CHORE_MR_GITLAB_TOKEN="glpat-xxxxxxxxxxxxxxxxxxxx" ./ci/dist_git_update_multi_mr.sh --clone --max-packages=10 --max-updates=3 --create-mrs # Create MRs only for clean packages (skip modified/independent) export CHORE_MR_GITLAB_TOKEN="glpat-xxxxxxxxxxxxxxxxxxxx" ./ci/dist_git_update_multi_mr.sh --clone --clean-only --create-mrs ``` ## Environment Variables - `CHORE_MR_GITLAB_TOKEN` - GitLab token with `write_repository` scope (required for `--create-mrs`) - `CHORE_MR_APPROVAL_GITLAB_TOKEN` - GitLab token with `api` scope for auto-approving MRs (used by CI) - `GITLAB_REMOTE_URL` - Target repo (default: ) ## Flags - `--clone` - Clone from GitLab to /tmp (safe for local testing, uses latest main branch) - `--max-packages=N` - Check only the first N packages (limits input set) - `--max-updates=N` - Stop after finding N updates (limits output MRs created) - `--create-mrs` - Actually create MRs (requires token) - `--only-package=NAME` - Check only the specified package (for testing/debugging specific packages) - `--clean-only` - Skip packages with `modification_status` of 'modified' or 'independent', only process clean packages - `--modified-only` - Skip packages with `modification_status` of 'clean' or 'independent', only process modified packages (mutually exclusive with `--clean-only`) ### Using --clean-only The `--clean-only` flag filters out packages marked as 'modified' or 'independent' before attempting updates. This is useful for: 1. **Better failure detection** - Exit code 1 indicates real update failures, not expected errors from modified/independent packages 2. **Cleaner output** - No error messages for packages that can't be auto-updated by design 3. **Efficient CI** - Focus on packages that should update automatically 4. **Performance** - Avoids invoking `dist_git.py` for packages that will fail Without `--clean-only`, the script attempts to update all packages. Modified/independent packages fail with: ```text ERROR: Cannot auto-update Status: modified/independent Reason: Use 'sync' to force update or 'mark-modified --clean' to allow updates ``` These expected failures can mask genuine update issues. Using `--clean-only` prevents these false failures. ### Using --modified-only The `--modified-only` flag filters to only process packages marked as 'modified', skipping clean and independent packages. This is useful for checking the merge logic, as well as getting an overview of current merge conflicts. ## Auto-Merge and Auto-Approval MRs created by this script are configured to: - **Auto-merge** when pipeline succeeds (set via `merge_request.merge_when_pipeline_succeeds`) - **Auto-approve** after 10 minutes via the `chore_mr_approval` CI job (gives Konflux time to post commit statuses) ## Version-Constrained Updates Packages with `track_upstream` set to a version prefix in their metadata are version-constrained. This is used for versioned packages like `golang1.26` that track a specific upstream version line. ### How It Works The `track_upstream` and `release_monitoring_project_id` metadata fields work together across two systems: **`dist_git.py update`** (dist-git sync from Fedora): When `track_upstream` is a version prefix (e.g., `"1.26"`), the update command uses prefix matching: - `1.26` matches `1.26`, `1.26.0`, `1.26.3` (allowed) - `1.26` does **not** match `1.27.0`, `2.0` (skipped) Skipped packages log a warning: ```text WARNING: Skipping golang1.26: upstream version 1.27.0 doesn't match tracked version 1.26 ``` **`check_upstream_versions.py`** (release-monitoring.org checks): When `release_monitoring_project_id` is a string, Anitya is queried using that name instead of the RPM package name. For example, `golang1.26` with `release_monitoring_project_id: "golang"` queries Anitya for `golang`. When it is an integer, the v2 API is queried directly by project ID. When `track_upstream` is a version prefix, the list of upstream versions returned by Anitya is filtered to only those matching the prefix. This means `check_upstream_versions.py check` will report `1.26.5` as the latest version for `golang1.26` even if Anitya reports `1.27.0` as the latest `golang` release. When `--update` is used, the script updates the spec file, downloads new sources, and commits the result. Packages that need custom update logic can provide a hooks file at `metadata/.update-hooks.yaml` to override the spec update, source download, or add a post-update step. See [Package Modification Tracking](../package-modification-tracking) for details. ### Setting Up Version Constraints ```bash # Constrain golang1.26 to only receive 1.26.x updates ./ci/dist_git.py set-upstream golang1.26 --project-id golang --track-version 1.26 # Remove the version constraint ./ci/dist_git.py set-upstream golang1.26 --no-track-version ``` ### Behavior - `dist_git.py`: Version constraint is checked before pre-release and Koji build checks - `dist_git.py`: The `sync` command bypasses the version constraint (explicit force operation) - `check_upstream_versions.py`: `release_monitoring_project_id` determines the Anitya lookup (int for project ID, string for name, absent for RPM name); `track_upstream` filters versions when set to a prefix - Batch operations continue processing other packages after skipping constrained ones ## Pre-Release Version Filtering By default, the update mechanism skips pre-release versions to prevent unstable packages from entering the repository automatically. Pre-release patterns include: - **Tilde notation**: `5.3.0~rc1`, `2.0~beta1`, `1.0~alpha` (RPM standard) - **Suffix notation**: `5.3.0-rc1`, `2.0.beta1`, `3.0-alpha`, `1.0.dev` - **Development markers**: `1.5git20240101`, `2024.01.snapshot`, `1.0dev` ### Manual Override To explicitly update to a pre-release version: ```bash # Update single package to pre-release version ./ci/dist_git.py update --allow-prerelease package-name --skip-build-check # Batch update allowing pre-releases ./ci/dist_git.py update --allow-prerelease --skip-build-check ``` ### Pre-release filtering behavior - Pre-release detection occurs before Koji build checks (saves API calls) - Skipped packages log a warning with the detected pattern - Batch updates continue processing other packages - The `sync` command bypasses this check (explicit force operation) -------------------------------------------------------------------------------- # Writing Documentation url: https://hummingbird-project.io/docs/operating/documentation/ description: How to contribute to the Project Hummingbird documentation. ## Overview New documentation is always welcome! This guide explains how to contribute to the Project Hummingbird documentation. ## Documentation Sites The Project Hummingbird documentation is **open by default** and available at two sites: - **Public documentation**: - **Internal documentation**: ## Documentation Repositories Documentation content comes from six repositories: 1. **[Containers repository][containers-repo]** - Documentation focused on container images (cross-cutting layout): - `documentation/contributing/` - How to add and modify container images - `documentation/background/` - Container image architecture and reference docs - `documentation/ci-scripts/` - CI script documentation - `documentation/operating/` - Operator runbooks for image maintenance - `documentation/using/` - End-user guides (custom CA certificates, etc.) - `README.md` - User guide - `CONTRIBUTING.md` - Quickstart guide 2. **[RPMs repository][rpms-repo]** - Documentation focused on RPM packages (cross-cutting layout): - `documentation/background/` - RPM pipeline architecture and Konflux deployment - `documentation/operating/` - Operator runbooks for package management - `CONTRIBUTING.md` - Contributing guide for RPM packages 3. **[Tools repository][tools-repo]** - Documentation for infrastructure tools and services (per-component layout): - `documentation/` - One page per tool/service (flat structure) 4. **[K8s test pipeline repository][k8s-test-pipeline-repo]** - Documentation for Kubernetes integration testing (per-component layout): - `documentation/` - Pipeline design, test format, and EaaS debugging 5. **[Public documentation repository][docs-repo]** - Site infrastructure and cross-cutting guides: - `content/docs/using/` - End-user guides for using container images - `content/docs/operating/` - Infrastructure operation guides (Konflux, GitLab, etc.) - `content/docs/background/` - General architecture and concepts 6. **[Infrastructure repository][infra-repo]** - Internal-only documentation (cross-cutting layout): - `documentation/operating/` - Internal operational docs (AWS, DNS, GitLab, etc.) - `documentation/background/` - Internal background docs (error budgets, etc.) - `documentation/presentations/` - Team presentations and talks ### How They Work Together The repositories are layered using Hugo Modules: 1. **Public documentation** imports specific directories from the containers, rpms, tools, and k8s-test-pipeline repositories 2. **Internal documentation** is built from the infrastructure repository's `internal-docs/` subtree. It imports the public documentation (which transitively includes the imported repository documentation) and mounts the infrastructure repo's `documentation/` directory as the Internal section **Hugo Module Mounts:** ```yaml # In documentation/config.yaml module: imports: - path: gitlab.com/redhat/hummingbird/containers mounts: - {source: 'README.md', target: content/snippets/readme.md} - {source: 'CONTRIBUTING.md', target: content/snippets/contributing.md} - {source: 'documentation/contributing', target: content/docs/contributing, files: "*.md"} - {source: 'documentation/background', target: content/docs/background/containers, files: "*.md"} - {source: 'documentation/ci-scripts', target: content/docs/background/containers/ci-scripts, files: "*.md"} - {source: 'documentation/operating', target: content/docs/operating, files: "*.md"} - {source: 'documentation/using', target: content/docs/using, files: "*.md"} - path: gitlab.com/redhat/hummingbird/rpms mounts: - {source: 'CONTRIBUTING.md', target: content/snippets/rpms-contributing.md} - {source: 'documentation/background', target: content/docs/background/rpms, files: "*.md"} - {source: 'documentation/operating', target: content/docs/operating, files: "*.md"} - path: gitlab.com/redhat/hummingbird/tools mounts: - {source: 'documentation', target: content/docs/background/tools, files: "*.md"} - path: gitlab.com/redhat/hummingbird/pipelines/k8s-test-pipeline mounts: - {source: 'documentation', target: content/docs/background/k8s-test-pipeline, files: "*.md"} ``` For README.md and CONTRIBUTING.md, front matter is provided via wrapper files. This creates a unified documentation site from multiple repositories without duplication. ### Documentation Layout Patterns Source repositories use one of two layout patterns depending on their content: **Cross-cutting layout** (containers, rpms): The repository documents cross-cutting concerns like contributing, operating, and background information. Documentation is organized into subdirectories matching the site sections: - `documentation/background/` - Explanation and reference material - `documentation/contributing/` - How to contribute - `documentation/operating/` - Operator runbooks - `documentation/using/` - End-user guides Each subdirectory is mounted into the matching site section, so operating docs from different repositories appear together under Operating. **Per-component layout** (tools, k8s-test-pipeline): The repository is a monorepo where each documentation page covers one tool or component. Documentation is a flat directory with one file per component: - `documentation/.md` The entire directory is mounted under `background//` on the site. ## Getting Started ### Quick Start Clone and start the documentation server: ```bash git clone https://gitlab.com/redhat/hummingbird/documentation cd documentation make serve # Available at http://localhost:1313/ ``` For the internal documentation: ```bash git clone https://gitlab.com/redhat/hummingbird/infrastructure cd infrastructure/internal-docs make serve # Available at http://localhost:1314/ ``` ### Recommended Setup For the best development experience, set up all repositories with [direnv]: ```bash mkdir -p ~/git/hummingbird cd ~/git/hummingbird # Clone all repositories git clone https://gitlab.com/redhat/hummingbird/containers git clone https://gitlab.com/redhat/hummingbird/rpms git clone https://gitlab.com/redhat/hummingbird/tools git clone https://gitlab.com/redhat/hummingbird/pipelines/k8s-test-pipeline git clone https://gitlab.com/redhat/hummingbird/documentation git clone https://gitlab.com/redhat/hummingbird/infrastructure # Set up environment variables cat > .envrc << 'EOF' export CONTAINERS_REPO_PATH=$(pwd)/containers export RPMS_REPO_PATH=$(pwd)/rpms export TOOLS_REPO_PATH=$(pwd)/tools export K8S_TEST_PIPELINE_REPO_PATH=$(pwd)/k8s-test-pipeline export DOCUMENTATION_REPO_PATH=$(pwd)/documentation export INFRASTRUCTURE_REPO_PATH=$(pwd)/infrastructure EOF direnv allow ``` This automatically uses your local repositories for imported content, making changes immediately visible in the documentation preview. ## Adding Documentation Add new documentation as markdown files inside the appropriate directory under `content/`. Every file should end in `.md`. Preview your changes: ```bash make serve # Starts local server with live reload ``` Check for problems: ```bash make check # Runs linters and validation ``` ## Link Format Documentation uses two types of internal links depending on the context: ### Same-Directory Links and Cross-Section Links in the Documentation Repository Use relative `.md` links for pages in the same directory, and for cross-section links in the documentation repository: ```markdown [Testing Guide](testing-images.md) [Adding Images](adding-images.md) ``` ### Cross-Section and Cross-Repository Links Use full URLs with `/l/` aliases for links across repositories or across sections in all repositories: ```markdown [Image Pipeline][image-pipeline] [Global Variables][global-vars] [Retrying Checks][retrying-checks] [image-pipeline]: https://hummingbird-project.io/l/image-pipeline [global-vars]: https://hummingbird-project.io/l/global-variables-reference [retrying-checks]: https://hummingbird-project.io/l/retrying-konflux-checks ``` **Why full URLs?** - Work in standalone contexts (GitHub, text editors, when files are copied) - Hugo's link render hook automatically converts them to relative links in the rendered site - Stable even when files are moved or reorganized - Clear and explicit about which page is being referenced ### Link Render Hook The documentation site includes a custom link render hook (`layouts/_default/_markup/render-link.html`) that: 1. Detects `https://hummingbird-project.io/l/...` URLs 2. Looks up the page with the matching alias 3. Converts to a relative link in the rendered HTML 4. Keeps external links as-is with `target="_blank"` This provides the best of both worlds: full URLs in markdown source files that work everywhere, and optimized relative links in the rendered site. ## Stable Link Aliases **All cross-referenced pages should have stable `/l/` aliases.** These aliases serve two purposes: 1. **Incoming links from external sources** (documentation, scripts, merge request comments) 2. **Cross-references within documentation** (across sections or repositories) ### Adding an Alias Add an alias in the page front matter: ```yaml --- title: "Konflux Integration" aliases: [/l/konflux-integration] --- ``` Now you can link to `https://hummingbird-project.io/l/konflux-integration` and it will continue working even if the page moves. ### When to Add Aliases Add `/l/` aliases to pages that: - Are referenced from other repositories (e.g., containers repo → docs repo) - Are linked to from CI scripts or infrastructure - Are shared in merge request comments or external documentation - Are referenced across documentation sections ### Naming Convention - Use `/l/` prefix for all stable links - Keep names short, memorable, and descriptive - Use kebab-case (lowercase with hyphens) - No nested paths (flat structure only) ## Submitting Changes 1. Fork the repository on GitLab 2. Clone your fork locally 3. Create a new branch for your changes 4. Make your changes and test them with `make serve` 5. Run `make check` to ensure there are no errors 6. Commit your changes with a descriptive commit message 7. Push to your fork 8. Create a merge request against the main repository ## Documentation Structure The public documentation is organized into several main sections: - **Using** - Guides for users of Project Hummingbird container images - **Contributing** - Guides for contributors to the project - **Operating** - Guides for infrastructure operators - **Background** - Technical background and architecture information The internal documentation adds: - **Internal** - Internal-only infrastructure documentation, organized into: - **Operating** - AWS, DNS, GitLab, OpenShift, and other service runbooks - **Background** - Error budgets, SLIs/SLOs - **Presentations** - Team talks and lightning talks When adding new documentation, place it in the most appropriate section. [containers-repo]: https://gitlab.com/redhat/hummingbird/containers [rpms-repo]: https://gitlab.com/redhat/hummingbird/rpms [tools-repo]: https://gitlab.com/redhat/hummingbird/tools [k8s-test-pipeline-repo]: https://gitlab.com/redhat/hummingbird/pipelines/k8s-test-pipeline [docs-repo]: https://gitlab.com/redhat/hummingbird/documentation [infra-repo]: https://gitlab.com/redhat/hummingbird/infrastructure [direnv]: https://direnv.net/ -------------------------------------------------------------------------------- # Adding Independent Packages url: https://hummingbird-project.io/docs/operating/adding-independent-packages/ description: How to add packages that originate in the Hummingbird repository Independent packages are packages that originate in the Hummingbird repository rather than being imported from Fedora dist-git. ## Steps to Add an Independent Package ### 1. Create Package Directory ```bash mkdir rpms/ cd rpms/ ``` ### 2. Add Package Files Create the following files in the package directory: - **`.spec`** - RPM spec file - **`sources`** - SHA512 checksums of source tarballs - Source tarballs (`.tar.gz`, `.tar.bz2`, `.tar.xz`) - Any patches or additional configuration files ### 3. Generate the `sources` File The `sources` file must contain SHA512 checksums in BSD-style format: ```bash sha512sum --tag oras-1.3.0.tar.gz oras-1.3.0-vendor.tar.bz2 > sources ``` This produces the correct format: ```text SHA512 (oras-1.3.0.tar.gz) = fb871c0577f621f7e1f56a54f249f96c659b29c771dad529f9a9e838379b2d56bca9cef03a90071915a568e28e266326989e67359e589a87908c4fc077b14689 SHA512 (oras-1.3.0-vendor.tar.bz2) = fa23324bf3910c3dc050eb3c0ebfef3224168489679cf1195d668c656c6c53beaac94b4786e45c2cf3782f9437b0d166da9c18e608a3b474b04957ff1dec5226 ``` **Important**: Use `sha512sum --tag` to generate the BSD-style format. Do not use the default GNU format. ### 4. Create Package Metadata Create `metadata/.json`: ```json { "modification_status": "independent", "release": "1", "upstream_repo": "https://github.com/example/project", "version": "1.3.0" } ``` Fields: - **`modification_status`**: Must be `"independent"` for packages not imported from Fedora (see [Package Metadata Fields](package-metadata-fields.md)) - **`upstream_repo`**: Canonical upstream git repository URL (required — CI enforces this). If no upstream repo exists, use `https://src.fedoraproject.org/rpms/` as a fallback. - **`version`**: Package version (must match spec file) - **`release`**: Base release number without dist tag (typically `1` or `0.1`; the `.hum1` suffix comes from `%{?dist}` in the spec `Release:` line — see [Package Metadata Fields](package-metadata-fields.md)) ### 5. Generate Konflux Resources Generate the Konflux Component and ImageRepository resources: ```bash make generate-host ``` Or directly: ```bash python3 ci/generate_resources.py all ``` This updates: - `konflux-templates/rendered.yml` - `.tekton/` pipeline files ### 6. Commit Changes ```bash git add rpms// metadata/.json konflux-templates/ .tekton/ git commit -m "Add -- Co-Authored-By: Claude Sonnet 4.5 " ``` **Note**: Source tarballs (`.tar.gz`, `.tar.bz2`, `.tar.xz`) are automatically ignored by `.gitignore` and should be uploaded to the lookaside cache instead. ## Example: Adding oras ```bash # 1. Create directory mkdir rpms/oras cd rpms/oras # 2. Add spec file and sources # ... create oras.spec ... # 3. Generate sources file sha512sum --tag oras-1.3.0.tar.gz oras-1.3.0-vendor.tar.bz2 > sources # 4. Create metadata cat > ../../metadata/oras.json << 'EOF' { "modification_status": "independent", "release": "1", "upstream_repo": "https://github.com/oras-project/oras", "version": "1.3.0" } EOF # 5. Generate Konflux resources make generate-host # 6. Commit git add rpms/oras/ metadata/oras.json konflux-templates/ .tekton/ git commit -m "Add oras-1.3.0-1.hum1" ``` ## See Also - [Package Metadata Fields](package-metadata-fields.md) - `modification_status` and `release` configuration - [Package Modification Tracking](package-modification-tracking.md) - Marking packages as modified vs clean - [Rebuilding Packages](rebuilding-packages.md) - Rebuilding existing packages - [Updating Dist-git Packages](updating-dist-git-packages.md) - Importing/updating from Fedora -------------------------------------------------------------------------------- # Pulp Access url: https://hummingbird-project.io/docs/operating/pulp-access/ ## Overview Pulp is used to host and distribute Hummingbird RPM repositories. Access to the Pulp API requires a service account with credentials stored in the vault. ## Existing credentials The public Pulp service account credentials are stored in the vault at `rhel-primitives/PULP_PUBLIC_RHEL_PRIMITIVES_CONFIG_FILE`. Retrieve the `cli.toml` field and save it locally (see [ci/pulp-setup/README.md](../../ci/pulp-setup/README.md) for usage). ## Creating a new Pulp service account When you need a dedicated service account (e.g., for private product repositories): 1. Go to 2. Log in with your `@redhat.com` account 3. Click **New Service Account** 4. Enter a username like `hummingbird--pulp-bot` 5. Enter a description like `bot account for hummingbird pulp operations` 6. Click **Create** 7. Capture the full username (in the format `|`) and the token ### Configure the CLI Create a `cli.toml` with the new credentials: ```toml [cli] base_url = "https://packages.redhat.com" api_root = "/api/pulp/" username = "|hummingbird--pulp-bot" password = "" verify_ssl = true format = "json" dry_run = false timeout = 0 verbose = 0 domain = "" ``` You can either save this to `~/.config/pulp/cli.toml` or pass it explicitly with `--config ./cli.toml` when running scripts. ### Store credentials in the vault Store the credentials in the vault at `apps/hummingbird` with the key `HUMMINGBIRD__PULP_BOT_CONFIG_FILE` (e.g., `HUMMINGBIRD_PRIVATE_PULP_BOT_CONFIG_FILE`). Create two subkeys: - `cli.toml` — the full config file contents - `value` — the token value ## Related - [Lookaside Cache Access](lookaside-cache-access.md) — AWS credentials for source tarball uploads - [Private RPM Repositories](private-rpm-repositories.md) — setting up private per-product Pulp repos - [ci/pulp-setup/README.md](../../ci/pulp-setup/README.md) — Pulp domain and repository creation script -------------------------------------------------------------------------------- # Rebuilding Packages url: https://hummingbird-project.io/docs/operating/rebuilding-packages/ > **AI Agent Note:** When asked to rebuild packages, use the `rebuild` command: > `./ci/dist_git.py rebuild --reason ""`. For rebuilding reverse dependencies > (e.g., "rebuild all Go packages"), use `rebuild-rev-deps --reason ""`. > Always ask the user for a ticket link or explanation first to use as the reason. If the command > fails with "uses macros in Release field", see the "Packages requiring manual rebuild" section > below for instructions. ## Overview This document covers three scenarios for triggering package builds: 1. **No-change rebuild**: Bump the Release field to rebuild with identical sources (e.g., to fix a faulty published RPM or pick up toolchain changes). 2. **Rebuilding reverse dependencies**: Rebuild all packages that depend on a changed package (e.g., rebuild all Go packages when golang updates). 3. **Backporting a patch**: Add an upstream patch that hasn't yet landed in Fedora to fast-track a fix or feature. All scenarios use the `.N` release suffix pattern to ensure our builds sort higher than the upstream Fedora release while remaining lower than the next upstream version. ## No-Change Rebuild ### Using the rebuild command (recommended) The `rebuild` command automates the Release field bump: ```bash # Rebuild a single package ./ci/dist_git.py rebuild --reason "" # Rebuild multiple packages (one commit per package) ./ci/dist_git.py rebuild ... --reason "" # Rebuild all packages (one commit per package) ./ci/dist_git.py rebuild --all --reason "" # Rebuild all packages except specific ones (requires --all) ./ci/dist_git.py rebuild --all --exclude , --reason "" ``` Examples: ```bash # Single package ./ci/dist_git.py rebuild ncurses --reason "published multiple times with different hashes" # Multiple packages ./ci/dist_git.py rebuild grep unzip sed --reason "fix faulty builds" # All packages (useful after toolchain updates) ./ci/dist_git.py rebuild --all --reason "toolchain update: GCC 15" # All packages except a few already rebuilt earlier in the same rollout ./ci/dist_git.py rebuild --all --exclude glibc,gcc,llvm --reason "toolchain update: GCC 15" ``` `--exclude` takes a comma-separated list of package names and only works together with `--all` — it is rejected when combined with an explicit package list, since you can simply omit the packages you don't want to rebuild in that case. The command: - Automatically bumps the Release field using the `.N` suffix pattern - Handles `%autorelease` by resolving and replacing with explicit values - Preserves macros in the Release field (e.g., `%{revision}`) - Creates a properly formatted commit message (one per package with `--all`) - Does **not** mark the package as modified (release-only changes are ephemeral) Supports `--dry-run` to preview changes without committing: ```bash ./ci/dist_git.py --dry-run rebuild --reason "test" ``` ### Creating MRs for rebuild commits After creating rebuild commits locally, use `rebuild_multi_mr.sh` to push each commit as its own merge request (one MR per package, auto-merge enabled): ```bash ./ci/rebuild_multi_mr.sh ``` By default the script compares against `origin/main`. For local development, use `--base` to point at a different ref: ```bash # Use local main branch as base (useful when origin/main is not up to date) ./ci/rebuild_multi_mr.sh --base main # Use a specific commit SHA as base ./ci/rebuild_multi_mr.sh --base abc1234 # Preview what would be created without pushing ./ci/rebuild_multi_mr.sh --dry-run # Limit to at most N MRs ./ci/rebuild_multi_mr.sh --max-updates=5 ``` The script: - Creates a `chore/rebuild-{package}` branch per commit and pushes it - Titles each MR `chore(rpms): Rebuild {package}: {reason}` - Enables auto-merge on all rebuild MRs - Leaves the current branch untouched - Skips branches that already exist on the remote (idempotent) - Only processes commits whose message matches `Rebuild {package}: {reason}`; other commits in the range are silently skipped **Full workflow example:** ```bash # 1. Create the rebuild commits ./ci/dist_git.py rebuild grep ncurses bash --reason "HUM-1234: toolchain update" # 2. Preview the MRs that would be created ./ci/rebuild_multi_mr.sh --base main --dry-run # 3. Create the MRs ./ci/rebuild_multi_mr.sh --base main ``` ### Packages requiring manual rebuild Some packages use complex macro systems that the automated rebuild command cannot handle. These require manual editing of the spec file. #### Macro indirection patterns These packages define the Release field using a macro, where the macro itself contains `%{?dist}`. The rebuild command cannot detect or manipulate these without expanding all macros, which would break the macro system. **nodejs packages (nodejs20, nodejs22, nodejs24, nodejs25):** ```spec %{load:%{_sourcedir}/nodejs.srpm.macros} %nodejs_define_version node 1:25.8.2-%{autorelease} -p ... Release: %{node_release} ``` The `%{node_release}` macro is defined by an external macro system loaded from `nodejs.srpm.macros`. The release component is embedded in the version definition. **How to rebuild:** Edit the `%nodejs_define_version node` line to bump the release component (e.g., change `-%{autorelease}` to `-1.1` or increment existing `.N`). **kernel-headers:** ```spec %define specrelease 59%{?buildid}%{?dist} ... Release: %{specrelease} ``` **How to rebuild:** Edit the `%define specrelease` line to add/increment the `.N` suffix before `%{?buildid}`: ```spec %define specrelease 59.1%{?buildid}%{?dist} ``` **krb5:** ```spec %global krb5_release 4%{?dist} ... Release: %{krb5_release} ``` **How to rebuild:** Edit the `%global krb5_release` line to add/increment the `.N` suffix: ```spec %global krb5_release 4.1%{?dist} ``` #### Why these can't be automated The rebuild command can handle: - ✅ Simple numeric: `Release: 5%{?dist}` - ✅ Macros ending with dist: `Release: %{baserelease}%{?dist}` (e.g., rpm, gcc) - ✅ Complex macros with dist: `Release: %{?snapver:0.%{snapver}.}%{baserelease}%{?dist}` - ✅ Content after dist: `Release: 11.1%{?dist} %{?extra_version:-e %{extra_version}}` (e.g., unbound) The rebuild command **cannot** handle: - ❌ Macros without `%{?dist}`: `Release: %{node_release}` - ❌ Macros where dist is inside the macro definition: `%{krb5_release}` contains `%{?dist}` This is because detecting and manipulating macros that contain dist internally would require expanding all macros (which changes the spec file semantically) or implementing RPM's full macro parser. ### Manual rebuild process If you need to rebuild manually or the automated command doesn't work for your use case, follow these steps: #### 1. Identify the package to rebuild Identify the source package name and locate its spec file in `rpms//.spec`. If you have a binary RPM name, the source package name may differ. Query the Hummingbird repos to get the source RPM name: ```bash podman run --rm quay.io/hummingbird-ci/hummingbird-builder:latest \ dnf5 repoquery --queryformat '%{SOURCERPM}' 2>/dev/null ``` Example: `ncurses-libs-6.5-8.20250614.hum1` -> SRPM `ncurses-6.5-8.20250614.hum1.src.rpm` -> spec file at `rpms/ncurses/ncurses.spec` #### 2. Determine the Release bump pattern The `.N` bump suffix must always appear **immediately before** `%{?dist}`. The `%{?dist}` suffix should always be the final component since it identifies the build environment. | Current Pattern | Example Before | Example After | | --------------- | -------------------------------- | ---------------------------------- | | Simple numeric | `Release: 3%{?dist}` | `Release: 3.1%{?dist}` | | Already bumped | `Release: 3.1%{?dist}` | `Release: 3.2%{?dist}` | | With macro | `Release: 8.%{revision}%{?dist}` | `Release: 8.%{revision}.1%{?dist}` | | autorelease | `Release: %autorelease` | `Release: 1.1%{?dist}` | For `%autorelease`, first resolve its value using `rpmspec`, then replace with the resolved value plus `.1`. In the Hummingbird monorepo, `%autorelease` always evaluates to `1`. > **Note:** If the Release field is missing `%{?dist}` entirely or looks unusual (e.g., `1build1` > instead of `1.1%{?dist}`), flag this to the user for resolution. Check the git history to > understand the original value: > > ```bash > git log -p -S "Release:" -- rpms//.spec > ``` > > This helps determine the correct fix when a previous bump was malformed. #### 3. Modify the spec file Use `sed` to edit only the `Release:` line, avoiding any unintended whitespace changes that text editors may introduce: ```bash sed -i 's/^Release: 3%{?dist}$/Release: 3.1%{?dist}/' rpms//.spec ``` Verify the change with `git diff` before committing: ```bash git diff rpms//.spec ``` The diff should show only the Release line change: ```diff - Release: 3%{?dist} + Release: 3.1%{?dist} ``` > **Important:** Only modify the Release line. Do not introduce any other changes such as whitespace > fixes or trailing newline modifications. If the diff shows additional changes, reset and retry > with `sed`. > > **Important:** Do not change the `release` field in `metadata/.json` during local > rebuilds or backports. That field is the current base release (Fedora/rawhide baseline, or a > local base such as `0.1` when ahead of Fedora); see > [Package Metadata Fields](package-metadata-fields.md). #### 4. Verify the bump is correct Use `rpm --eval` to confirm the new release sorts higher than the original: ```bash # Returns -1 if first < second (correct), 1 if first > second (wrong) rpm --eval '%{lua:print(rpm.vercmp("3.hum1", "3.1.hum1"))}' # Expected output: -1 ``` #### 5. Commit the change Use this commit message format: ```text Rebuild : ``` Example: ```text Rebuild ncurses: published multiple times with different hashes HUM-1234 ``` #### 6. Verify the commit After committing, verify only the Release line was changed: ```bash git show --stat HEAD ``` Expected output should show exactly 1 insertion and 1 deletion: ```text rpms//.spec | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) ``` If the commit shows more changes, amend or reset and redo the change using `sed`. ### Important notes about rebuilds #### Modification status Rebuilds **do not** change a package's `modification_status`. Spec `Release:` bumps are ephemeral and do not make a package `modified` versus `clean`. See [Package Metadata Fields](package-metadata-fields.md) for how `modification_status` and metadata `release` relate to rebuilds. During updates, Release lines are normalized to avoid conflicts, and the automation ignores Release-only changes, so automatic Fedora updates continue normally after a rebuild. If you want to explicitly prevent automatic updates (e.g., you're investigating an issue), you can manually mark the package as modified: ```bash ./ci/dist_git.py mark-modified --modified --reason "Investigating build issue" ``` Note: This will block automatic Fedora updates until you mark it clean again. ## Rebuilding Reverse Dependencies When a compiler, runtime, or toolchain package changes, you may need to rebuild all packages that depend on it. The `rebuild-rev-deps` command automates finding and rebuilding reverse dependencies. ### When to use Common scenarios for rebuilding reverse dependencies: - **Golang/Python runtime updates**: When updating `golang1.26`, `python3.14`, etc., rebuild all Go/Python packages - **Toolchain changes**: When updating `gcc`, rebuild packages that BuildRequire it - **Library ABI changes**: When a library's ABI changes, rebuild packages that BuildRequire it ### Usage ```bash ./ci/dist_git.py rebuild-rev-deps --reason "" ``` The command: 1. Finds all packages that have `BuildRequires: ` in their spec files 2. Rebuilds each package (bumps Release field and commits) 3. Reports a summary of successful/failed rebuilds Examples: ```bash # Rebuild all Go packages when golang updates ./ci/dist_git.py rebuild-rev-deps golang1.26 --reason "golang 1.26.2 update" # Rebuild all Python packages when python updates ./ci/dist_git.py rebuild-rev-deps python3.14 --reason "python 3.14.1 update" # Rebuild packages that depend on a specific library ./ci/dist_git.py rebuild-rev-deps openssl --reason "openssl 3.4.0 update" ``` ### Virtual BuildRequires (golang/python) For virtual BuildRequires like `golang1.25`, `golang1.26`, `python3.13`, `python3.14`, the command automatically handles translation to the actual BuildRequires target: ```bash # These all work the same way: ./ci/dist_git.py rebuild-rev-deps golang1.26 --reason "..." ./ci/dist_git.py rebuild-rev-deps go-rpm-macros --reason "..." ``` **Important:** Only the **latest** version triggers rebuilds. This is because: 1. Multiple golang versions exist: `golang1.25`, `golang1.26` 2. They all provide the same virtual package: `Provides: golang = ` 3. DNF always picks the highest version to satisfy `Requires: golang` Therefore: - ✅ `rebuild-rev-deps golang1.26` → Rebuilds all Go packages (latest version) - ❌ `rebuild-rev-deps golang1.25` → **Error**: Not the latest version This ensures rebuilds only happen when the active runtime actually changes. ### Creating MRs for reverse dependency rebuilds After creating rebuild commits, use `rebuild_multi_mr.sh` to push each commit as its own MR: ```bash ./ci/rebuild_multi_mr.sh --base main ``` See the "Creating MRs for rebuild commits" section above for full details. ### Dry-run mode Preview which packages would be rebuilt without making changes: ```bash ./ci/dist_git.py --dry-run rebuild-rev-deps golang1.26 --reason "test" ``` This shows: - Which packages have the BuildRequires dependency - What the new Release values would be - Does not commit any changes ## Backporting a Patch Use this workflow when you need to fast-track an upstream fix or feature that hasn't yet been released in Fedora. ### 1. Obtain the patch Fetch the patch from the upstream repository. For GitHub PRs, append `.patch` to the PR URL: ```bash curl -L https://github.com///pull/.patch \ > rpms//-.patch ``` Name the patch file with a numeric prefix matching the next available `PatchN:` slot in the spec file (e.g., `0004-fix-foo.patch` if Patch1-3 already exist). ### 2. Add the patch to the spec file Add a `PatchN:` declaration after the existing patches: ```spec Patch3: 0003-existing-patch.patch Patch4: 0004-fix-foo.patch ``` The patch will be applied automatically if the spec uses `%autosetup -p1`. If the spec uses explicit `%patchN` macros, add the corresponding apply line in the `%prep` section. ### 3. Check for a gorget source-pipeline If `metadata/.source-pipeline.yaml` exists, check whether its `transform:` step applies patches or otherwise runs against patched source (e.g. resolving a lockfile with `yarn install`, `pnpm fetch`, `npm ci`): ```bash # run from the root of the rpms repo checkout grep -n -E "patch -p[0-9]|git apply" metadata/.source-pipeline.yaml ``` This catches the common forms, but isn't exhaustive — if the transform applies patches some other way (a different tool, or a script that wraps `patch`/`git apply`), read the `transform:` step directly rather than trusting a `grep` miss. If it does, and your new patch touches a file that step depends on (a lockfile or manifest such as `yarn.lock`, `package.json`, `pnpm-lock.yaml`, `go.sum`, `Cargo.lock`), add the same `patch -p1 < "${PACKAGE_DIR}/.patch"` line there too, **in the same relative order** as its `PatchN:` slot in the spec. The patch file itself needs no extra fetch step — it's already in `${PACKAGE_DIR}/` because you committed it there in step 2. The pipeline's `transform:` step and the spec's `%prep`/`%goprep` are two independent lists of patches to apply — nothing keeps them in sync automatically. A patch declared only in the spec is invisible to whatever artifact the transform step generates (e.g. an offline yarn/pnpm cache built from the lockfile), and the generated artifact silently drifts from what `%build` actually applies. See "Known sharp edge: patch-list duplication" in [the source-pipeline design doc][sharp-edge] for the grafana12.4/grafana13.1 incident this rule comes from. `test/test_source_pipeline_patches.py` checks for this drift, but treat it as a safety net, not a substitute for updating the pipeline yourself here. ### 4. For Go packages: check go-vendor-tools.toml's pre_commands This applies whether or not the package uses gorget — `rpms//go-vendor-tools.toml`'s `[archive] pre_commands` (sed edits, `go get` bumps, `go mod tidy`) only ever run against the vendor archive's own checkout, never against the plain source tarball (`Source0`), which is fetched separately: ```bash grep -n -A2 "pre_commands" rpms//go-vendor-tools.toml ``` If your new patch targets a vendored Go module (or you're editing `pre_commands` directly instead of adding a patch — don't), and any `pre_commands` entry mutates `go.mod`/`go.sum` (directly, via `go get`, or via `go mod tidy`/`edit`), your patch must apply the *same* `go.mod`/`go.sum` change to the actual source tree, not just to `go-vendor-tools.toml`. `pre_commands` cannot substitute for this: they require live network access to run `go get`, which `%prep` doesn't have (Konflux builds are hermetic). Compute the resulting `go.mod`/`go.sum` diff offline (e.g. clone upstream, apply the same edits, run `go mod tidy` if `pre_commands` does) and commit that as the patch. Skipping this leaves `go.mod` in the build tree and `vendor/modules.txt` in the generated vendor archive silently requiring different versions of the same package, which `go build -mod=vendor` rejects as inconsistent vendoring. See [the source-pipeline design doc][sharp-edge] for the trivy incident this rule comes from. `test/test_govendortools_gomod_patch_sync.py` checks for this drift, same caveat as above. ### 5. Bump the Release Follow the same `.N` suffix pattern as no-change rebuilds: ```diff - Release: 3%{?dist} + Release: 3.1%{?dist} ``` ### 6. Commit the patch Use this commit message format: ```text : backport Backport: ``` Example: ```text dnf5: backport reproducible build sorting fix Backport: https://github.com/rpm-software-management/dnf5/pull/2522 ``` ### 7. Mark package as modified Mark the package as modified to prevent automatic Fedora updates from overwriting your backport: ```bash ./ci/dist_git.py mark-modified --modified \ --reason "Backport fix for " ``` Example: ```bash ./ci/dist_git.py mark-modified dnf5 --modified \ --reason "Backport reproducible build sorting fix from upstream PR#2522" ``` This ensures the package won't be automatically updated from Fedora until the backported patch lands upstream and you explicitly mark it clean again. ### 8. Test the build locally (optional) Build the package locally to verify the patch applies cleanly: ```bash ./ci/build_rpms.sh ``` Built RPMs will be in `builds//RPMS/`. ## Related Operations - [Excluding Packages from Images][exclude] - temporarily block faulty packages in container builds [exclude]: https://hummingbird-project.io/l/excluding-packages-from-images [sharp-edge]: https://hummingbird-project.io/l/source-pipeline-tool#known-sharp-edge-patch-list-duplication -------------------------------------------------------------------------------- # Setting Up Pulp Repositories url: https://hummingbird-project.io/docs/operating/setting-up-pulp-repositories/ description: How to create and set up Pulp domain and repositories ## Overview * This document and script should be used to initialize and setup a pulp domain and a set of repositories * We choose the domain prefix of `public-` to ensure that the repositories are publicly available ## Setup * make sure you have followed the local development setup instructions in the [infrastructure repository](https://gitlab.com/redhat/hummingbird/infrastructure) * install pulp cli via dnf install pulp-cli * setup the local pulp environment via: ```bash cki_secret PULP_PUBLIC_RHEL_PRIMITIVES_CONFIG_FILE > ~/.config/pulp/cli.toml # or vault kv get -mount=apps -field cli.toml hummingbird/PULP_PUBLIC_RHEL_PRIMITIVES_CONFIG_FILE > ~/.config/pulp/cli.toml ``` * Run the [creation script] [creation script]: https://gitlab.com/redhat/hummingbird/rpms/-/blob/main/ci/pulp-setup/create-pulp-resources.sh ## Create ```bash ./create-pulp-resources.sh public-hummingbird "source,x86_64,s390x,ppc64le,aarch64" ``` -------------------------------------------------------------------------------- # Package Metadata Fields url: https://hummingbird-project.io/docs/operating/package-metadata-fields/ ## Overview Each package has a metadata file at `metadata/.json`. Two fields control how Hummingbird treats local changes and rebuilds relative to Fedora: | Field | Purpose | | ----- | ------- | | `modification_status` | Whether the package may be auto-updated from Fedora | | `release` | Base release without dist tag — Fedora/rawhide baseline, or a local base (independents / ahead-of-Fedora) | This page is the canonical reference for configuring those fields. For day-to-day workflows (marking packages, rebuilding, importing), see the related docs at the end. ## `modification_status` `modification_status` records whether a package still matches its Fedora upstream import, has local source-level changes, or is Hummingbird-independent (not from Fedora). | Status | Meaning | Auto-updates from Fedora | | ------------- | ------------------------------------------------- | ------------------------ | | `clean` | Unmodified Fedora import | Allowed | | `modified` | Local changes (patches, spec modifications) | Blocked | | `independent` | Hummingbird-independent package (not from Fedora) | Blocked | ### When to use each value - **`clean`**: Default for packages imported from Fedora with no local source changes. Automatic Fedora updates are allowed. Conflicts with `modification_reason` (must be absent). - **`modified`**: Use after backports, custom patches, or other spec/source edits that must not be overwritten by an automatic update. Requires `modification_reason`. - **`independent`**: Required for packages that are not imported from Fedora. These packages have no `source` / `branch` / `sha` fields. Conflicts with `modification_reason` (must be absent). Imports set the status automatically (`clean` for Fedora imports, `independent` for Hummingbird-independent imports). After local source changes, set status with `dist_git.py mark-modified` — see [Package Modification Tracking](package-modification-tracking.md). Rebuilds that only bump the spec `Release:` line do **not** change `modification_status`. Release-only changes are ephemeral and are ignored when deciding whether a package is modified versus clean. ### `modification_reason` When `modification_status` is `modified`, `modification_reason` is required. It should be a short explanation of why the package cannot be auto-updated (for example, a CVE backport or a custom spec change). Clear the reason when marking the package clean again. `modification_reason` applies only to `modified` packages. Do not add it for `clean` or `independent` packages — see [When not to update these fields](#when-not-to-update-these-fields). ## `release` The `release` field in metadata is **not** the same as the `Release:` line in the package spec. It is overloaded: the same field stores one of two kinds of base release (always without a dist tag such as `.fc42` or `.hum1`): 1. **Fedora/rawhide base** — the upstream Fedora release this package was imported or last synced from (dist suffix stripped). Used while Hummingbird still tracks a Fedora build of the current version. 2. **Local base** — a Hummingbird-chosen base for packages with no Fedora upstream (`independent`), or for Fedora-imported packages that Hummingbird has version-bumped ahead of Fedora. In the ahead-of-Fedora case this is typically `0.1` — a locally invented placeholder so a later Fedora import with `Release >= 1` sorts higher; it is **not** a value confirmed from an actual Fedora build. `dist_git.py rebuild` uses this base (when a Fedora `source` is present) to compute the next `.N` micro-bump on the spec `Release:` line. | Location | What it represents | | -------- | ------------------ | | `metadata/.json` → `release` | Base release without dist tag (Fedora/rawhide baseline **or** local base — see above) | | Spec `Release:` | The release used for the Hummingbird build (includes `%{?dist}` / `.hum1`, and may include local `.N` rebuild suffixes) | ### Who writes metadata `release` | Writer | When | Value stored | | ------ | ---- | ------------ | | `dist_git.py import` / `update` / `sync` | Fedora import or refresh | Resolved Fedora release with dist suffix stripped (e.g. `5` from `5.fc42`) | | Independent package creation | Adding a package not from Fedora | Initial local base (e.g. `1` or `0.1`) | | `check_upstream_versions.py` | Local upstream version bump (`check --update`) | Resolved base from the updated spec — typically the local `0.1` placeholder, not a Fedora-confirmed release | ### Fedora-imported packages For packages with a Fedora `source`: - `release` is set on import and refreshed on `dist_git.py update` / `sync` from the upstream Fedora package (dist suffixes like `.fc42` are stripped). That is the Fedora/rawhide base. - When `check_upstream_versions.py check --update` bumps the package ahead of Fedora, it resets the spec to `Release: 0.1%{?dist}` (unless `%autorelease`) and writes that local base (`0.1`) into metadata `release`. Until Fedora ships the same version and `update` / `sync` runs again, metadata no longer holds a Fedora-confirmed baseline. - Do **not** change metadata `release` during local rebuilds or backports. Rebuilds bump only the spec `Release:` line; metadata continues to record the current base (Fedora or local) so `rebuild` can compute the next `.N` suffix correctly. - When Fedora’s baseline is `Release: %autorelease`, metadata `release` stores the **resolved** numeric release from MDAPI (not the literal `%autorelease`). Import/update replace `%autorelease` in the local spec with that value plus `%{?dist}`. For rebuilds that still see `%autorelease`, follow [Rebuilding Packages](rebuilding-packages.md). ### Independent packages For Hummingbird-independent packages (`modification_status: "independent"`, no `source` field): - There is no Fedora upstream release. Set `release` to a local base such as `1` or `0.1` when adding the package (no dist tag — same storage convention as Fedora-imported packages). - The `.hum1` suffix comes from the spec `Release:` line via `%{?dist}` (for example `Release: 1%{?dist}`), not from metadata `release`. - `dist_git.py rebuild` does not treat metadata `release` as an upstream Fedora baseline for independent packages (it only uses that baseline when a `source` field is present). ### Spec `Release:` vs metadata `release` No-change rebuilds and reverse-dependency rebuilds change the **spec** `Release:` only. See [Rebuilding Packages](rebuilding-packages.md). When a Fedora update for the same version lands later, `dist_git.py update` / `sync` replaces the spec `Release:` from upstream and restores metadata `release` to the new Fedora/rawhide baseline. ## When not to update these fields Most day-to-day package work changes the spec, sources, or patches — not metadata `modification_status`, `modification_reason`, or `release`. Leave those fields alone unless the package's relationship to Fedora actually changed. | Situation | Do not change | Why | | --------- | ------------- | --- | | Independent package gets a CVE patch, backport, or other source edit | `modification_status`, `modification_reason` | Status stays `independent`. There is no Fedora auto-update to block, so do not switch to `modified` or add a `modification_reason`. | | No-change rebuild (spec `Release:` bump only) | `modification_status`, `modification_reason`, metadata `release` | Rebuilds are ephemeral. Status stays `clean` (or whatever it was); metadata `release` still records the current base (Fedora/rawhide or local). | | Fedora-imported package gets a local patch or backport | metadata `release` | Mark the package `modified` with a reason, but keep metadata `release` as the current base (last Fedora baseline, or the local `0.1` placeholder if already ahead of Fedora). Only bump the **spec** `Release:` if needed. | | Package is already `modified` and you add another local change | `modification_status` | Leave status as `modified`. Update `modification_reason` only if the existing reason no longer describes why auto-updates must stay blocked. | | Upstream Fedora update lands and you want auto-updates again | (do not leave stale fields) | Mark clean (clears `modification_reason`). Do not hand-edit `release`; let `dist_git.py update` refresh version/release from Fedora. | ### Examples **Independent package + CVE fix:** Edit the spec and add the patch. Keep: ```json { "modification_status": "independent", "release": "1", "version": "1.3.0" } ``` Do not add `modification_reason`, and do not change status to `modified`. **Fedora package + CVE backport:** After the patch, mark modified so auto-updates stay blocked: ```bash ./ci/dist_git.py mark-modified --modified \ --reason "Backport CVE-2024-12345" ``` Do not edit metadata `release` as part of that change. **Rebuild only:** Use `dist_git.py rebuild` (or bump the spec `Release:`). Leave metadata `modification_status` and `release` unchanged. ## Related documentation - [Package Modification Tracking](package-modification-tracking.md) — mark packages modified/clean, check status, validation, and update hooks - [Rebuilding Packages](rebuilding-packages.md) — bump spec `Release:` for no-change rebuilds and backports - [Adding Independent Packages](adding-independent-packages.md) — create metadata for packages not from Fedora - [Updating Dist-git Packages](updating-dist-git-packages.md) — import and update from Fedora -------------------------------------------------------------------------------- # Debugging Build Failures url: https://hummingbird-project.io/docs/operating/debugging-build-failures/ ## Overview When `./ci/build_rpms.sh` fails to build a package, you can use the `--shell-after` flag to drop into an interactive debugging environment. This preserves the build state and allows you to investigate failures, modify files, and re-run the build without starting over. ## Quick Start ```bash # Run build and drop into shell after completion (even on failure) ./ci/build_rpms.sh --shell-after ``` After the build completes (success or failure), you'll be dropped into a mock shell inside the build environment. ## Understanding the Build Environment The build happens in two layers: 1. **Outer container**: The rpm-build-pipeline podman container (Red Hat build environment) 2. **Mock shell**: An isolated RPM build environment inside the container (similar to a chroot) When you use `--shell-after`, you're placed directly in the **mock shell**. ### Key directories in the outer container - `/results/build.log` - Full build output with all commands that were run - `/sources/` - Package source files (spec, patches, etc.) - `/config/` - Mock configuration - `/repo/` - Your local rpms repository (read-only mount) ### Key directories in the mock shell - `/builddir/build/BUILD/-*/` - Unpacked source tree (modify and re-run build/test commands here) - `/builddir/build/originals/.spec` - The spec file (modify to test spec changes) - `/builddir/build/SOURCES/` - Source tarballs and patches - `/builddir/build/BUILDROOT/` - Install root (where %install places files) ## Debugging Workflow ### 1. Connect to the outer container To get a second shell without killing your existing mock shell session, connect to the outer container: ```bash # In another terminal, find the running container podman ps | grep rpm-build-pipeline # Get a shell in the outer container podman exec -it bash ``` ### 2. Examine the build failure From the outer container, check the build log to understand what failed: ```bash # View the end of the build log tail -100 /results/build.log # Search for specific errors grep -i error /results/build.log ``` ### 3. Enter the mock shell To investigate or modify the build environment, you can get an additional shell in the mock chroot, particularly for agents which cannot use the initial interactive `--shell-after` from above. ```bash # From the outer container, chroot into the mock build environment chroot /var/lib/mock/local-x86_64/root /bin/bash ``` ### 4. Iterate on the fix The recommended approach for debugging is to extract and re-run specific commands directly: #### Fast iteration: Extract and re-run specific commands (recommended) For quick iteration while developing a fix, extract the exact build or test command from the build log and run it directly: ```bash # From the outer container, find what command the failing phase runs grep -A 30 'Executing(%check)' /results/build.log # Look for lines starting with "+ " that show the actual commands # Example output: # + cd /builddir/build/BUILD/dnf5-5.4.0.0-build/dnf5-5.4.0.0 # + /usr/bin/ctest --test-dir redhat-linux-build --output-on-failure ... # Run that command in the mock chroot podman exec -u root chroot /var/lib/mock/local-x86_64/root \ su mockbuild -c "cd /builddir/build/BUILD/-*/... && " ``` **For build failures during %build:** Look for `Executing(%build)` in the log, find commands like `make` or `ninja`, then re-run them: ```bash # Example: extract the build command grep -A 30 'Executing(%build)' /results/build.log | grep -E '^\+ (make|ninja|cmake)' # Re-run in the build directory podman exec -u root chroot /var/lib/mock/local-x86_64/root \ su mockbuild -c "cd /builddir/build/BUILD/-*/ && make -j14" ``` **For test failures during %check:** Look for `Executing(%check)` and extract the test command: ```bash # Example: extract the test command grep -A 30 'Executing(%check)' /results/build.log | grep -E '^\+ (ctest|make check|pytest)' # Re-run tests podman exec -u root chroot /var/lib/mock/local-x86_64/root \ su mockbuild -c "cd /builddir/build/BUILD/-*/... && ctest --output-on-failure" ``` This approach is fast because it skips rpmbuild overhead and goes straight to the failing command. It works with the preserved BUILD directory from `--shell-after` and is ideal when you're modifying source files and want quick feedback. **Modifying source files:** Edit files directly in `/builddir/build/BUILD/-*/` and re-run the command to test your changes quickly. **Modifying the spec file:** You can also test spec file changes by editing `/builddir/build/originals/.spec` in the mock chroot: ```bash # From the host, edit the spec file in the mock chroot podman exec -it -u root chroot /var/lib/mock/local-x86_64/root \ vim /builddir/build/originals/.spec # Run rpmbuild to see the change podman exec -u root chroot /var/lib/mock/local-x86_64/root \ env HOME=/builddir LANG=C.UTF-8 su mockbuild -c \ 'rpmbuild -bb --target x86_64 --nodeps /builddir/build/originals/.spec' ``` #### Full validation: Apply fixes and rebuild with build_rpms.sh Manual `rpmbuild` in the preserved environment may not work reliably for all packages due to complex build dependencies and environment requirements. Once you've identified a fix using the fast iteration approach: 1. **Exit the debug environment** and apply your fix to the actual source files or spec file in `rpms//` 2. **Test the complete build** using the build script: ```bash ./ci/build_rpms.sh ``` This ensures your fix works through the entire build process in a clean environment. ## Related Operations - [Rebuilding Packages](rebuilding-packages.md) - Bump release or backport patches - [Package Modification Tracking](package-modification-tracking.md) - Mark packages as modified -------------------------------------------------------------------------------- # Package Modification Tracking url: https://hummingbird-project.io/docs/operating/package-modification-tracking/ ## Overview The RPMs repository tracks whether packages have been locally modified from their Fedora upstream source. This tracking prevents automatic updates from overwriting local changes like backported patches or custom modifications. ## Modification Status Each package metadata file (`metadata/.json`) has a `modification_status` of `clean`, `modified`, or `independent`. That field (and related `modification_reason` / `release` configuration) is documented in [Package Metadata Fields](package-metadata-fields.md). This page covers the workflows for checking status, marking packages, viewing diffs, and configuring update hooks. An optional `track_upstream` string field controls whether a package is checked by `check_upstream_versions.py` for new upstream releases (via release-monitoring.org). Set it to `"latest"` to track the latest version, or to a version prefix like `"1.26"` to constrain updates to that series. Its presence enables tracking; omit the field to disable it. The `check` subcommand only checks packages with `track_upstream` set when no explicit package arguments are given. The `list` subcommand shows all packages regardless of this field. ## Checking Package Status View a package's modification status: ```bash jq .modification_status metadata/.json ``` View reason for modification (if modified): ```bash jq .modification_reason metadata/.json ``` List all modified packages: ```bash for f in metadata/*.json; do status=$(jq -r .modification_status "$f" 2>/dev/null) if [ "$status" = "modified" ]; then pkg=$(basename "$f" .json) reason=$(jq -r .modification_reason "$f" 2>/dev/null) echo "$pkg: $reason" fi done ``` ## Viewing Package Differences To see what changes exist in a modified package compared to upstream Fedora: ```bash # Show full diff for a package ./ci/dist_git.py diff bash # Show summary statistics ./ci/dist_git.py diff bash --stat # Show only which files changed ./ci/dist_git.py diff bash --name-only # Show raw diff (includes Release: bumps and whitespace) ./ci/dist_git.py diff bash --raw # Diff all modified packages ./ci/dist_git.py diff --all ``` **What's shown:** - By default, the diff ignores Release: number changes (no-change rebuilds) - Trailing whitespace and blank line changes are ignored - Use `--raw` to see absolutely everything, including Release: bumps **Package types:** - **Modified packages**: Shows the differences - **Clean packages**: Shows nothing (useful for verification) - **Independent packages**: Skips with message "no upstream to diff against" ## Marking Packages ### Mark as Modified Use this when you make local changes to a package (backports, custom patches, etc.): ```bash ./ci/dist_git.py mark-modified --modified \ --reason "Brief explanation of why" ``` Examples: ```bash # After backporting a patch ./ci/dist_git.py mark-modified gcc --modified \ --reason "Backport CVE-2024-12345 fix from upstream" # After custom spec change ./ci/dist_git.py mark-modified systemd --modified \ --reason "Add custom service unit for Hummingbird" ``` The reason field is **required** and should be concise but descriptive. It helps future maintainers understand why the package can't be auto-updated. ### Mark as Clean Use this to re-enable automatic updates after confirming your changes are no longer needed (e.g., the fix landed in Fedora): ```bash ./ci/dist_git.py mark-modified --clean ``` This removes the `modified` status and allows the package to receive automatic updates from Fedora again. ### Configure Upstream Tracking Use `set-upstream` to configure upstream tracking settings for a package. Each flag independently sets or clears one metadata field. At least one flag is required; omitted flags leave their fields untouched. ```bash ./ci/dist_git.py set-upstream [flags] ``` | Flag | Sets field | Clears with | | ------------------------ | ------------------------------------- | -------------------- | | `--track-version latest` | `track_upstream: "latest"` | `--no-track-version` | | `--track-version VER` | `track_upstream: "VER"` | `--no-track-version` | | `--project-id ID` | `release_monitoring_project_id` (int) | `--no-project-id` | | `--project-id NAME` | `release_monitoring_project_id` (str) | `--no-project-id` | Each set/clear pair is mutually exclusive (can't pass `--track-version` and `--no-track-version` together). **Examples:** ```bash # Enable upstream version tracking (any version) ./ci/dist_git.py set-upstream bash --track-version latest # Disable upstream version tracking ./ci/dist_git.py set-upstream bash --no-track-version # Set upstream name with version constraint for versioned packages ./ci/dist_git.py set-upstream golang1.26 --project-id golang --track-version 1.26 # Set release-monitoring.org project ID (integer) ./ci/dist_git.py set-upstream python3.11 --track-version 3.11 --project-id 13254 # Remove project ID (reverts to RPM name lookup) ./ci/dist_git.py set-upstream python3.11 --no-project-id # Combine multiple flags in one call ./ci/dist_git.py set-upstream golang1.26 \ --track-version 1.26 --project-id 13254 ``` **Metadata fields:** | Field | Description | Example | | ------------------------------- | ----------------------------------------------------------------- | ---------------------------------------- | | `track_upstream` | `"latest"` or version prefix to constrain updates | `"latest"`, `"1.26"` | | `release_monitoring_project_id` | Anitya project ID (int) or upstream name (str) | `13254`, `"golang"` | | `version_suffix_strip` | Suffix to strip from Anitya-reported versions | `"-RELEASE"` | | `upstream_version_transform` | Named transform from Anitya version to RPM scheme | `"openjdk_to_rpm"` | | `source_availability_check` | Named source checker to HEAD-probe before selecting a version | `"openjdk_osci"` | These fields affect two systems: - **`dist_git.py update`**: When `track_upstream` is a version prefix, skips upstream versions that don't match. For example, `track_upstream: "1.26"` allows `1.26`, `1.26.0`, `1.26.3` but rejects `1.27.0`. - **`check_upstream_versions.py`**: When `release_monitoring_project_id` is an integer, queries the v2 API directly by Anitya project ID. When it is a string, queries release-monitoring.org using that name instead of the RPM package name (e.g., looks up `golang` instead of `golang1.26`). When absent, uses the RPM package name. When `track_upstream` is a version prefix, filters the reported upstream versions to only those matching the prefix. Only packages with `track_upstream` set are checked by `check_upstream_versions.py check` when no explicit package arguments are given. Find release-monitoring.org project IDs by searching on . The project ID can be combined with a version prefix to filter versions returned by the project ID lookup: ```json { "release_monitoring_project_id": 13254, "track_upstream": "3.11" } ``` Some upstream projects tag releases with a suffix that is not part of the RPM version (e.g., `swift-6.3.3-RELEASE`). After Anitya strips the version prefix, the reported version still contains the suffix (`6.3.3-RELEASE`), which is incompatible with RPM's `Version:` field (hyphens are not allowed). Use `version_suffix_strip` to remove it before comparison and update: ```json { "release_monitoring_project_id": 21267, "version_suffix_strip": "-RELEASE" } ``` Some upstream projects publish source tarballs independently of tag creation, and Anitya may report a version before the tarball is available. Use `source_availability_check` to name a checker function (registered in `SOURCE_AVAILABILITY_CHECKERS` in `check_upstream_versions.py`) that HEAD-probes the source URL before selecting a version. Versions whose source returns 404 are skipped in favour of the next available version. Available checkers: - `openjdk_osci` — probes `https://openjdk-sources.osci.io/openjdk{feature}/openjdk-{version}.tar.xz` ```json { "release_monitoring_project_id": 369281, "track_upstream": "21", "upstream_version_transform": "openjdk_to_rpm", "source_availability_check": "openjdk_osci" } ``` ### `version_from_ref` (fixed-ref, no-tagged-release packages) Most gorget `source-pipeline.yaml` fetch steps are version-templated (`ref: "v${VERSION}"`), so `dist_git.py update` can invoke gorget with `metadata/.json`'s own `version` directly. A few packages instead pin a fixed git-snapshot commit with no tagged upstream release at all (gcc, glibc, libyuv, php-patchwork-jsqueeze, vim) -- their `version` field doesn't change when the pinned commit does, so it can't drive gorget's `--version` argument. `version_from_ref` tells `dist_git.py update` how to derive gorget's `--version` string from a newly-pinned commit, instead of requiring someone to compute it by hand every time (the previous process, documented as a manual step in each such package's `source-pipeline.yaml` comments). Currently one `type` is supported: - **`commit-date`** (used by gcc): `"-"`, where the date comes from `git log -1 --format=%cd --date=format:%Y%m%d ` run against the newly-pinned commit. ```json { "version_from_ref": {"type": "commit-date"} } ``` **Currently gcc-only.** `dist_git.py update` only fully automates a fixed-ref pin refresh for gcc's simple case (a full 40-char commit SHA lives directly in a spec `%global`). The other four packages pin via a `git describe`-style string (e.g. glibc's `%{glibcsrcdir}` macro, `glibc-2.43-47-gbc95068f5f`) whose trailing hash is only a 10-character abbreviation -- resolving that back to a full SHA needs a local clone containing the commit object, real extra network work with its own failure modes (ambiguous hashes, upstream unavailability), not yet implemented (see HUM-4621's "Remaining work"). For those four, `dist_git.py update` detects when the pipeline's pinned `ref:` no longer matches the merged spec and refuses to auto-commit -- it's routed to a draft, `no-test`-labeled MR for manual resolution instead, the same way a real git merge conflict is, rather than silently committing a `sources` file gorget was never actually run against. ## Per-Package Update Hooks When `check_upstream_versions.py check --update` updates a package, by default it sets `Version:` to the new upstream version and `Release:` to `0.1%{?dist}` (unless `%autorelease` is used), adds a changelog entry, and downloads new sources from the URLs declared in the spec. Some packages need custom logic (e.g. generating stripped tarballs or patching macro-based version lines). A per-package hooks file lets you override or extend these default phases without changing `check_upstream_versions.py` itself. ### Hooks file location ```text metadata/.update-hooks.yaml ``` For example, `metadata/nodejs25.update-hooks.yaml`. ### Hook phases The YAML file supports three optional keys. Each value is a shell command string executed with `bash -eo pipefail -c` in the package directory as the working directory. | Phase | Behaviour | | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `update_spec` | **Replaces** the default update that sets `Version:` to the new upstream version and `Release:` to `0.1%{?dist}`. A changelog entry is still added automatically. | | `download_sources` | **Replaces** the default URL-based source download. Must print one filename per line to stdout for files to upload to the lookaside cache. Redirect any other output to stderr (`>&2`). | | `post_update` | **Additive** — runs after spec + sources are ready. No default equivalent. | Omitting a phase means the default logic runs for that phase. Packages without a hooks file behave identically to before. Unknown phase keys in the YAML cause a `ValueError` (fail-fast). ### Environment variables Every hook receives these environment variables: | Variable | Example | | --------------------- | ---------------------------------------- | | `UPDATE_PACKAGE` | `nodejs25` | | `UPDATE_OLD_VERSION` | `25.6.1` | | `UPDATE_NEW_VERSION` | `25.8.2` | | `UPDATE_SPEC_FILE` | `/home/rpms/rpms/nodejs25/nodejs25.spec` | | `UPDATE_PACKAGE_DIR` | `/home/rpms/rpms/nodejs25` | | `UPDATE_SOURCES_FILE` | `/home/rpms/rpms/nodejs25/sources` | | `UPDATE_ROOT_DIR` | `/home/rpms` | ### Example See [`metadata/nodejs25.update-hooks.yaml`](../../../metadata/nodejs25.update-hooks.yaml) for a working example that uses all three hook phases. ## How Auto-Updates Work The `./ci/dist_git.py update` command (used by automation) checks modification status before updating packages: - **clean packages**: Updated automatically when new Fedora versions are available - **modified packages**: Automatically merged with upstream changes (conflicts create draft MRs) - **independent packages**: Update blocked (not sourced from Fedora) - **version-constrained packages**: Skipped if upstream version doesn't match `track_upstream` prefix To force-update a modified package (discarding local changes): ```bash ./ci/dist_git.py sync ``` The `sync` command bypasses the modification check and force-updates to the latest upstream version. After syncing, the package is automatically marked clean. ## Resolving Merge Conflicts When modified packages are updated from Fedora, `dist_git.py update` attempts to automatically merge local changes with the new upstream version using git's three-way merge. When conflicts occur, the update still succeeds but creates a commit with conflict markers, and the automation files a draft merge request labeled with `CONFLICT:` for manual resolution. ### Understanding Conflict Markers Git uses this conflict marker structure: ```text <<<<<<< HEAD Fedora's version (new upstream) ======= Hummingbird's local modifications >>>>>>> hummingbird-local ``` **ALL THREE markers must be removed** for a clean resolution. ### Update branch/MR structure MRs are created on branches following the pattern: ```text chore/dist-git-update-PACKAGENAME ``` These branches are automatically created by the `dist_git_update` GitLab schedule. If they have conflicts, they result in draft MRs with: - Title prefix: `CONFLICT: chore(rpms): Update ...` - Description listing the conflicting files - `no-test` label to skip CI tests (saves resources since conflicts need manual resolution) ### Resolution Process 1. **Check out the conflict branch:** ```bash git fetch origin git checkout origin/chore/dist-git-update-PACKAGENAME ``` 2. **Examine the conflict:** ```bash # Find all files with conflict markers git grep -nE '^<{7} .+|^={7}$|^>{7} .+' -- rpms/PACKAGENAME/ # View the specific conflict git show HEAD:rpms/PACKAGENAME/PACKAGENAME.spec | grep -B5 -A10 "^<<<<<<< HEAD" ``` 3. **Understand the local changes:** ```bash # Review commit history to understand why changes were made git log --oneline -- rpms/PACKAGENAME/ git log -p -- rpms/PACKAGENAME/ # With diffs # Check the modification reason jq -r .modification_reason metadata/PACKAGENAME.json ``` Understand the context for correct resolution: - What was the original purpose of the local change? Is it transient or permanent? - Is it a workaround for a bug, a security patch, or a configuration difference? - Does it affect other packages (e.g., nss builds nspr as a subpackage)? - Check spec file comments (e.g., NOTE: comments) for packaging details Decide which version to accept: - **Accept HEAD (Fedora)** for: release number lags, fixed workarounds that Fedora improved or addressed differently - **Keep hummingbird-local** for: security patches not in Fedora, FIPS requirements, critical fixes, and other permanent modifications - **Merge both** for: test skip lists, independent changes that don't conflict logically - **When in doubt:** Accept Fedora's version for packaging metadata (Release:, subpackage versions), keep Hummingbird's version for functional changes (patches, dependencies, build options) 4. **Resolve the conflict:** Edit the file to choose the appropriate version (HEAD, hummingbird-local, or merge both). Verify no markers remain: ```bash git grep -nE '^<{7} .+|^={7}$|^>{7} .+' -- rpms/PACKAGENAME/ ``` 5. **Validate the resolution:** Check that local modifications are preserved: ```bash # Check the diff against upstream (works on working tree, staging not required) ./ci/dist_git.py diff PACKAGENAME # Compare with previous modification commits to verify git log -p -- rpms/PACKAGENAME/ ``` The diff should show only the intended local modifications (ignoring Release: bumps). This confirms the merge preserved your changes correctly. Note: `dist_git.py diff` compares the filesystem working tree against upstream, so it works before or after staging. 6. **Amend the commit:** Record the original conflicted commit SHA, then amend: ```bash # Record the original conflicted commit SHA ORIGINAL_SHA=$(git rev-parse HEAD) # Stage the resolved files and amend the commit git add rpms/PACKAGENAME/ git commit --amend -m "$(git log -1 --format=%B | head -n -1) Conflicted-Update: $ORIGINAL_SHA" ``` This preserves the original commit message while adding a `Conflicted-Update:` trailer that records which commit contained the conflict markers. This helps track the resolution history and can be useful for auditing or debugging later. 7. **Push the resolution:** ```bash git push origin HEAD:chore/dist-git-update-PACKAGENAME --force-with-lease --push-option merge_request.unlabel=no-test ``` This removes the `no-test` label from the MR, which triggers CI tests to run and verifies the resolution works correctly. Some developers might have `origin` as read-only remote, and a different writable remote (e.g. `originw`). ### Common Conflicts #### nss: Subpackage Release Numbers The `nss` package builds `nspr` as a subpackage with its own release number offset. **BACKGROUND:** - `nss` builds both `nss` and `nspr` RPMs from the same source - `nspr_release` uses an offset (`%[%baserelease+n]`) to avoid NVR clashes - The spec file NOTE explains: reset to 1 when `nspr_version` changes, increment when only `nss` changes - Fedora manages these offsets in their ecosystem to prevent conflicts **CONFLICT EXAMPLE:** ```text <<<<<<< HEAD %global nspr_release %[%baserelease+3] ======= %global nspr_release %[%baserelease+1] >>>>>>> hummingbird-local ``` **REASONING:** When updating to a new upstream `nss` version from Fedora: - Accept Fedora's `nspr_release` offset (HEAD) - they manage NVR clashes - Our local offset was specific to Hummingbird rebuilds - New upstream version should reset to Fedora's packaging values - Don't try to "calculate" what it should be - trust Fedora's packaging **RESOLUTION:** Accept HEAD (Fedora's value) ## Special Case: Rebuild-Only Changes Release-only changes (no-change rebuilds) are automatically ignored by the modification detection logic. This means: - Bumping `Release: 3%{?dist}` → `Release: 3.1%{?dist}` does **not** mark the package as modified - The package can still receive automatic Fedora updates - The Release bump will be preserved if the update doesn't change the upstream Release field You **do not need** to mark packages as modified for rebuild-only changes, unless you want to explicitly prevent automatic updates for other reasons. ## CI Validation The CI pipeline validates modification status consistency using `make check`, which runs: ```bash ./ci/validate_package_modifications.py --all ``` This validation ensures: 1. All packages have a `modification_status` field 2. The value is one of: `clean`, `modified`, `independent` 3. If `track_upstream` is present, it must be a string (`"latest"` or a version prefix) 4. Modified packages have a `modification_reason` 5. Independent packages do not have source/branch/sha fields (Hummingbird-independent only) 6. Git commit history matches the declared modification status The validation runs on every merge request and push to main, failing the build if metadata is inconsistent. For local development, run the full validation: ```bash ./ci/validate_package_modifications.py --all ``` Or validate specific packages: ```bash ./ci/validate_package_modifications.py bash glibc gcc ``` ### Validation Modes The validation script has two modes: **Fast mode (default)**: Checks git commit history patterns ```bash ./ci/validate_package_modifications.py --all ``` This validates that all commits since the last Sync follow standard patterns (have Upstream: trailers). Runs in less than a minute for all packages. **Thorough mode**: Clones upstream repos and compares filesystems ```bash ./ci/validate_package_modifications.py --all --thorough ``` This performs full filesystem comparisons with upstream Fedora repositories. Slow and unreliable (hundreds of upstream dist-git clones) but authoritative - validates actual state regardless of git commit history. For CI and daily development, fast mode is sufficient. Use thorough mode when: - Debugging discrepancies between metadata and actual state - Auditing the entire repository for hidden modifications - Investigating why a package can't be updated ## Workflow Examples ### Backporting a Patch 1. Add patch file and modify spec (see [Rebuilding Packages](../rebuilding-packages)) 2. Commit the changes 3. **Mark as modified:** ```bash ./ci/dist_git.py mark-modified dnf5 --modified \ --reason "Backport reproducible build fix (upstream PR#2522)" ``` 4. Package is now protected from automatic Fedora updates ### Re-enabling Auto-Updates When your backported fix lands in Fedora: 1. Verify the fix is in the latest Fedora version: ```bash ./ci/dist_git.py update dnf5 # This will fail with "modified" error ``` 2. Mark the package clean: ```bash ./ci/dist_git.py mark-modified dnf5 --clean ``` 3. Update from Fedora: ```bash ./ci/dist_git.py update dnf5 # Now succeeds ``` ### Importing New Packages When importing packages, modification status is set automatically: ```bash # Fedora package → marked as "clean" ./ci/dist_git.py import fedora/neofetch # Hummingbird-independent package → marked as "independent" ./ci/dist_git.py import hummingbird/custom-tool ``` No manual marking needed for imports. ## Troubleshooting ### CI Fails: "Missing modification_status field" This means a metadata file is missing the required field. This means that the package was not imported properly. ### CI Fails: "Marked as clean but package has modifications" The package has local changes but metadata says it's clean. To fix: 1. Check what changed: ```bash git log -p -- rpms// ``` 2. Mark as modified with the appropriate reason: ```bash ./ci/dist_git.py mark-modified --modified --reason "..." ``` ### CI Fails: "Marked as modified but package is actually clean" The package has no local changes but is marked modified. To fix: 1. Verify it's actually clean: ```bash ./ci/dist_git.py update # Check if upstream matches ``` 2. If confirmed clean, remove the modified status: ```bash ./ci/dist_git.py mark-modified --clean ``` ### Update Blocked: "Cannot auto-update ``" This is expected for modified packages. Options: 1. **Wait for fix to land in Fedora**, then mark clean and update 2. **Force-sync** to discard local changes: ```bash ./ci/dist_git.py sync ``` 3. **Keep blocked** if the local changes are still needed ## Related Documentation - [Package Metadata Fields](package-metadata-fields.md) - `modification_status` and `release` configuration - [Rebuilding Packages](../rebuilding-packages) - How to rebuild and backport patches - [Updating Dist-git Packages](../updating-dist-git-packages) - How automatic updates work -------------------------------------------------------------------------------- # Private RPM Repositories url: https://hummingbird-project.io/docs/operating/private-rpm-repositories/ > **AI Agent Note:** When asked to route a package to a private repository, add > `private_product: ` to the package's entry in `ci/package-overrides.yaml`, > then run `make generate` to regenerate resources. The product name must match an > existing RPA entry in `ci/konflux_rpa_config.yml`. ## Overview Hummingbird LTS products (e.g., LTS OpenJDK, LTS DotNet) require subscription-gated RPMs that must not be publicly accessible. Each LTS product gets its own private Pulp repository with independent credentials, enabling per-product subscription enforcement. The private RPM pipeline uses the same build infrastructure as public RPMs but routes packages to separate Pulp domains via dedicated Konflux Applications and ReleasePlanAdmissions (RPAs). ### How it works 1. Each private product has a dedicated **Konflux Application** (e.g., `private--rpms-main`) separate from the public `rpms-main` application. 2. A **ReleasePlan** connects the private Application to its RPA, enabling auto-releases when builds complete. 3. Packages are assigned to a private product via `private_product` in `ci/package-overrides.yaml`. 4. A dedicated **RPA** per product routes those packages to private Pulp repositories. 5. Packages with `private_product` are automatically excluded from the public RPA. ### Architecture ```text package-overrides.yaml konflux_rpa_config.yml : rpas: private_product: - name: ...public... - name: ...private-... private_product: | | v v Component resource ReleasePlanAdmission application: private- targets: private--rpms-main -rpms-main pulp_signed_domain: private-hummingbird- ^ | ReleasePlan application: private--rpms-main releasePlanAdmission: hummingbird-rpms-private- ``` ## Assigning a Package to a Private Product Add `private_product` to the package's entry in `ci/package-overrides.yaml`: ```yaml : private_product: timeout_hours: 8 ``` Then regenerate all resources: ```bash make generate ``` This will: - Set the package's Konflux Component to use `application: private--rpms-main` - Include the package in the private product's RPA component list - Automatically exclude the package from the public RPA ### Verifying the assignment After running `make generate`, verify the changes: ```bash # Check the component's application assignment grep -A5 '-main' konflux-templates/rendered.yml | grep application # Check the package appears in the private RPA grep '-main' releng/hummingbird-rpms-private-.yaml # Check the package is excluded from the public RPA grep '-main' releng/hummingbird-rpms-tech-preview-staging.yaml # (should return no results) ``` ## Onboarding a New Private Product ### Automated onboarding Use the `add-private-product` subcommand to automate Pulp setup, RPA configuration, package assignment, and resource regeneration in a single step: ```bash ./ci/dist_git.py add-private-product \ --packages pkg1,pkg2 \ --infra-repo \ --pulp-config ``` This handles Pulp infrastructure (step 2), RPA config (step 5), package assignment (step 6), infrastructure repo templates (step 4 file creation), and resource regeneration. You still need to manually complete: credentials secret deployment (step 3), merging the infrastructure repo MR created by this tool (step 4), and content guard setup (step 7). Use `--dry-run` to preview changes without modifying anything. ### Manual onboarding Adding a new LTS product (beyond existing ones) requires these steps: service account, Pulp infrastructure, credentials secret, Konflux Application, RPA configuration, resource regeneration, and content guard setup. ### 1. Create a Pulp service account (optional) If you want isolated credentials for private repo operations (recommended), create a dedicated service account before setting up the Pulp infrastructure. See [Pulp Access](pulp-access.md) for instructions on creating a service account, configuring the CLI, and storing credentials in the vault. Otherwise, you can reuse the existing public Pulp credentials. ### 2. Create Pulp infrastructure Use the existing Pulp setup script to create the private domains, RPM repositories, and file repositories (for SBOMs and attestations). See [ci/pulp-setup/README.md](../../ci/pulp-setup/README.md#prerequisites) for the required `pulp-cli-console` plugin install before running these commands: ```bash # Unsigned (staging) - RPM repos ./ci/pulp-setup/create-pulp-resources.sh \ --domain private-hummingbird--unsigned # Unsigned (staging) - file repos (SBOM/attestations) ./ci/pulp-setup/create-pulp-resources.sh \ --domain private-hummingbird--unsigned \ --type file # Signed (production) - RPM repos ./ci/pulp-setup/create-pulp-resources.sh \ --domain private-hummingbird- # Signed (production) - file repos (SBOM/attestations) ./ci/pulp-setup/create-pulp-resources.sh \ --domain private-hummingbird- \ --type file ``` If using a dedicated service account, add `--config ` to each command. This creates per-architecture RPM repositories (`source`, `x86_64`, `s390x`, `ppc64le`, `aarch64`) with distributions, and file repositories (`metadata`, `rpm-catalog`) for SBOM and attestation storage. ### 3. Deploy the Pulp credentials secret If using a dedicated service account (step 1), add a Kubernetes Secret template to the **infrastructure repo** so the credentials are deployed to the Konflux cluster: 1. Create `kubernetes/setup-konflux/47-pulp-private-hummingbird-config-file-secret.yml.j2`: ```yaml --- kind: Secret apiVersion: v1 metadata: name: pulp-private-hummingbird-config-file-secret stringData: cli.toml: «{ lookup_secret("HUMMINGBIRD_PRIVATE_PULP_BOT_CONFIG_FILE[deployed]:cli.toml") | forceescape }» type: Opaque ``` 2. Add an entry to `secrets.yml` for the vault key: ```yaml HUMMINGBIRD_PRIVATE_PULP_BOT_CONFIG_FILE: backend: hv meta: active: true created_at: '' deployed: true ``` The secret name (`pulp-private-hummingbird-config-file-secret`) must match the `pulp_secret_name` in the private RPA configuration (step 5). 3. Add an ExternalSecret to the **konflux-release-data repo** (`rhtap-release-data`) so the credentials are available to the release pipeline. Create the ExternalSecret in the cluster-specific directory (e.g., `tenants-config/cluster/kflux-prd-rh03/managed/rhtap-releng-tenant/`): ```yaml --- apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: name: hummingbird-private-pulp-bot-production-secret annotations: argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true argocd.argoproj.io/sync-wave: "0" spec: dataFrom: - extract: key: releng/konflux/rhtap-releng-tenant/public-network/hummingbird-private-pulp-bot-production refreshInterval: 1h secretStoreRef: kind: SecretStore name: releng-vault target: creationPolicy: Owner deletionPolicy: Delete name: hummingbird-pulp-credentials-private-production-secret ``` Add it to the cluster's `kustomization.yaml` and regenerate auto-generated files with `tenants-config/build-manifests.sh`. 4. Add the secret to the **releng vault** at `https://vault.devshift.net` under the path `releng/konflux/rhtap-releng-tenant/public-network/hummingbird-private-pulp-bot-production` with the following keys: | Key | Value | | --- | --- | | `cli.toml` | Full contents of the Pulp CLI config file | | `expires-on` | `na` | | `owner` | Your Kerberos username | ### 4. Create the Konflux Application The Application resource is managed in the **infrastructure repo**, not this repo. Create it there following the same pattern as `rpms-main`: 1. Create a new directory `kubernetes/private--rpms-main/` in the infrastructure repo. 2. Add `00-application.yml.j2` with the standard Application template: ```yaml --- apiVersion: appstudio.redhat.com/v1alpha1 kind: Application metadata: name: {{ env["PROJECT_NAME"] }} spec: appModelRepository: {url: ""} displayName: {{ env["PROJECT_NAME"] }} gitOpsRepository: {url: ""} ``` 3. Add the new project to the CI matrix in `infrastructure/.gitlab-ci.yml`. Find the `PROJECT_NAME` list under the `konflux-rh03/hummingbird-tenant` context and add the new application name: ```yaml - PROJECT_NAME: - rpms-main - private--rpms-main # add this line PROJECT_CONTEXT: - konflux-rh03/hummingbird-tenant ``` 4. Add `01-release-plans.yml.j2` with a ReleasePlan that references the private RPA. Copy from `kubernetes/rpms-main/01-release-plans.yml.j2` and update the `releasePlanAdmission` label to match the private RPA name: ```yaml metadata: name: hummingbird-rpm-release-private- labels: release.appstudio.openshift.io/auto-release: "true" release.appstudio.openshift.io/standing-attribution: "true" release.appstudio.openshift.io/releasePlanAdmission: hummingbird-rpms-private- spec: application: {{ env["PROJECT_NAME"] }} target: rhtap-releng-tenant # ... copy remaining spec from rpms-main/01-release-plans.yml.j2 ``` The ReleasePlan connects the private Application to its ReleasePlanAdmission. Without it, builds in the private Application will not trigger releases. 5. Copy `10-integration-test-scenarios-testing-farm.yml.j2` from `kubernetes/rpms-main/`. The template is fully parameterized with `{{ env["PROJECT_NAME"] }}` so no edits are needed. This may be needed to prevent the Konflux PR group integration test from blocking MRs that touch private product packages. Merge this MR in the infrastructure repo first — the CI pipeline will deploy the Application, ReleasePlan, and IntegrationTestScenarios to the cluster. Components and RPAs in this repo reference it by name, so the Application must exist before they are applied. ### 5. Add RPA configuration Add a new entry to the `rpas:` list in `ci/konflux_rpa_config.yml`: ```yaml rpas: # Existing public RPA (private packages are automatically excluded) - name: hummingbird-rpms-tech-preview-staging # ... existing config ... # New private product RPA - name: hummingbird-rpms-private- application_prefix: private--rpms release_org: registry.stage.redhat.io/hummingbird-tech-preview single_component_mode: true service_account_name: hummingbird-rpm-release-staging pulp_unsigned_domain: private-hummingbird--unsigned pulp_signed_domain: private-hummingbird- pulp_secret_name: hummingbird-pulp-credentials-private-production-secret pipeline_revision: pipeline_url: https://github.com/scoheb/release-service-catalog.git private_product: component_filter: path_prefix: rpms/ ``` Key fields: | Field | Purpose | | --- | --- | | `name` | Kubernetes resource name for the RPA | | `application_prefix` | Combined with branch to form the Konflux Application name | | `private_product` | Matches the `private_product` value in `package-overrides.yaml` | | `pulp_unsigned_domain` | Pulp domain for unsigned RPMs (staging) | | `pulp_signed_domain` | Pulp domain for signed RPMs (production) | | `pulp_secret_name` | Kubernetes secret containing Pulp publishing credentials | ### 6. Assign packages and regenerate Add `private_product: ` to each package in `ci/package-overrides.yaml`, then regenerate: ```bash make generate ``` This produces: - Updated `konflux-templates/rendered.yml` with per-component application assignments - A new RPA file at `releng/hummingbird-rpms-private-.yaml` - Updated public RPA excluding private packages ### 7. Set up Pulp content guard To restrict access to the private Pulp repositories, a content guard must be configured so only customers with the correct subscription can access the content: 1. Obtain a **SKU** for the private product's subscription offering. 2. Create a **feature** for that SKU in the Feature service. 3. Contact the **Pulp team** to: - Map the organization ID to the feature - Set up the content guard on the private Pulp domain(s) This step is required before customers can access the private repositories. Without it, the repositories are created but have no access control. ### Pulp credentials If using a dedicated service account (step 1), see [Pulp Access](pulp-access.md) for vault storage instructions. If using the shared service account, private RPAs reuse the same Pulp publishing credentials as the public RPA (`hummingbird-pulp-credentials-production-secret`). Per-product access control for customers is handled downstream by Red Hat's subscription entitlement system, not at the Pulp publishing layer. ## Public/Private Routing Packages with `private_product` set in `ci/package-overrides.yaml` are automatically excluded from the public RPA and included only in their product's private RPA. Removing the `private_product` field returns the package to public-only publishing. ## Related Files | File | Purpose | | --- | --- | | `ci/package-overrides.yaml` | Per-package `private_product` assignment | | `ci/konflux_rpa_config.yml` | RPA definitions (public + private) | | `ci/generate_resources.py` | Generates Components, RPAs, and PipelineRuns | | `ci/pulp-setup/create-pulp-resources.sh` | Creates Pulp domains and repositories | | `konflux-templates/macros/releng/release-plan-admission.yml.j2` | RPA template | | `konflux-templates/macros/component.yml.j2` | Component template (per-component application) | | `konflux-templates/macros/image-repository.yml.j2` | ImageRepository template (per-component application) | -------------------------------------------------------------------------------- # Reporting CVE Data Issues url: https://hummingbird-project.io/docs/operating/reporting-cve-data-issues/ ## Overview Sometimes CVE data published on [cve.org](https://www.cve.org/) or [NIST NVD](https://nvd.nist.gov/) is incorrect -- wrong affected package, wrong severity score, wrong affected version range, or missing fix information. When this happens, the incorrect data propagates into vulnerability scanners and can cause false positives (or false negatives) for Hummingbird container images. This document describes how to report these issues to get the data corrected at the source. ## When to Report Report a CVE data issue when you find any of the following: - **Wrong affected package** -- the CVE Record lists a package that is not actually affected - **Wrong version range** -- the affected or fixed version boundaries are incorrect - **Wrong severity** -- the CVSS score or severity rating does not match the actual impact - **Missing fix information** -- a fix exists upstream but the CVE Record does not reflect it - **Wrong CPE match** -- the NVD CPE configuration matches products that are not affected ## How to Report ### Reporting to the CVE Program (cve.org) The CVE Program accepts corrections through the CVE Numbering Authority (CNA) that owns the record. 1. Find the CVE Record at `https://www.cve.org/CVERecord?id=CVE-YYYY-XXXXX`. 2. Identify the CNA listed in the record (shown in the "Assigning CNA" field). 3. Contact the CNA directly to request a correction. Most CNAs accept reports via: - Their security reporting email (often listed in their CNA page) - GitHub issues if the CNA is an open source project 4. If the CNA is unresponsive, use the [CVE Program Request form](https://cveform.mitre.org/) to dispute the record. ### Reporting to NIST NVD NIST NVD enriches CVE Records with CVSS scores and CPE match data. To request corrections: 1. Navigate to the NVD entry at `https://nvd.nist.gov/vuln/detail/CVE-YYYY-XXXXX`. 2. Click the "Contact" link or email `nvd@nist.gov` with: - The CVE ID - The specific field that is incorrect - Evidence of the correct data (links to upstream commits, release notes, etc.) ### Reporting to Red Hat Product Security If the incorrect CVE data is causing issues in Red Hat's vulnerability tracking: 1. File a Jira ticket in the HUM project with the CVE ID and a description of the data issue. 2. Red Hat Product Security can update the VEX feed to reflect the correct disposition for Hummingbird, even before the upstream CVE data is corrected. ## Tracking the Correction After reporting, track the status of the correction: - CVE Record updates are published at `https://www.cve.org/CVERecord?id=CVE-YYYY-XXXXX` - NVD updates appear at `https://nvd.nist.gov/vuln/detail/CVE-YYYY-XXXXX` (NVD may take days to weeks to process corrections) - Update the HUM Jira ticket with the correction status so the team is aware ## Requesting information on the fix status of an existing CVE File a support request at . -------------------------------------------------------------------------------- # Lookaside Cache Access url: https://hummingbird-project.io/docs/operating/lookaside-cache-access/ description: How to access the S3-based lookaside cache for RPM source tarballs ## Overview Source tarballs for RPM packages are stored in an S3-based lookaside cache rather than in git. To upload files to the cache (e.g., when updating a package to a new upstream version), you need AWS credentials with the appropriate permissions. The `ci/upload-to-lookaside-cache.sh` script handles uploads directly, and `ci/check_upstream_versions.py --update` calls it automatically when downloading new source archives. ## Prerequisites You must be a **poweruser** in the `arr-cloud-aws-core` group (`it-cloud-aws-727920394381-poweruser`). Request access to this group if you do not already have it. ## Obtaining AWS Credentials There are two ways to authenticate. ### Option A: Browser-based login Run `aws login`, which opens a browser for authentication: ```bash aws login ``` ### Option B: Kerberos-based login via container Use the CKI tools container to obtain credentials via Kerberos: ```bash $ podman run --rm -it \ -e KRB5CCNAME=FILE:/tmp/krb5cc_$(id -u) \ -v $HOME/.aws:/cki/.aws:U,Z \ quay.io/cki/cki-tools:latest bash-5.2# kinit @IPA.REDHAT.COM bash-5.2# AWS_IDP_URL=https://auth.redhat.com/auth/realms/EmployeeIDP/protocol/saml/clients/itaws \ cki_aws_login --duration 43200 \ --account 727920394381 --role poweruser ``` Replace `` with your Kerberos user ID. This writes credentials to `$HOME/.aws` on the host (bind-mounted into the container). **Note:** This overwrites the AWS default profile credentials. If you use the default profile for other purposes, back up `$HOME/.aws/credentials` before running this command. ## Uploading Files Once credentials are configured in `$HOME/.aws`, you can upload files to the lookaside cache. ### Manual upload ```bash ./ci/upload-to-lookaside-cache.sh -f -p ``` Example: ```bash ./ci/upload-to-lookaside-cache.sh -f rpms/tar/tar-1.35.tar.xz -p tar ``` ### Automated upload via version checker `check_upstream_versions.py --update` downloads new source archives and uploads them to the lookaside cache automatically: ```bash ./ci/check_upstream_versions.py check --update ``` The `check` subcommand only processes packages with `"track_upstream": true` in their metadata when no explicit package arguments are given. To see all packages and their upstream status, use `check_upstream_versions.py list`. See [Package Modification Tracking](package-modification-tracking.md) for how to enable tracking. ## See Also - [Adding Independent Packages](adding-independent-packages.md) - Adding new packages with source tarballs - [Rebuilding Packages](rebuilding-packages.md) - Rebuilding existing packages -------------------------------------------------------------------------------- # Upstream Diff Analysis url: https://hummingbird-project.io/docs/operating/upstream-diff-analysis/ ## Overview The upstream diff analysis tool classifies locally modified RPM packages by how their changes relate to upstream Fedora. This helps the team systematically decide which modifications should be proposed upstream, which are Hummingbird-specific, and which need more complex handling. The primary interface is the `/upstream-diff` Claude Code skill, which orchestrates the full workflow. The skill delegates all deterministic operations (diffing, metadata lookup, caching) to `ci/upstream_diff.py` and applies its own judgment only for the classification step. ## Classification Categories Each modified package is classified into one of five categories: | Category | Meaning | Action | | ---------------------- | -------------------------------------------------------------- | --------------------------------------- | | `upstreamable` | Changes can be submitted to Fedora as-is | File upstream PR | | `hummingbird-specific` | Changes are intentional and specific to Hummingbird | Keep locally, no upstream action needed | | `complex` | Changes need investigation or partial upstreaming | Requires human decision | | `mixed` | Some changes are upstreamable, others are Hummingbird-specific | Split and handle separately | | `no-diff` | Package is marked modified but has no actual diff | Consider unmarking as modified | ## Workflow Invoke the skill in Claude Code with `/upstream-diff`. When called with no arguments, it offers three modes: ### Analyze Prepares the next batch of unanalyzed or stale packages, reviews each diff and any upstream PR activity, then classifies each package into a category with reasoning and a recommended action. Results are cached locally so subsequent runs skip already-analyzed packages. You can also target specific packages by name (e.g., `/upstream-diff analyze bash glibc`). ### View Results Displays the cached analysis as a summary table. Supports filtering by category and showing details for a single package. Use this to get a quick overview of where things stand across all modified packages. ### JIRA Creates or updates a JIRA issue for a specific package to track the upstream proposal. The skill generates a pre-filled description and analysis comment from the cached data, previews it for confirmation, and then files or updates the issue. The JIRA issue key is recorded in the cache so it appears in the view output. ## Script Reference The skill uses `ci/upstream_diff.py` under the hood. The script can also be called directly for automation, scripting, or when working outside Claude Code. Note that `prepare` and `save` are two halves of a classification pipeline: `prepare` gathers raw data (diffs, metadata, upstream PRs, category definitions), but its output requires human or LLM judgment to select a category and write the reasoning fields before calling `save`. The skill's **Analyze** mode bridges this gap automatically. ### Prepare Gather metadata, diffs, upstream PR status, and the classification schema — this is the data the skill's **Analyze** mode feeds into its classification step: ```bash # Specific packages ./ci/upstream_diff.py prepare bash glibc # Next batch of unanalyzed packages ./ci/upstream_diff.py prepare --batch 10 ``` ### Save Store a classification result. The skill's **Analyze** mode calls this after classifying each package: ```bash ./ci/upstream_diff.py save \ --category \ --changes-summary "One-line summary of changes" \ --reasoning "Why this category was chosen" \ --recommendation "Recommended next action" \ --upstream-prs ... # No related upstream PRs ./ci/upstream_diff.py save \ --category hummingbird-specific \ --changes-summary "Custom Hummingbird macros for FIPS" \ --reasoning "FIPS build flags are specific to Hummingbird" \ --upstream-prs "" ``` The `--hummingbird-macros` flag can be added when changes use Hummingbird-specific RPM macros. Results are cached in `.cache/upstream-diff-analysis.json`. ### View Display cached results — this is what the skill's **View Results** mode calls: ```bash # Summary table ./ci/upstream_diff.py view # Single package detail ./ci/upstream_diff.py view # Filter by category ./ci/upstream_diff.py view --category upstreamable # Include unanalyzed modified packages ./ci/upstream_diff.py view --all # Markdown-formatted links (for JIRA or rendered contexts) ./ci/upstream_diff.py view --markdown # Raw JSON ./ci/upstream_diff.py view --json ``` ### Check Upstream PRs Check whether local changes have already been submitted or merged in the upstream Fedora dist-git repository: ```bash ./ci/upstream_diff.py check-prs ... ``` Shows open and recently merged (last 90 days) pull requests from the upstream Fedora dist-git repo. ### Update Related PRs After initial analysis, update which upstream PRs are related to a package's local changes: ```bash # Set related PRs ./ci/upstream_diff.py save --set-upstream-prs ... # Clear related PRs ./ci/upstream_diff.py save --set-upstream-prs ``` ### JIRA Templates Generate pre-filled JIRA content from cached analysis — this is what the skill's **JIRA** mode uses: ```bash ./ci/upstream_diff.py jira-template [--epic HUM-1613] ``` Record a JIRA issue key after creating an issue: ```bash ./ci/upstream_diff.py save --set-jira HUM-XXXX ``` ## Related Documentation - [Package Modification Tracking](package-modification-tracking.md) — how packages are marked modified and how diffs are computed - [Rebuilding Packages](rebuilding-packages.md) — how to rebuild and backport patches -------------------------------------------------------------------------------- # GPG Source Verification url: https://hummingbird-project.io/docs/operating/gpg-source-verification/ description: How upstream signing keys are stored, and how to add, rotate, or revoke them ## Overview Some packages verify the authenticity of their upstream source tarball with a detached OpenPGP signature where upstream signs each release with a private key and publishes a `.asc`/`.sig` alongside the tarball. During source fetching, the [gorget source-pipeline tool][source-pipeline-tool] checks that signature against the project's **public** key before the bytes are ever used in a build. The trusted public keys live centrally in **`metadata/gpg-keys/.gpg`**, one keyring file per upstream project. Centralized storage means: - the full set of signers we trust is auditable in one directory, and - rotating or revoking a key is a single, reviewable commit. The directory is handed to gorget as `--gpg-keys-dir metadata/gpg-keys` (see `ci/check_upstream_versions.py`), so a pipeline's `keyring:` field is just a filename within it. ## How the pieces fit together Three things wire up verification for a package: 1. **The keyring** — `metadata/gpg-keys/.gpg`, the trusted public key(s). 2. **The pipeline step** — a `verify:` entry in `metadata/.source-pipeline.yaml`: ```yaml fetch: - type: url url: "https://curl.se/download/curl-${VERSION}.tar.xz" - type: url url: "https://curl.se/download/curl-${VERSION}.tar.xz.asc" verify: - type: gpg-signature target: "curl-${VERSION}.tar.xz" # the artifact to verify signature: "curl-${VERSION}.tar.xz.asc" # its detached signature keyring: "curl.gpg" # filename in metadata/gpg-keys/ ``` gorget imports `keyring` into a fresh, throwaway GPG homedir per check, then runs the equivalent of `gpg --verify `. A bad or missing signature fails the fetch. 3. **CI validation** — `test/test_gpg_keys.py` (run by `make check`) confirms every file in `metadata/gpg-keys/` is a parseable public key, and that every `keyring:` referenced by a pipeline actually exists. ## Adding a key for a new package Prerequisite: `gpg` (from the `gnupg2` package). It ships in the CI image, run the commands below inside a container, e.g. `podman run --rm -it -v "$PWD:$PWD:z" -w "$PWD" quay.io/hummingbird-ci/gitlab-ci:latest bash`. 1. **Obtain the upstream public key.** Prefer a key you can already trust: many packages already ship the maintainer's key next to their spec (e.g. `rpms/curl/mykey.asc`, `rpms/bash/chet-gpgkey.asc`). Otherwise download it from the project's official key page. 2. **Store it as a keyring in `metadata/gpg-keys/`.** The keyring may be ASCII-armored or binary; this repo standardizes on binary `.gpg`. Convert an armored key with `--dearmor`: ```bash gpg --dearmor < rpms//.asc > metadata/gpg-keys/.gpg ``` Name the file after the upstream **project**, not the RPM (so multiple versioned packages, e.g. `python3.11`/`python3.12`, can share one keyring). 3. **Verify the key is what you expect.** Print its fingerprints and confirm they match the fingerprints published on the upstream's official channel: ```bash gpg --show-keys --with-fingerprint metadata/gpg-keys/.gpg ``` 4. **Wire up the pipeline.** Add (or extend) `metadata/.source-pipeline.yaml` with the `fetch` steps for the tarball + signature and the `verify: [{type: gpg-signature, ...}]` step shown above. 5. **Prove the whole chain end to end** before committing import the keyring into a throwaway homedir and verify a real release signature against it: ```bash V= curl -fsSLO "https:///-$V.tar.xz" curl -fsSLO "https:///-$V.tar.xz.asc" export GNUPGHOME=$(mktemp -d) gpg --import metadata/gpg-keys/.gpg gpg --verify "-$V.tar.xz.asc" "-$V.tar.xz" # expect "Good signature" ``` 6. **Run the checks:** `make check` (validates the keyring and the pipeline reference). ## Rotating or replacing a key Upstream may roll to a new signing key (expiry, policy, new maintainer). Because the trusted set is just files in one directory, rotation is a single commit: 1. Obtain the new public key from the upstream's official channel and confirm its fingerprint out of band. 2. Replace the contents of `metadata/gpg-keys/.gpg` (re-run the `--dearmor` step). If upstream signs a transition period with both keys, you may keep both by importing them into the same keyring: ```bash gpg --dearmor < old-key.asc > metadata/gpg-keys/.gpg gpg --dearmor < new-key.asc >> metadata/gpg-keys/.gpg ``` 3. Re-run the end-to-end verification (step 5 above) against the latest release, then `make check`. 4. Commit with a message recording *why* the key changed and how you confirmed the new fingerprint, this file is the audit trail for what we trust. ## Revoking / removing a key - If a project drops GPG verification, delete both `metadata/gpg-keys/.gpg` and the `verify:` step from its pipeline in the same commit. (CI fails a pipeline that references a missing keyring, and if the key is orphaned the reverse is easy to spot.) - If a key is compromised, remove it immediately and replace it with the upstream's revocation / replacement key. ## Troubleshooting - **`gpg: no valid OpenPGP data found`** — the file isn't a real keyring (empty, truncated, or you saved an HTML error page). Re-download and re-`--dearmor`. `test_gpg_keys.py` catches this in CI. - **`gpg: Can't check signature: No public key`** — the signature was made with a key that isn't in the keyring. Upstream likely rotated keys, follow *Rotating or replacing a key* above. - **`BAD signature`** — the tarball does not match the signature. Do **not** paper over this. It means a corrupted download or, in the worst case, tampering. Re-fetch from the canonical source and if it persists, escalate rather than accepting the artifact. [source-pipeline-tool]: https://hummingbird-project.io/l/source-pipeline-tool -------------------------------------------------------------------------------- # Rebasing the Buildroot to a New Fedora Release url: https://hummingbird-project.io/docs/operating/rebasing-buildroot-to-new-fedora/ description: Step-by-step runbook for moving mock.cfg, the core toolchain, and the rest of the package set to a new Fedora release > **AI Agent Note:** This document is the procedural runbook for a Fedora buildroot rebase (e.g. > F43 -> F44). It was written after executing the F44 rebase (see > [HUM-2018](https://redhat.atlassian.net/browse/HUM-2018) and its child stories HUM-3369, > HUM-3370, HUM-4536, HUM-4537 for the detailed history) and should be followed, and updated, the > next time this operation is performed (e.g. F44 -> F45). Read this whole document before > starting; the ordering between steps matters and several steps have hard dependencies on the one > before it. ## Overview Periodically, Fedora ships a new stable release and Hummingbird needs to move its buildroot (`mock/mock.cfg`) and the packages that track a specific Fedora branch (rather than `rawhide`) onto it. This is a multi-day, multi-phase effort touching the core toolchain, CI/testing configuration, and potentially every package in the repository (since changing the compiler/linker toolchain can affect any build). The F44 rebase took about 2 calendar weeks. ## Pre-flight checks Before starting, confirm: - The target Fedora release is actually stable (not just branched) and its package repos are reachable from the build environment. - Which packages currently track the outgoing release branch explicitly (i.e. `branch` is a versioned Fedora branch like `f43`, not `rawhide`) via: ```bash # List every package whose branch is pinned to a specific Fedora release (not rawhide) ./ci/dist_git.py list | xargs -I{} sh -c 'echo -n "{}: "; python3 -c "import json; print(json.load(open(\"metadata/{}.json\"))[\"branch\"])"' 2>/dev/null | grep -E ': f[0-9]+$' ``` For each, decide up front whether it should move to the new branch, move to `rawhide`, or stay pinned (see "Packages that intentionally stay behind" below) — don't leave this undecided until a package fails to build. - Whether any of those packages carry FIPS-validation-critical modifications. Packages whose `modification_reason` references FIPS certification should generally **not** be rebased to new branch content, even though they still need to be rebuilt against the new toolchain. Moving them would risk invalidating the FIPS-validated source. (See HUM-4989 for a related, more general proposal to record *why* a package is pinned in its metadata rather than relying on the modification reason text.) - Whether any package whose version *is* the "which version is default" signal for an adjacent language/tooling ecosystem (e.g. `python-rpm-macros`) needs the same branch-pinning treatment as the interpreter itself — especially if that ecosystem is mid-transition to a new major version upstream at the same time as this rebase. See "Known gotchas: python3dist / rawhide-tracked toolchain-adjacent packages" below for why this matters and what already went wrong once. - Whether to pause the routine automated dist-git sync bot for the duration of the Step 4 mass rebuild. See "Known gotchas: automation racing a manual mass rebuild" below for the tradeoff. ## Step 1 — Buildroot config + core toolchain (separate MRs) Do **not** bundle the buildroot config change with the toolchain rebuilds in one MR — split them so the buildroot config lands and validates quickly, independent of the (much slower) toolchain rebuilds. 1. **`mock/mock.cfg`**: update `dist`, `releasever`, `bootstrap_image`, `description`, and the `[fedora]`/`[fedora-updates]` repo `baseurl`s and names to the new release. Merge this on its own or bundled only with `fedora-repos` (see next point) and `binutils` if it's a clean (unmodified) package — do not bundle `glibc`/`gcc`/`llvm` here. 2. **Testing/CI fallback config** (`rpms/ci/repos/fedora-43.repo` → `fedora-44.repo`, and `ci/default-tests/tests-rpm.yml`'s repo copy + GPG key import): these reference the buildroot's Fedora version directly, and there's exactly one of each — bump them in the same MR as `mock.cfg` (or immediately alongside it), not later. During the F44 rebase this was deferred and wasn't caught until `php` failed its install test days later in Step 4, against a fallback repo that still pointed at the outgoing release. (This is distinct from the *per-package* test fixture gotcha below, which recurs throughout every step — see "Known gotchas: package-specific test fixtures with hardcoded Fedora-version references.") 3. **`fedora-repos`**: rebase to the new release. If Fedora's spec uses a conditional ELN-style macro for `Release:`, evaluate whether to keep it or hardcode a plain `Release: N%{?dist}` — this is a judgment call each time depending on what's simplest to maintain; document whichever choice is made in the package's `modification_reason`. 4. **`binutils`** (if it carries local CVE backport patches): before assuming the patches still apply, check whether the CVEs they address have already been fixed upstream in the new Fedora version. If so: - Pull Fedora's current spec as the new baseline (don't try to merge patch-by-patch on top of the old spec). - Re-add only the local patches that are **not** already fixed upstream. - For a quick validation build, it's sufficient to confirm the patch set applies and compilation *begins* — you don't need to wait for a full build to complete just to validate the patch set is sane. Kill the build once compilation starts. 5. **`glibc`**: this is the highest-risk package in this phase because of a circular bootstrap dependency (see "Known gotcha: glibc/gcc bootstrap circularity" below). Expect this to require a temporary workaround, tracked as a `modified` package, to be reverted in Step 2. 6. **`gcc`**: reapply any local modification (e.g. disabling non-essential language frontends) against the new spec. Expect this to be one of the longest-running builds in the whole rebase (multiple hours per architecture) — plan CI capacity accordingly. 7. **`llvm`**: rebase and rebuild; this can run in parallel with `gcc` and typically finishes faster. ## Step 2 — Bootstrap cleanup Once the new compiler (from Step 1) is published to Pulp: 1. **Rebuild `annobin`** against the new compiler. This must happen *before* the next step, or reverting the glibc bootstrap workaround will immediately re-trigger the same circular dependency. 2. **Revert the glibc bootstrap workaround** from Step 1, now that the new compiler and annobin are both available. Clear the `modified` status/reason that was recording the workaround. 3. **Rebuild `libtool`** against the new compiler. This is easy to miss (it was missed during the F44 rebase and had to be done out-of-band mid-mass-rebuild) but several packages (`apr`, `audit`, `authselect`, `automake`, `avahi`, `bind`, `catatonit`, `cryptsetup`, `find`, and likely others) will fail to build against the new toolchain without it. **Do this here, not in Step 4.** ## Step 3 — Toolchain-adjacent, release-pinned packages Rebase the remaining packages that were explicitly tracking the outgoing Fedora branch and don't carry a permanent reason to stay behind (see "Packages that intentionally stay behind"). For each: - Decide whether it should move to the new versioned branch (e.g. `f44`) or to `rawhide` — moving to `rawhide` can simplify build-dependency resolution for clean (unmodified) packages, but ties future updates to rawhide's churn. - For packages with local modifications, check whether the modification is still needed (e.g. a security backport that may have landed upstream by now) before reapplying it blindly. - **Check `test/rpms/.yml` for a hardcoded Fedora-version reference** before opening this package's MR (see "Known gotchas: package-specific test fixtures with hardcoded Fedora-version references" below) — fix it in this same MR if present. ## Step 4 — Mass rebuild of everything else Rebuild every remaining package against the new toolchain so that linking, annobin annotations, and compiler hardening are consistent across the whole package set. 1. **Build the exclusion list first**: everything already rebuilt in Steps 1–3, plus anything merged in the hours immediately before starting this step (check recent merge history). Use `dist_git.py rebuild --all`'s `--exclude` option ([HUM-5183](https://redhat.atlassian.net/browse/HUM-5183)): ```bash ./ci/dist_git.py rebuild --all --exclude pkg1,pkg2,... --reason " toolchain rebuild" ``` During the F44 rebase this flag didn't exist yet, so the exclusion list had to be hand-rolled with `./ci/dist_git.py list | grep -v -E '^(pkg1|pkg2|...)$'` piped into `rebuild` — no longer necessary now that `--exclude` has landed. 2. **Watch for packages whose `Release:` field doesn't fit the simple `.N` bump pattern** (e.g. packages using build-time macros for `Release:`) — these will fail to commit cleanly and need to be handled in a separate pass. 3. **Submit in batches**, not one giant MR wave — use `./ci/rebuild_multi_mr.sh --dry-run` to preview, then `./ci/rebuild_multi_mr.sh --max-updates=N` to submit and auto-merge-on-green in controlled increments. Plan for several days; a run of ~460 packages took about a week of elapsed time across 5 batches during the F44 rebase, gated by CI/Testing Farm throughput more than build time itself. 4. **Check `test/rpms/.yml` for a hardcoded Fedora-version reference before each package's rebuild MR merges** (see "Known gotchas: package-specific test fixtures with hardcoded Fedora-version references" below) — this is what let `php`'s failure slip through undetected for days during the F44 rebase, so don't rely on remembering it only once at the end. 5. **Expect a small number of packages to need bespoke fixes** that a routine rebuild can't resolve automatically — license metadata regressions surfaced by newer scanners, compiler/linker regressions in vendored dependencies (e.g. a `bindgen` version incompatible with a newer clang), confirmed upstream compiler-triggered bugs in JIT/JVM-style software, or stale version-gates in local patches that assumed an older toolchain. Budget time for a handful of these near the end of the mass rebuild. ## Known gotchas ### glibc/gcc bootstrap circularity A new Fedora release's `glibc` spec may use a feature of the new `gcc` (e.g. a new compiler flag), while the new `gcc` may in turn require symbols only present in the new `glibc`. If Hummingbird's own Pulp repos still serve the *old* toolchain at a higher priority than the new Fedora repos during the transition window, this becomes a real chicken-and-egg problem, not just a theoretical one. The pragmatic fix used for F44 was to drop the new-toolchain-specific flag from `glibc` temporarily (if it's a defensive/hardening-only flag whose absence doesn't break correctness), rebuild `glibc` and `gcc` against each other with it disabled, then re-enable it once both are published and `annobin`/`libtool` have also been rebuilt against the new compiler (Step 2 above). ### Package-specific test fixtures with hardcoded Fedora-version references Beyond the single global fallback config bumped in Step 1, some packages carry their own `test/rpms/.yml` test suite that independently hardcodes a Fedora repo/GPG-key reference for setting up a test chroot (during F44 this was `glib2`, `hummingbird-release`, `ca-certificates`, `crypto-policies`, and `openssl`). Don't wait for a one-time catch-all audit to find these — fix each one in the same MR as that package's own rebuild/update, whenever it happens to land during the rebase (Step 1, 3, or 4, whichever touches that specific package — see the reminders in each of those steps above). Relying on an exhaustive upfront sweep of every `test/rpms/*.yml` is fragile and is exactly what let `php`'s failure slip through undetected for days during the F44 rebase — its own MR was fine, but a *different* package's stale GPG key/repo reference broke shared test infrastructure `php` depended on. When you open the rebuild/update MR for any package, grep `test/rpms/.yml` for the outgoing release string first. See [HUM-5184](https://redhat.atlassian.net/browse/HUM-5184) for tracking a structural fix (single source of truth or a CI lint check) so this class of drift is caught automatically instead of relying on either of the above being remembered. ### Automation racing a manual mass rebuild The routine automated dist-git sync bot keeps running on its normal schedule regardless of an in-progress manual mass rebuild. If both touch the same packages in the same window, you'll get a wave of merge-conflicted bot MRs that need manual cleanup — during F44 this was roughly 70 MRs at once. Decide before starting Step 4 (see "Pre-flight checks" above) whether to pause routine sync automation for the duration of the mass rebuild, or accept the cleanup cost. ### External CI capacity constraints Large rebuild waves can coincide with unrelated capacity crunches on shared infrastructure (e.g. Testing Farm). These are outside Hummingbird's control but can stall a batch for a day; check external status pages when a batch is unexpectedly slow (for reference, the F44 rebase hit ). ### python3dist / rawhide-tracked toolchain-adjacent packages If any packages central to a specific language ecosystem's "what version is default" (e.g. `python-rpm-macros`) are tracking `rawhide` rather than a pinned branch, a routine automated sync during this window can silently pull in an experimental version bump (e.g. Fedora bootstrapping the *next* language version) that corrupts auto-generated package metadata across dozens of downstream packages without any build failing immediately. See HUM-4988 and HUM-4989 for the full incident report and proposed guardrails from the F44 rebase. Decide during "Pre-flight checks" above whether any "default version" declaration packages need the same branch-pinning treatment as the interpreter itself before you start. ## Packages that intentionally stay behind Not every package needs to move to the new branch. Document (in `modification_reason` and/or the epic tracking the rebase) any package that's staying pinned to the old branch, and why — otherwise this looks like unfinished work to the next person. Reasons seen so far: - **FIPS validation**: the package carries a modification tied to FIPS certification, and moving to newer upstream content risks invalidating that certification. The package is still rebuilt against the new toolchain — it just doesn't pick up newer Fedora source content. - **Package no longer exists upstream**: Fedora sometimes drops a package entirely in a new release (e.g. an older LTS language runtime superseded by a newer one). If Hummingbird still needs to carry it, it has no newer branch to rebase to and simply stays on its last available branch, rebuilt against the new toolchain. ## Reference: commands used during the F44 rebase These are shown verbatim from the F44 rebase for concreteness. When reusing them for the next rebase (e.g. F45), remember to substitute the actual target release everywhere `F44` appears below — including the exclusion list of already-rebuilt packages, which will differ each time. ```bash # Preview a rebuild plan across all packages ./ci/rebuild_multi_mr.sh --dry-run # Submit rebuild MRs in controlled batches, auto-merge on green ./ci/rebuild_multi_mr.sh --max-updates=50 # Rebuild everything except a known set of already-handled packages, using the # --exclude flag (HUM-5183, landed after the F44 rebase — see Step 4 above) ./ci/dist_git.py rebuild --all --exclude glibc,gcc,llvm,annobin,binutils,fedora-repos,python3.13,python-gitlab --reason "F44 toolchain rebuild" ``` During the F44 rebase itself, `--exclude` didn't exist yet, so the equivalent exclusion list had to be hand-rolled: ```bash ./ci/dist_git.py rebuild $(./ci/dist_git.py list | grep -v -E '^(glibc|gcc|llvm|annobin|binutils|fedora-repos|python3.13|python-gitlab)$') --reason "F44 toolchain rebuild" ``` ## Related tickets - [HUM-2018](https://redhat.atlassian.net/browse/HUM-2018) — F44 buildroot rebase epic (full history) - [HUM-3369](https://redhat.atlassian.net/browse/HUM-3369) / [HUM-3370](https://redhat.atlassian.net/browse/HUM-3370) / [HUM-4536](https://redhat.atlassian.net/browse/HUM-4536) / [HUM-4537](https://redhat.atlassian.net/browse/HUM-4537) — Steps 1-4, each with a detailed what/why/fix writeup - [HUM-4988](https://redhat.atlassian.net/browse/HUM-4988) / [HUM-4989](https://redhat.atlassian.net/browse/HUM-4989) — python-rpm-macros/python3dist regression incident and guardrails - [HUM-5182](https://redhat.atlassian.net/browse/HUM-5182) — `bump_release()` double-suffix bug - [HUM-5183](https://redhat.atlassian.net/browse/HUM-5183) — `dist_git.py rebuild --exclude` option (implemented) - [HUM-5184](https://redhat.atlassian.net/browse/HUM-5184) — generalize the testing-infra Fedora-version drift fix -------------------------------------------------------------------------------- # Lot's of background information url: https://hummingbird-project.io/docs/background/ Background and explanation documentation for understanding Project Hummingbird systems and concepts. -------------------------------------------------------------------------------- # RPMs repository url: https://hummingbird-project.io/docs/background/rpms/ description: Background documentation and contributing guide for Project Hummingbird RPM packages. Thank you for your interest in contributing! This repository contains RPM packaging, minimal tests, and CI plumbing used by Hummingbird to build, lint, and validate RPMs in containers and Testing Farm. This guide is adapted from the Hummingbird containers contribution guide and aligned with our workflows and tooling. See the original for broader context: [Hummingbird Containers CONTRIBUTING](https://gitlab.com/redhat/hummingbird/containers/-/blob/main/CONTRIBUTING.md). ## Code of Conduct Be respectful and constructive. By participating, you agree to uphold a professional and inclusive environment. ## Repository layout - `rpms//` – RPM dist-gits (spec, sources), mostly auto-imported - `ci/` – helper scripts and default tests - `build_rpms.sh` – build RPMs using Mock in the Konflux-compatible container - `run_tests_rpm.sh` – run rpmlint and install/rebuild tests in a container - `default-tests/tests-rpm.yml` – default tests included for all packages - `run_tests_rpm.fmf` – tmt/Testing Farm entry point - `konflux-templates/` – Konflux/PAC resources - `mock/` – mock configuration - `test/rpms/.yml` – package-specific tests ## Prerequisites - Fedora or RHEL-like environment - Podman - rpmlint - tmt (optional, for local TF-style runs) - jq, python3, python3-yaml (used by `run_tests_rpm.sh`) ## Build locally Build a package's SRPM and RPMs using the Konflux-aligned environment: ```bash ./ci/build_rpms.sh # Results are written to: /tmp/konflux-build--*/results/ ``` ### Building in Lima VM When building inside a Lima VM (macOS users), use the `--build-dir` flag to specify a directory on the VM's native filesystem. This avoids permission issues with mock's bootstrap process on macOS mounts: ```bash limactl shell fedora bash -c 'cd /path/to/repo && ./ci/build_rpms.sh --build-dir /tmp/rpm-build ' ``` The build directory must be on the VM's native filesystem (not a macOS mount) to ensure Linux file ownership, permissions, and symlinks work correctly with `/var/lib/mock`. ### Interactive repository debugging To investigate package/dependency/installability issues, you can run an interactive shell in the same environment used by the package builds: ```bash ./ci/build_rpms.sh --shell-before setup ``` This will show you the `mock` command that would be used to build the `setup` package, and drop you into a shell in the same environment before executing the build, with `dnf` available. You can then run the command manually and inspect the environment, repositories, and package metadata. If you want it to actually build the local package first, so that you can include it in your investigation, use `--shell-after` instead: ```bash ./ci/build_rpms.sh --shell-after setup ``` ## Test locally (containerized) Run the default and package-specific tests against one or more built RPMs: ```bash # Basic usage (binary rpm) ./ci/run_tests_rpm.sh --rpm /path/to/pkg-1.2-1.fcXX.x86_64.rpm # Include source RPM tests ./ci/run_tests_rpm.sh \ --rpm /path/to/pkg-1.2-1.fcXX.x86_64.rpm \ --src-rpm /path/to/pkg-1.2-1.fcXX.src.rpm \ # Test all RPMs in the build output directory after build_rpms.sh ./ci/run_tests_rpm.sh $(printf -- '--rpm %s ' builds//RPMS/*.rpm) \ --repo-dir builds//RPMS/ \ --src-rpm builds//SRPMS/*.src.rpm \ ``` Notes: - Tests run inside a Podman container. - The test image defaults to `quay.io/hummingbird/core-runtime:latest-builder`. You can override via `TEST_IMAGE`: ```bash TEST_IMAGE=quay.io/hummingbird/core-runtime:specific-tag ./ci/run_tests_rpm.sh --rpm /path/to/pkg.rpm ``` - Container execution is performed as root with `HOME=/root` to avoid XDG state permission issues in dnf5. ## Test in tmt/Testing Farm locally You can drive the same FMF test locally with tmt. The FMF test expects an OCI artifact that contains RPMs, referenced via `IMAGE_URL` (Testing Farm sets this automatically). For local trials you can point to any compatible OCI artifact or skip the ORAS pull logic and directly call the script. ```bash # If using an OCI artifact with RPMs IMAGE_URL="oci://registry/namespace/artifact:tag_or_digest" tmt run -a # Or run the script directly with built RPMs (bypassing the FMF wrapper) ./ci/run_tests_rpm.sh --rpm /path/to/pkg.rpm --src-rpm /path/to/pkg.src.rpm ``` If you need longer time in Testing Farm, the FMF test includes `duration`, which you can adjust in `ci/run_tests_rpm.fmf`. ## Dist-git Imports The `ci/dist_git.py` tool imports Fedora/CentOS dist-git packages into the `rpms/` directory. We expect most packages to not have (permanent) Hummingbird specific changes, so most of them will keep syncing with upstream dist-gits. In most cases that will be Fedora rawhide, but for some packages we may pick a different upstream, e.g. stable Fedora or even CentOS Stream. The status of all imports is tracked in [imports.json](./imports.json). Active Fedora releases are tracked in [upstream-releases.json](./upstream-releases.json), which is updated from the Bodhi API. Note that `rawhide` is automatically resolved to the highest numbered Fedora release at runtime and is not stored in the JSON file. See `./ci/dist_git.py --help` for all available options. Some examples: - Update upstream-releases.json from Bodhi API (should be done periodically, e.g., when new Fedora versions are released): ```bash ./ci/dist_git.py update-releases ``` This fetches the latest Fedora releases from Bodhi. The `rawhide` branch automatically resolves to the highest numbered Fedora version (e.g., if f43 and f44 are available, rawhide uses f44). - Import a new package. This requires specifying the dist-git URL (with `fedora/` being a shortcut for the Fedora dist-git URL) and optionally a branch (default: rawhide): ```bash ./ci/dist_git.py import fedora/bash ./ci/dist_git.py import --branch f42 fedora/glibc ./ci/dist_git.py import --branch c10s https://gitlab.com/redhat/centos-stream/rpms/postfix.git ``` - Update all or a single package: ```bash ./ci/dist_git.py update ./ci/dist_git.py update bash ``` This only imports changes if these were actually built in Koji, to ensure we only import changes which are meant to be released. You can disable this check with `--skip-build-check`. - Re-sync a package to upstream, discarding any local modifications. We use this after Fedora adopted our change, or it is no longer relevant: ```bash ./ci/dist_git.py sync bash ``` All of these commands automatically commit changes with descriptive commit messages including the upstream SHA. To avoid that, you can use the `--dry-run` option. - Enable upstream version tracking for a package (used by `check_upstream_versions.py check`). These commands modify the metadata file but do not create a git commit: ```bash ./ci/dist_git.py set-upstream bash --track ./ci/dist_git.py set-upstream bash --no-track ``` - Check upstream version status. The `check` subcommand checks tracked packages; `list` shows all packages in a table: ```bash ./ci/check_upstream_versions.py check ./ci/check_upstream_versions.py list ./ci/check_upstream_versions.py list --json ``` ## Package-specific overrides Per-package build configuration can be customized in `ci/package-overrides.yaml`. If a package is not listed, it uses default settings. Available options: | Option | Description | Default | | ----------------- | ------------------------------------------------------- | ----------------- | | `timeout_hours` | Build timeout in hours | 4 | | `build_platforms` | List of MPLs (instance sizes) for multi-platform builds | Pipeline defaults | Example configuration: ```yaml # Long-running package with custom timeout setup: timeout_hours: 12 # Package requiring larger build instances llvm: timeout_hours: 12 build_platforms: - "linux-d160-c8xlarge/arm64" - "linux-d160-c8xlarge/amd64" ``` All available MPLs can be found in the [Konflux multi-platform builds documentation](https://konflux.pages.redhat.com/docs/users/getting-started/multi-platform-builds.html). After modifying overrides, regenerate the pipeline files: ```bash make generate ``` This updates `.tekton/rpms-on-push.yaml` and `.tekton/rpms-on-pull-request.yaml` with the new configuration. ## Branching and pull requests - Create feature branches from the default branch. - Keep edits focused and small; separate unrelated changes into separate PRs. - Include meaningful commit messages (why + what). Reference related issues if applicable. - PRs must pass CI (build + tests). Fix lint/test failures or mark known failures properly. ## Commit message conventions - First line: short imperative summary (≤ 72 chars) - Body (optional): context, rationale, and user/ops impact - Reference issues using standard notation (e.g. "Fixes: #123") ## Spec guidelines - Don't use `%changelog` for Hummingbird specific changes. It's just a point of conflict with Fedora imports, commit messages are good enough. - Hummingbird specific changes increase `Release:` number in steps of 0.1 (e.g. 2 → 2.1 → 2.2). This avoids colliding with Fedora's Release number namespace and allows us to sync again to a future -3. ## Packaging and testing guidelines - Spec files should be reproducible and minimal. - Prefer pinned container digests for CI images where feasible. If a tag and digest are both present (`name:tag@sha256:`), the digest is authoritative for content selection. - Default tests in `ci/default-tests/tests-rpm.yml` must be fast, deterministic, and safe for all packages. - Package-specific tests (`rpms//tests-rpm.yml`) can override or extend defaults. Keep them bounded in runtime and dependencies. - When possible, capture flaky or environmental issues under `known_issues` in test YAML with clear matching patterns and descriptions. ## Style - Shell: bash with `set -euo pipefail`, readable variable names, and early returns where reasonable. - YAML: consistent indentation and quoting; prefer explicitness over magic. - Comments: add when necessary to explain non-obvious rationale; avoid restating the code. ## Security and supply chain - Avoid embedding credentials or secrets; use environment variables or CI secret stores. - Prefer pulling images by digest for stability and repeatability in CI. - Validate inputs and sanitize any paths used by scripts. ## Resolving CVEs When *manually* fixing a CVE (e.g., adding a patch or performing a manual update), a `Fixes: CVE-YYYY-XXXX` must be added to the MR description and/or a commit. This will indicate to the build system that a CVE fix has been submitted for review. ## Reporting issues Open an issue describing the problem, reproduction steps, and environment. Attach logs (build/test) when possible. If the failure is intermittent, call that out explicitly. ## Licensing Ensure files include appropriate licenses and that any third-party content is compatible with the project's license. ## Thank you Your contributions make the Hummingbird RPMs better for everyone. We appreciate your time and feedback! -------------------------------------------------------------------------------- # RPM Pipeline url: https://hummingbird-project.io/docs/background/rpms/rpm-pipeline/ description: How an RPM spec change flows through the build pipeline to the package repository Explanation of the RPM build pipeline, from spec file changes through validation, building, signing, and publishing to the Hummingbird package repository. ## Overview The RPM pipeline consists of five main stages: 1. **Spec Change** - A package spec file is modified in `rpms//` 2. **Merge Request & Validation** - CI validates the change 3. **Build** - Konflux builds RPMs per architecture via mock in hermetic mode 4. **Signing** - Built RPMs are cryptographically signed 5. **Publishing** - Signed RPMs are uploaded to the Hummingbird Pulp repository After publishing, updated RPMs are picked up by the [container image pipeline][image-pipeline] via lockfile updates. [image-pipeline]: https://hummingbird-project.io/l/image-pipeline ```mermaid flowchart TD A["Spec Change
rpms/<package>/"] --> B["Merge Request"] B --> C["CI Validation
(check, tree_status)"] C --> D["Merge to main"] D --> E["Konflux RPM Build
Tekton PipelineRun per package
mock hermetic build per arch"] E --> F["RPM Signing
Kerberos-based"] F --> G["Publish to Pulp
packages.redhat.com
/public-hummingbird/<arch>/"] G --> H["Container lockfile updates
(see Image Pipeline)"] ``` ## Stage 1: Spec Change An RPM change begins as a commit in `rpms//`. The change typically includes a modified `.spec` file (updated version, new patch, or Release bump) and any associated source files or patches. ### How changes originate | Method | Description | Modification Status | | -------- | ------------- | --------------------- | | Automated Fedora sync | `ci/dist_git.py update` merges upstream Fedora changes | No `independent` packages | | Upstream version update | `ci/check_upstream_versions.py --update` bumps to new upstream release | No `independent` packages | | Manual patch backport | Developer adds a CVE patch and references it in the `.spec` | Marked `modified` afterward | | No-change rebuild | `ci/dist_git.py rebuild ` bumps the Release field | Status unchanged | | Reverse dependency rebuild | `ci/dist_git.py rebuild-rev-deps ` rebuilds all dependents | Status unchanged | See [Rebuilding Packages][rebuilding] and [Updating Dist-git Packages][updating] for detailed workflows. [rebuilding]: /l/rebuilding-packages [updating]: /l/updating-dist-git-packages ### Package metadata Each package has a metadata file at `metadata/.json` that tracks its relationship to upstream: ```json { "source": "https://src.fedoraproject.org/rpms/curl.git", "branch": "rawhide", "sha": "abc123...", "version": "8.12.1", "release": "1", "modification_status": "clean", "upstream_repo": "https://github.com/curl/curl" } ``` Key fields: | Field | Purpose | | ------- | --------- | | `modification_status` | `clean` (auto-updates enabled), `modified` (auto-updates blocked), or `independent` (no upstream) | | `modification_reason` | Explanation of local modifications (when `modified`) | | `release` | Base release without dist tag — Fedora/rawhide baseline, or a local base (independents / ahead-of-Fedora) | | `upstream_repo` | Canonical upstream git repository URL | | `track_upstream` | Version prefix constraint (e.g., `"1.26"` for golang1.26) | | `version_transform` | Version mapping rule for CVE analysis (e.g., `dotnet_sdk_to_runtime`) | | `cve_product` | CVE vendor/product override (string, or a list for multiple products; e.g. `"Oracle Corporation / Oracle Java SE"`) | See [Package Metadata Fields][metadata-fields] for `modification_status` and `release` configuration, and [Package Modification Tracking][mod-tracking] for managing modification status day to day. [metadata-fields]: /l/package-metadata-fields [mod-tracking]: /l/package-modification-tracking ## Stage 2: Merge Request & Validation Changes reach main via merge requests. Automated MRs are created using: - **`ci/create_mr.sh`** - Single MR with configurable options - **`ci/rebuild_multi_mr.sh`** - One MR per package with auto-merge enabled - **`ci/dist_git_update_multi_mr.sh`** - Automated Fedora update MRs (scheduled CI job) - **`ci/upstream_update_multi_mr.sh`** - Automated upstream version update MRs (scheduled CI job) ### CI validation GitLab CI runs validation jobs on every merge request: - **`check`** - Linting, type checking, and spec validation - **`check_nevr_conflicts`** - Predicts the NEVR(A) each changed package's spec would build (using `rpmspec` with the real dist-tag macros, without actually building) and checks it against the public Pulp repo. Catches the case where a Konflux build succeeds and the MR merges, but the resulting NEVR was already published (typically two independent Release bumps based on stale `main`, e.g. two `rebuild` MRs), which fails the post-merge build+sign+publish pipeline since Pulp treats each NEVRA as unique and immutable. Only scheduled for MRs that touch `rpms/**/*`, and within that, only checks packages whose `rpms//` directory itself changed -- matching Konflux's own build trigger -- not packages whose only change is under `metadata/`, since a metadata-only change (e.g. `ci/dist_git.py mark-modified`) never causes a rebuild/republish. Run locally with `make check-nevr-conflicts ARGS=''`, or against real built RPMs with `make check-nevr-conflicts ARGS='--rpms-dir builds/'` after `ci/build_rpms.sh`. See `ci/check_nevr_conflicts.py`'s module docstring for full details. Known limitations: only the public signed Pulp domain is checked (packages routed to a `private_product` are noted but not fully verified, since their domains require Pulp API credentials this tool doesn't have); and since it reflects Pulp's state at job-run time, it reduces but cannot fully close the race where another MR publishes the same NEVR between this check passing and this MR's eventual merge -- only a hard check inside the actual Konflux publish pipeline (a separate pipeline-bundle repository, `quay.io/hummingbird-ci/rpmbuild-pipeline`, not covered by this job) could close that window completely; and in spec mode some `OK` results can be for a package name a real build never actually produces (e.g. a base package with no top-level `%files` section, roughly 14% of packages in this repo) -- expected and harmless, and never a reason to discount a `CONFLICT` found elsewhere in the same run. - **`tree_status`** - Repository consistency checks (metadata integrity, spec file presence) - **Testing Farm** - Integration tests run via an IntegrationTestScenario that exercises built RPMs on a real RHEL compose (see [Testing](#testing) below) ### Auto-approval for chore MRs Automated update MRs (`chore/*` branches) follow an accelerated path: 1. MR is created with auto-merge enabled 2. Konflux builds the package and posts commit statuses 3. After a delay, the `chore_mr_approval` CI job auto-approves the MR 4. GitLab merges the MR once the pipeline succeeds ## Stage 3: Build After merging to main, Konflux builds RPMs automatically. ### Build triggers Each package has a Tekton PipelineRun definition generated from its directory in `rpms//` (see [`.tekton/rpms-on-pull-request.yaml.j2`][pr-template] and the rendered [`.tekton/rpms-on-pull-request.yaml`][pr-pipeline]). [PipelinesAsCode][pac] triggers a build when changes to that package's directory are pushed to main. When a single push touches multiple packages, one PipelineRun is triggered per affected package directory. Merge request pushes also trigger builds for validation. The `setup` package acts as a canary: changes to `ci/` or `mock/` directories also trigger a `setup` build on pull requests, validating build infrastructure changes before they reach main. [pac]: https://pipelinesascode.com/ [pr-template]: ../../.tekton/rpms-on-pull-request.yaml.j2 [pr-pipeline]: ../../.tekton/rpms-on-pull-request.yaml ### Build process The build pipeline (`build-rpm-package`) executes for each target architecture: 1. Checks out the repository at the merge commit 2. Calculates build dependencies 3. Runs `mock` in **hermetic mode** (network-isolated with pre-fetched dependencies) 4. Produces RPM artifacts and build logs 5. Generates an SBOM (Software Bill of Materials) 6. Stores artifacts as Trusted Artifacts in the Konflux OCI registry ### Build architectures Packages are built for the architectures specified in their PipelineRun definition. Most packages build for x86_64 and aarch64. Some packages also build for s390x and ppc64le. ### Build output Build artifacts are stored in the Konflux OCI registry: ```text quay.io/redhat-user-workloads/hummingbird-tenant/--main ``` ### Konflux resources Build infrastructure is defined across two repositories: - **rpms repo** (`konflux-templates/`) - Per-package Component and ImageRepository resources, ReleasePlanAdmission, EnterpriseContractPolicy - **infrastructure repo** (`kubernetes/rpms-main/`) - Application, ReleasePlan, IntegrationTestScenario, ServiceAccount, and Secret resources See [Konflux Resource Deployment][konflux-deploy] for details on how these resources are managed and deployed. [konflux-deploy]: konflux-resource-deployment.md ### Testing RPM packages are validated through integration tests that run on Testing Farm infrastructure. Tests run via [Testing Farm][testing-farm] on RHEL-9-Nightly systems for both `x86_64` and `aarch64` architectures. Test results appear as external jobs in GitLab CI pipelines, providing pass/fail status and links to the Konflux PipelineRun. [testing-farm]: https://docs.testing-farm.io/ #### Test triggering Tests are only triggered on merge requests, not on main branch builds. For a package at `rpms//`, tests are triggered for all changes below that directory. #### Integration Test Scenario Tests are triggered via an `IntegrationTestScenario` resource defined in the [infrastructure repository][infrastructure]: **Configuration**: `infrastructure/kubernetes/rpms-main/10-integration-test-scenarios-testing-farm.yml.j2` The scenario uses the upstream [Testing Farm pipeline for Konflux CI][integrations-konflux] and is parameterized as follows: [infrastructure]: https://gitlab.com/redhat/hummingbird/infrastructure [integrations-konflux]: https://gitlab.com/testing-farm/integrations-konflux **Scenario parameters**: - `COMPOSE: RHEL-9-Nightly` - Test environment OS/compose - `PIPELINE_MODE: rpm` - Configures RPM testing mode (vs. container) - `ARCH: x86_64,aarch64` - Architectures to test (creates separate test runs per arch) - `PASS_SNAPSHOT_TO_TF: false` - Don't pass full snapshot JSON (package info comes via `IMAGE_NAME`/`IMAGE_URL` instead) - `IMAGE_TAG:` - Version of the tmt-via-testing-farm pipeline bundle **Key characteristics**: - **Single scenario for all packages** - Not per-package, runs for every package build - **Context-based triggering** - Runs on `pull_request` snapshots only - **Package identification** - Konflux automatically provides `IMAGE_NAME` (e.g., `openssl-main`) and `IMAGE_URL` (OCI image with built RPMs) as environment variables #### FMF test plan The root folder of the `rpms` repository is marked with a `.fmf` directory to identify it as an FMF metadata tree for tmt. The test plan in `ci/run_tests_rpm.fmf` sets up the test instance and runs default and package-specific tests via `ci/run_tests_rpm.sh`. #### Test environment These environment variables are available to the tmt test execution: | Variable | Description | | ------------- | ------------------------------------------------------------------------ | | `IMAGE_NAME` | Component name (e.g., `openssl-main`) | | `IMAGE_URL` | OCI image URL with built RPMs (e.g., `quay.io/.../openssl-main@sha...`) | | `SNAPSHOT` | Snapshot metadata (if `PASS_SNAPSHOT_TO_TF` is true) | | `COMPOSE` | OS/compose being tested (`RHEL-9-Nightly`) | | `ARCH` | Architectures being tested (`x86_64,aarch64`) | | `TMT_VERSION` | tmt version running the tests | ## Stage 4: Signing The Konflux Release Service signs built RPMs before publishing: - **Method**: Kerberos-based signing via `konflux-release-signing-prod@IPA.REDHAT.COM` - **Pipeline**: Signing runs as part of the release pipeline, before the Pulp upload step - **Signing image**: `quay.io/konflux-ci/signing:latest` *(verify pinned digest in ReleasePlanAdmission)* ## Stage 5: Publishing to Pulp After signing, RPMs are uploaded to the Hummingbird Pulp repository. ### Release pipeline The `push-rpms-to-pulp` release pipeline handles the upload. Configuration is defined in the ReleasePlanAdmission resource (`releng/hummingbird-rpms-tech-preview-staging.yaml`): ```yaml mapping: rpm-repositories: - name: x86_64 repository_id: public-hummingbird-x86_64-rpms - name: aarch64 repository_id: public-hummingbird-aarch64-rpms - name: src repository_id: public-hummingbird-source-rpms # See ReleasePlanAdmission for the full list of published architectures ``` ### Pulp repository Published RPMs are available in the [Pulp content index](https://packages.redhat.com/api/pulp-content/public-hummingbird/): Pulp automatically regenerates repository metadata (repodata) after each upload, making new packages immediately resolvable by DNF/YUM clients. ## What Happens Next Once RPMs are available in Pulp, the [container image pipeline][image-pipeline] picks them up via lockfile updates. See the [Image Pipeline][image-pipeline] documentation in the containers repo for the full container lifecycle. ## Related Documentation - [Rebuilding Packages][rebuilding] - No-change rebuilds, reverse dependency rebuilds, and patch backports - [Updating Dist-git Packages][updating] - Automated Fedora sync workflow - [Package Metadata Fields][metadata-fields] - `modification_status` and `release` configuration - [Package Modification Tracking][mod-tracking] - Managing modification status and version constraints - [Konflux Resource Deployment][konflux-deploy] - How Konflux resources are defined and deployed - [Image Pipeline][image-pipeline] - Container image pipeline (continues from where this document ends) -------------------------------------------------------------------------------- # Konflux Resource Deployment url: https://hummingbird-project.io/docs/background/rpms/konflux-resource-deployment/ description: How Konflux resources are defined and deployed across repositories Konflux resources for RPM packages are split between two repositories: - **[rpms]**: Per-package resources generated from package directories - **[infrastructure]**: Application-level resources and deployment to the Konflux cluster [rpms]: https://gitlab.com/redhat/hummingbird/rpms [infrastructure]: https://gitlab.com/redhat/hummingbird/infrastructure This separation allows independent iteration on each concern. ## Deployment Matrix | Resource Type | Source | RPMs MR | RPMs Push | Infra MR | Infra Push | | ------------------------ | -------------- | ------- | --------- | -------- | ---------- | | Component | rpms | manual | manual | - | - | | ImageRepository | rpms | manual | manual | - | - | | ReleasePlanAdmission | rpms | - | automatic | manual | automatic | | EnterpriseContractPolicy | rpms | - | automatic | manual | automatic | | Application | infrastructure | - | - | manual | automatic | | ReleasePlan | infrastructure | - | - | manual | automatic | | IntegrationTestScenario | infrastructure | - | - | manual | automatic | | ServiceAccount, Secret | infrastructure | - | - | manual | automatic | **Legend**: - **RPMs MR**: Downstream pipeline triggered from rpms MR (deploys from MR commit) - **RPMs Push**: Downstream pipeline triggered from rpms push to main - **Infra MR**: Infrastructure MR pipeline (manual trigger) - **Infra Push**: Infrastructure push to main or web pipeline ## Resource Locations and Rationale ### RPMs Repo Resources are defined in `konflux-templates/` and rendered to `konflux-templates/rendered.yml`. **Component and ImageRepository**: 1. **Ownership**: Components are managed by the rpms repo. The infrastructure pipeline uses `ONLY_DOWNSTREAM` so only explicit downstream triggers from rpms deploy changes, giving the rpms repo full control over the component lifecycle. 2. **Dynamic generation**: Generated from `rpms/*/` directories via `ci/generate_konflux_resources.sh`, depending on package presence and configuration. 3. **Timing**: Must be deployed early during MR review so Konflux can build and test new packages. **ReleasePlanAdmission**: 1. **Static configuration**: Unlike containers (where RPA contains per-image tag mappings), the RPMs ReleasePlanAdmission is static—it configures the Pulp publishing pipeline without per-package data. It stays in the rpms repo for consistency with the containers pattern. 2. **Main branch only**: Deployed only on push to main to maintain deployment consistency. **EnterpriseContractPolicy**: 1. **Consistency**: Follows the containers repo pattern of keeping policy definitions alongside the resources they govern. ### Infrastructure Repo **Application, ReleasePlan, and IntegrationTestScenario** are defined in `kubernetes/rpms-main/`: 1. **Independent iteration**: Changes are decoupled from rpms repo activity—they can be modified, test-deployed via manual trigger in an infrastructure MR, verified, and merged without touching the rpms repo. 2. **Testable before merge**: If defined in the rpms repo, these would only deploy after merging to main, making iteration difficult. 3. **Static configuration**: These resources don't depend on per-package data. **ServiceAccount and Secret** for releases are defined in `kubernetes/setup-konflux/`: 1. **Security**: Secret specifications (names, structure, credential references) should not be exposed in the rpms repo. 2. **Independent iteration**: Like other infrastructure resources, these can be modified and test-deployed without touching the rpms repo. The service account is referenced by ReleasePlanAdmission to authorize publishing RPMs to Pulp. -------------------------------------------------------------------------------- # Source Pipeline Tool url: https://hummingbird-project.io/docs/background/rpms/source-pipeline-tool/ ## Source Pipeline Tool **Codename: Gorget** — the iridescent throat patch that makes hummingbirds distinctive. In ornithology, the gorget is the defining feature used to identify species; in this project, it represents the verification layer that distinguishes independently-sourced packages from unverified ones. ## Problem Statement RPM-based distributions that derive packages from upstream sources face a common set of supply chain challenges: 1. **Supply chain trust gap.** Distributions that consume source tarballs from another distribution's lookaside cache (e.g., Fedora's) inherit an unverified trust boundary. Package maintainers download upstream release artifacts, may modify them, and upload to a lookaside cache. Downstream consumers use those tarballs without verifying they match upstream. Any maintainer could, intentionally or unintentionally, introduce modifications that compromise packages. 2. **Version dependency on upstream packagers.** Downstream distributions are blocked on upstream maintainers to package new releases. If upstream ships a critical fix, downstream must wait for the intermediate distribution to update before consuming it. 3. **Non-durable security transforms.** When a distribution applies a security fix to vendored dependencies (e.g., patching a lockfile to bump a vulnerable transitive dependency), a future upstream update regenerates vendor artifacts from the upstream baseline. The fix silently disappears. The update succeeds, CI passes, and the vulnerable dependency is back. 4. **Unverified patches.** Patches (`.patch` files) from upstream distributions flow into packages without verification. There is no check whether patches correspond to upstream commits, whether they are needed for the target build environment, or whether new patches have been introduced. ## Tool Overview A **containerized pipeline tool** that reads a declarative per-package YAML definition, fetches source tarballs directly from upstream, applies transforms, verifies integrity, enforces policies, and emits artifacts ready for a lookaside cache. ```text ┌──────────────────────────────┐ │ source-pipeline tool │ │ (container image) │ │ │ ┌───────────┐ │ ┌───────┐ ┌───────────┐ │ ┌───────────────┐ │ package │ │ │ fetch │───▸│ transform │ │ │ tarballs │ │ dir │──────────▸│ └───────┘ └─────┬─────┘ │───▸│ sources │ │ (spec, │ │ │ │ │ report │ │ patches, │ │ ┌───────────┐ │ │ └───────┬───────┘ │ config) │ │ │ verify │◂─────┘ │ │ │ │ │ └─────┬─────┘ │ ▼ └───────────┘ │ │ │ ┌───────────────┐ │ ┌─────▼─────┐ │ │ lookaside │ ┌───────────┐ │ │ enforce │ │ │ cache │ │ pipeline │ │ │ policy │ │ └───────────────┘ │ yaml │──────────▸│ └─────┬─────┘ │ │ │ │ │ │ └───────────┘ │ ┌─────▼─────┐ │ │ │ emit │ │ │ └───────────┘ │ └──────────────────────────────┘ ``` ### Design principles - **Runs anywhere.** Same container image via podman — locally for development, in CI for automation, in SLSA-attested build environments for provenance. - **Declarative.** Per-package behavior is defined in YAML, not imperative scripts. The YAML serves as an attestation artifact — anyone can read it to understand exactly how a package's sources are produced. - **Fail-closed.** Verification failures and policy violations fail the pipeline. No silent fallbacks. On failure, the package is skipped (stays at its current version) and the automation continues to the next package. - **Ecosystem-aware.** Built-in support for Go (vendor archives), npm (node_modules), cargo (vendor), and Composer (PHP) workflows, with escape hatches for custom transforms. ### Container interface ```bash podman run --rm \ -v ./:/package:ro \ -v ./pipeline.yaml:/pipeline.yaml:ro \ -v ./gpg-keys:/gpg-keys:ro \ -v ./output:/output \ source-pipeline:latest \ --version \ [--old-version ] \ [--verify-patches /patches-from-upstream] \ [--dry-run] ``` **Inputs (mounted read-only):** - `/package` — the package directory (spec file, patches, existing `sources` file) - `/pipeline.yaml` — the declarative pipeline definition - `/gpg-keys` — centralized GPG keyring directory **Outputs (written to `/output`):** - Source tarballs (ready for lookaside upload) - `sources` — updated sources manifest with checksums - `report.json` — verification and policy results (pass/fail per check, patch classifications) **Exit codes:** - `0` — success, all checks passed - `1` — error (download failure, tool error) - `2` — policy violation (verification or constraint failure) `--dry-run` runs all stages through Verify and Policy but skips Emit — no tarballs are written to `/output`. Exit codes and `report.json` behave identically, so developers can preview what would happen without producing artifacts. Useful for validating a new pipeline YAML before committing. On any non-zero exit, `report.json` is still written with the failure details (stage, error type, message). The calling automation uses this to log the failure and skip the package — no commit is created, the package stays at its current version, and the automation continues to the next package. Transient failures (exit 1) self-heal on the next scheduled run; verification and policy failures (exit 2) require human intervention. ## Pipeline Stages ### 1. Fetch Downloads source artifacts directly from upstream. - Parses `Source:` URLs from the spec file (resolving RPM macros like `%{version}`, `%{name}`, `%{url}`) - Downloads each source from the upstream URL - Supports fetching from git repositories at a tag, branch, or commit - For packages without a pipeline YAML, uses a default fetch-from-spec behavior (covering the "trivial" package case) ### 2. Transform Applies per-package source modifications. - **Vendor archive generation** — runs ecosystem-specific tooling (e.g., `go_vendor_archive` for Go, `npm pack` for Node.js, `cargo vendor` for Rust) - **Vendor dependency pinning** — modifies lockfiles (`go.mod`/`go.sum`, `package-lock.json`, `Cargo.lock`) to bump specific dependencies to required versions before vendoring. This is the enforcement counterpart to policy's validation: pins are applied during transform, then policy confirms the result. Solves the non-durable security transforms problem — a declarative pin is re-applied on every source generation, so upstream updates cannot silently revert a CVE fix. - **Source stripping** — removes content that cannot be distributed (crypto, bundled pre-built binaries, non-free assets) - **UI asset builds** — builds JavaScript/TypeScript UI assets from source (for packages like Prometheus, Jaeger, Grafana that currently vendor pre-built UI) - **Custom transforms** — escape hatch for arbitrary commands when built-in stages are insufficient - **Toolchain versioning** — packages can declare required toolchain versions (Node.js, Go, Rust, etc.) via a `toolchain:` section. The container ships defaults; per-package overrides ensure that `build-ui`, `vendor`, and `run:` steps use the correct toolchain without requiring separate container images per package #### Known sharp edge: patch-list duplication Two independent mechanisms hand-apply changes outside of `%prep`, and neither reads its list from the spec's `PatchN:` declarations — so both can silently drift out of sync with what the spec actually declares. This is the canonical writeup; `rebuilding-packages.md` and the `/cve` skill point back here instead of re-telling the story. **Custom `run:` transforms** that hand-apply patches before resolving a lockfile (`patch -p1 < ...` followed by `yarn install`/`pnpm fetch`/etc., as grafana's yarn-cache generation does) maintain their own copy of "which patches touch this source tree" rather than reading it from the spec's `PatchN:` declarations. Nothing enforces these two lists stay in sync. A patch added to the spec through the normal backport workflow (see [Rebuilding Packages](https://hummingbird-project.io/l/rebuilding-packages)) is invisible to the transform unless a human also updates the pipeline YAML — and if that patch touches a file the transform's install command resolves against (a lockfile or manifest), the generated artifact silently drifts from what `%build` actually applies. This broke grafana12.4 and grafana13.1: four separate CVE backports bumped `yarn.lock` without updating the pipeline's yarn-cache generation step, and the mismatch didn't surface until the next automated version bump re-ran the transform from a pristine checkout. `test/test_source_pipeline_patches.py` checks for this drift across all packages with a pipeline definition, but it's a safety net for a design gap, not a fix for it: a built-in transform primitive that reads `PatchN:` from the spec directly, instead of requiring the YAML to hand-duplicate the list, would close this class of bug at the source rather than relying on the check to catch it after the fact. **`go-vendor-tools.toml`'s `[archive] pre_commands`** have the same shape of problem for Go packages, whether or not they're migrated to gorget: `pre_commands` (sed edits, `go get` bumps, `go mod tidy`) only ever run against the vendor archive's own checkout, never against the plain source tarball (`Source0`), which is fetched separately. A patch that bumps a vendored dependency's version must be mirrored into `go.mod`/`go.sum` by a spec patch — nothing keeps `pre_commands` and the spec's patches in sync. This broke trivy: a CVE backport added `go get` calls to `pre_commands` to bump vendored dependencies (CVE-2026-15788/15792/56852) without a matching spec patch, so `go.mod` in the build tree and `vendor/modules.txt` in the generated vendor archive ended up requiring different versions of the same package — `go build -mod=vendor` rejected it as inconsistent vendoring. The mismatch sat latent for over a week until an unrelated version bump's `%check` run finally caught it. `test/test_govendortools_gomod_patch_sync.py` checks for this drift, same caveat as above. ### 3. Verify Validates integrity and authenticity of fetched sources. - **GPG signature verification** — downloads signature files (`.asc`, `.sig`) declared in the spec and verifies against upstream keys stored in a centralized keyring directory. Keys are organized by upstream project (e.g., `gpg-keys/curl.gpg`). Centralized storage means the full set of trusted keys is auditable in one directory, and key rotation or revocation is a single-commit operation. - **Checksum verification** — compares against published checksums where available - **Reproducibility check** — for packages with existing tarballs from another distribution, optionally compares the independently-fetched tarball to identify divergences - **Re-publication detection** — if a previously committed checksum exists in the `sources` file for the same version and the freshly downloaded artifact does not match, the pipeline fails (exit code 2). This catches upstream projects that silently re-publish release artifacts under the same version. The pipeline will continue to fail on automated retries until a human explicitly updates an `accepted-checksums` entry in the pipeline YAML with the new hash and a reason. This forces investigation, prevents automated retries from silently accepting changed content, and provides a committed audit trail of what changed and why. ### 4. Enforce policy Validates the final artifacts. Acts as a safety net for `vendor-pin` (confirms pins took effect) and catches violations in packages that don't use `vendor-pin`. - **Vendor dependency constraints** — ensures vendored dependencies meet version requirements (e.g., `sanitize-html >= 2.17.5` for a CVE fix) - **Ecosystem-specific checks** — Go module verification, npm audit, cargo audit - **License compliance** — flags vendored dependencies with incompatible licenses - Fail-closed: any policy violation exits with code 2 ### 5. Emit Produces final artifacts. - Writes tarballs to the output directory - Generates `sources` file in dist-git format (`SHA512 (filename) = hash`) - Writes `report.json` with verification results, policy check results, and patch classification ## Pipeline Schema Per-package pipeline definitions are YAML files that declare the full source generation workflow. Packages without a pipeline definition use a built-in default that fetches sources from the spec's `Source:` URLs — no transforms, no verification, no policy. ### Schema definition ```yaml # Spec preparation (optional) # Runs before fetch — fixes version macros that the specfile library cannot # trace, so Source URLs resolve correctly. spec-update: # Macro substitutions applied before Source URL resolution macros: - name: go_patch # %global go_patch value: "${VERSION_PATCH}" # extracted from VERSION (e.g., 1.25.3 → 3) - name: k8s_ver # %global k8s_ver value: "${VERSION}" # Reset Release to 0.1%{?dist} on version bump (common for versioned packages) reset-release: true # Source fetching fetch: sources: # Fetch Source0 from the URL declared in the spec - spec-source: 0 # Fetch Source1 (e.g., a GPG signature file) - spec-source: 1 # Or fetch from an explicit URL (for sources not in the spec) - url: "https://example.com/extra-source-${VERSION}.tar.gz" # Or fetch from a git repository at a tag/commit - git: repo: "https://github.com/example/project" ref: "v${VERSION}" # tag, branch, or commit hash include-history: false # include .git dir in tarball (default: false) # Vendor archive generation (optional) vendor: ecosystem: go # go | npm | cargo | composer config: go-vendor-tools.toml # ecosystem-specific config file in the package dir source-dir: . # directory to vendor from (default: extracted source root) submodules: # (optional) vendor multiple Go submodules independently - server - etcdctl - etcdutl # Source transforms (optional, ordered) transform: # Built-in transform types - strip-tarball: source: "node-v${VERSION}.tar.gz" remove: - "deps/openssl/" - "deps/ngtcp2/ngtcp2/crypto/" output: "node-v${VERSION}-stripped.tar.gz" # Pin vendored dependency versions before vendor archive generation. # Modifies lockfiles (go.mod/go.sum, package-lock.json, Cargo.lock) to # bump specific dependencies, then re-resolves the dependency graph. # Runs before the vendor stage so the pinned versions are included in # the vendor archive. Re-applied on every source generation, so upstream # updates cannot silently revert a security fix. # # Version constraints are always minimum versions — the version field # means "at least this version." The tool translates to ecosystem-native # operations: # Go: go get @v (minimum version selection) # npm: npm install @">= " # Cargo: set dependency requirement to ">= " in Cargo.toml - vendor-pin: - package: golang.org/x/crypto ecosystem: go version: "0.31.0" reason: "CVE-2024-45337" - package: sanitize-html ecosystem: npm version: "2.17.5" reason: "CVE-2024-XXXXX" - package: tokio ecosystem: cargo version: "1.38.1" reason: "CVE-2024-YYYYY" # Custom command (escape hatch) - run: "./packaging/make-tarball.sh ${VERSION}" outputs: - "package-${VERSION}-stripped.tar.gz" # UI asset build - build-ui: ecosystem: npm # npm | yarn source-dir: "web/ui" output: "${PACKAGE}-${VERSION}-ui.tar.gz" # Toolchain requirements (optional) # Declares the toolchain versions a package needs during source generation. # The container ships default toolchain versions; this section overrides them # when a package requires something different. Any primitive that invokes a # toolchain (vendor, build-ui, run) uses the versions declared here. # # Packages that don't declare a toolchain section get the container defaults. # If a required toolchain is not available in the container image, the pipeline # fails with exit code 1 and a clear message naming the missing tool. toolchain: node: "20" # Node.js version for build-ui, npm vendor, and run: steps go: "1.23" # Go version for go vendor and run: steps rust: "1.80" # Rust version for cargo vendor and run: steps python: "3.12" # Python version for run: steps # Integrity verification (optional) verify: gpg: signature-source: 1 # spec Source index containing the .asc/.sig keyring: "curl.gpg" # key name in the GPG keys directory checksums: url: "https://example.com/SHA256SUMS" algorithm: sha256 # Override for upstream re-publications. Required when upstream re-publishes # a release artifact with different content under the same version. The # pipeline refuses to accept changed content automatically — a human must # add the new checksum here after investigation. accepted-checksums: - file: "example-1.2.3.tar.gz" sha512: "abc123..." reason: "Upstream re-published with corrected LICENSE file (verified via upstream issue #456)" # Policy enforcement (optional) # Validates the final artifacts. vendor-constraints acts as a safety net: # if vendor-pin (above) is used, policy confirms the pin took effect; if # vendor-pin is not used, policy catches violations that need manual action. # Uses the same minimum version semantics as vendor-pin. policy: vendor-constraints: - package: sanitize-html ecosystem: npm version: "2.17.5" reason: "CVE-2024-XXXXX" - package: golang.org/x/crypto ecosystem: go version: "0.31.0" reason: "CVE-2024-45337" # Patch verification (optional) patches: verify: true # diff-detect new/changed patches classify: true # attempt to classify by origin upstream-repo: "https://github.com/curl/curl" fail-on-unverified: false # warn-only by default # Patch lifecycle rules (optional) # Declares version-scoped applicability for patches. The pipeline tool reads # these rules from patch headers (preferred) or from this YAML, and enforces # them during updates: patches outside their valid range are flagged or dropped. lifecycle: - file: "fix-memory-leak.patch" applies-to: "< 1.5.0" # drop this patch at version 1.5.0+ reason: "Fixed upstream in 1.5.0 (commit abc123)" action: drop # drop | warn (default: warn) - file: "cve-2024-45337.patch" applies-to: "< 0.31.0" reason: "CVE-2024-45337 — fixed upstream in golang.org/x/crypto 0.31.0" action: drop - file: "distro-branding.patch" applies-to: "*" # carry forward unconditionally reason: "Distribution-specific branding, always required" # Post-update spec modifications (optional) # Runs after fetch/transform — extracts metadata from downloaded sources and # patches it into the spec. post: # Extract bundled dependency versions and splice into spec between markers - bundled-provides: modules-txt: "vendor/modules.txt" # Go modules.txt path inside extracted source start-marker: "# --- bundled-deps.sh ---" end-marker: "# --- end bundled-deps.sh ---" # Run a custom command (escape hatch for complex metadata extraction) - run: "./packaging/fill-versions.sh ${SPEC_FILE} source-v${VERSION}-stripped.tar.gz" ``` ### Variable substitution The following variables are available in all string values: | Variable | Value | | -------- | ----- | | `${VERSION}` | New upstream version (e.g., `1.25.3`) | | `${VERSION_MAJOR}` | Major version component (e.g., `1`) | | `${VERSION_MINOR}` | Minor version component (e.g., `25`) | | `${VERSION_PATCH}` | Patch version component (e.g., `3`) | | `${OLD_VERSION}` | Previous version | | `${PACKAGE}` | Package name (directory name) | | `${SPEC_FILE}` | Path to the spec file | ### Version constraint semantics All `version:` fields in `vendor-pin` and `policy.vendor-constraints` use minimum-version semantics: the value means "at least this version." The tool translates this into ecosystem-native operations: - **Go**: `go get @v` (minimum version selection via MVS) - **npm**: `npm install @">= "` - **Cargo**: sets the dependency requirement to `>= ` in `Cargo.toml` ### Default behavior (no pipeline YAML) When no pipeline YAML is provided, the tool applies a built-in default: 1. Parse all `Source:` URLs from the spec 2. Download each from the upstream URL 3. Generate `sources` file with SHA512 checksums 4. No transforms, no verification, no policy enforcement This covers the common case where the upstream distribution's tarball is identical to upstream's release artifact. ## Patch Verification When packages are updated from an upstream distribution, patch files (`.patch`) may arrive without inspection. The pipeline tool can optionally verify and manage patches. ### Verification approach The pipeline tool can optionally verify patches when invoked with `--verify-patches`: 1. **Diff detection.** Compare the set of `.patch` files in the updated package directory against the previous version. Identify new patches, removed patches, and modified patches. 2. **Header parsing.** Extract metadata from patch headers: - `From:` — author identity - `Subject:` — description - Commit hash references (e.g., `From `, `cherry picked from commit `) - `Bug:` / `CVE:` references 3. **Classification.** Attempt to classify each patch: - **Upstream backport** — references a commit hash that exists in the upstream repo - **CVE fix** — references a CVE identifier - **Build/packaging fix** — modifies build system files (Makefile, configure, CMakeLists) - **Distribution-specific** — modifies paths, branding, or distribution-specific integration - **Unclassified** — cannot be automatically categorized 4. **Upstream verification.** For patches claiming to be backports, check whether the referenced commit exists in the upstream repo (via `git ls-remote` or the upstream API). 5. **Reporting.** Output classification and verification results in `report.json`. Optionally fail on unverified patches (controlled by `patches.fail-on-unverified` in the pipeline YAML). ### Patch lifecycle enforcement Patches have version-scoped lifetimes. A CVE backport is only valid until the upstream version that includes the fix. A build system workaround may only apply to a specific major version. Without enforcement, a patch that should have been dropped at version 2.0 silently persists, and a patch that must be carried forward can be accidentally removed during an update. The pipeline tool enforces patch lifecycle rules declared either in patch headers or in the pipeline YAML's `patches.lifecycle` section. **Header-based declaration (preferred).** Patch authors add structured keywords to the patch header: ```text From: maintainer@example.com Subject: Backport fix for CVE-2024-45337 Applies-To: < 0.31.0 Lifecycle-Action: drop Lifecycle-Reason: Fixed upstream in golang.org/x/crypto 0.31.0 --- ``` **YAML-based declaration (fallback).** For patches from upstream distributions that cannot have headers modified, rules are declared in `patches.lifecycle` in the pipeline YAML (see schema above). Header-based rules take precedence when both exist for the same patch. **Enforcement behavior:** 1. During an update to version `${VERSION}`, the tool evaluates each patch's `applies-to` range against the new version. 2. If a patch is outside its valid range: - `action: drop` — the patch file is deleted from the package directory and the corresponding `Patch:` declaration and `%patch` / `%autopatch` application directives are removed from the spec. If the patch is applied inside a conditional block (`%if`), the tool flags it for manual intervention instead of attempting removal. The tool reports all actions in `report.json`. - `action: warn` (default) — the patch is flagged in `report.json` but not removed. The update proceeds. 3. If a patch has `applies-to: *`, it is always carried forward. 4. Patches without any lifecycle declaration are treated as having no version constraint (equivalent to `applies-to: *`, `action: warn`). **Carry-forward enforcement.** The inverse case is also important: some patches (branding, distribution-specific integration) must never be dropped. If a patch marked `applies-to: *` is missing after an update (e.g., removed by an upstream distribution sync), the tool flags it as an error. ### Example report output ```json { "patches": { "new": [ { "file": "fix-memory-leak.patch", "classification": "upstream-backport", "upstream_commit": "abc123def456", "verified": true } ], "removed": ["old-workaround.patch"], "unchanged": ["distro-paths.patch"], "lifecycle": [ { "file": "cve-2024-45337.patch", "applies_to": "< 0.31.0", "current_version": "0.31.0", "action": "drop", "result": "removed — version 0.31.0 is outside applies-to range" }, { "file": "cve-2024-99999.patch", "applies_to": "< 2.0.0", "current_version": "2.0.0", "action": "drop", "result": "flagged — patch is applied inside a conditional %if block; manual removal required" }, { "file": "distro-branding.patch", "applies_to": "*", "action": "carry-forward", "result": "present" } ] } } ``` ## Examples ### Trivial package (curl) No pipeline YAML needed. The default behavior fetches `Source0` from the URL in the spec. If you want GPG verification: ```yaml fetch: sources: - spec-source: 0 - spec-source: 1 verify: gpg: signature-source: 1 keyring: "curl.gpg" ``` ### Transformed package (Node.js) Strip bundled OpenSSL, verify upstream checksums, extract component versions into spec: ```yaml spec-update: macros: - name: nodejs_define_version node value: "${VERSION}" reset-release: true toolchain: node: "22" fetch: sources: - url: "https://nodejs.org/dist/v${VERSION}/node-v${VERSION}.tar.gz" transform: - strip-tarball: source: "node-v${VERSION}.tar.gz" remove: - "deps/openssl/" output: "node-v${VERSION}-stripped.tar.gz" verify: checksums: url: "https://nodejs.org/dist/v${VERSION}/SHASUMS256.txt" algorithm: sha256 post: - run: "./packaging/fill-versions.sh ${SPEC_FILE} node-v${VERSION}-stripped.tar.gz" ``` ### Simple Go vendor (caddy) A pattern shared by nats-server, oauth2-proxy, and similar Go projects: ```yaml fetch: sources: - git: repo: "https://github.com/caddyserver/caddy" ref: "v${VERSION}" vendor: ecosystem: go ``` ### Multi-submodule Go vendor (etcd) ```yaml fetch: sources: - url: "https://github.com/etcd-io/etcd/archive/v${VERSION}/etcd-${VERSION}.tar.gz" vendor: ecosystem: go config: go-vendor-tools.toml submodules: - server - etcdctl - etcdutl ``` ### Code generation + Go vendor (opentelemetry-collector-contrib) Built-in primitives handle fetch and vendor; the code generation step requires `run:`: ```yaml fetch: sources: - url: "https://github.com/open-telemetry/opentelemetry-collector-releases/archive/v${VERSION}/opentelemetry-collector-releases-${VERSION}.tar.gz" vendor: ecosystem: go config: go-vendor-tools.toml source-dir: _build transform: - run: | curl -fSL -o ocb "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2Fv${VERSION}/ocb_${VERSION}_linux_amd64" curl -fSL -o ocb_checksums.txt "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2Fv${VERSION}/ocb_${VERSION}_checksums.txt" sha256sum -c --ignore-missing ocb_checksums.txt chmod +x ocb tar -xzf "opentelemetry-collector-releases-${VERSION}.tar.gz" cd "opentelemetry-collector-releases-${VERSION}" if [ -f scripts/prepare-obi.sh ]; then bash scripts/prepare-obi.sh otelcol-contrib fi ../ocb --skip-compilation --config distributions/otelcol-contrib/manifest.yaml outputs: - "opentelemetry-collector-contrib-${VERSION}-generated.tar.bz2" ``` ### Git snapshot (libXtst) A pattern shared by libX11, libXext, libXi, libXrender: ```yaml fetch: sources: - git: repo: "https://gitlab.freedesktop.org/xorg/lib/libXtst" ref: "libXtst-${VERSION}" include-history: true ``` ## Built-in Primitives Coverage An audit of 48 existing tarball/vendor scripts across 25+ packages identified the following coverage: **Fully declarative (no `run:` needed) — ~30 script files (15 unique patterns):** | Built-in primitive | Packages covered | | ------------------ | ---------------- | | `vendor: {ecosystem: go}` | caddy, nats-server×2, oauth2-proxy | | `vendor: {ecosystem: go, submodules: [...]}` | etcd (3 submodules) | | `strip-tarball: {remove: [...]}` | nodejs×5, cyrus-sasl, perl-libnet, java in-tree libs×2 | | `fetch: {git: {repo, ref}}` | oniguruma, php-jsqueeze, libX11/Xext/Xi/Xrender/Xtst | | `fetch` + `verify: {checksums}` | nodejs download+verify | | `vendor: {ecosystem: composer}` | composer | **Partially declarative (generic primitives + 1–2 `run:` steps):** | Package | Generic part | Custom `run:` step | | ------- | ------------ | ------------------ | | otel-collector, otel-collector-contrib | `vendor: {ecosystem: go}` | OCB binary code generation | | java-openjdk×2 | `strip-tarball`, `fetch: {git}` | `./configure` + `make store-source-revision` (needs boot JDK) | | selinux-policy | `fetch: {git}` (×3 repos) | Multi-repo selective archiving | **Genuinely custom (`run:` required) — 7 packages:** | Package | Why it can't be declarative | | ------- | --------------------------- | | nss-fips | Container-based RPM download with subscription-manager credentials + QEMU cross-arch | | openssl-fips-provider | Container-based SRPM download + nested RPM extraction | | ca-certificates | Interactive multi-source crypto trust data merging | | gcc | GCC-specific changelog/PR extraction from git history | | gdb | Interactive patch management tooling (stgit) | | python3.14 | Koji task-specific JIT stencil artifact extraction | | erlang27 | Complex patch reformatting + spec rewriting | The `run:` escape hatch exists for these ~7 packages. All other packages should use built-in primitives to maintain the declarative contract. ## Open Questions 1. **Implementation approach.** The pipeline YAML schema (ordered transform stages, `run:` escape hatches, variable substitution) resembles a bespoke Ansible without the ecosystem. The implementation choice also determines the language. Alternatives to consider: - **Custom tool (Python or Go)** — Python is consistent with existing tooling; Go produces a single static binary. Either way we own the full stack. - **Tekton StepActions** — already in some build ecosystems. Source generation could be a parameterized Tekton pipeline rather than a custom tool. Downside: harder to run locally. - **Shared shell function library** — the majority of scripts decompose into 3–4 operations (`vendor_go`, `strip_tarball`, `fetch_git`). A thin shell library called from per-package Makefiles may be more honest than YAML that serializes shell commands. - **Minimal declarative YAML + driver** — keep the YAML as a pure declaration of intent (what to fetch, what to strip, what to vendor) with no `run:` blocks, no ordering, no conditionals. A thin driver interprets it by calling shell functions. Packages that can't be expressed this way keep their shell scripts. This preserves the attestation value of the YAML without building a workflow engine. --- ## Hummingbird Integration This section describes how the source pipeline tool integrates with Hummingbird's existing infrastructure. The tool itself is distribution-agnostic; this section covers the Hummingbird-specific wiring. ## Current Architecture ### Two update systems | System | Packages | Source of tarballs | Hook system | | ------ | -------- | ------------------ | ----------- | | `dist_git.py update` | 450 (clean + modified) | Fedora lookaside cache (via `sources` file copied from Fedora dist-git) | None | | `check_upstream_versions.py` | 22 (independent) | Upstream URLs (via `download_sources` hooks or default spec URL download) | Yes: `update_spec`, `download_sources`, `post_update` | **`dist_git.py update` flow:** 1. Clones Fedora dist-git for the package 2. Checks version, Koji build status, pre-release filtering 3. Copies entire Fedora dist-git checkout into `rpms//` via `shutil.copytree` — this includes the spec, patches, `.gitignore`, and `sources` file 4. For modified packages, performs a 3-way git merge to preserve local changes 5. Commits everything (spec, patches, `sources` file, metadata) The `sources` file is committed to git as a text manifest (format: `SHA512 (filename) = hash`). Actual tarballs are excluded by `.gitignore` and stored in a lookaside cache. **`check_upstream_versions.py` flow:** 1. Queries release-monitoring.org for new upstream versions 2. Runs three hook phases per package: - `update_spec` — updates the spec's Version/Release (default: `specfile.update_version()`) - `download_sources` — downloads source tarballs (default: fetches from spec Source URLs) - `post_update` — additional steps (default: no-op) 3. Uploads downloaded tarballs to Hummingbird's lookaside cache 4. Updates the `sources` file with new checksums 5. Commits Hooks are defined in `metadata/.update-hooks.yaml`. 12 packages currently have hooks. ### Three lookaside cache backends Configured in `mock/dist-git-client.ini`: | Backend | URL pattern | Used by | | ------- | ----------- | ------- | | Fedora | `src.fedoraproject.org/repo/pkgs/rpms/{name}/{filename}/{hashtype}/{hash}/{filename}` | ~410 packages (default) | | CentOS Stream | `sources.stream.centos.org/sources/rpms/{name}/...` | 1 package (`rust-rpm-sequoia`) | | Hummingbird | `d1766whheab9hg.cloudfront.net/rpms/{name}/...` (S3: `arr-hummingbird-prod-dist-git-cache`) | ~40 packages (independent + forked) | The `forked_from` field in `ci/package-overrides.yaml` determines which cache a package uses at build time. Packages without `forked_from` default to Fedora's cache. ### How Konflux builds consume sources Tekton PipelineRuns (generated by `ci/generate_resources.py` from Jinja2 templates) pass parameters to the `build-rpm-package` pipeline bundle (`quay.io/hummingbird-ci/rpmbuild-pipeline`): - `monorepo-subdir: rpms/` — locates the spec, patches, and `sources` file - `package-name: ` — upstream name for lookaside URL construction - `forked-from: ` — (optional) selects the lookaside backend - `dist-git-client-configdir: mock/` — (optional) points to the custom `dist-git-client.ini` The pipeline uses `dist-git-client` to read the `sources` file and download tarballs from the appropriate cache, then runs `mock` to build RPMs. ### Existing source generation patterns Some independent packages already have tarball generation scripts: - `rpms/opentelemetry-collector-contrib/create-vendor-tarball.sh` — downloads upstream release, runs OCB to generate source code, vendors Go dependencies - `rpms/caddy/create-vendor-tarball.sh` — downloads upstream tarball, generates Go vendor archive - `rpms/nodejs25/packaging/make-nodejs-tarball.sh` — downloads upstream, strips bundled content These scripts are wired into the update path via `download_sources` hooks in `metadata/.update-hooks.yaml`. The pattern is: script runs, prints output filenames to stdout, automation uploads them to the Hummingbird lookaside cache. ## Integration with `dist_git.py update` The current flow copies everything from Fedora dist-git (including the `sources` file) via `shutil.copytree`. The pipeline tool inserts after this step: ```text Current: 1. Clone Fedora dist-git 2. copytree into rpms// ← sources file points to Fedora lookaside 3. Commit Proposed: 1. Clone Fedora dist-git 2. copytree into rpms// ← sources file points to Fedora lookaside 3. Run source-pipeline tool ← fetches from upstream, replaces sources file 4. Upload tarballs to Hummingbird lookaside 5. Commit ← sources file now points to Hummingbird lookaside ``` Implementation: - After the `shutil.copytree` (or merge for modified packages), check if `metadata/.source-pipeline.yaml` exists - If it exists: invoke the pipeline tool via podman, collect outputs, upload to lookaside, replace the `sources` file - If it does not exist: run the built-in default (fetch from spec URLs, upload, replace `sources`) - A `--skip-pipeline` flag allows falling back to the current behavior during migration - Update `ci/package-overrides.yaml` to set `forked_from` to hummingbird for each migrated package (so Konflux builds fetch from the Hummingbird cache) ## Integration with `check_upstream_versions.py` The pipeline tool replaces all three hook phases from `*.update-hooks.yaml`, consolidating per-package update behavior into a single `*.source-pipeline.yaml` file: ```text Current: 1. update_spec hook/default ← *.update-hooks.yaml 2. download_sources hook/default ← *.update-hooks.yaml 3. post_update hook/default ← *.update-hooks.yaml Proposed: 1. spec-update (pipeline YAML) ← replaces update_spec hooks 2. fetch + transform (pipeline YAML) ← replaces download_sources hooks 3. post (pipeline YAML) ← replaces post_update hooks ``` This reduces per-package metadata from three files (`metadata/.json`, `*.update-hooks.yaml`, `*.source-pipeline.yaml`) to two (`metadata/.json` for identity/tracking, `*.source-pipeline.yaml` for all update behavior). Existing `*.update-hooks.yaml` files are migrated to `*.source-pipeline.yaml` definitions. During migration, the hook system remains as a legacy fallback: if a package has a `*.update-hooks.yaml` but no pipeline YAML, the hooks run as before. Once all 12 hook files are migrated, the hook system is removed. ## Integration with Konflux Initially, the pipeline tool runs pre-build (during the update automation in GitLab CI). The tarballs it produces are uploaded to the Hummingbird lookaside, and the existing Tekton build pipeline consumes them via `dist-git-client` as it does today. Future: the pipeline tool could run as a Tekton task within the Konflux build pipeline itself. This would move source generation into the SLSA-attested build environment, strengthening the provenance chain. The container image is already compatible — it just needs a Tekton Task definition. ## Hummingbird-specific configuration Pipeline YAML files live at `metadata/.source-pipeline.yaml`. GPG keys live at `metadata/gpg-keys/.gpg`. The container image is published as `quay.io/hummingbird-ci/source-pipeline:latest`. ## Local development Developers can run the tool directly: ```bash podman run --rm \ -v ./rpms/curl:/package:ro \ -v ./metadata/curl.source-pipeline.yaml:/pipeline.yaml:ro \ -v ./metadata/gpg-keys:/gpg-keys:ro \ -v /tmp/output:/output \ quay.io/hummingbird-ci/source-pipeline:latest \ --version 8.21.0 # Outputs in /tmp/output/: # curl-8.21.0.tar.xz # curl-8.21.0.tar.xz.asc # sources # report.json ``` To preview without producing artifacts: ```bash podman run --rm \ -v ./rpms/curl:/package:ro \ -v ./metadata/curl.source-pipeline.yaml:/pipeline.yaml:ro \ -v ./metadata/gpg-keys:/gpg-keys:ro \ quay.io/hummingbird-ci/source-pipeline:latest \ --version 8.21.0 \ --dry-run # Runs Fetch → Transform → Verify → Policy but skips Emit. # Exit code and report.json reflect what would happen. # No /output mount needed. ``` ## Migration Path ### Phase 1: Independent Go packages Migrate the 22 independent packages that already fetch from upstream. Convert their existing `create-vendor-tarball.sh` scripts and `download_sources` hooks into pipeline YAML definitions. Validates the tool against known-good packages. **Scope:** 7 packages with existing scripts + 15 without. Of the 7 scripted packages, 4 (caddy, nats-server×2, oauth2-proxy) use an identical clone + `go mod vendor` + tar pattern that maps directly to `vendor: {ecosystem: go}`. etcd requires multi-submodule vendor support. The 2 otel-collector packages need a `run:` block for OCB code generation, but their vendor step is generic. ### Phase 2: Trivial clean packages Enable the default pipeline behavior (fetch from spec URLs) for the ~351 clean packages where Fedora's tarball is identical to upstream. This is the highest-impact, lowest-effort phase. **Prerequisites:** - Audit to classify packages as trivial vs. transformed (HUM-4620) - `dist_git.py update` integration complete (HUM-4621) **Per-package steps:** 1. Run the pipeline tool, compare output against existing Fedora-sourced tarball 2. If identical: add `forked_from: hummingbird` to `package-overrides.yaml` 3. If different: flag for Phase 3 ### Phase 3: Transformed packages Write pipeline YAML definitions for packages where Fedora modifies the tarball. Based on the script audit, most transforms decompose into built-in primitives: - **Strip + repack** (nodejs×5, cyrus-sasl, perl-libnet, java in-tree libs): `strip-tarball` - **Git snapshots** (libX11 family×5, oniguruma, php-jsqueeze): `fetch: {git: ...}` - **Composer vendor** (composer): `vendor: {ecosystem: composer}` Only ~7 packages (nss-fips, openssl-fips-provider, ca-certificates, gcc, gdb, python3.14, erlang27) require `run:` blocks for genuinely custom logic. **Scope:** determined by Phase 2 audit. ### Phase 4: Policy enforcement Add `policy` sections to pipeline YAMLs for packages with known vendor dependency constraints. Initially driven by CVE fixes that need to survive upstream updates. ### Phase 5: Patch verification Enable `patches.verify` and `patches.classify` for packages. Start with warn-only (`fail-on-unverified: false`), gather data on classification accuracy, then selectively enable fail-on-unverified for high-risk packages. ## Operational Health ### Escape hatch ratio The primary health metric for the source pipeline is the **escape hatch ratio**: the percentage of pipeline YAML definitions that contain `run:` blocks (in `transform` or `post` sections) relative to the total number of pipeline YAML definitions. A `run:` block means the package needs custom shell commands that the built-in primitives cannot express. Every concern that accumulates over time -- ecosystem coverage gaps, upstream behavioral drift, lock file format churn, new language ecosystems -- ultimately manifests the same way: a package that cannot be expressed declaratively gets a `run:` block. This makes the ratio a single number that aggregates all pressures on the declarative model. **Interpretation:** | Ratio | Signal | | ----- | ------ | | < 15% | Healthy. The primitive set covers real-world needs. | | 15-25% | Watch. Look for repeated patterns in `run:` blocks that should become primitives. | | > 25% | Action needed. The YAML is becoming a workflow engine. Either expand the primitive set or reconsider the abstraction. | **When the ratio climbs,** examine the `run:` blocks for repeated patterns. If three or more packages use `run:` to do the same thing (e.g. a new vendoring ecosystem, a common tarball repack operation), that pattern should become a built-in primitive. The ratio climbing is not inherently bad -- it is bad only if the response is to keep adding one-off scripts instead of investing in the primitive set. ### CI enforcement The `ci/validate_pipeline_health.py` script scans all `metadata/*.source-pipeline.yaml` files, calculates the escape hatch ratio, and reports the result. It runs as part of `make check`. - **During migration (Phases 1-3):** warn mode. The script prints the ratio and flags if it exceeds 25%, but does not fail CI. This avoids blocking legitimate migration work where some packages temporarily need `run:` blocks before new primitives are added. - **After Phase 3:** enable `--fail` mode. Once the primitive set is established and migration is complete, exceeding 25% is a hard CI failure that forces a conversation before a `run:` block is added. -------------------------------------------------------------------------------- # Agent-Friendly Documentation url: https://hummingbird-project.io/docs/background/doc-agent-support/ description: How AI agents and LLMs can discover and consume the Project Hummingbird documentation site. The documentation site provides several features that make it easier for AI agents and LLMs to discover, index, and consume its content. ## Discovery and access methods ### llms.txt The site publishes an [llms.txt][llms-txt] file at the root, following the [llmstxt.org][llmstxt-spec] specification. This file is a Markdown-formatted index of all documentation pages, grouped by section, with links to each page's markdown URL. It serves as the primary entry point for agents discovering the site. ### llms-full.txt The site also publishes [llms-full.txt][llms-full-txt] -- a full content dump of all documentation pages in a single file. This is intended for LLMs with large context windows that want to ingest the entire site in a single request without crawling individual pages. Pages whose rendered content exceeds a size threshold (currently 50,000 characters) are replaced with a placeholder that includes the page title, description, and a link to the individual page's markdown URL for on-demand fetching. This keeps the file at a manageable size while ensuring no content is completely inaccessible. ### Per-page markdown output Every page on the site is available in clean markdown format by appending `/index.md` to its URL. For example: - HTML: `https://hummingbird-project.io/docs/contributing/quickstart/` - Markdown: `https://hummingbird-project.io/docs/contributing/quickstart/index.md` The markdown output resolves all shortcodes (such as file includes from other repositories) while preserving the content as markdown rather than converting to HTML. Each markdown page includes links to both `llms.txt` and `llms-full.txt` so agents can discover the site-wide indexes from any page. ## Specifications and tools The site aims to follow the emerging standards for agent-friendly documentation: - [llmstxt.org][llmstxt-spec] -- the llms.txt specification - [Agent-Friendly Documentation Spec][afdocs-spec] -- broader specification for documentation sites that serve AI agents - [AFDocs][afdocs-tool] -- scoring tool that evaluates a site's compliance with the Agent-Friendly Documentation Spec ## Planned features ### Content negotiation (HUM-6570) A CloudFront content-negotiation layer will allow agents to request `text/markdown` via the HTTP `Accept` header and receive the markdown variant without knowing the `/index.md` URL convention. This is a requirement of the Agent-Friendly Documentation Spec. ### HTML llms-directive (HUM-6571) A Docsy upgrade to v0.17.0 will embed a pointer to `llms.txt` inside the HTML `` of every page, so agents landing on any HTML page can discover the LLMS index without prior knowledge of the site structure. ## Documentation quality evaluation (HUM-6575) A [promptfoo][promptfoo]-based evaluation harness will measure how accurately LLMs can answer questions about Hummingbird using the published documentation. This provides a feedback loop for identifying documentation pages that need improvement for agent consumption. [llms-txt]: /llms.txt [llms-full-txt]: /llms-full.txt [llmstxt-spec]: https://llmstxt.org/ [afdocs-spec]: https://agentdocsspec.com/ [afdocs-tool]: https://afdocs.dev/ [promptfoo]: https://github.com/promptfoo/promptfoo -------------------------------------------------------------------------------- # Containers repository url: https://hummingbird-project.io/docs/background/containers/ Background documentation sourced from the containers repository. -------------------------------------------------------------------------------- # Image Pipeline url: https://hummingbird-project.io/docs/background/containers/image-pipeline/ description: How the container image pipeline works from source to release Explanation of the complete container image pipeline, from RPM dependency updates through source templates, generation, build, testing, release, and advisory publication. ## Overview The container image pipeline consists of six main stages: 1. **Source Templates** - Jinja2 templates in `images/*/` define container images 2. **Generation** - Templates are rendered into Containerfiles and Konflux resources 3. **Build** - Konflux builds multi-architecture container images 4. **Testing** - Testing Farm runs integration tests via Konflux 5. **Enterprise Contract Validation** - Conforma validates policy compliance before release 6. **Release** - Images are published to registries and advisories are created RPM packages are built and published separately via the [RPM Pipeline][rpm-pipeline]. This document covers what happens after RPMs are available in the Hummingbird package repository. [rpm-pipeline]: https://hummingbird-project.io/l/rpm-pipeline ## RPM Dependency Updates Container images pin exact RPM versions via lockfiles. When new RPMs are published to the Hummingbird Pulp repository, lockfiles must be regenerated for images to pick up the updates. ### Lockfile structure Each image variant has two RPM-related files: | File | Purpose | |---------------------------------------------------------|------------------------------------------| | `images////rpms/rpms.in.yaml` | Declares required packages | | `images////rpms/rpms.lock.yaml` | Pins exact versions, URLs, and checksums | The lockfile contains every RPM (direct and transitive dependencies) with its exact version, architecture, URL pointing to the Pulp repository, and checksum. During build, the locked RPMs are downloaded in hermetic mode — no network access to external repositories. ### Automatic lockfile updates **MintMaker / Renovate** — A custom Renovate instance (forked `rpm-lockfile` manager from `redhat-exd-rebuilds/renovate`) runs as a Kubernetes CronJob. It scans each `rpms.in.yaml` file individually, resolves dependencies against Pulp repositories, and opens auto-merge MRs — one per image group and distro (e.g., "Refresh RPM lockfiles for go-1-25/hummingbird"). The underlying tool is [rpm-lockfile-prototype][], which wraps the DNF dependency solver. For a given `rpms.in.yaml`, it resolves all direct and transitive dependencies against the yum repositories defined in `yum-repos/*.repo` and outputs a complete `rpms.lock.yaml` with pinned versions, download URLs, and checksums. Locally, `generate_rpms_lock.py` adds hash-based optimization: it hashes the input file and skips regeneration when the hash matches the existing lockfile, avoiding expensive solver runs when nothing has changed. [rpm-lockfile-prototype]: https://github.com/konflux-ci/rpm-lockfile-prototype ### Manual lockfile updates To refresh the lockfile for a single image variant: ```bash make images////rpms/rpms.lock.yaml FORCE_REFRESH=true ``` For example: ```bash make images/caddy/hummingbird/default/rpms/rpms.lock.yaml FORCE_REFRESH=true ``` To regenerate all lockfiles and open MRs for the changes: ```bash # Regenerate all rpms.lock.yaml files make all-host FORCE_REFRESH=true # Open MRs for any changed lockfiles ci/create_lockfile_update_mrs.sh ``` ### From lockfile MR to container build When a lockfile MR merges to main, PipelinesAsCode triggers a container build (Stage 3). Each image variant has CEL expressions in its PipelineRun definition that match on file paths, so only images whose lockfiles changed get rebuilt. ## Stage 1: Source Templates Each container image is defined by templates in `images//`: - **`properties.yml`** - Image configuration (packages, variants, tags, etc.) - **`Containerfile.j2`** - Jinja2 template for the container build - **`README.md.j2`** - Documentation template - **`tests-container.yml`** - Integration test definitions Templates use reusable macros from `macros/*.yml.j2`: - `setup_newroot()` - Configures DNF and filesystem - `install_newroot()` - Installs packages - `cleanup_newroot()` - Cleans up files - `final_stage()` - Creates scratch-based final image Shared configuration is defined in `images/variables.yml`. See [Global Variables Reference][global-vars] for details. ## Stage 2: Generation Templates are rendered into concrete artifacts that drive the pipeline: - **Containerfiles** - Build instructions for each image variant - **README documentation** - Image documentation for Quay.io - **Konflux resources** - CI/CD pipeline definitions ### Containerfile Generation Containerfiles are generated from templates for each image variant using Make's incremental build system: ```bash make ``` This combines: - Reusable macros from `macros/` - Service-specific templates from `images/*/Containerfile.j2` - Configuration from `properties.yml` (see [Image Configuration Reference][image-config]) - Variables from `images/variables.yml` (see [Global Variables Reference][global-vars]) - RPM versions from `rpms.lock.yaml` files - Git submodule information from `.gitmodules` Output: `images///Containerfile`, along with `VERSION` and `TAGS` files The build system uses timestamp-based dependency tracking, so only changed files are regenerated. ### README Generation After Containerfiles are generated, README documentation is generated from `README.md.j2` templates: 1. Tag values are extracted from the generated Containerfile labels 2. README is rendered using macros from `macros/readme.yml.j2` 3. Generated README includes actual version tags from the Containerfile Output: `images//README.md` ### Konflux Resource Generation Konflux CI/CD resources are generated from templates in `konflux-templates/`: ```bash make ``` This generates: - **Components** - Define what to build (one per image variant) - **ImageRepositories** - Define where to push images - **ReleasePlanAdmissions** - Define how to release images Output: `konflux-templates/rendered.yml` These resources must be deployed to Konflux before builds can run. See the [Konflux Resource Deployment](konflux-resource-deployment.md) guide for how and when resources are deployed. ## Stage 3: Build Konflux builds container images automatically when changes are pushed to GitLab. ### Build Triggers - **Merge Requests**: Builds all changed images and triggers tests - **Main Branch**: Builds all changed images (tests do not run on main) ### Build Process For each image variant: 1. Konflux Component watches the GitLab repository 2. On changes, Konflux triggers a build PipelineRun 3. The build uses the generated Containerfile from `images///Containerfile` 4. Images are built for multiple architectures (x86_64 and aarch64) 5. Built images are pushed to the development registry ### Build Output Development images are pushed to the Red Hat User Workloads registry: ```text quay.io/redhat-user-workloads/hummingbird-tenant/----main ``` Merge request builds are tagged with: ```text quay.io/redhat-user-workloads/hummingbird-tenant/----main:on-mr-- ``` Examples: - `quay.io/redhat-user-workloads/hummingbird-tenant/curl--default--main` - `quay.io/redhat-user-workloads/hummingbird-tenant/nginx--builder--main:on-mr-123-abc1234` ### SBOM Generation Each per-architecture build also produces an SPDX 2.3 Software Bill of Materials (SBOM), attached as an OCI artifact to the image. The SBOM is assembled from two independent scans, merged into a single document: ```mermaid flowchart LR syft["Syft\n(buildah-remote-oci-ta)"] --> mobster["Mobster\n(buildah-remote-oci-ta)"] hermeto["Hermeto\n(prefetch-dependencies-oci-ta)"] --> mobster mobster --> sbom["Per-arch SPDX SBOM"] sbom --> index["Mobster\n(build-image-index)"] index --> oci["Index SBOM\n(.sbom OCI artifact)"] ``` - **[Syft][]** runs as the `sbom-syft-generate` step inside the [buildah-remote-oci-ta][] Tekton task. It scans the RPM database of the built per-arch image and finds installed binary RPMs plus non-RPM packages (Go modules, pip packages, etc.). One SBOM is produced per architecture. - **[Hermeto][]** runs inside the [prefetch-dependencies-oci-ta][] Tekton task. It records all build-time dependencies from lockfiles. A single Hermeto SBOM covers all architectures and source RPMs. - **[Mobster][]** runs as the `prepare-sboms` step inside [buildah-remote-oci-ta][], merging the Syft and Hermeto SBOMs into one SPDX document per architecture. A second Mobster invocation in the [build-image-index][] task generates an index-level SBOM and attaches it as an OCI artifact. See [Security Labels and Metadata](security-labels-and-metadata.md#software-bill-of-materials-sbom) for the SBOM entry structure and how to access SBOMs from the registry. ## Stage 4: Testing Images are validated through two types of integration tests: - **Container tests** (`tests-container.yml`) - Run with Podman and Docker via [Testing Farm][testing-farm] on RHEL-9-Nightly systems - **K8s tests** (`tests-k8s.yml`) - Run in Konflux ephemeral Kubernetes namespaces Test results appear as external jobs in GitLab CI pipelines, providing pass/fail status and links to the Konflux PipelineRun. [testing-farm]: https://docs.testing-farm.io/ [infrastructure]: https://gitlab.com/redhat/hummingbird/infrastructure ### Test Triggering Tests are only triggered on merge requests, not on main branch builds. For an image group at `images//`, tests are triggered for all changes below that directory, excluding documentation-only changes. Any changes below `.tekton/` or `ci/` will trigger tests for the `caddy` image to ensure infrastructure changes work before merging. ### Test Execution For each image group, pipelines are created per variant. If there is more than one variant in an image group, additional group pipelines are created. See [Konflux group snapshot documentation][group-snapshots] for details. [group-snapshots]: https://konflux-ci.dev/docs/testing/integration/snapshots/group-snapshots/ Tests are selected based on the `variants` field in test files. Additionally, global tests from `ci/{variant}-tests/` are included. Group pipelines run tests that specify `variants: [group]`, allowing validation across multiple variants. ### Reverse Dependency Testing When a base image like `core-runtime` changes, dependent images (rust, xcaddy, etc.) need to be tested to ensure compatibility. Reverse dependency testing is enabled by default. Images can opt-out by setting `reverse_dependency_tests: false` in their `properties.yml`. This is recommended for images like "curl" which are widely used and only have a small API which their own tests cover well enough. For container tests, dependent images are rebuilt locally in the testing environment to prevent version skew and ensure dependent images are in sync with the repository status in the merge request under test. ### Integration Test Scenarios Integration tests are triggered via `IntegrationTestScenario` resources defined in the [infrastructure repository][infrastructure]: | Application | Purpose | |--------------------------------------|----------------------------------------------------------------------| | `containers-hummingbird` | Red Hat supported Hummingbird images | | `containers-community-hummingbird` | Community images (`support_level: community`) | | `containers-ci-hummingbird` | Experimental/infrastructure images (`support_level: experimental`) | | `containers-rawhide` | All Rawhide-based images | ### Container Testing Container tests run via [Testing Farm][testing-farm] on RHEL-9-Nightly systems for both `x86_64` and `aarch64` architectures. #### Container Testing Flow 1. Developer opens MR modifying `images/nginx/` 2. GitLab CI pipeline and Konflux pipelines start in parallel 3. Konflux builds `nginx` image variants 4. `IntegrationTestScenario` triggers Testing Farm job 5. tmt discovers fmf plan (`ci/run_tests_container.fmf`) 6. Testing Farm provisions machines (`x86_64` and `aarch64` with RHEL-9-Nightly) 7. tmt sets up the testing environment 8. tmt runs tests via `ci/run_tests_container.sh` #### Container Test ITS Configuration The container tests use the upstream [Testing Farm pipeline for Konflux CI][integrations-konflux]: [integrations-konflux]: https://gitlab.com/testing-farm/integrations-konflux ```yaml kind: IntegrationTestScenario spec: contexts: - {name: pull_request} params: - {name: COMPOSE, value: RHEL-9-Nightly} - {name: ARCH, value: x86_64|aarch64} # one for each - {name: IMAGE_TAG, value: v3.2} resolverRef: resolver: bundles params: - {name: bundle, value: quay.io/testing-farm/tmt-via-testing-farm:$(params.IMAGE_TAG)} - {name: name, value: tmt-via-testing-farm} - {name: kind, value: pipeline} ``` #### FMF Test Plan The root folder of the `containers` repository is marked with a `.fmf` directory to enable Testing Farm support. The actual test plan in `ci/run_tests_container.fmf` runs the following steps: 1. Install `podman` for container testing 2. Install Docker and start the Docker daemon on the host for Docker integration tests 3. Fix git submodules until [TFT-3991] is resolved 4. For regular pipelines: - Verify the image built by Konflux is reproducible using `ci/test_rebuild.sh` - If needed, build reverse dependency images locally via Buildah - Run Podman and Docker tests via `ci/run_tests_container.sh --component-name` 5. For group pipelines: - Run Podman and Docker group tests via `ci/run_tests_container.sh --group-component-name` [TFT-3991]: https://issues.redhat.com/browse/TFT-3991 #### Container Test Environment Testing Farm provides these environment variables to the test plan: | Variable | Description | |-----------------|------------------------------------------------| | `IMAGE_NAME` | Single component (e.g., `curl--default--main`) | | `IMAGE_NAMES` | Multiple components for group pipelines | | `IMAGE_URL` | Image URL from Konflux | | `IMAGE_URL_...` | Image URLs from Konflux for group pipelines | | `SNAPSHOT_b64` | Snapshot metadata (base64 encoded) | ### K8s Testing K8s tests run in Konflux ephemeral namespaces provisioned via EaaS (Environment as a Service). #### K8s Testing Flow 1. Developer opens MR modifying an image in the containers repository (e.g., `nginx`) 2. GitLab CI pipeline and Konflux pipelines start in parallel 3. Konflux builds image variants 4. `IntegrationTestScenario` triggers K8s test pipeline 5. Pipeline checks for `tests-k8s.yml`; skips with SUCCESS if not found 6. Pipeline provisions ephemeral namespace via Konflux EaaS (tied to PipelineRun lifecycle) 7. Pipeline fetches source via Trusted Artifacts 8. Tests run via `ci/run_tests_k8s.sh` with kubeconfig for ephemeral namespace 9. Pipeline fails if any test reports non-SUCCESS, ensuring GitLab sees correct status #### K8s Test ITS Configuration The K8s tests use the [k8s-test-pipeline][k8s-pipeline]: [k8s-pipeline]: https://gitlab.com/redhat/hummingbird/pipelines/k8s-test-pipeline ```yaml kind: IntegrationTestScenario spec: contexts: - {name: pull_request} resolverRef: resolver: bundles params: - {name: bundle, value: quay.io/hummingbird-ci/k8s-test-pipeline:latest} - {name: name, value: k8s-test} - {name: kind, value: pipeline} ``` ## Stage 5: Enterprise Contract Validation Before images can be released, they must pass [Enterprise Contract][ec] (also known as [Conforma][conforma]) policy validation. This ensures images meet security, compliance, and build quality standards. These checks can also be [run locally][local-conforma-checks] against Konflux-built images. [ec]: https://enterprisecontract.dev/ [conforma]: https://conforma.dev/ [local-conforma-checks]: https://hummingbird-project.io/l/running-conforma-checks-locally ### Policy Validation Enterprise Contract validates that: - Images are built using trusted, verified Tekton tasks - Builds are hermetic (network-isolated with pre-fetched dependencies) - Required security tests have passed - Images have proper metadata and labels - Build artifacts meet supply chain security requirements ### Policy Configuration Policies are defined as `EnterpriseContractPolicy` resources in [`konflux-templates/macros/policy.yml.j2`][policy-macro]: [policy-macro]: https://gitlab.com/redhat/hummingbird/containers/-/blob/main/konflux-templates/macros/policy.yml.j2 - **`containers-hummingbird`** / **`containers-rawhide`** - Strict policies for production images - **`containers-community-hummingbird`** - Policy for community-supported images in the containers repository (same base exclusions as production policies) These policies use the [`@redhat` rule collection][ec-redhat] from the [ec-release-policy][ec-policy]. [ec-redhat]: https://conforma.dev/docs/policy/release_policy.html#redhat [ec-policy]: https://github.com/enterprise-contract/ec-policies ### Policy Exclusions The following checks are excluded from the default `@redhat` policy set. When modifying exclusions, update the [policy macro][policy-macro] and this documentation. #### Test Package The [test package][ec-test] verifies that each build was subjected to a set of tests and that those tests all passed. Snyk SAST checks (`test.required_tests_passed:sast-snyk-check`, `test.no_skipped_tests:sast-snyk-check`, `test.required_tests_passed:sast-snyk-check-oci-ta`, `test.no_skipped_tests:sast-snyk-check-oci-ta`) are excluded because Hummingbird images are currently not supported by Snyk. Red Hat certification preflight checks (`test.no_failed_tests:ecosystem-cert-preflight-checks`, `test.no_erred_tests:ecosystem-cert-preflight-checks`) are excluded because Hummingbird images are not yet published to the Red Hat certified container registry. Informative test failures (`test.no_failed_informative_tests`) are excluded because these produce warnings for advisory purposes only and are explicitly non-blocking. Deprecated image warnings (`test.no_test_warnings:deprecated-image-check`) are excluded because final images are built `FROM scratch`, and the builder image is updated via Renovate like all other images. [ec-test]: https://conforma.dev/docs/policy/packages/release_test.html #### Trusted Task Package The [trusted_task package][ec-trusted-task] verifies that all Tekton Tasks involved in building the image are trusted by comparing Task references with a pre-defined list of trusted Tasks. The `trusted_task.current` check warns when newer versions of tasks are available. This is excluded because we use stable pinned task versions and control upgrade timing via Renovate rather than requiring the latest version at all times. [ec-trusted-task]: https://conforma.dev/docs/policy/packages/release_trusted_task.html #### RPM Repos Package The [rpm_repos package][ec-rpm-repos] confirms that all RPM packages listed in SBOMs specify a known and permitted repository ID. The `rpm_repos.ids_known` check is excluded because images use the internal [hummingbird repository](../../yum-repos/hummingbird.repo) and Fedora repositories, which are not in the upstream [known_rpm_repositories.yml][known-repos] list (that file only contains Red Hat official repositories). [ec-rpm-repos]: https://conforma.dev/docs/policy/packages/release_rpm_repos.html [known-repos]: https://github.com/release-engineering/rhtap-ec-policy/blob/main/data/known_rpm_repositories.yml #### Labels Package The [labels package][ec-labels] checks if the image has the expected labels set, including required and optional labels for Red Hat container certification. Both `labels.required_labels` and `labels.optional_labels` are excluded because images currently only include basic labels (`maintainer`, `license_terms`, `name`, `cpe`) and version labels, not the full set of Red Hat certification labels (`vendor`, `version`, `release`, `summary`, `description`, `url`, etc.) required for Red Hat Ecosystem Catalog publishing. [ec-labels]: https://conforma.dev/docs/policy/packages/release_labels.html #### Buildah Build Task Package The [buildah_build_task package][ec-buildah] verifies buildah build task parameters. The `buildah_build_task.privileged_nested_param` check verifies that PRIVILEGED_NESTED is not set to true. This is excluded because images use the [`dnf-installroot`][dnf-installroot] helper from the builder image to build all containers, including the builder image itself. This script requires privileged operations (`unshare`, `mount -t tmpfs`, `mount --bind` for `/proc` and `/dev/*`) to set up the install root environment (see [commit 3668af16][privileged-commit]). [ec-buildah]: https://conforma.dev/docs/policy/packages/release_buildah_build_task.html [dnf-installroot]: https://gitlab.com/redhat/hummingbird/containers/-/blob/main/images/hummingbird-builder/dnf-installroot.sh [privileged-commit]: https://gitlab.com/redhat/hummingbird/containers/-/commit/3668af16 #### Schedule Package The [schedule package][ec-schedule] verifies that releases conform to a given schedule, including weekday restrictions. The `schedule.weekday_restriction` check is excluded to allow releases any day including weekends. The `@redhat` policy restricts weekend releases, but this project needs the ability to ship urgent CVE fixes immediately regardless of the day of week. [ec-schedule]: https://conforma.dev/docs/policy/packages/release_schedule.html #### CVE Package The [cve package][ec-cve] checks for blocking and non-blocking CVEs in container images. The `cve.cve_blockers` check is excluded because blocking on known CVEs would prevent releasing images that fix other CVEs. A VEX feed could suppress false positives for RPM-level CVEs, but CVEs can also originate from other artifact types where VEX does not apply (see [MR !2227][cve-mr]). [ec-cve]: https://conforma.dev/docs/policy/packages/release_cve.html [cve-mr]: https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/2227 #### Hermetic Task Package (CI-Only) The [hermetic_task package][ec-hermetic] verifies that tasks were invoked with the proper parameters to perform a hermetic (network-isolated) execution. All Konflux applications in the containers repository enforce the hermetic task check. All applications allow `quay.io/hummingbird-ci/` as a base image source because images build `FROM quay.io/hummingbird-ci/hummingbird-builder` via `setup_newroot.yml.j2`. [ec-hermetic]: https://conforma.dev/docs/policy/packages/release_hermetic_task.html ## Stage 6: Release After images pass testing in merge requests and are merged to main, they are released to public registries. ### Registry Organization Images are published to different registries based on distro and purpose: | Registry | Purpose | |----------------------------------|---------------------------------------------------| | `registry.access.redhat.com/hi/` | Red Hat supported Hummingbird images (production) | | `quay.io/hummingbird/` | Red Hat supported Hummingbird images (mirror) | | `quay.io/hummingbird-community/` | Community-supported Hummingbird images | | `quay.io/hummingbird-rawhide/` | All Rawhide-based images | | `quay.io/hummingbird-ci/` | CI and tools images | **Production Registry:** Red Hat supported Hummingbird images are published to `registry.access.redhat.com/hi/` via the `rh-advisories` release pipeline and mirrored to `quay.io/hummingbird/`. **Community Registry (`hummingbird-community`):** Publishes community-supported images (those with `support_level: community` in `properties.yml`). Examples include `minio`, `minio-client`, and `bootc-os`. **CI/tools Registry (`hummingbird-ci`):** Publishes tooling and infrastructure images. This includes tools-repo images (built outside the containers repository) and experimental images from the containers repository (`support_level: experimental`), such as `hummingbird-builder`. ### Release Process 1. Merge request passes all tests 2. Merge request is merged to main branch 3. Konflux builds images from main 4. ReleasePlanAdmission resources trigger the release pipeline 5. Images are signed via Cosign for supply chain attestation 6. Images are copied from the Konflux registry to target registries 7. Tags are applied based on `properties.yml` configuration 8. For production releases: images are registered in the Red Hat container catalog via Pyxis ### Release Mechanism Releases are configured via ReleasePlanAdmission (RPA) resources in the containers repo and ReleasePlan resources in the infrastructure repo: - **ReleasePlanAdmission** - Defines per-registry release configuration (target registry, tags, visibility settings) - **ReleasePlan** - Triggers the release pipeline for a specific application and registry Each distro/registry combination has its own RPA. #### Production release pipeline The production release uses the `rh-advisories` pipeline from the [release-service-catalog][]. The `push-snapshot` task pushes images from the development registry to: - Production: `registry.access.redhat.com/hi/:` Configuration is defined in `releng/hummingbird-containers-prod.yaml`. [release-service-catalog]: https://github.com/konflux-ci/release-service-catalog #### Image signing Images are signed using Cosign as part of the release pipeline. Signing provides supply chain attestation, allowing consumers to verify image provenance. #### Pyxis catalog registration The release pipeline registers images in the [Red Hat container catalog][catalog] via the Pyxis API: - Product: Red Hat Hardened Images (Product ID 1071) - Metadata: version, categories, layer information, SBOM references - Configuration: `releng/pyxis-hummingbird.yaml` Pyxis registration makes images discoverable in the Red Hat Ecosystem Catalog and links them to security advisories. [catalog]: https://catalog.redhat.com/ ### Release Output Released images are published to registries based on distro and support level: ```text registry.access.redhat.com/hi/: quay.io/hummingbird/: quay.io/hummingbird-community/: quay.io/hummingbird-rawhide/: quay.io/hummingbird-ci/: ``` Examples: - `registry.access.redhat.com/hi/curl:8` - Production Red Hat supported image - `quay.io/hummingbird/nodejs:20` - Red Hat supported Hummingbird image - `quay.io/hummingbird-community/minio:latest` - Community-supported image - `quay.io/hummingbird-ci/hummingbird-builder:latest` - Hummingbird Builder image - `quay.io/hummingbird-rawhide/curl:latest` - Rawhide image ### Release Tags Tags are extracted from Containerfile labels as defined in `properties.yml`: - `latest` - Latest version of the image - `` - Major version (e.g., `20` for Node.js 20.x) - `.` - Major.minor version (e.g., `20.11`) - `` - Complete version with release (e.g., `20.11.1-1.fc42`) - `` - Build timestamp (production releases only) Non-default variants receive a `-` suffix (e.g., `latest-builder`). ### Advisory Creation When the production release pipeline runs, advisories are created: 1. The Release Service generates an Advisory YAML from ReleasePlan and ReleasePlanAdmission metadata 2. The advisory is pushed to the advisories repo on CEE GitLab 3. GitLab CI validates the advisory against the schema and enforces field-level permissions 4. On merge to main, the advisory is published via UMB and Kafka Advisory types: | Type | Meaning | |------|----------------------------------------| | RHSA | Security Advisory (contains CVE fixes) | | RHBA | Bug Fix Advisory | | RHEA | Enhancement Advisory | After publication, the Pyxis API links each image record to its advisory via `image_advisory_id`. ### VEX Feed Update After advisory publication, Red Hat SDEngine generates public vulnerability data: - **CSAF VEX documents** (per-CVE) published at `https://security.access.redhat.com/data/csaf/v2/vex-feed/` - **CSAF Advisory documents** (per-advisory) published at `https://security.access.redhat.com/data/csaf/v2/advisories/` The `hummingbird-vex-feed` repository maintains CPE mappings that associate Hummingbird package repositories with `cpe:/a:redhat:hummingbird:1`. ### Image Documentation When a commit to main changes a `README.md` file, the content is automatically pushed to quay.io as the image description via the `update_quay_description` job. ## References - [RPM Pipeline][rpm-pipeline] - How RPM spec changes flow through the build pipeline to the package repository - [Development Workflow][dev-workflow] - Practical guide for contributing - [Adding Images][adding-images] - Step-by-step guide for adding new images - [Testing Guide][testing] - How to run and write tests locally - [Image Configuration Reference][image-config] - Complete `properties.yml` reference [global-vars]: global-variables-reference.md [image-config]: https://hummingbird-project.io/l/image-configuration-reference [dev-workflow]: https://hummingbird-project.io/l/development-workflow [adding-images]: https://hummingbird-project.io/l/adding-images [testing]: https://hummingbird-project.io/l/testing-images [Syft]: https://github.com/anchore/syft [Hermeto]: https://github.com/hermetoproject/hermeto [Mobster]: https://github.com/konflux-ci/mobster [buildah-remote-oci-ta]: https://github.com/konflux-ci/build-definitions/tree/main/task/buildah-remote-oci-ta [prefetch-dependencies-oci-ta]: https://github.com/konflux-ci/build-definitions/tree/main/task/prefetch-dependencies-oci-ta [build-image-index]: https://github.com/konflux-ci/build-definitions/tree/main/task/build-image-index -------------------------------------------------------------------------------- # Global Variables Reference url: https://hummingbird-project.io/docs/background/containers/global-variables-reference/ description: Complete reference for global configuration in images/variables.yml Complete reference for global configuration settings that apply to all container images. ## Overview The `images/variables.yml` file contains global configuration that applies to all container images in the repository. These settings define project-wide defaults, security parameters, and variant-specific behavior. **Location:** `images/variables.yml` (in the containers repository root) **Scope:** All images inherit these settings unless overridden in image-specific `properties.yml` ## Configuration Reference ### cpe - **Type**: String (CPE 2.2 formatted string) - **Default**: `"cpe:/a:redhat:hummingbird:1"` - **Description**: Common Platform Enumeration (CPE) identifier for all Hummingbird containers. - **Usage**: Automatically added as a label to all generated container images and passed to `inject-source-info` during the build process. - **Format**: CPE 2.2 URI format: `cpe:/part:vendor:product:version` ### default_user - **Type**: String (numeric UID) - **Default**: `"65532"` - **Description**: The default unprivileged user ID that all containers run as. - **Usage**: Referenced by the `{{ set_user() }}` macro when `user: default` is specified (or when `user:` is omitted, as `default` is the default value). - **Value**: `65532` is chosen as a high, non-conflicting UID that: - Avoids conflicts with system users (typically < 1000) - Avoids conflicts with regular users (typically 1000-60000) ### default_distros - **Type**: Array of strings - **Default**: `["rawhide", "hummingbird"]` - **Description**: Defines the distribution variants (distros) for which images are built. Each distro represents a different base package source. - **Directory Structure**: Images are organized as `images////` - **Common Values**: - `rawhide` - Uses Fedora Rawhide repositories only - `hummingbird` - Uses both Fedora Rawhide and Hummingbird repositories - **Usage**: Scripts iterate over distros and variants to find all image variants to build. Each distro/variant combination produces a separate container image. - **See Also**: [default_variant_repos](#default_variant_repos) for repository configuration per distro ### default_variants - **Type**: Array of strings - **Default**: `["default", "builder"]` - **Description**: Defines the default set of variants to generate for each image within each distro when not explicitly specified in the image's `properties.yml`. - **Directory Structure**: Variants are subdirectories within each distro: `images////` - **Common Values**: - `default` - The minimal runtime variant - `builder` - Extended variant with build tools and package managers - **Extending**: Images can use `additional_variants` in `properties.yml` to add extra variants while keeping the defaults (preferred approach) - **Override**: Images can specify `variants` in `properties.yml` to replace these defaults entirely (use when you need to exclude default variants) - **See Also**: [Image Configuration Reference - variants][variants-field] ### default_rpm_packages - **Type**: Object with variant names as keys, arrays of package names as values - **Default**: ```yaml default_rpm_packages: builder: - bash - dnf5 - shadow-utils ``` - **Description**: Defines packages that are automatically included in specific variants across all images. These are added in addition to packages defined in image-specific `properties.yml`. - **Variants**: Currently only `builder` is defined, but other variants can be added - **Usage**: Use this to provide variant-specific packages that should be available in all images using that variant. For example, builder variants include package management tools for development workflows. ### default_variant_repos - **Type**: Object with distro names as keys, arrays of repository filenames as values - **Default**: ```yaml default_variant_repos: rawhide: - fedora-44.repo hummingbird: - fedora-43.repo - hummingbird.repo ``` - **Description**: Defines which yum repositories each distro uses by default for package installation and lockfile generation. The Rawhide distro uses the latest branched Fedora development repo, while Hummingbird uses a stable Fedora release plus the Hummingbird package repository. - **Repository Files**: References files in the `yum-repos/` directory - **Usage**: This allows different distros to use different package sources. The `hummingbird` distro includes additional repositories that provide Hummingbird-specific packages. - **Override**: Image-specific `additional_repos` in `properties.yml` are appended to the distro-specific repos - **See Also**: [Image Configuration Reference - additional_repos][additional_repos] ### oscap - **Type**: Object - **Default**: ```yaml oscap: enabled: true profiles: cis: true stig: variants: - "*fips*" exclude_rules: - id: xccdf_org.ssgproject.content_rule_root_path_no_dot reason: "False positive: environmentvariable58 probe requires /proc which is unavailable in offline chroot scanning" ``` - **Description**: Global defaults for OpenSCAP compliance scanning. Scanning is enabled by default (`enabled: true`). Images can opt out by setting `enabled: false` in their `properties.yml`. - **Fields**: - `enabled` - Whether compliance scanning is active (overridden per image) - `profiles` - Which compliance profiles run per variant. Each profile can be `true` (all variants), `false` (disabled), or `{variants: [...]}` with glob patterns. Default: CIS for all variants, STIG for FIPS variants. - `exclude_rules` - Global rule exclusions applied to all oscap-enabled images. Image-specific exclusions in `properties.yml` are concatenated with these (not replaced). - **Merge Behavior**: When an image overrides `oscap` fields: - `enabled` (scalar) is replaced by the image value - `profiles` (dict) is recursively merged, so an image can override individual profiles without affecting others - `exclude_rules` (list) is concatenated, so image rules are added after global rules - **See Also**: [Image Configuration Reference - Compliance Scanning][oscap-config] ### readme_targets - **Type**: Object with target names as keys, configuration objects as values - **Default**: See the actual file - **Description**: Defines target-specific configuration for generating multiple README files from a single template. Each target represents a different build/distribution channel (e.g., Hummingbird project vs Red Hat product). - **Target Configuration Fields**: - `filename` - Output filename for this target's README - `registry` - Container registry URL for image references - `product_name` - Full product name for documentation - `product_name_short` - Short product name for headings and titles - `doc_base_url` - Base URL for documentation links - **Usage**: Templates access these values via `{{ readme_targets[target].registry }}` and similar expressions. The build system generates one README file per target from the same `README.md.j2` template. - **See Also**: [Image Pipeline - README Generation][readme-generation] ## Next Steps - [Image Configuration Reference][image-config] - Per-image configuration - [Image Variants][image-variants] - Understanding variant types - [Adding Images][adding-images] - How to add a new container image [additional_repos]: https://hummingbird-project.io/l/image-configuration-reference#additional_repos [image-config]: https://hummingbird-project.io/l/image-configuration-reference [image-variants]: https://hummingbird-project.io/l/image-variants [adding-images]: https://hummingbird-project.io/l/adding-images [variants-field]: https://hummingbird-project.io/l/image-configuration-reference#variants [readme-generation]: https://hummingbird-project.io/l/image-pipeline/#readme-generation [oscap-config]: https://hummingbird-project.io/l/image-configuration-reference#compliance-scanning -------------------------------------------------------------------------------- # Konflux Resource Deployment url: https://hummingbird-project.io/docs/background/containers/konflux-resource-deployment/ description: How Konflux resources are defined and deployed across repositories Konflux resources for container images are split between two repositories: - **[containers]**: Per-image resources generated from `properties.yml` - **[infrastructure]**: Application-level resources and deployment to the Konflux cluster [containers]: https://gitlab.com/redhat/hummingbird/containers [infrastructure]: https://gitlab.com/redhat/hummingbird/infrastructure This separation allows independent iteration on each concern. ## Deployment Matrix | Resource Type | Source | Containers MR | Containers Push | Infra MR | Infra Push | | ------------------------ | -------------- | ------------- | --------------- | -------- | ---------- | | Component | containers | manual | manual | - | - | | ImageRepository | containers | manual | manual | - | - | | ReleasePlanAdmission | containers | - | automatic | manual | automatic | | EnterpriseContractPolicy | containers | - | automatic | manual | automatic | | Application | infrastructure | - | - | manual | automatic | | ReleasePlan | infrastructure | - | - | manual | automatic | | IntegrationTestScenario | infrastructure | - | - | manual | automatic | | ServiceAccount, Secret | infrastructure | - | - | manual | automatic | **Legend**: - **Containers MR**: Downstream pipeline triggered from containers MR (deploys from MR commit) - **Containers Push**: Downstream pipeline triggered from containers push to main - **Infra MR**: Infrastructure MR pipeline (manual trigger) - **Infra Push**: Infrastructure push to main or web pipeline ## Resource Locations and Rationale ### Containers Repo Resources are defined in `konflux-templates/` and rendered to `konflux-templates/rendered.yml`. **Component and ImageRepository**: 1. **Ownership**: Components are managed by the containers repo. The infrastructure pipeline uses `ONLY_DOWNSTREAM` so only explicit downstream triggers from containers deploy changes, giving the containers repo full control over the component lifecycle. 2. **Dynamic generation**: Generated from `images/*/properties.yml` via `ci/internal/generate_konflux_resources.sh`, depending on per-image configuration (variants, tags, repository names). 3. **Timing**: Must be deployed early during MR review so Konflux can build and test new images. **ReleasePlanAdmission**: 1. **Dynamic generation**: Includes per-image tag mappings extracted from `properties.yml`. 2. **Main branch only**: Prevents race conditions—deploying from a feature branch risks undoing other images' onboarding. If branch A adds image-foo and branch B (created earlier) adds image-bar, deploying B's RPA would remove image-foo from the release config. **EnterpriseContractPolicy**: 1. **Documentation co-location**: Policy exclusions are documented in [Image Pipeline](image-pipeline.md). Keeping policy and documentation together eases maintenance. ### Infrastructure Repo **Application, ReleasePlan, and IntegrationTestScenario** are defined in `kubernetes/containers-*/` directories (one per Konflux application): 1. **Independent iteration**: Changes are decoupled from containers repo activity—they can be modified, test-deployed via manual trigger in an infrastructure MR, verified, and merged without touching the containers repo. 2. **Testable before merge**: If defined in the containers repo, these would only deploy after merging to main (like RPAs), making iteration difficult. 3. **Static configuration**: These resources don't depend on per-image data from `properties.yml`. **ServiceAccount and Secret** for releases are defined in `kubernetes/release-quay-hummingbird*/` directories (one per Quay organization): 1. **Security**: Secret specifications (names, structure, credential references) should not be exposed in the containers repo. 2. **Independent iteration**: Like other infrastructure resources, these can be modified and test-deployed without touching the containers repo. The service accounts are referenced by ReleasePlanAdmissions to authorize pushing images to each `quay.io/hummingbird*` organization. ## See Also - [Image Pipeline](image-pipeline.md) – Build, test, and release stages -------------------------------------------------------------------------------- # Security Labels and Metadata url: https://hummingbird-project.io/docs/background/containers/security-labels-and-metadata/ description: Container labels, embedded metadata, and SBOMs for vulnerability scanning Hummingbird container images include labels, embedded metadata, and SBOMs that enable security scanners to perform container-first vulnerability reporting. ## Overview Security scanners need a way to determine which vulnerabilities are applicable to a container image. Three mechanisms provide this information: 1. **Container Labels** - OCI image labels (`name`, `cpe`) set in the Containerfile 2. **Embedded Metadata** - A `labels.json` file written to the container filesystem during build 3. **Software Bill of Materials (SBOM)** - An SPDX document listing all packages, attached as an OCI artifact alongside the image Labels and embedded metadata identify the *product*; the SBOM identifies the *packages* within it. Labels and embedded metadata provide the same core information but serve different access paths: labels are accessible via container inspection tools, while embedded metadata is accessible to scanners with only filesystem access. ## Applicability The `cpe` label is only added to released **Hummingbird** distro images (no CPE identifier for non-product images). All other labels are added to all images. The `labels.json` embedded metadata file contains all Containerfile LABEL values plus auto-computed fields. SBOMs are attached to all released images (both Rawhide and Hummingbird). ## Container Labels ### name - **Type**: String - **Format**: `/` or `/-` - **Examples**: `hummingbird/nodejs-24`, `hummingbird/nodejs-24-builder`, `hummingbird/caddy` - **Description**: Canonical name for the container image. This is the name that appears in VEX (Vulnerability Exploitability eXchange) statements and is used by scanners to map vulnerabilities to specific images. The canonical name is derived from the image directory name (which includes the version, e.g. `nodejs-24`) and the variant. For default variants, the name is `hummingbird/` (e.g. `hummingbird/nodejs-24`). For non-default variants, the variant is appended with a hyphen (e.g. `hummingbird/nodejs-24-builder`). Note that the canonical name may differ from the registry repository path. For example, `nodejs-24` and `nodejs-20` are both pushed to the `hummingbird/nodejs` registry repository with different tags, but their canonical names are `hummingbird/nodejs-24` and `hummingbird/nodejs-20` respectively. See [container-image-labels.md](container-image-labels.md) for the full `name` label format across all image types. ### cpe - **Type**: String (CPE 2.2 formatted) - **Format**: `cpe:/a:redhat:hummingbird:1` - **Description**: Common Platform Enumeration (CPE) identifier for the container. CPE is a standardized naming scheme for software products that enables correlation with vulnerability databases. The CPE value is defined globally in `images/variables.yml` and applied to all Hummingbird images. Containers with the same CPE are considered the same software product for vulnerability reporting purposes. ### org.opencontainers.image.created - **Type**: String (RFC3339 timestamp) - **Format**: `2025-01-15T12:00:00Z` - **Description**: Creation timestamp of the container image. Used by scanners to determine if an image predates or postdates a vulnerability fix. If `SOURCE_DATE_EPOCH` is set during the build, that timestamp is used instead of the actual build time, supporting reproducible builds. ## Embedded Metadata (labels.json) The `labels.json` file is written to `/usr/share/buildinfo/labels.json` inside the container filesystem by the `inject-source-info` script during build. It contains all Containerfile LABEL values plus auto-computed fields like `architecture` and `org.opencontainers.image.created`. ### Schema The file follows the [embedded_metadata.v1 schema][schema] published by Red Hat Product Security. ### Fields The [embedded_metadata.v1 schema][schema] defines the minimum fields required by security scanners: | Field | Type | Description | |------------------------------------|--------|----------------------------------------------------------| | `name` | string | Canonical container name (e.g., `hummingbird/nodejs-24`) | | `cpe` | string | CPE identifier (e.g., `cpe:/a:redhat:hummingbird:1`) | | `architecture` | string | Target architecture (e.g., `amd64`, `arm64`) | | `org.opencontainers.image.created` | string | RFC3339 creation timestamp | The file also includes all other Containerfile LABEL values (e.g., `description`, `summary`, `vendor`, `version`). See [container-image-labels.md](container-image-labels.md) for the complete label reference. ### Example ```json { "name": "hummingbird/caddy", "cpe": "cpe:/a:redhat:hummingbird:1", "architecture": "amd64", "org.opencontainers.image.created": "2025-01-15T12:00:00Z" } ``` ## Software Bill of Materials (SBOM) Each released image has a per-architecture SPDX 2.3 SBOM attached as an OCI artifact. See [SBOM Generation](image-pipeline.md#sbom-generation) for how the production SBOMs are built from Syft, Hermeto, and Mobster. ### Accessing SBOMs SBOMs are stored as OCI artifacts alongside each per-architecture image. To download one, resolve the per-arch digest and use [cosign][]: ```bash IMAGE="quay.io/hummingbird/caddy" TAG="latest" ARCH="amd64" # Get per-architecture digests from the image index skopeo inspect --raw "docker://${IMAGE}:${TAG}" \ | jq '.manifests[] | {digest, platform}' # Download the SBOM for the chosen architecture ARCH_DIGEST="$(skopeo inspect --raw "docker://${IMAGE}:${TAG}" \ | jq -r --arg arch "${ARCH}" \ '.manifests[] | select(.platform.architecture == $arch) | .digest')" cosign download sbom "${IMAGE}@${ARCH_DIGEST}" > sbom.json ``` ### Entry Sources The merged SBOM contains entries from two tools: | Source | Content | Scope | |---------|------------------------------------------------|---------------------------------| | Syft | Installed binary RPMs + non-RPM packages | Image architecture only | | Hermeto | Build-time dependencies from lockfiles | All architectures + source RPMs | A package installed in the final image may have entries from both Syft and Hermeto (with different metadata), since it was both a build dependency and is present at runtime. ### Distinguishing Entry Sources After the Mobster merge, Syft and Hermeto entries carry mutually exclusive markers: | Marker | Syft | Hermeto | |---------------------------------|------|---------| | `sourceInfo` field present | ✓ | | | `licenseDeclared` set | ✓ | | | CPE references | ✓ | | | `upstream=` in PURL | ✓ | | | Annotation containing "hermeto" | | ✓ | | `repository_id=` in PURL | | ✓ | As a general rule, any entry with an annotation containing the word "hermeto" (case-insensitive) originates from Hermeto (build-time provenance). All other entries originate from Syft (runtime image scan). ### PURL Format Syft and Hermeto use different PURL qualifier sets for the same package. Examples from caddy:latest (Hummingbird, amd64): **Syft** (runtime): ```text pkg:rpm/hummingbird/caddy@2.10.2-1.hum1?arch=x86_64&distro=hummingbird-20251124&upstream=caddy-2.10.2-1.hum1.src.rpm ``` **Hermeto** (build, binary): ```text pkg:rpm/caddy@2.10.2-1.hum1?arch=x86_64&checksum=sha256:ca02a0...&repository_id=public-hummingbird-x86_64-rpms ``` **Hermeto** (build, source): ```text pkg:rpm/caddy@2.10.2-1.hum1?arch=src&checksum=sha256:42912d...&repository_id=public-hummingbird-source-rpms ``` Key differences: - **Distro namespace**: Syft includes the distro in the PURL path (`pkg:rpm/hummingbird/...`). Hermeto omits it (`pkg:rpm/...`). - **EVR**: Syft populates `versionInfo` with the full epoch:version-release. Hermeto sets `versionInfo` to the bare upstream version; the full EVR is only in the PURL `@version`. - **Architecture**: Syft entries match the image architecture. Hermeto entries carry an `arch=` PURL qualifier that may be the image arch, a different arch (cross-arch), `noarch`, or `src` (source RPMs). - **Source RPM**: Syft entries carry `upstream=` in the PURL. Hermeto entries have separate `arch=src` entries instead. ### Hermeto Annotation Format Hermeto entries carry annotations with JSON-encoded metadata: ```json { "annotationDate": "2026-03-02T11:58:12Z", "annotationType": "OTHER", "annotator": "Tool: hermeto:jsonencoded", "comment": "{\"name\": \"hermeto:found_by\", \"value\": \"hermeto\"}" } ``` ### Entry Breakdown Example Typical counts for caddy:latest (Hummingbird, amd64); exact counts vary as image dependencies change: | Source | Typical count | Content | |---------|---------------|-----------------------------------| | Syft | ~53 | Binary RPMs (x86_64 + noarch) | | Syft | 1 | Go module (stdlib) | | Syft | 2 | OCI image metadata | | Hermeto | ~41 | Binary RPMs (x86_64) | | Hermeto | ~41 | Binary RPMs (aarch64, cross-arch) | | Hermeto | ~11 | Binary RPMs (noarch) | | Hermeto | ~36 | Source RPMs (arch=src) | ## How Scanners Use This Metadata 1. **CPE Matching**: Scanners use the `cpe` value to look up applicable VEX statements for the product 2. **Name Matching**: The `name` identifies which specific container the VEX statements apply to 3. **Version Comparison**: The creation timestamp enables comparison between the scanned image and fixed versions reported in VEX statements 4. **Package Enumeration**: Scanners use the SBOM to enumerate all packages in the image and correlate them with vulnerability databases via PURLs and CPEs ## Related Files | File | Purpose | |------------------------------------------------------|-----------------------------------------| | `documentation/background/container-image-labels.md` | Complete reference for all image labels | | `documentation/background/image-pipeline.md` | SBOM generation pipeline (Stage 3) | | `images/variables.yml` | Defines the global `cpe` value | | `images/hummingbird-builder/inject-source-info.sh` | Script that creates `labels.json` | | `macros/inject_source_info_labels.yml.j2` | Macro that adds LABEL to Containerfile | | `macros/install_newroot.yml.j2` | Macro that invokes `inject-source-info` | ## See Also - [Container Image Labels](container-image-labels.md) -- complete reference for all image labels across all categories - [Image Pipeline -- SBOM Generation](image-pipeline.md#sbom-generation) -- how the production SBOMs are built from Syft, Hermeto, and Mobster ## References - [Embedded Metadata Schema][schema] - [Container-First Vulnerability Reporting][konflux-feature] (Konflux feature specification) - [VEX (Vulnerability Exploitability eXchange)][vex] - [Syft][syft] -- SBOM generation tool - [Hermeto][hermeto] -- build-time dependency provenance tool - [Mobster][mobster] -- SBOM merging tool - [SPDX Specification][spdx] [schema]: https://github.com/RedHatProductSecurity/security-data-guidelines/blob/main/schema/embedded_metadata.v1.schema.json [konflux-feature]: https://issues.redhat.com/browse/KONFLUX-6210 [vex]: https://security.access.redhat.com/data/csaf/v2/vex/ [syft]: https://github.com/anchore/syft [hermeto]: https://github.com/hermetoproject/hermeto [mobster]: https://github.com/konflux-ci/mobster [cosign]: https://github.com/sigstore/cosign [spdx]: https://spdx.github.io/spdx-spec/v2.3/ -------------------------------------------------------------------------------- # Container Image Labels url: https://hummingbird-project.io/docs/background/containers/container-image-labels/ description: Complete reference for all container image labels Hummingbird container images carry labels from multiple standards and namespaces. This page is the single reference for every label. Standards: **C** = [Conforma][conforma-labels] ([rule dataset][rule-data]), **O** = [OCI Image Spec][oci-spec], **H** = Hummingbird project, **S** = [Security schema][security-schema] ## Labels | Label | Aliases | Value | C | O | H | S | |----------------------------------------------|----------------------------------|--------------------------------------------|---|---|---|---| | `architecture` | | Host architecture (e.g., `x86_64`) | | | | ✓ | | `com.redhat.component` | | `hummingbird` | ✓ | | | | | `com.redhat.license_terms` | | [UBI EULA][ubi-eula] ¹ | | | | | | `cpe` | | CPE identifier (Hummingbird only) ² | ✓ | | | ✓ | | `distribution-scope` | | `public` | ✓ | | | | | `io.hummingbird-project.containerfile` | | Containerfile path relative to repo root ⁹ | | | ✓ | | | `io.hummingbird-project.major-minor-version` | | Major.minor from tags (e.g., `2.10`) ³ | | | ✓ | | | `io.hummingbird-project.major-version` | | Major from tags (e.g., `2`) ³ | | | ✓ | | | `io.hummingbird-project.repository` | | Publishing name (e.g., `caddy`) ⁴ | | | ✓ | | | `io.hummingbird-project.stream` | | Version stream (e.g., `2`) ⁴ | | | ✓ | | | `io.hummingbird-project.variant` | | Variant name (e.g., `fpm-builder`) ⁸ | | | ✓ | | | `io.hummingbird-project.variant.base` | | Base specialization (e.g., `fpm`) ⁸ | | | ✓ | | | `io.hummingbird-project.variant.builder` | | `true` when builder (absent otherwise) ⁸ | | | ✓ | | | `io.hummingbird-project.variant.description` | | Base variant description (no modifiers) ⁸ | | | ✓ | | | `io.hummingbird-project.variant.fips` | | `true` when FIPS (absent otherwise) ⁸ | | | ✓ | | | `io.k8s.description` | | Long description ⁵ | ✓ | | | | | `maintainer` | | `Project Hummingbird / Red Hat` | ✓ | | | | | `name` | | `hummingbird/[-]` ² | ✓ | | | ✓ | | `org.opencontainers.image.created` | `build-date` | RFC3339 build timestamp | ✓ | ✓ | | ✓ | | `org.opencontainers.image.description` | `description` | Long description ⁵ | ✓ | ✓ | | | | `org.opencontainers.image.revision` | `vcs-ref` | Full git commit SHA | ✓ | ✓ | | | | `org.opencontainers.image.source` | | GitLab repository URL | | ✓ | | | | `org.opencontainers.image.title` | | Image name (e.g., `caddy`) | | ✓ | | | | `org.opencontainers.image.url` | `url` | Upstream project URL ⁵ | ✓ | ✓ | | | | `org.opencontainers.image.vendor` | `vendor` | `Red Hat, Inc.` ⁶ | ✓ | ✓ | | | | `org.opencontainers.image.version` | `version` | Full version from tags (e.g., `2.10.2`) ³ | ✓ | ✓ | | | | `release` | | `SOURCE_DATE_EPOCH` (commit timestamp) ⁷ | ✓ | | | | | `summary` | | Short one-liner ⁵ | ✓ | | | | | `vcs-type` | | `git` | ✓ | | | | ## Notes ### ¹ Licenses The `com.redhat.license_terms` label is not required by Conforma, OCI, or the security schema. It is a Red Hat convention present on all Red Hat container images (UBI, language runtimes, etc.), pointing to the UBI EULA. The `org.opencontainers.image.licenses` label is intentionally not set. Per-package license data is derived from [SBOMs](security-labels-and-metadata.md#software-bill-of-materials-sbom) stored as OCI artifacts alongside each image. A manually-set SPDX expression would be incomplete compared to the SBOM-derived data available in the image catalog. ### ² Name and identity The `name` label uses the registry organization prefix (`hummingbird/`, `hummingbird-rawhide/`, `hummingbird-community/`, or `hummingbird-ci/`) followed by the image name. For non-default variants, the variant is appended with a hyphen (e.g., `hummingbird/nodejs-24-builder`). The `cpe` label is only set for Hummingbird distro images (not Rawhide). See [Security Labels and Metadata](security-labels-and-metadata.md) for details on the scanning workflow and `labels.json`. ### ³ Version labels Three version granularities are derived from the image's tags: `version` (full, e.g., `2.10.2`), `major-minor-version` (e.g., `2.10`), and `major-version` (e.g., `2`). The full version is the OCI/Conforma `org.opencontainers.image.version`; the two coarser granularities are Hummingbird project labels used for tag aliasing. There is no `io.hummingbird-project.version` label — it would duplicate the OCI label. ### ⁴ Repository and stream The `io.hummingbird-project.repository` and `io.hummingbird-project.stream` fields together form the release identity pair. Both are defined in `properties.yml`. See [image-configuration-reference.md](../contributing/image-configuration-reference.md) for field definitions. ### ⁵ Description, summary, and URL The `description`, `summary`, and `url` fields are defined in `properties.yml`. Description and summary target different display contexts: - **summary**: One-liner (~40-80 chars) for table/list views - **description**: Short paragraph (~100-250 chars, 1-2 sentences) for card views and `podman inspect` Style rules: - Do not start `summary` or `description` with the image name - Use `>-` YAML scalar for multi-line readability in `properties.yml` - Avoid embedded double quotes and backslashes (no escaping in templates) ### ⁶ Vendor All images use `Red Hat, Inc.` as vendor (the distributing entity). The OCI `org.opencontainers.image.vendor` uses `Red Hat` (without ", Inc.") per OCI convention. Both values are set on all distros. ### ⁷ Release The `release` label uses the commit timestamp as a Unix epoch (matching Red Hat convention). In Konflux this is set via the `generate-labels` pipeline task. ### ⁸ Variant labels Each variant name is decomposed into a **base specialization** and **cross-cutting modifiers** (builder, fips). The naming convention is `[-fips][-builder]`, where modifier order does not matter. A bare modifier like `builder` has base `default`. Examples: `fpm-builder` → base=`fpm`, builder=yes; `fips-builder` → base=`default`, builder=yes, fips=yes. The `.description` label stores only the base description (e.g., "PHP FastCGI process manager"). Modifier display is handled by consumers using the `.builder` and `.fips` boolean labels. Base descriptions come from `variant_descriptions` in `images/variables.yml` (for `default`) and image-specific `properties.yml` (for bases like `fpm`, `runtime`). ### ⁹ Containerfile path The `io.hummingbird-project.containerfile` label contains the path to the Containerfile source relative to the repository root (e.g., `images/caddy/hummingbird/default/Containerfile`). A full URL to the source can be constructed from `org.opencontainers.image.source` + `org.opencontainers.image.revision` + this path. ## Embedded Metadata (labels.json) All labels are written to `/usr/share/buildinfo/labels.json` inside the container filesystem, providing filesystem-level access for security scanners. See [Security Labels and Metadata](security-labels-and-metadata.md) for schema details. ## Related Files | File | Purpose | |----------------------------------------------------|------------------------------------------------| | `images//properties.yml` | Per-image metadata fields | | `ci/internal/generate_jinja2.py` | Builds `inject_labels` dict from properties | | `macros/image_metadata_labels.yml.j2` | Emits LABEL instructions from `inject_labels` | | `macros/inject_source_info_labels.yml.j2` | Emits `name` and `cpe` LABEL instructions | | `macros/install_newroot.yml.j2` | Invokes `inject-source-info.sh` with labels | | `images/hummingbird-builder/inject-source-info.sh` | Writes `labels.json` to container filesystem | | `ci/build_images.sh` | Adds build-time labels | | `ci/check_release_fields.py` | Validates label field values in properties.yml | [conforma-labels]: https://conforma.dev/docs/policy/packages/release_labels.html [rule-data]: https://github.com/release-engineering/rhtap-ec-policy/blob/main/data/rule_data.yml [oci-spec]: https://github.com/opencontainers/image-spec/blob/main/annotations.md [security-schema]: https://github.com/RedHatProductSecurity/security-data-guidelines/blob/main/schema/embedded_metadata.v1.schema.json [ubi-eula]: https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI -------------------------------------------------------------------------------- # CI Scripts url: https://hummingbird-project.io/docs/background/containers/ci-scripts/ This section documents the CI scripts used in the Project Hummingbird container build and test pipeline. These scripts can be run locally for development and testing, and are also used by the automated CI/CD pipeline. -------------------------------------------------------------------------------- # build_images.sh url: https://hummingbird-project.io/docs/background/containers/ci-scripts/build_images/ description: Build container images using buildah with support for multiple architectures and container engines ## Purpose Build container images using buildah with support for multiple architectures and container engines. ## Usage ```text Usage: ci/build_images.sh [OPTIONS] [GROUP_NAMES...] [-- BUILDAH_ARGS...] OPTIONS: --verbose, -v Enable verbose output during build --arch ARCHITECTURE Specify target architecture (e.g., amd64, arm64, arm/v7) --engine ENGINE Specify runtime engine for testing/export (podman or docker) --setup Set up Docker-in-Docker environment before exporting --component-name NAME Parse component name (format: group--distro--variant) Example: curl--rawhide--default → curl/rawhide/default --local-rpms-dir DIR Directory containing custom RPMs to use during build. These have a higher priority and thus override the standard repositories. --build-deps Build all dependencies of specified groups, but NOT the specified groups themselves. Collects forward, reverse, and transitive dependencies automatically. --pull Pull images from registry instead of building locally. Fast alternative for local development. Uses published images from quay.io. --dryrun Show what would be built (or pulled with --pull) without actually building/pulling. --help, -h Show this help message ``` ## Examples ```bash # Build single image group (all distro/variants) ci/build_images.sh nginx ci/build_images.sh nodejs-20 # Build specific distro/variant only ci/build_images.sh nginx/rawhide/builder ci/build_images.sh nodejs-20/hummingbird/default # Build multiple image groups at once ci/build_images.sh nginx curl git ci/build_images.sh nginx/rawhide/default curl/hummingbird/builder # Build with options ci/build_images.sh --verbose curl ci/build_images.sh --arch arm64 nginx ci/build_images.sh --engine docker nginx ci/build_images.sh --engine podman --verbose nginx # Build multiple image groups with options ci/build_images.sh --verbose nginx curl git ci/build_images.sh --arch arm64 nginx postgresql # Build all dependencies of dotnet-runtime-10-0 (forward + reverse + transitive) ci/build_images.sh --build-deps dotnet-runtime-10-0 # Pull dependencies from registry instead of building (faster for local dev) ci/build_images.sh --build-deps --pull dotnet-runtime-10-0 # Pull a specific image from registry ci/build_images.sh --pull nginx/rawhide/default # Build from CI component name format (used in CI environments) ci/build_images.sh --component-name curl--rawhide--default # Build with custom RPMs (for testing modified packages) ci/build_images.sh --local-rpms-dir ../rpms/builds/hostname/RPMS git/rawhide/builder ``` **Note:** Some images require git submodules initialized (use `git init --recurse-submodules` when cloning or `git submodule update --init` if already cloned). When building for foreign architectures, make sure `qemu-user-static` is available. ## Dependency Building The `--build-deps` flag builds all dependencies needed for testing an image, but not the image itself. This is used in CI to ensure all required images are available before running tests. ### Pulling vs Building Dependencies For local test development, use `--pull` with `--build-deps` to pull pre-built images from the registry instead of building them locally. This is much faster and uses the same dependency collection logic: ```bash # Slow: Build all dependencies locally ci/build_images.sh --build-deps nginx # Fast: Pull all dependencies from registry ci/build_images.sh --build-deps --pull nginx ``` The `--pull` flag works without `--build-deps` too: ```bash # Pull a specific image instead of building it ci/build_images.sh --pull caddy/rawhide/default ``` ### How Dependency Collection Works When you run `ci/build_images.sh --build-deps `, it performs a 2-level expansion: 1. **Level 1**: Collects forward and reverse dependencies of the specified image - **Forward dependencies**: Images that the specified image's tests depend on (detected by `TEST_IMAGES[...]` references in test files) - **Reverse dependencies**: Images that depend on the specified image (filtered by `reverse_dependency_tests: true` in `properties.yml`) 2. **Level 2**: Collects forward dependencies of the reverse dependencies 3. **Stops**: No further expansion (avoids infinite graph traversal) All dependencies are automatically deduplicated to ensure each image is built exactly once. ### Relationship to Testing The `reverse_dependency_tests` property in `properties.yml` affects both building and testing: - **Build phase** (`--build-deps`): Filters which reverse dependencies to build - **Test phase** (`ci/run_tests_container.sh` and `ci/run_tests_k8s.sh` with `--include-reverse-deps`): Filters which reverse dependencies to test Set `reverse_dependency_tests: false` for images like `curl` that are used pervasively but don't need reverse dependency workflows. ### Dependency Building Examples ```bash # Build dependencies for dotnet-runtime-10-0 # Builds: dotnet-sdk-10-0 (forward dependency) ci/build_images.sh --build-deps dotnet-runtime-10-0 # Pull dependencies for dotnet-runtime-10-0 (faster alternative) # Pulls: dotnet-sdk-10-0 from quay.io/hummingbird-rawhide ci/build_images.sh --build-deps --pull dotnet-runtime-10-0 # Build dependencies for core-runtime # Builds: xcaddy, go, rust (reverse deps with reverse_dependency_tests: true) # Plus their forward dependencies ci/build_images.sh --build-deps core-runtime # Use --dryrun to see what would be built/pulled without actually doing it ci/build_images.sh --dryrun --build-deps core-runtime ci/build_images.sh --dryrun --build-deps --pull core-runtime ``` ## Building with custom RPMs The `--local-rpms-dir` option enables testing container images with custom-built RPM packages, for iterating on package changes before committing to the RPM repository. ### Workflow 1. **Build custom RPMs** in the `rpms` repository (see its documentation for the complete workflow): ```bash cd ../rpms # modify a package ci/build_rpms.sh packagename # Built RPMs will be in builds/packagename/RPMS/ ``` 2. **Build container image** using the custom RPMs: ```bash cd ../containers ci/build_images.sh --local-rpms-dir ../rpms/builds/packagename/RPMS imagename/builder ``` 3. **Verify the custom package** was installed: ```bash podman run --rm --entrypoint '' quay.io/hummingbird/imagename:latest-builder rpm -qa ``` -------------------------------------------------------------------------------- # run_tests_container.sh url: https://hummingbird-project.io/docs/background/containers/ci-scripts/run_tests_container/ description: Run tests for image groups using containerized test environment with support for both Docker and Podman engines ## Purpose Run tests for image groups using containerized test environment with support for both Docker and Podman engines. ## Usage Run `ci/run_tests_container.sh --help` for full usage information. ```text Usage: ci/run_tests_container.sh [OPTIONS] [GROUP_NAMES...] OPTIONS: --verbose, -v Enable verbose output during testing --engine ENGINE Specify container engine (podman or docker) --setup Set up Docker-in-Docker environment before running tests --pause, -p Pause failed tests before cleanup to allow debugging Prints a message and waits for Enter before cleaning up containers --hermetic Use hermetic builds (--pull=never, only use prefetched/built images) Default for CI. Without this, missing images are pulled on demand. --component-name NAME Parse component name (format: group--distro--variant) Example: curl--rawhide--default → curl/rawhide/default --group-component-name NAME Parse component name and add /group variant Example: curl--rawhide--default → curl/rawhide/group --include-reverse-deps Also test groups that depend on specified groups --help, -h Show this help message ``` **Note**: If no distro/variant is specified, all combinations will be tested. To test only a specific variant, use `//`. To test only a specific test, use `///`. **Engine Selection**: Use `--engine` to specify the container engine (`podman` or `docker`). When using Docker, add `--setup` for automatic Docker-in-Docker environment setup. **Hermetic vs Non-Hermetic Testing**: - **Local development** (default): Without `--hermetic`, the test runner allows pulling missing images from the registry. This is convenient for testing individual images without building all dependencies first. - **CI environment**: Use `--hermetic` to enforce that tests only use prefetched or locally-built images (via `--pull=never`). This ensures reproducible builds and prevents accidentally using images from the registry that differ from what Konflux built. ## Building and Testing For local development, you'll typically want to build the images first, then test them: ```bash # Build the images ci/build_images.sh [/distro/variant] ci/build_images.sh # Build multiple image groups # Test the images ci/run_tests_container.sh [/distro/variant] ci/run_tests_container.sh # Test multiple image groups ``` The build script uses the same syntax as the test script, making it easy to build and test the same image/variant combination. When working on base images (like `core-runtime`) that other images depend on, build the base image and its dependencies before running reverse-dependency tests: ```bash # Build core-runtime, then its forward and reverse dependencies ci/build_images.sh core-runtime ci/build_images.sh --build-deps core-runtime # Then test them all ci/run_tests_container.sh --include-reverse-deps core-runtime ``` The `-p`/`--pause` option pauses the script on failed tests before cleaning up to allow interactive and efficient debugging. ## Testing with Specific Image Builds To reproduce CI failures, you can run the test with mapping the image under test to the Konflux build. Set `IMAGE_URL_____` environment variables (uppercase, hyphens/dots → underscores, `__` separates parts): ```bash # Test with a specific CI build IMAGE_URL_TOMCAT_10__HUMMINGBIRD__BUILDER='quay.io/redhat-user-workloads/.../tomcat-10--hummingbird--builder@sha256:...' \ ci/run_tests_container.sh tomcat-10/hummingbird/builder ``` ## Automatic Retries for Transient Infrastructure Failures The test runner automatically retries tests that fail with certain transient infrastructure errors. This helps avoid false test failures caused by temporary issues with external services like container registries or network problems. **Retry Behavior**: - Failed tests are checked against a list of retriable error patterns - If a match is found, the test is automatically retried - If the test still fails after all attempts, it's reported as a normal failure ## Examples ```bash # Test single group (all distro/variants) ci/run_tests_container.sh curl ci/run_tests_container.sh nginx # Test single group (specific distro/variant) ci/run_tests_container.sh curl/rawhide/default ci/run_tests_container.sh nodejs-20/hummingbird/builder # Test multiple groups (all distro/variants) ci/run_tests_container.sh curl nginx mariadb # Test multiple groups (mixed variants) ci/run_tests_container.sh curl/rawhide/default nginx mariadb/hummingbird/builder # Test with different engines ci/run_tests_container.sh --engine docker --setup git ci/run_tests_container.sh --engine podman --verbose nginx # Test with verbose output for debugging ci/run_tests_container.sh --verbose nginx mariadb dotnet-runtime-8-0 # Test specific tests ci/run_tests_container.sh curl/rawhide/default/version ci/run_tests_container.sh nginx/rawhide/default/port ci/run_tests_container.sh curl/rawhide/default/version nginx/hummingbird/default/port # Also test groups with reverse dependencies ci/run_tests_container.sh --include-reverse-deps core-runtime # Test from CI component name format (used in CI environments) ci/run_tests_container.sh --component-name curl--rawhide--default # Test across variants from CI component name format ci/run_tests_container.sh --group-component-name curl--rawhide--default ``` Reverse dependency testing is enabled by default for all images. When enabled, changes to an image will automatically: - Find images that depend on it by scanning for `TEST_IMAGES[group/distro/variant]` references - Run tests for both the changed image and all its dependents This catches breaking changes in base images early. Images can opt-out by setting `reverse_dependency_tests: false` in `properties.yml` (e.g., `curl`, which is widely used but typically doesn't need reverse dependency testing). -------------------------------------------------------------------------------- # run_tests_k8s.sh url: https://hummingbird-project.io/docs/background/containers/ci-scripts/run_tests_k8s/ description: Run K8s tests for image groups using kubectl with support for local development and CI environments ## Purpose Run K8s tests for image groups using kubectl. Tests execute in a real Kubernetes environment, validating that images work correctly in Kubernetes. ## Usage Run `ci/run_tests_k8s.sh --help` for full usage information. ```text Usage: ci/run_tests_k8s.sh [OPTIONS] [GROUP_NAMES...] OPTIONS: --context CONTEXT K8s context to use (REQUIRED unless --kubeconfig set) --kubeconfig FILE Path to kubeconfig file (REQUIRED unless --context set) --push-image IMAGE Push local image to OpenShift internal registry before testing Sets TEST_IMAGE to the internal registry reference --verbose, -v Show full output for passing tests --pause, -p Pause on test failure before cleanup to allow debugging --component-name NAME Parse component name (format: group--distro--variant) Example: curl--rawhide--default → curl/rawhide/default --group-component-name NAME Parse component name and add /group variant Example: curl--rawhide--default → curl/rawhide/group --include-reverse-deps Also test groups that depend on specified groups --output FILE Write JSON test results to FILE (for CI integration) When set, exits 0 after writing results regardless of test outcome --help, -h Show this help message ``` **Safety**: The script requires explicit `--context` or `--kubeconfig` to prevent accidental operations on production clusters. **Note**: If no distro/variant is specified, all combinations will be tested. To test only a specific variant, use `//`. To test only a specific test, use `///`. ## Local Development Workflow For local development, build images and push them to the internal registry: ```bash # Build the image locally podman build -t my-nginx:dev images/nginx/hummingbird/default/ # Push to internal registry and test ci/run_tests_k8s.sh --context mpp-preprod --push-image my-nginx:dev nginx/hummingbird/default ``` The `--push-image` flag: - Compares local image digest with remote - Skips push if image already exists with same digest - Uses port-forward to registry-proxy for pushing - Sets `TEST_IMAGE` to the internal registry reference **Prerequisites:** - `oc login` to the target cluster - `oc project ` to set the target namespace - Port-forward access to `registry-proxy` in `hummingbird--internal` ## Testing with Published Images Test published images without building locally: ```bash # Test using published images (resolved via IMAGE_URL/IMAGE_NAME) IMAGE_URL=quay.io/hummingbird/nginx:latest \ IMAGE_NAME=nginx--hummingbird--default \ ci/run_tests_k8s.sh --context mpp-preprod nginx/hummingbird/default ``` ## Environment Variables Tests have access to these environment variables: | Variable | Description | |--------------------|-------------------------------------------------------------------------------| | `TEST_IMAGE` | Container image under test (with digest) | | `TEST_IMAGES` | Associative array with `group/variant` image URLs (uses TEST_DISTRO context) | | `TEST_IMAGES_PATH` | Path to file containing serialized `TEST_IMAGES` array | | `TEST_GROUP` | The image group being tested | | `TEST_DISTRO` | The distro being tested (e.g., `rawhide`) | | `TEST_VARIANT` | The variant being tested (e.g., `default`) | | `TEST_VERBOSE` | Show test command output (`true` or `false`) | | `TEST_RUN_ID` | Unique ID for this test run (for resource naming) | | `TEST_RUN_LABEL` | Label selector for cleanup (`hum-k8s-test=`) | ### Cluster Access The `kubectl` command is pre-configured with the context/kubeconfig from CLI args, so tests can use it directly without additional configuration. ### Helper Function | Function | Description | |-------------|-------------------------------------------| | `test_fail` | Fail the test with a custom error message | ## Examples ```bash # Test single group with explicit context (all distro/variants) ci/run_tests_k8s.sh --context mpp-preprod nginx # Test specific distro/variant ci/run_tests_k8s.sh --context mpp-preprod nginx/hummingbird/default # Test specific test ci/run_tests_k8s.sh --context mpp-preprod nginx/hummingbird/default/readiness-probe # Test with verbose output for debugging ci/run_tests_k8s.sh --verbose --context mpp-preprod nginx # Test with kubeconfig file (CI environments) ci/run_tests_k8s.sh --kubeconfig /workspace/kubeconfig nginx/hummingbird/default # Build locally and push to internal registry podman build -t my-nginx:dev images/nginx/hummingbird/default/ ci/run_tests_k8s.sh --context mpp-preprod --push-image my-nginx:dev nginx/hummingbird/default # Test from CI component name format ci/run_tests_k8s.sh --kubeconfig /workspace/kubeconfig --component-name nginx--hummingbird--default # Write JSON results for CI integration ci/run_tests_k8s.sh --output /tmp/results.json --kubeconfig /workspace/kubeconfig nginx/hummingbird/default ``` ## Resource Cleanup Tests should label resources with `TEST_RUN_LABEL` for automatic cleanup: ```bash kubectl create configmap my-config --from-literal=key=value kubectl label configmap my-config "${TEST_RUN_LABEL}" ``` The test runner automatically cleans up all resources with the test run label after each test and at script exit. ## CI Integration In CI environments (Konflux), the script receives: - `--kubeconfig` pointing to the ephemeral namespace kubeconfig - `--component-name` or `--group-component-name` for component identification - `--output` for structured JSON results The `--output` flag writes results in Konflux-compatible format and exits 0, allowing the pipeline to read results without relying on exit codes. -------------------------------------------------------------------------------- # retrigger_failed_checks.py url: https://hummingbird-project.io/docs/background/containers/ci-scripts/retrigger_failed_checks/ description: Retrigger failed Konflux CI checks for a given GitLab merge request ## Purpose Retrigger failed Konflux CI checks by posting retest commands and waiting for the checks to start running. ## Usage Run `ci/retrigger_failed_checks.py --help` for full usage information. ```text Usage: ci/retrigger_failed_checks.py [OPTIONS] OPTIONS: -h, --help show this help message and exit --dry-run just print what comments would be posted --token-path, -t PATH path to file containing GitLab API token Authentication (in order of precedence): 1. --token-path: Path to file containing GitLab API token (highest priority) 2. GITLAB_TOKEN_PATH: Environment variable with path to token file 3. GITLAB_TOKEN: Environment variable with GitLab API token (fallback) ``` ## Behavior The script will: 1. Extract failed pipeline runs from the commit statuses on the MR 2. Generate `/retest {pipeline-name}` commands for each failed run 3. Post the retest commands as MR comments (unless --dry-run) 4. Wait for the pipeline runs to start (unless --dry-run) ## Examples ```bash # Using --token-path option (highest priority) ci/retrigger_failed_checks.py --token-path /path/to/token https://gitlab.com/group/project/-/merge_requests/1234 # Using GITLAB_TOKEN_PATH environment variable GITLAB_TOKEN_PATH=/path/to/token ci/retrigger_failed_checks.py https://gitlab.com/group/project/-/merge_requests/1234 # Using GITLAB_TOKEN environment variable (legacy) GITLAB_TOKEN=your-token-here ci/retrigger_failed_checks.py https://gitlab.com/group/project/-/merge_requests/1234 # Dry-run mode (show what would be done, don't post or wait) ci/retrigger_failed_checks.py --token-path /path/to/token --dry-run https://gitlab.com/group/project/-/merge_requests/1234 ``` -------------------------------------------------------------------------------- # gitlab_sync.py url: https://hummingbird-project.io/docs/background/containers/ci-scripts/gitlab_sync/ description: Sync a generated file to an external GitLab repository via merge request, with automatic merge wait, pipeline retry, and preemption support ## Purpose Sync a file from a public source URL to a target GitLab repository via merge request. The script always waits for the MR to be merged and for the post-merge pipeline to complete before returning. It is invoked by infrastructure CI jobs to keep external repos (`pyxis-repo-configs`, `konflux-release-data`) in sync with generated files from the source repos. ## Usage Run `gitlab_sync.py --help` for full usage information. ```text usage: gitlab_sync.py [-h] --source-url SOURCE_URL --target-project TARGET_PROJECT --target-file TARGET_FILE --sync-branch SYNC_BRANCH --mr-title MR_TITLE --gitlab-url GITLAB_URL [--merge-timeout MERGE_TIMEOUT] [--post-merge-timeout POST_MERGE_TIMEOUT] [--mr-description MR_DESCRIPTION] [--squash-on-merge] [--dry-run] Authentication: GITLAB_TOKEN Environment variable with GitLab API token (required) Monitoring: SENTRY_DSN Sentry DSN for alerting on failures (optional) ``` ## How It Works 1. **Fetch source** from the public `--source-url` (with HTTP retries) 2. **Determine comparison ref**: use the sync branch if it exists, otherwise the project's default branch 3. **Compare** source content against the comparison ref (stripped whitespace) 4. **Early exit** if content matches and the comparison ref is the default branch (nothing to deploy) 5. **Commit** the source to the sync branch using the GitLab Commits API with `force: true` (creates a single-commit branch on top of the default branch). Also auto-squashes if the branch has accumulated multiple commits. 6. **Create or update MR** targeting the default branch 7. **Self-approve** the MR (best-effort; continues if already approved) 8. **Post takeover comment** with `CI_JOB_ID` for preemption tracking 9. **Wait for MR merge** (polling every 30s, up to `--merge-timeout`) 10. **Wait for post-merge pipeline** to succeed (up to `--post-merge-timeout`) ## Exit Codes | Code | Meaning | |------|-----------------------------------------------------------| | 0 | Success (MR merged) or no changes needed | | 1 | Failure (timeout, pipeline failure, unrecoverable error) | | 42 | Preempted by a newer CI job managing the same MR | ## Preemption When multiple CI jobs target the same sync branch, the script uses MR comments to coordinate. Each job posts a comment containing its `CI_JOB_ID` (monotonically increasing within a GitLab instance). Before each merge attempt, the script checks for comments with a higher job ID. If found, it yields by exiting with code 42. The caller should handle exit 42 to stop processing (a newer job will handle all remaining syncs). When running outside CI (`CI_JOB_ID` not set), preemption is disabled. ## Error Recovery - **Pipeline failure**: Retries the MR pipeline with exponential backoff (60s, 300s, 900s). Sends a Sentry alert if all retries are exhausted. - **Merge conflict**: Rebases the MR and re-approves (rebase resets approvals in GitLab). Up to 3 attempts before failing with a Sentry alert. - **Post-merge pipeline failure**: Retries once, then returns exit code 1. This blocks downstream sync steps (e.g., Pyxis must succeed before RPA). - **Draft MR**: Keeps polling without attempting to merge (allows manual intervention). ## Examples ```bash # Dry run: show diff without modifying anything GITLAB_TOKEN=$TOKEN gitlab_sync.py \ --source-url "https://gitlab.com/redhat/hummingbird/containers/-/raw/main/releng/pyxis-hummingbird.yaml" \ --target-project "releng/pyxis-repo-configs" \ --target-file "products/hummingbird/hummingbird.yaml" \ --sync-branch "hummingbird/sync-containers-pyxis" \ --mr-title "chore: Update hummingbird Pyxis config" \ --gitlab-url "https://gitlab.cee.redhat.com" \ --dry-run # Sync Pyxis config (requires squash merge) gitlab_sync.py \ --gitlab-url "https://gitlab.cee.redhat.com" \ --source-url "https://gitlab.com/redhat/hummingbird/containers/-/raw/main/releng/pyxis-hummingbird.yaml" \ --target-project "releng/pyxis-repo-configs" \ --target-file "products/hummingbird/hummingbird.yaml" \ --sync-branch "hummingbird/sync-containers-pyxis" \ --mr-title "chore: Update hummingbird Pyxis config" \ --squash-on-merge # Sync RPM RPA to konflux-release-data gitlab_sync.py \ --gitlab-url "https://gitlab.cee.redhat.com" \ --source-url "https://gitlab.com/redhat/hummingbird/rpms/-/raw/main/releng/hummingbird-rpms-staging.yaml" \ --target-project "releng/konflux-release-data" \ --target-file "config/kflux-prd-rh03.nnv1.p1/product/ReleasePlanAdmission/hummingbird/hummingbird-rpms-staging.yaml" \ --sync-branch "hummingbird/sync-rpms-rpa" \ --mr-title "Update hummingbird RPM ReleasePlanAdmission" ``` Preemption wrapper for sequential syncs in a CI job: ```bash gitlab_sync.py --gitlab-url ... --source-url ... --squash-on-merge rc=$? if [ "$rc" -eq 42 ]; then echo "Preempted by newer job, stopping" exit 0 elif [ "$rc" -ne 0 ]; then exit "$rc" fi # Proceed to next sync only on success gitlab_sync.py --gitlab-url ... --source-url ... ``` ## Environment Variables | Variable | Required | Description | |----------------|----------|---------------------------------------------------------| | `GITLAB_TOKEN` | Yes | API token with Developer+ access on the target project | | `CI_JOB_ID` | No | Set automatically in CI; enables preemption detection | | `SENTRY_DSN` | No | Sentry DSN for failure alerting | ## Development The script and `gitlab-ci` tool image are maintained in the [Hummingbird tools repository](https://gitlab.com/redhat/hummingbird/tools), not this repository. Make implementation and test changes there. -------------------------------------------------------------------------------- # K8s Test Pipeline url: https://hummingbird-project.io/docs/background/k8s-test-pipeline/ Background documentation sourced from the K8s test pipeline repository, covering pipeline design, test format, and EaaS debugging for Kubernetes integration tests. -------------------------------------------------------------------------------- # Pipeline Design url: https://hummingbird-project.io/docs/background/k8s-test-pipeline/pipeline-design/ description: Design guidelines and architecture of the K8s test pipeline. ## Task/Step Overview | Task | Type | Steps | Results | Retries | Timeout | | ------------------------------- | --------------------------------------------- | ------------------------------------------------------------------------------- | -------------------------- | ------- | ------- | | `check-for-tests` | Inline `taskSpec` | write-snapshot, resolve-component, fetch-source, check-for-tests, write-results | `HAS_TESTS`, `TEST_OUTPUT` | 2 | 10m | | `provision-namespace` | External bundle (`task-eaas-provision-space`) | (managed by task) | `secretRef` | 2 | 10m | | `run-tests` | Inline `taskSpec` | write-snapshot, resolve-component, fetch-source, run-tests, write-results | `TEST_OUTPUT` | 2 | 24h | | `fail-pipeline-on-test-failure` | Inline `taskSpec` (finally) | check-results | — | 0 | 10m | Both `check-for-tests` and `run-tests` independently fetch source — each task gets its own `emptyDir` volume, so no state carries across tasks. ## Design Guidelines ### 1. Filesystem over results Pass data between steps via files on a shared `emptyDir`, not Tekton results. Tekton results have a 4KB termination message limit ([tektoncd/pipeline#4060](https://github.com/tektoncd/pipeline/issues/4060)) and large `$(params.*)` substitutions hit `ARG_MAX`. Each task mounts an `emptyDir` at `/workdir`. Steps communicate via files: `snapshot.json`, `component`, `image_names`, `src/`, `kubeconfig`, `result`, `error`. Zero `$(params.*)` references remain inside `script:` blocks — all param values are passed via env vars or written to `/workdir`. ### 2. Distinguish retryable from non-retryable errors Use `exit 1` for transient failures (network, registry) that benefit from retry. Use `exit 0` + a sentinel file for permanent failures (bad data) to prevent wasted retries. Non-retryable errors write Konflux-format JSON to `/workdir/error` and `exit 0`. Subsequent steps check for this sentinel and skip. The `write-results` step copies it to `TEST_OUTPUT`. Tasks default to `retries: 2`, but `fail-pipeline-on-test-failure` has `retries: 0` — test failures should not be retried. ### 3. Fail fast Every shell script enables `set -euo pipefail` so unexpected command failures surface immediately rather than cascading silently. ### 4. Separate data from status Report test results as structured data for the CI system to consume, and separately translate results into pipeline pass/fail for source control integration. Konflux reads `TEST_OUTPUT` results as data but does not fail the pipeline based on them. The `fail-pipeline-on-test-failure` finally-task reads both `TEST_OUTPUT` values and `exit 1` on non-SUCCESS. This is needed because GitLab PAC integration requires the pipeline itself to fail for MR status reporting. ### 5. Gate expensive work behind cheap checks Run a fast, cheap check before provisioning resources. Skip the expensive path when there is nothing to do. `provision-namespace` and `run-tests` are gated on `check-for-tests.results.HAS_TESTS == "true"` via Tekton `when` expressions. If no `tests-k8s.yml` exists, the pipeline completes with just `check-for-tests`, skipping EaaS provisioning. This is the common case for components without K8s tests. ### 6. Make failures reproducible Preserve all diagnostic output (do not swallow stderr). Log exact commands so developers can copy-paste to reproduce locally. Stderr from cosign/jq/oras is not swallowed. The full `run_tests_k8s.sh` command line is logged. ### 7. Set task timeouts deliberately Tekton applies a global default TaskRun timeout (typically 1h from `default-timeout-minutes` in `config-defaults`) to every task that has no explicit `timeout`, regardless of pipeline-level timeouts. This means omitting a task timeout does NOT let the pipeline timeout control the bound — the task gets killed at 1h. Every task in this pipeline has an explicit timeout. Bounded operations (`check-for-tests`, `provision-namespace`, `fail-pipeline-on-test-failure`) use `10m`. The variable-duration `run-tests` task uses `24h` so the pipeline-level timeout (controlled per-application via ITS annotations like `test.appstudio.openshift.io/pipeline_timeout`) is the effective bound, following the Testing Farm pipeline pattern. ### 8. Define shared logic once When multiple tasks need identical steps, use YAML anchors to define them once and reuse via aliases. Tekton pipelines have no function/import mechanism, so anchors are the only DRY tool available within a single pipeline file. Without this, step logic drifts between tasks when one copy is updated but not the other. `write-snapshot`, `resolve-component`, and `fetch-source` are defined as anchors (`&write-snapshot-step`, etc.) in `check-for-tests` and reused via `*write-snapshot-step` in `run-tests`. Both tasks independently execute the same steps to fetch and process source, staying in sync. ## Trusted Artifacts Source Fetch Source code is fetched via the Konflux Trusted Artifacts chain: 1. `cosign download attestation` — retrieves build attestations (handles multiple JSONL attestations) 2. Extract `SOURCE_ARTIFACT` from `.predicate.buildConfig.tasks[].results[]` 3. `oras blob fetch` — downloads the source tarball 4. `tar -xzf` — extracts to `/workdir/src` The `oci:` prefix is stripped from the artifact URI. `--no-same-owner` avoids permission issues during extraction. -------------------------------------------------------------------------------- # Test Format url: https://hummingbird-project.io/docs/background/k8s-test-pipeline/test-format/ description: Pipeline-level test discovery, environment variables, parameters, and group snapshot handling. ## Test Discovery The pipeline locates `tests-k8s.yml` using `source.git.context` from the Konflux Snapshot. For example, if `source.git.context` is `images/curl`, the pipeline looks for `images/curl/tests-k8s.yml` in the source tree. When `source.git.context` is not populated (currently the case for the triggering component in PR-triggered pipelines, tracked as [KONFLUX-12674](https://issues.redhat.com/browse/KONFLUX-12674)), a heuristic fallback derives the context from the component name: it tries `images/` and then ``. The actual `tests-k8s.yml` format is opaque to the pipeline — it is passed to the repository's test runner script, which defines the structure and semantics. See the respective repository documentation for the YAML format. ## Environment Variables The test runner receives these variables from the pipeline: | Variable | Description | | ------------------ | ---------------------------------------------------------- | | `TEST_IMAGE` | Container image under test (with digest) | | `TEST_GROUP` | Image group name (e.g., `git`, `postgresql`) | | `TEST_VARIANT` | Variant being tested (e.g., `default`, `builder`) | | `KUBECONFIG` | Path to kubeconfig for the ephemeral namespace | | `TEST_IMAGES` | Bash associative array of all images in the snapshot | | `TEST_IMAGES_PATH` | Path to source `TEST_IMAGES` in external scripts | | `TEST_RUN_ID` | Unique ID for this test run (for resource naming) | | `TEST_RUN_LABEL` | Label selector for cleanup (e.g., `hum-k8s-test=`) | | `test_fail` | Shell function — call `test_fail "message"` to fail a test | ## Pipeline Parameters | Parameter | Description | Default | | ----------------------------- | ------------------------------------------------------------------------------------ | ------------------------ | | `SNAPSHOT` | Konflux snapshot JSON or snapshot name (with `snapshot-param-as-name` annotation) | — | | `EXTRA_SINGLE_COMPONENT_ARGS` | Additional arguments appended to the test runner for single-component (PR) snapshots | `--include-reverse-deps` | ## Group Snapshot Handling For group snapshots (post-merge), the pipeline exports: - `IMAGE_NAMES` — space-separated list of all component names - `IMAGE_URL_` — image URL for each component (name uppercased, dashes replaced with underscores) The test runner script is called once with multiple `--group-component-name` arguments. Test discovery checks all components for `tests-k8s.yml`. Source is fetched from the first component's build attestation (shared source tree). -------------------------------------------------------------------------------- # EaaS and Debugging url: https://hummingbird-project.io/docs/background/k8s-test-pipeline/eaas/ description: How the pipeline uses Konflux Environment as a Service, and how to debug test failures. ## Environment as a Service (EaaS) EaaS is a Konflux-managed service that provisions ephemeral OpenShift namespaces on member clusters for integration testing. ### Provisioning Flow 1. The `provision-namespace` task creates an `eaas.konflux-ci.dev/v1alpha1 Namespace` claim, owned by the PipelineRun. 2. The EaaS controller allocates a namespace on a member cluster (currently `kflux-prd-es01`) and writes a scoped kubeconfig into a Kubernetes Secret. 3. The `run-tests` step reads the kubeconfig from the Secret to get cluster access. 4. When the PipelineRun is deleted, the owner reference triggers garbage collection of the claim, namespace, and Secret. The member cluster may change — it is dynamically assigned by the EaaS controller. ## Debugging Test Failures ### Using Kubearchive for Historical PLRs Kubearchive stores historical PipelineRun, TaskRun, and Pod data. The [containers repo](https://gitlab.com/redhat/hummingbird/containers) includes a helper script at `ci/internal/k8s_helper.py`. Example: find PLRs for a specific PR: ```bash cd /path/to/rprm-containers python3 -c " from ci.internal import k8s_helper h = k8s_helper.K8sHelper('https://konflux-ui.apps.kflux-prd-rh03.nnv1.p1.openshiftapps.com/ns/hummingbird-tenant/') endpoint = h._tekton_endpoint('pipelineruns') items = h._fetch_from_both_sources(endpoint, 'pac.test.appstudio.openshift.io/pull-request=421,test.appstudio.openshift.io/scenario=tools-k8s-test') for plr in items: print(plr['metadata']['name'], plr['status']['conditions'][0]['reason']) " ``` Useful label selectors: - `test.appstudio.openshift.io/scenario=` — filter by ITS name - `pac.test.appstudio.openshift.io/pull-request=` — filter by MR - `appstudio.openshift.io/component=` — filter by component ### Accessing the EaaS Namespace During a Live PLR While a PipelineRun is running, you can access the ephemeral namespace: 1. Find the provision secret name from the `provision-namespace` task results (visible in the PLR status or Konflux UI). 2. Extract the kubeconfig: ```bash kubectl get secret -n hummingbird-tenant \ -o jsonpath='{.data.kubeconfig}' | base64 -d > /tmp/eaas-kubeconfig ``` 3. Use it to inspect the namespace: ```bash export KUBECONFIG=/tmp/eaas-kubeconfig kubectl get pods kubectl get events --sort-by='.lastTimestamp' kubectl logs ``` ### Finding Which Cluster EaaS Uses To determine the current EaaS member cluster, extract the server URL from a kubeconfig provisioned by EaaS: ```bash kubectl get secret -n hummingbird-tenant \ -o jsonpath='{.data.kubeconfig}' | base64 -d | grep server ``` As of this writing, the EaaS member cluster is `kflux-prd-es01.1ion.p1.openshiftapps.com`. This may change as the EaaS controller dynamically assigns clusters. -------------------------------------------------------------------------------- # Tools repository url: https://hummingbird-project.io/docs/background/tools/ Background documentation sourced from the tools repository, covering infrastructure components like event forwarders, the message bus, and monitoring tools. -------------------------------------------------------------------------------- # Message Bus Architecture url: https://hummingbird-project.io/docs/background/tools/message-bus/ The Hummingbird message bus is an event-driven architecture built on AWS SNS. Events from multiple sources flow through a central topic, enabling subscribers to filter and process only the events they need. ## Architecture ```mermaid flowchart TD subgraph Publishers GL[GitLab Webhooks] K8S[Kubernetes Clusters] end subgraph MessageBus [Message Bus] SNS[(SNS Topic)] end subgraph Archiver ARCH[SNS S3 Archiver] S3[(S3 Bucket)] end subgraph StatusDB [Status Database] SQS1[SQS Queue] WORKER[hummingbird-status] PG[(PostgreSQL)] end subgraph ConsumerQueue [Consumer Queue] SQS2[SQS Queue] end subgraph Catalog [Container Catalog] SYNC[SyncFunction] DDB[(DynamoDB)] end subgraph Consumers CONSUMER[Event Consumer] end GL -->|gitlab-event-forwarder| SNS K8S -->|kubernetes-event-forwarder| SNS SNS --> ARCH ARCH --> S3 SNS --> SQS1 SQS1 --> WORKER WORKER --> PG SNS -->|"FilterPolicy\nkind=Release"| SYNC SYNC -->|fetch manifests| Registry[(OCI Registry)] SYNC -->|read/write| DDB SNS --> SQS2 SQS2 -->|new events| CONSUMER S3 -.->|historic events| CONSUMER PG -.->|aggregated status| CONSUMER ``` ## Components | Component | Role | Description | |----------------------------------|----------------|------------------------------------------------------------------| | [hummingbird-events-topic][het] | Infrastructure | Central SNS topic for all events | | [gitlab-event-forwarder][gef] | Publisher | Receives GitLab webhooks, publishes to SNS | | [kubernetes-event-forwarder][kef]| Publisher | Watches K8s resources, publishes changes to SNS | | [sns-s3-archiver][ssa] | Subscriber | Archives all events to S3 for querying/replay | | [hummingbird-status][hs] | Subscriber | Ingests events to PostgreSQL for structured queries | | [container-catalog][cc] | Subscriber | Incrementally syncs image metadata to DynamoDB on Release events | ## Message Format All messages include standard attributes for filtering: | Attribute | Description | Examples | |--------------|---------------|----------------------------------------------| | `source` | Event origin | `gitlab`, `kubernetes` | | `event_type` | Type of event | `push`, `merge_request`, `ADDED`, `MODIFIED` | Additional attributes vary by source - see individual publisher docs for details. ### GitLab Events Published by [gitlab-event-forwarder][gef]: | Attribute | Description | Example | |----------------|-------------------|---------------------------------| | `project_path` | Full project path | `redhat/hummingbird/containers` | | `group_path` | Full group path | `redhat/hummingbird` | ### Kubernetes Events Published by [kubernetes-event-forwarder][kef]: | Attribute | Description | Example | |---------------|------------------|----------------------------| | `cluster` | API server URL | `https://api.cluster:6443` | | `namespace` | Object namespace | `production` | | `kind` | Resource kind | `Pod`, `Deployment` | | `api_version` | API version | `v1`, `apps/v1` | | `object_name` | Resource name | `nginx-7d8c4c9d6f` | ## Subscription Filtering SNS filter policies enable subscribers to receive only relevant events: ```json { "source": ["gitlab"], "event_type": ["push", "merge_request"] } ``` ```json { "source": ["kubernetes"], "kind": ["Deployment"], "event_type": ["MODIFIED"] } ``` Kubernetes Release events (used by [container-catalog][cc] sync Lambda): ```json { "kind": ["Release"] } ``` See [hummingbird-events-topic][het] for subscription setup instructions. ## Event Flow Example 1. Developer pushes to GitLab repository 2. GitLab sends webhook to gitlab-event-forwarder Lambda 3. Lambda validates token, extracts metadata, publishes to SNS 4. SNS delivers to all matching subscribers: - sns-s3-archiver stores event in S3 - Event consumers process new events in real-time ## Consuming Events Consumers can receive events from multiple sources: - **New events**: Subscribe to SNS topic for real-time processing - **Historic events**: Download from S3 archive for replay or catch-up - **Structured queries**: Query PostgreSQL via [hummingbird-status][hs] for pipeline status, component state, and cross-referenced data The S3 archive enables consumers to bootstrap state, then switch to live SNS events. The PostgreSQL database provides a queryable view of pipeline status with relationships between pushes, builds, snapshots, and releases. ## Replaying Events The [sns-s3-archiver][ssa] stores complete SNS records with decoded payloads, enabling easy replay: ```bash # Download archived events aws s3 sync s3://bucket-name/2025/12/13/ ./local-events/ # Browse events zcat ./local-events/2025/12/13/12/34/*.json.gz | jq ``` ```python import gzip import json from pathlib import Path # Replay archived events to handler for path in sorted(Path("./local-events").rglob("*.json.gz")): with gzip.open(path, "rt") as f: sns_record = json.load(f) event = {"Records": [{"Sns": sns_record}]} my_handler.lambda_handler(event, None) ``` See [sns-s3-archiver documentation][ssa] for details on storage format and the `_decode_message` pattern for handlers. [het]: hummingbird-events-topic.md [gef]: gitlab-event-forwarder.md [kef]: kubernetes-event-forwarder.md [ssa]: sns-s3-archiver.md [hs]: hummingbird-status.md [cc]: container-catalog.md -------------------------------------------------------------------------------- # Alloy CloudWatch url: https://hummingbird-project.io/docs/background/tools/alloy-cloudwatch/ An AWS CloudFormation/SAM stack that provisions a read-only IAM user for Grafana Alloy's `prometheus.exporter.cloudwatch` component. Grants access to CloudWatch Metrics APIs so selected AWS service metrics (Lambda, SQS, SNS, DynamoDB) can be ingested into Mimir for alerting. ## Features - **Read-Only IAM User**: Dedicated user for Alloy CloudWatch metrics exporter - **CloudWatch Metrics**: `GetMetricData`, `GetMetricStatistics`, `ListMetrics` - **Resource Discovery**: `tag:GetResources` for filtering by `app-code=RPRM-001` - **Account Discovery**: `sts:GetCallerIdentity` and `ec2:DescribeRegions` for YACE auto-discovery ## Architecture Single SAM template that creates: 1. **IAM User** - Read-only service account (`${ResourcePrefix}-user`) 2. **IAM Policy** - CloudWatch Metrics read, tag and region discovery, STS caller identity The user's access keys are created manually after deployment and stored in Vault. The companion [infrastructure repo][infra-repo] injects the credentials into the Alloy Hub pod on MPP via a Kubernetes Secret. ```mermaid flowchart LR subgraph aws [AWS] iam["IAM User\n(read-only)"] cwMetrics["CloudWatch Metrics"] end subgraph mpp [MPP Hub] alloy["Alloy Hub"] mimir["Mimir"] end alloy -->|"GetMetricData"| cwMetrics alloy -->|"remote_write"| mimir alloy -.->|"credentials from Vault"| iam ``` ## IAM Permissions All permissions are read-only with `Resource: "*"` (CloudWatch does not support resource-level restrictions for most read operations). | Category | Actions | | -------- | ------- | | Metrics | `GetMetricData`, `GetMetricStatistics`, `ListMetrics` | | Discovery | `ec2:DescribeRegions`, `tag:GetResources`, `sts:GetCallerIdentity` | ## Prerequisites - AWS CLI configured with appropriate credentials (IAM permissions for CloudFormation) - Podman or Docker (for containerized SAM build/deploy) ## Deployment Build and deploy using containerized AWS SAM CLI: ```bash cd alloy-cloudwatch sam deploy --guided # First deployment sam deploy # Subsequent deployments ``` After deployment, create an access key for the IAM user and store it in Vault using `cki_tools.credentials.manager`. ## Related - **Alloy Hub deployment**: `kubernetes/alloy-hub/` in the [infrastructure repo][infra-repo] - **Monitoring documentation**: `documentation/monitoring.md` in the [infrastructure repo][infra-repo] ## License This project is licensed under the GNU General Public License v3.0 or later - see the [LICENSE][license] file for details. [infra-repo]: https://gitlab.com/redhat/hummingbird/infrastructure [license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE -------------------------------------------------------------------------------- # Deployment Bot url: https://hummingbird-project.io/docs/background/tools/deployment-bot/ An AWS CloudFormation/SAM stack that provisions IAM resources for automated AWS deployments from GitLab CI. Creates a service account with permissions to deploy SAM applications including Lambda functions, API Gateway, SNS topics, and related resources. ## Features - **IAM Service Account**: Dedicated user for CI/CD pipelines - **Scoped Permissions**: Least-privilege access for SAM deployments - **Resource Naming**: All managed resources use consistent prefix ## Architecture Single SAM template that creates: 1. **IAM User** - Service account for deployments (`${ResourcePrefix}-deployment-bot`) 2. **IAM Policy** - Permissions for SAM deployment operations ## Prerequisites - AWS CLI configured with admin credentials - Podman or Docker (for containerized SAM build/deploy) ## Deployment Deploy using containerized AWS SAM CLI: ```bash cd deployment-bot sam build sam deploy --guided # First deployment (interactive) sam deploy # Subsequent deployments ``` After deployment, create access keys for the IAM user and store them securely in HashiCorp Vault. ### SAM Parameters | Parameter | Description | Default | |----------------------------|------------------------------------------------------|--------------| | `ResourcePrefix` | Prefix for resources | `myapp-prod` | | `AdditionalResourcePrefix` | Additional prefix this bot can manage (e.g. staging) | `""` | **Resource naming:** IAM resources follow `{ResourcePrefix}-deployment-bot` pattern (e.g., `myapp-prod-deployment-bot`). Use `AdditionalResourcePrefix` to allow the same bot to manage resources in another environment. ## Permissions The deployment bot has permissions to manage: | Service | Scope | |----------------|------------------------------------------------------------| | CloudFormation | Stacks matching `${ResourcePrefix}-*`, SAM bootstrap stack | | S3 | SAM CLI managed buckets | | Lambda | Functions matching `${ResourcePrefix}-*` | | API Gateway | All REST APIs and domains | | SNS | Topics matching `${ResourcePrefix}-*` | | IAM | Users/policies/roles matching `${ResourcePrefix}-*` | | CloudWatch | Log groups for managed Lambda functions | | Route53 | Record changes (for custom domains) | | ACM | Certificate management | ## Usage ### Bootstrap (One-Time) The deployment bot must be deployed first using admin credentials: ```bash # With admin AWS credentials cd deployment-bot sam build sam deploy --guided ``` Create access keys for the new IAM user and store in HashiCorp Vault. ### Automated Deployments Once bootstrapped, the deployment bot credentials are used by `aws_deploy.sh` in the infrastructure repository for all subsequent SAM deployments: ```bash export PROJECT_NAME=gitlab-event-forwarder ./aws_deploy.sh ``` GitLab CI runs this automatically via the `aws` matrix job. ## Development This project contains only a SAM template with no application code. ## Security - Access keys should be stored in HashiCorp Vault - Permissions are scoped to resources with the configured prefix - Route53 and ACM have broader permissions due to API limitations ## License This project is licensed under the GNU General Public License v3.0 or later - see the [LICENSE][license] file for details. [license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE -------------------------------------------------------------------------------- # Grafana CloudWatch url: https://hummingbird-project.io/docs/background/tools/grafana-cloudwatch/ An AWS CloudFormation/SAM stack that provisions a read-only IAM user for Grafana's CloudWatch datasource plugin. Grants access to CloudWatch Logs and Metrics APIs so Lambda execution logs and AWS service metrics are browsable directly in Grafana. ## Features - **Read-Only IAM User**: Dedicated user for Grafana CloudWatch datasource - **CloudWatch Logs**: Browse Lambda and AWS service log groups via Logs Insights - **CloudWatch Metrics**: Browse metrics across all AWS namespaces (Lambda, SQS, SNS, DynamoDB, etc.) - **Resource Discovery**: `tag:GetResources` and `ec2:DescribeRegions` for Grafana plugin autodiscovery ## Architecture Single SAM template that creates: 1. **IAM User** - Read-only service account (`${ResourcePrefix}-user`) 2. **IAM Policy** - CloudWatch Logs read, Metrics read, tag and region discovery The user's access keys are created manually after deployment and stored in Vault. The companion [infrastructure repo][infra-repo] injects the credentials into the Grafana pod on MPP via a Kubernetes Secret. ```mermaid flowchart LR subgraph aws [AWS] iam["IAM User\n(read-only)"] cwLogs["CloudWatch Logs"] cwMetrics["CloudWatch Metrics"] end subgraph mpp [MPP Hub] grafana["Grafana\n(CloudWatch datasource)"] end grafana -->|"Logs Insights API"| cwLogs grafana -->|"Metrics API"| cwMetrics grafana -.->|"credentials from Vault"| iam ``` ## IAM Permissions All permissions are read-only with `Resource: "*"` (CloudWatch does not support resource-level restrictions for most read operations). | Category | Actions | | -------- | ------- | | Logs | `DescribeLogGroups`, `DescribeLogStreams`, `GetLogEvents`, `FilterLogEvents`, `StartQuery`, `StopQuery`, `GetQueryResults`, `DescribeQueries` | | Metrics | `GetMetricData`, `GetMetricStatistics`, `ListMetrics`, `DescribeAlarms`, `DescribeAlarmsForMetric` | | Discovery | `ec2:DescribeRegions`, `tag:GetResources` | ## Prerequisites - AWS CLI configured with appropriate credentials (IAM permissions for CloudFormation) - Podman or Docker (for containerized SAM build/deploy) ## Deployment Build and deploy using containerized AWS SAM CLI: ```bash cd grafana-cloudwatch sam deploy --guided # First deployment sam deploy # Subsequent deployments ``` After deployment, create an access key for the IAM user and store it in Vault using `cki_tools.credentials.manager`. ## Related - **Grafana deployment**: `kubernetes/grafana/` in the [infrastructure repo][infra-repo] - **CloudWatch datasource**: `grafana_data/datasource/` in the [infrastructure repo][infra-repo] - **Monitoring documentation**: `documentation/monitoring.md` in the [infrastructure repo][infra-repo] ## License This project is licensed under the GNU General Public License v3.0 or later - see the [LICENSE][license] file for details. [infra-repo]: https://gitlab.com/redhat/hummingbird/infrastructure [license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE -------------------------------------------------------------------------------- # Hummingbird Events Topic url: https://hummingbird-project.io/docs/background/tools/hummingbird-events-topic/ An SNS topic for all Hummingbird project events. This topic serves as a central hub for distributing events from multiple sources (GitLab webhooks, Kubernetes, etc.) to multiple subscribers, enabling event-driven architectures and integrations. ## Features - **Central Event Hub**: Single SNS topic for all project events from multiple sources - **Multiple Subscribers**: Supports Lambda, SQS, HTTP endpoints, email, SMS, and more - **Event Filtering**: Subscribers can filter events using SNS subscription filter policies ## Prerequisites - AWS CLI configured with appropriate credentials (IAM permissions for SNS, CloudFormation) - Podman or Docker (for containerized SAM build/deploy) ## Deployment Build and deploy using containerized AWS SAM CLI: ```bash cd hummingbird-events-topic make build # Build SAM application make deploy # First deployment (interactive/guided) make redeploy # Subsequent deployments (non-interactive) ``` **Deployment outputs:** - `TopicArn` - SNS topic ARN (for event publishers) - `TopicName` - SNS topic name ### Parameters | Parameter | Description | Default | |-------------|----------------|---------------------| | `TopicName` | SNS topic name | `myapp-prod-events` | **Resource naming:** The SNS topic uses the provided `TopicName` parameter. ## Usage ### Publishing Events Event publishers need the topic ARN to publish messages: ```bash # Get topic ARN from CloudFormation stack aws cloudformation describe-stacks \ --stack-name \ --query 'Stacks[0].Outputs[?OutputKey==`TopicArn`].OutputValue' \ --output text ``` **Event publishers:** - [gitlab-event-forwarder][gef-docs] - Publishes GitLab webhook events - [kubernetes-event-forwarder][kef-docs] - Publishes Kubernetes resource changes - Custom event sources ### Subscribing to Events Subscribe services to receive events: **Via AWS Console:** 1. Open SNS console → Topics 2. Select the topic 3. Create subscription (choose protocol: Lambda, SQS, HTTP, Email, etc.) 4. Add subscription filter policy (optional) **Via AWS CLI:** ```bash aws sns subscribe \ --topic-arn \ --protocol lambda \ --notification-endpoint ``` **Add filter policy:** ```bash aws sns set-subscription-attributes \ --subscription-arn \ --attribute-name FilterPolicy \ --attribute-value '{"source": ["gitlab"], "event_type": ["push"]}' ``` ### Subscription Filter Examples GitLab push events from specific project: ```json { "source": ["gitlab"], "event_type": ["push"], "project_path": ["redhat/hummingbird/containers"] } ``` All merge request events: ```json { "source": ["gitlab"], "event_type": ["merge_request"] } ``` All events from GitLab: ```json { "source": ["gitlab"] } ``` **Event metadata:** See publisher documentation for available metadata: - [gitlab-event-forwarder][gef-docs] - GitLab webhook event metadata - [kubernetes-event-forwarder][kef-docs] - Kubernetes resource event metadata ## Development This is a pure infrastructure project (no application code). See the main [README][readme] for SAM build/deploy commands. ## Security & Limitations **Security:** - SNS topic follows least privilege principle - Access controlled via IAM policies - Supports server-side encryption (optional) - Publishers require `sns:Publish` permission - Subscribers require appropriate protocol permissions **Limitations:** - Message size: Up to 256 KB - Maximum subscriptions: 12,500,000 per topic - Message retention: Not supported (use SQS for durable queuing) - Delivery retries: Protocol-dependent ## License This project is licensed under the GNU General Public License v3.0 or later - see the [LICENSE][license] file for details. [gef-docs]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/documentation/gitlab-event-forwarder.md [kef-docs]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/documentation/kubernetes-event-forwarder.md [readme]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/README.md [license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE -------------------------------------------------------------------------------- # Vertex AI Cost Metrics url: https://hummingbird-project.io/docs/background/tools/vertex-metrics/ This document defines the shared cost-observability contract for all Hummingbird services that call Vertex AI. It covers two obligations: 1. **Prometheus metrics** — estimated cost counters scraped into Mimir. 2. **Vertex request labels** — billing metadata attached to every API call. All services that make Vertex AI calls MUST implement both. ## 1. Prometheus metrics (estimated cost) Every Vertex AI caller MUST emit the following counters on its scraped `/metrics` endpoint. Using identical names and label sets across services enables a single Grafana dashboard without per-app query logic. ### Counters | Name | Unit | Labels | Description | | --------------------------------------- | -------- | -------------------------------- | ---------------------------------------------- | | `hummingbird_vertex_cost_dollars_total` | USD | `model`, `workflow` | Estimated cost based on MODEL_PRICING x tokens | | `hummingbird_vertex_tokens_total` | tokens | `model`, `workflow`, `direction` | Token counts per API call | | `hummingbird_vertex_requests_total` | requests | `model`, `workflow`, `status` | API call attempts | ### Labels Applications MUST set these labels. Additional labels are permitted if cardinality remains bounded. | Label | Values | | ----------- | --------------------------------------------------------------------------- | | `model` | Model name as sent to Vertex (e.g. `gemini-2.5-flash`, `claude-sonnet-4-6`) | | `workflow` | Workflow or operation name (e.g. `code-review`, `renovate-babysit`) | | `direction` | `input`, `output`, `cache_read` | | `status` | `success`, `error`, `retry` | Applications MUST NOT set `cluster`, `namespace`, `app`, `service`, or `pod` — Alloy adds these automatically at scrape time via relabel rules. ### Cardinality - `model`: bounded by deployed model set (currently 2-4) - `workflow`: bounded by workflow config entries (~10 for agent, 1 for dashboard) - `direction`: fixed set of 3 - `status`: fixed set of 3 Total series per app: model x workflow x max(direction, status) = 4 x 10 x 3 = 120. Well within Mimir per-metric limits. ## 2. Vertex request labels (billed cost) Every Vertex API call MUST include billing labels so that GCP Cloud Billing rows can be attributed to a specific application and workflow once BigQuery export is enabled. ### Required labels | Key | Value | Example | | ---------- | ----------------------------------------- | --------------------------------- | | `app` | Application name (matches Kubernetes app) | `agent`, `dashboard` | | `workflow` | Workflow or operation name within the app | `code-review`, `analyze-failures` | Callers MUST set both keys on every `generateContent`, `streamGenerateContent`, or `rawPredict` request. ### How to attach | Endpoint | Mechanism | | --------------------------------- | ------------------------------------- | | `generateContent` (Gemini) | Top-level `"labels"` key in JSON body | | `rawPredict` / `streamRawPredict` | `X-Vertex-AI-Labels` HTTP header | For `generateContent`, add the labels object to the request body: ```json { "contents": [...], "labels": {"app": "agent", "workflow": "code-review"} } ``` For `rawPredict`, base64-encode the labels JSON and pass as a header (shown decoded for clarity): ```python import base64 import json labels = {"app": "dashboard", "workflow": "analyze-failures"} headers["X-Vertex-AI-Labels"] = base64.b64encode( json.dumps(labels).encode(), ).decode() ``` ### Constraints (GCP requirements) - Keys and values: lowercase letters, numbers, underscores, dashes only - Max 63 characters each - Keys MUST start with a letter - Labels are only forwarded for PayGo consumption (Provisioned Throughput silently ignores them) ## Reference ### Example PromQL Total estimated spend yesterday (all apps): ```promql sum(increase(hummingbird_vertex_cost_dollars_total[1d])) ``` Per-app daily cost: ```promql sum by (app)(increase(hummingbird_vertex_cost_dollars_total[1d])) ``` Per-workflow breakdown for the agent: ```promql sum by (workflow, model)( increase(hummingbird_vertex_cost_dollars_total{app="hummingbird-agent"}[1d]) ) ``` Token consumption by direction: ```promql sum by (app, direction)(increase(hummingbird_vertex_tokens_total[1d])) ``` ### Implementing apps | Application | Port | Workflow values | | --------------------- | ---- | ------------------------------- | | hummingbird-agent | 9090 | from `wf_cfg.name` per workflow | | hummingbird-dashboard | 8080 | `analyze-failures` | -------------------------------------------------------------------------------- # SNS S3 Archiver url: https://hummingbird-project.io/docs/background/tools/sns-s3-archiver/ A Lambda function that archives all messages from the Hummingbird Events SNS topic to S3. Events are stored as clean JSON with decoded payloads, enabling easy browsing with `zcat | jq` and replay to handlers. ## Features - **S3 Storage**: Events stored with time-hierarchical key structure for efficient time-range queries - **Clean JSON**: Decodes gzip+base64 Message payload for human-readable storage - **Replay Support**: Preserves all MessageAttributes with updated `content_encoding` for replay fidelity - **Easy Access**: Download via `aws s3 sync`, browse with `zcat | jq` ## Prerequisites - AWS CLI configured with appropriate credentials (IAM permissions for Lambda, S3, SNS, CloudFormation) - Podman or Docker (for containerized SAM build/deploy) - Python 3.11 or later (for development) ## Deployment Build and deploy using containerized AWS SAM CLI: ```bash cd sns-s3-archiver make build # Build SAM application make deploy # First deployment (interactive/guided) make redeploy # Subsequent deployments (non-interactive) ``` **Deployment outputs:** - `BucketName` - S3 bucket name for archived events - `BucketArn` - S3 bucket ARN - `FunctionArn` - Lambda function ARN ### Parameters | Parameter | Description | Default | |------------------|------------------------------------------|--------------| | `ResourcePrefix` | Prefix for all resource names | `myapp-prod` | | `SnsTopicArn` | ARN of the SNS topic to subscribe to | (required) | | `SentryDsn` | Optional Sentry DSN for error tracking | `""` | | `RetentionDays` | Days to retain events (0=indefinite) | `0` | **Resource naming:** Bucket and Lambda names are derived from ResourcePrefix: - S3 bucket: `{ResourcePrefix}-s3-events` - Lambda function: `{ResourcePrefix}-lambda-s3-archiver` **Prerequisites:** Requires an existing SNS topic. Deploy [hummingbird-events-topic][het-docs] first to create the topic, then use its ARN for the `SnsTopicArn` parameter. ## S3 Key Structure Events are stored with a time-hierarchical key structure optimized for time-range queries: ```text sns/YYYY/MM/DD/HH/MM/TIMESTAMP#MSGID::source::kind::name.json.gz ``` Examples: ```text sns/2025/12/13/12/34/2025-12-13T12:34:56.789Z#abc12345::kubernetes::Snapshot::my-snapshot.json.gz sns/2025/12/13/12/34/2025-12-13T12:34:56.789Z#def67890::gitlab::push::abc123def456.json.gz ``` The `sns/` prefix separates the new format from legacy data, enabling incremental migration. The `::` delimiter separates metadata fields (avoiding conflicts with K8s names that contain `--`). The resource name in the key enables efficient "latest state per resource" queries without downloading all files. The minute-level subdirectory (`/MM/`) provides write distribution for high-throughput scenarios. ## Storage Format Each S3 object contains a gzip-compressed JSON record: ```json { "MessageId": "abc12345-1234-5678-9abc-def012345678", "Timestamp": "2025-12-13T12:34:56.789Z", "Message": { "apiVersion": "appstudio.redhat.com/v1alpha1", "kind": "Snapshot", "metadata": { ... }, "spec": { ... } }, "MessageAttributes": { "source": { "Type": "String", "Value": "kubernetes" }, "kind": { "Type": "String", "Value": "Snapshot" }, "content_encoding": { "Type": "String", "Value": "json" } } } ``` Key points: - `Message` is a decoded JSON object (not the original base64+gzip string from SNS) - `content_encoding` is set to `"json"` to indicate the decoded format ## Usage ### Download Events Locally ```bash # Sync entire bucket aws s3 sync s3://bucket-name/ ./local-events/ # Sync specific time range aws s3 sync s3://bucket-name/2025/12/13/ ./local-events/2025/12/13/ # Sync with filtering aws s3 sync s3://bucket-name/ ./local-events/ --exclude "*" --include "*kubernetes*Snapshot*" ``` ### Browse Events ```bash # List events for a specific hour aws s3 ls s3://bucket-name/2025/12/13/12/ --recursive # View a single event aws s3 cp s3://bucket-name/2025/12/13/12/34/event.json.gz - | zcat | jq # Local browsing zcat ./local-events/2025/12/13/12/34/event.json.gz | jq ``` ### Replay Events Consumer handlers should support both live SNS format and archived format: ```python def _decode_message(sns_record: dict) -> dict: """Decode Message - handles live SNS and archived formats.""" message = sns_record.get("Message", "") encoding = sns_record.get("MessageAttributes", {}).get("content_encoding", {}).get("Value") # Archived format: Message is already decoded if encoding == "json" or isinstance(message, dict): return message if isinstance(message, dict) else json.loads(message) # Live SNS format: Message is gzip+base64 if encoding == "gzip+base64": return json.loads(gzip.decompress(base64.b64decode(message))) # Fallback: plain JSON string return json.loads(message) ``` ## Compression Flow ```text Forwarder → SNS → Archiver → S3 1. Forwarder: gzip(payload) → base64 → SNS Message 2. SNS: Passes through with content_encoding="gzip+base64" 3. Archiver: Decodes Message, sets content_encoding="json", gzip(record) → S3 4. S3: Stores clean JSON record, gzip-compressed at object level ``` ## Development See the main [README][readme] for development workflows. ```bash make setup # Install dependencies make check # Lint code (ruff) make fmt # Format code make test # Run unit tests make coverage # Run tests with coverage ``` ## S3 Lifecycle Policies The bucket has two lifecycle rules: - **AbortIncompleteUploads**: Automatically cleans up incomplete multipart uploads after 1 day (prevents storage cost from failed uploads) - **ExpireOldEvents**: When `RetentionDays > 0`, automatically deletes objects older than the specified retention period. Disabled when `RetentionDays = 0` (indefinite retention, the default). ## Security & Limitations **Security:** - S3 bucket blocks all public access by default - S3 objects encrypted at rest (AES256) - Lambda follows least privilege principle (PutObject only) - CloudWatch logs capture all archive operations (7-day retention) - Sentry integration for error tracking **Limitations:** - Lambda timeout: 30 seconds - Lambda memory: 256 MB - S3 object key length: max 1024 bytes - Single-region deployment (follows SNS topic region) ## License This project is licensed under the GNU General Public License v3.0 or later - see the [LICENSE][license] file for details. [readme]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/README.md [license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE [het-docs]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/documentation/hummingbird-events-topic.md -------------------------------------------------------------------------------- # Hummingbird Tools url: https://hummingbird-project.io/docs/background/tools/hummingbird-tools/ Database backup and restore tools for the Hummingbird status database. Creates compressed PostgreSQL dumps, uploads to S3 with rotation, and supports restore from the latest backup. ## Features - **Streaming Backup**: Streams pg_dump through gzip to temp file, then S3 multipart upload - handles arbitrarily large databases - **Streaming Restore**: Downloads backup to temp file, streams through psql - no memory constraints - **Automatic Rotation**: Keeps configurable number of backups per group (daily/weekly/monthly) - **Separate IAM Users**: Write user for production backups, read-only user for staging restore ## Prerequisites - AWS CLI configured with appropriate credentials - PostgreSQL client tools (pg_dump, psql) - Python 3.11 or later - Access to the hummingbird-status PostgreSQL database ## Deployment The SAM template creates: - S3 bucket for database backups - IAM user with write permissions (for production backup CronJobs) - IAM user with read-only permissions (for staging restore CronJob) Build and deploy using containerized AWS SAM CLI: ```bash cd hummingbird-tools make build # Build SAM application make deploy # First deployment (interactive/guided) make redeploy # Subsequent deployments (non-interactive) ``` **Note:** Only deployed in production - staging restore reads from the production backup bucket. ### Parameters | Parameter | Description | Default | |------------------|-------------------------------|----------------| | `ResourcePrefix` | Prefix for all resource names | `myapp-prod` | **Resource naming:** - S3 bucket: `{ResourcePrefix}` - Write IAM user: `{ResourcePrefix}-write-user` - Read IAM user: `{ResourcePrefix}-read-user` ## Usage ### Backup Run a backup with rotation: ```bash python3 -m hummingbird_tools.backup ``` Arguments: - `group`: Backup group name (e.g., `daily`, `weekly`, `monthly`) - `rotation`: Number of backups to keep for this group Environment variables: | Variable | Description | |----------------------|--------------------------------| | `S3_BUCKET` | S3 bucket name | | `S3_PREFIX` | Optional prefix for S3 keys | | `POSTGRESQL_HOST` | Database hostname | | `POSTGRESQL_USER` | Database username | | `POSTGRESQL_PASSWORD`| Database password | | `POSTGRESQL_DATABASE`| Database name | Example: ```bash export S3_BUCKET=myapp-prod-db-backups export S3_PREFIX=mydb/ export POSTGRESQL_HOST=myapp-postgres export POSTGRESQL_USER=postgres export POSTGRESQL_PASSWORD=secret export POSTGRESQL_DATABASE=mydb python3 -m hummingbird_tools.backup daily 7 ``` This creates a backup like `mydb/2026-01-16-02-30.daily.sql.gz` and removes any daily backups beyond 7. ### Restore Restore from the latest backup: ```bash python3 -m hummingbird_tools.restore ``` Environment variables are the same as for backup. The restore process: 1. Lists all backups matching `*.{daily,weekly,monthly}.sql.gz` 2. Selects the latest by filename sort (most recent timestamp) 3. Downloads to temp file 4. Drops all existing tables in the `public` schema 5. Streams the backup through psql ## CronJob Schedule Deployed via Kubernetes CronJobs in `kubernetes/hummingbird-status/`: | CronJob | Schedule | Environment | Command | |---------|---------------|-------------|--------------------------------------------------| | daily | `30 2 * * *` | production | `python3 -m hummingbird_tools.backup daily 7` | | weekly | `30 6 * * 0` | production | `python3 -m hummingbird_tools.backup weekly 4` | | monthly | `30 10 1 * *` | production | `python3 -m hummingbird_tools.backup monthly 12` | | restore | (suspended) | staging | `python3 -m hummingbird_tools.restore` | To manually trigger a staging restore: ```bash kubectl create job --from=cronjob/hummingbird-status-restore manual-restore-$(date +%s) ``` ## S3 Key Structure Backups are stored with a flat key structure per database: ```text {database}/{timestamp}.{group}.sql.gz ``` Examples: ```text mydb/2026-01-16-02-30.daily.sql.gz mydb/2026-01-12-06-30.weekly.sql.gz mydb/2026-01-01-10-30.monthly.sql.gz ``` Multiple databases can share the same bucket using different prefixes. ## Development See the main [README][readme] for development workflows. ```bash make hummingbird-tools/setup # Install dependencies make check # Lint code (ruff) make fmt # Format code make test # Run unit tests make coverage # Run tests with coverage ``` ## Security - S3 bucket blocks all public access - S3 objects encrypted at rest (AES256) - Separate IAM users with least privilege: - Write user: `s3:PutObject`, `s3:DeleteObject`, `s3:ListBucket` - Read user: `s3:GetObject`, `s3:ListBucket` - Database passwords passed via environment variables (Kubernetes secrets) ## License This project is licensed under the GNU General Public License v3.0 or later - see the [LICENSE][license] file for details. [readme]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/README.md [license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE -------------------------------------------------------------------------------- # ProdSec RPM Catalog url: https://hummingbird-project.io/docs/background/tools/hummingbird-tools-prodsec-catalog/ Builds an RPM catalog from the Hummingbird Pulp repos and publishes it to the metadata distribution for ProdSec. Also includes a monitoring companion that prints the last publication timestamp. ## Features - **DNF-Based Repo Query**: Reads package metadata from all 5 Hummingbird repos (source, x86_64, aarch64, ppc64le, s390x) using the DNF API - **Pulp REST API Upload**: Creates artifacts, file content, publications, and updates the distribution in a single automated workflow - **Artifact Deduplication**: Skips upload when content SHA-256 matches an existing artifact - **Async Task Polling**: Handles Pulp's asynchronous tasks with configurable timeout (10 minutes) - **Sentry Error Reporting**: Errors are reported to Sentry when `SENTRY_DSN` is set - **Monitoring Check**: Companion module prints the publication timestamp for automated freshness monitoring ## Prerequisites - Python 3.11 or later - `dnf` system library (available on Fedora/RHEL) - Network access to `packages.redhat.com` (repo metadata) - Pulp API access with client certificate or username/password auth - A file repository and distribution named `metadata` in the `public-hummingbird` Pulp domain ## Usage ### Build and Publish Catalog ```bash python3 -m hummingbird_tools.prodsec_catalog ``` No arguments. All configuration is via environment variables. ### Check Last Publication Time ```bash python3 -m hummingbird_tools.prodsec_catalog_check ``` Prints the timestamp of the most recent catalog publication in local timezone. Uses the same authentication as the catalog builder. ## Configuration ### Authentication Client certificate auth (preferred for CronJob): | Variable | Description | |------------------------------------|------------------------------------------| | `HUMMINGBIRD_PULP_BOT_CERTIFICATE` | Path to client certificate PEM file | | `HUMMINGBIRD_PULP_BOT_KEY` | Path to private key PEM file (optional) | | `HUMMINGBIRD_PULP_BOT_PASSWORD` | Optional passphrase for the key | Basic auth (alternative for local use): | Variable | Description | |-----------------|----------------------| | `PULP_USERNAME` | Pulp API username | | `PULP_PASSWORD` | Pulp API password | Credentials can also be configured in `~/.config/pulp/cli.toml` (same format as the pulp CLI). Set `PULP_CONFIG` to override the config file path (useful in containers where the home directory may vary). ### Pulp API | Variable | Description | |-----------------|------------------------------------------------------------------| | `PULP_BASE_URL` | Pulp API base URL (or read from `cli.toml`) | | `PULP_CONFIG` | Path to pulp CLI config file (default `~/.config/pulp/cli.toml`) | ### Error Reporting | Variable | Description | |--------------|----------------------------------------| | `SENTRY_DSN` | Optional Sentry DSN for error tracking | ## CronJob Schedule Deployed via Kubernetes CronJob in `kubernetes/hummingbird-status/`: | CronJob | Schedule | Environment | Command | |------------------|-------------|-------------|------------------------------------------------------| | prodsec-catalog | `0 4 * * *` | production | `python3 -m hummingbird_tools.prodsec_catalog` | ## Catalog Format The catalog is a tab-separated file with one RPM per line: ```text {name}-{version}-{release}.{arch}.rpm\t{repo}\t{build_timestamp} ``` Example: ```text nginx-1.28.0-1.hum1.x86_64.rpm x86_64 2026-01-15 14:30:00 kernel-6.12.5-1.hum1.src.rpm source 2026-01-10 08:00:00 ``` Published at: `https://packages.redhat.com/api/pulp-content/public-hummingbird/metadata/hummingbird-rpm-catalog.txt` ## Development See the main [README][readme] for development workflows. ```bash make hummingbird-tools/setup # Install dependencies make check # Lint code (ruff) make test # Run unit tests ``` ## License This project is licensed under the GNU General Public License v3.0 or later - see the [LICENSE][license] file for details. [readme]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/README.md [license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE -------------------------------------------------------------------------------- # CVE Analysis url: https://hummingbird-project.io/docs/background/tools/hummingbird-cve-analysis/ Content not included (62550 characters, exceeds 50000 limit). Fetch the full page: https://hummingbird-project.io/docs/background/tools/hummingbird-cve-analysis/index.md -------------------------------------------------------------------------------- # Dashboard MR Linker url: https://hummingbird-project.io/docs/background/tools/dashboard-mr-linker/ An AWS Lambda function that posts internal notes on newly opened GitLab merge requests linking to the Hummingbird dashboard status page. This provides an easy way for developers to check Konflux build status directly from their MRs. ## Features - **Internal Notes**: Posts an internal note with dashboard link when MRs are opened (visible to project members only) - **Configurable Projects**: Only monitors specified projects via space-delimited list - **SNS Integration**: Subscribes to GitLab events via SNS with filter policy ## Architecture The Lambda subscribes to the existing SNS topic (from gitlab-event-forwarder) with a filter for `merge_request` events: ```text GitLab Webhook → gitlab-event-forwarder → SNS Topic → dashboard-mr-linker → GitLab API ``` When a new MR is opened on a configured repository, the Lambda posts an internal note: ```markdown :robot: **Hummingbird Status** View Konflux build status for this MR: [org/group/repo!123](https://dashboard.example.com/mr/org/group/repo/123) ``` ## Prerequisites - AWS CLI configured with appropriate credentials (IAM permissions for Lambda, SNS, CloudFormation, CloudWatch Logs) - Podman or Docker (for containerized SAM build/deploy) - Python 3.11 or later (for development) - GitLab API token with `api` scope (Reporter role or higher on monitored projects) ## Deployment Build and deploy using containerized AWS SAM CLI: ```bash cd dashboard-mr-linker make build # Build Lambda package make deploy # First deployment (interactive/guided) make redeploy # Subsequent deployments (non-interactive) ``` ### Parameters | Parameter | Description | Default | |--------------------|----------------------------------------------|----------------------| | `ResourcePrefix` | Prefix for resources | `myapp-prod` | | `SnsTopicArn` | SNS topic ARN | (required) | | `GitLabToken` | GitLab API token (`api` scope, Reporter+) | (required) | | `GitLabUrl` | GitLab instance URL | `https://gitlab.com` | | `DashboardBaseUrl` | Base URL of the dashboard | (required) | | `GitLabProjects` | Space-delimited list of GitLab project paths | (required) | | `SentryDsn` | Optional Sentry DSN | `` | **Resource naming:** Lambda follows `{ResourcePrefix}-lambda-linker` pattern. **Prerequisites:** Requires an existing SNS topic. Deploy [hummingbird-events-topic][het-docs] first to create the topic, then use its ARN for the `SnsTopicArn` parameter. [het-docs]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/documentation/hummingbird-events-topic.md ## Usage Configure the `GITLAB_PROJECTS` parameter with space-delimited project paths: ```text org/group/containers org/group/rpms org/other/project ``` Only MRs opened on these projects will receive dashboard link notes. ## Development See the main [README][readme] for development workflows. ```bash make setup # Install dependencies make check # Lint code (ruff) make fmt # Format code make test # Run unit tests make coverage # Run tests with coverage ``` ## Configuration Lambda function receives configuration via environment variables (automatically set by CloudFormation): | Variable | Description | |----------------------|----------------------------------------------| | `GITLAB_TOKEN` | GitLab API token (`api` scope, Reporter+) | | `GITLAB_URL` | GitLab instance URL | | `DASHBOARD_BASE_URL` | Base URL of the dashboard | | `GITLAB_PROJECTS` | Space-delimited list of GitLab project paths | | `SENTRY_DSN` | Optional Sentry DSN | | `AWS_REGION` | AWS region (auto-set) | ## Security & Limitations **Security:** - GitLab API token should have minimal required scopes (`api`) - Token stored as CloudFormation parameter with `NoEcho: true` - SNS subscription uses filter policy to only receive relevant events - Notes are posted as internal (visible to project members only) - CloudWatch logs capture all note posts (7-day retention) - Sentry integration for error tracking **Limitations:** - Lambda timeout: 30 seconds - Lambda memory: 256 MB - Only processes `action: open` events (not updates or other actions) ## License This project is licensed under the GNU General Public License v3.0 or later - see the [LICENSE][license] file for details. [readme]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/README.md [license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE -------------------------------------------------------------------------------- # GitLab Event Forwarder url: https://hummingbird-project.io/docs/background/tools/gitlab-event-forwarder/ An AWS Lambda function that receives GitLab webhook events and forwards them to an SNS topic with structured metadata for filtering. Validates webhook signatures and extracts minimal metadata (source, event type, project/group paths) as SNS message attributes, enabling downstream subscribers to filter events precisely. The original GitLab JSON payload is forwarded unchanged as the SNS message body. ## Features - **Webhook Signature Validation**: Verifies authenticity using GitLab's `X-Gitlab-Token` header - **Structured Metadata**: Extracts event type, project/group paths as SNS message attributes - **SNS Subscription Filtering**: Enables precise event routing to subscribers - **Custom Domain**: Optional custom domain with automatic TLS certificate management via ACM and Route53 ## Architecture Single Lambda function handles the webhook processing: 1. **Handler** - API Gateway endpoint (`/webhook`) that validates GitLab webhook signatures, extracts metadata, and publishes to SNS ## Prerequisites - AWS CLI configured with appropriate credentials (IAM permissions for Lambda, API Gateway, SNS, CloudFormation, CloudWatch Logs, and optionally Route53/ACM for custom domain) - Podman or Docker (for containerized SAM build/deploy) - Python 3.11 or later (for development) ## Deployment Build and deploy using containerized AWS SAM CLI: ```bash cd gitlab-event-forwarder make build # Build Lambda package make deploy # First deployment (interactive/guided) make redeploy # Subsequent deployments (non-interactive) ``` **Deployment outputs:** `ApiEndpoint` - API Gateway endpoint URL (``) ### Custom Domain Optional custom domain with automatic TLS certificate management (ACM + Route53). Requires a Route53 hosted zone. Deploy with `CustomDomainName` and `HostedZoneId` parameters - CloudFormation handles certificate creation, DNS validation, and configuration. Certificate validation takes 5-30 minutes; allow up to 1 hour for DNS propagation. ### Parameters | Parameter | Description | Default | |-----------------------|--------------------------------|----------------| | `ResourcePrefix` | Prefix for resources | `myapp-prod` | | `SnsTopicArn` | SNS topic ARN | (required) | | `GitLabWebhookTokens` | GitLab webhook tokens (JSON) | (required) | | `SentryDsn` | Optional Sentry DSN | `` | | `CustomDomainName` | Custom domain name | `` | | `HostedZoneId` | Route53 hosted zone | `` | **Resource naming:** Lambda and API resources follow `{ResourcePrefix}-{type}-{name}` pattern (e.g., `myapp-prod-lambda-handler`, `myapp-prod-api`). **Prerequisites:** Requires an existing SNS topic. Deploy [hummingbird-events-topic][het-docs] first to create the topic, then use its ARN for the `SnsTopicArn` parameter. [het-docs]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/documentation/hummingbird-events-topic.md ## Usage Configure GitLab projects or groups to send webhooks to ``: ```yaml webhooks: : token: push_events: true merge_requests_events: true pipeline_events: true ``` **SNS Subscription Filter Examples:** Push events from a specific project: ```json { "source": ["gitlab"], "event_type": ["push"], "project_path": ["redhat/hummingbird/containers"] } ``` All merge request events: ```json { "source": ["gitlab"], "event_type": ["merge_request"] } ``` Member events from a group: ```json { "source": ["gitlab"], "event_type": ["member"], "group_path": ["redhat/hummingbird"] } ``` ## Development See the main [README][readme] for development workflows. ```bash make setup # Install dependencies make check # Lint code (ruff) make fmt # Format code make test # Run unit tests make coverage # Run tests with coverage ``` ## Configuration Lambda function receives configuration via environment variables (automatically set by CloudFormation): | Variable | Description | |-------------------------|--------------------------------------| | `SNS_TOPIC_ARN` | SNS topic ARN | | `GITLAB_WEBHOOK_TOKENS` | GitLab webhook tokens (JSON array) | | `SENTRY_DSN` | Optional Sentry DSN | | `AWS_REGION` | AWS region (auto-set) | ## Event Metadata The Lambda function extracts minimal metadata from GitLab webhook events and adds them as SNS message attributes: | Attribute | Description | Example | |----------------|--------------------| ------------------------------------| | `source` | Always `"gitlab"` | `gitlab` | | `event_type` | From `object_kind` | `push`, `merge_request` | | `project_path` | Full project path | `redhat/hummingbird/containers` | | `group_path` | Full group path | `redhat/hummingbird` | ## Security & Limitations **Security:** - Webhook token validation using `X-Gitlab-Token` header - Supports multiple active tokens for zero-downtime rotation - Invalid/missing tokens return HTTP 401 - SNS topic follows least privilege principle - CloudWatch logs capture all webhook deliveries (7-day retention) - Sentry integration for error tracking **Limitations:** - Lambda timeout: 30 seconds - Lambda memory: 256 MB - Webhook payload size: Up to 6 MB (API Gateway limit) ## License This project is licensed under the GNU General Public License v3.0 or later - see the [LICENSE][license] file for details. [readme]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/README.md [license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE -------------------------------------------------------------------------------- # Hummingbird MR Human Tracker url: https://hummingbird-project.io/docs/background/tools/hummingbird-mr-human-tracker/ A CLI tool that detects human intervention on automation-opened GitLab merge requests (Renovate, Dependabot, etc.) and links them to a per-sprint Jira epic for tracking. ## Features - **Bot MR Detection**: Identifies MRs authored by bot accounts across multiple GitLab projects using configurable username patterns - **Human Activity Detection**: Finds MRs where humans have commented or pushed commits (via GitLab system notes) - **Per-Sprint Jira Epic**: Automatically finds or creates an epic per sprint to track automation fix work (e.g. "Automation MR fixes for sprint Hum S16") - **Remote Link Tracking**: Adds affected MR URLs as remote links on the Jira epic, with deduplication to avoid duplicates on repeated runs - **Merged/Closed MR Scanning**: `--since` flag catches MRs that were merged or closed before the next scheduled run - **Dry Run**: Preview what would be linked without making any changes ## Prerequisites - Python 3.11 or later - GitLab API token with read access to target projects (including MR notes) - Jira API token with permission to create epics and remote links in the HUM project ## Installation ```bash pip install -e hummingbird-mr-human-tracker ``` ## Usage ```bash mr-human-tracker --jira-user user@redhat.com --dry-run -v ``` ### Common invocations ```bash # Preview what would be linked (no changes made) mr-human-tracker --jira-user user@redhat.com --dry-run -v # Run for real, including MRs merged in the last 2 days mr-human-tracker --jira-user user@redhat.com --since 2d # Scan only specific projects mr-human-tracker --jira-user user@redhat.com --projects redhat/hummingbird/tools # Verbose output for debugging mr-human-tracker --jira-user user@redhat.com --since 1w -v ``` ### Output On completion, the tool prints a summary of linked MRs: ```text INFO hummingbird_mr_human_tracker.tracker: Active sprint: Hum S16 [June 11 - June 25] (id=68851) INFO hummingbird_mr_human_tracker.tracker: Using epic HUM-2602 INFO hummingbird_mr_human_tracker.tracker: Linked 2 MR(s) to HUM-2602: INFO hummingbird_mr_human_tracker.tracker: https://gitlab.com/redhat/hummingbird/rpms/-/merge_requests/2395 INFO hummingbird_mr_human_tracker.tracker: https://gitlab.com/redhat/hummingbird/containers/-/merge_requests/8476 ``` ## Configuration All configuration is via CLI arguments and environment variables. ### CLI arguments | Argument | Description | Default | |-------------------|----------------------------------------------------|----------------------------| | `--gitlab-url` | GitLab instance URL | `GITLAB_URL` env | | `--gitlab-token` | GitLab private token | `GITLAB_TOKEN` env | | `--jira-url` | Jira base URL | `JIRA_URL` env | | `--jira-token` | Jira API token | `JIRA_TOKEN` env | | `--jira-user` | Jira email for Basic auth (omit for Bearer) | None | | `--projects` | GitLab project paths to scan | tools, rpms, containers | | `--board-id` | Jira board ID for sprint lookup | `1489` | | `--sprint-prefix` | Sprint name prefix to match | `Hum S` | | `--since` | Also scan merged/closed MRs updated within window | None (open MRs only) | | `--dry-run` | Report without making changes | False | | `-v, --verbose` | Enable debug logging | False | ### Environment variables | Variable | Description | |----------------|--------------------------------| | `GITLAB_URL` | GitLab instance URL | | `GITLAB_TOKEN` | GitLab private token | | `JIRA_URL` | Jira base URL | | `JIRA_TOKEN` | Jira API token | ### Duration format for `--since` The `--since` flag accepts a number followed by a unit: | Unit | Meaning | |------|---------| | `h` | Hours | | `d` | Days | | `w` | Weeks | Examples: `12h`, `2d`, `1w` ## How it works 1. **Fetch active sprint** from the Jira Agile API (board 1489, prefix "Hum S") 2. **Find or create** a Jira epic named "Automation MR fixes for sprint {sprint_name}" in the HUM project, assigned to the active sprint 3. **Scan GitLab projects** for open MRs authored by bot accounts; if `--since` is set, also scan recently merged/closed MRs 4. **Detect human activity** on each bot MR by checking: - Non-bot, non-system comments - System notes indicating a human pushed commits ("added N commit") - Reopen events by non-bot users 5. **Add remote links** on the Jira epic for each human-touched MR (skipping URLs already linked) ### Bot detection A username is considered a bot if it matches any of: - Contains `bot_` or `bot-` (e.g. `renovate_bot`) - Ends with `_bot` (e.g. `some_bot`) - Ends with `[bot]` (e.g. `renovate[bot]`, `dependabot[bot]`) - Starts with `project__bot` or `group__bot` (GitLab service accounts) ## Development ```bash make setup # Install dependencies make check # Lint code (ruff) make fmt # Format code make test # Run unit tests make coverage # Run tests with coverage ``` ## License This project is licensed under the GNU General Public License v3.0 or later - see the [LICENSE][license] file for details. [license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE -------------------------------------------------------------------------------- # MR Auto-Approver url: https://hummingbird-project.io/docs/background/tools/mr-auto-approver/ An AWS Lambda function that auto-approves GitLab merge requests based on configurable per-project rules. Replaces CI/CD-based approval jobs and moves approval tokens out of GitLab CI/CD variables. ## Features - **Rules-Based Config**: YAML config with per-project ordered rules matching on user IDs, username patterns, branch patterns, and Konflux pipeline status - **First-Match-Wins**: Rules evaluated in order; first match determines action (approve or deny) - **Konflux Integration**: Optional check that Konflux pipelines have posted commit statuses before approving - **Fork Rejection**: Unconditionally rejects MRs from forked projects - **User Verification**: Matches against the authenticated webhook event user, not forgeable Git commit metadata - **Stateless & Idempotent**: No queues or stored state; re-approving is a no-op ## Architecture The Lambda subscribes to the existing SNS topic (from gitlab-event-forwarder) with a filter for `merge_request` and `pipeline` events: ```text GitLab Webhook → gitlab-event-forwarder → SNS Topic → mr-auto-approver → GitLab API ``` **Merge request events** (`open`, `update`): evaluate rules using the authenticated pusher from the webhook payload. If matched and no Konflux check needed, approve immediately. If Konflux check needed, check statuses now and approve only if present and not failed. **Pipeline events** (`success`, `failed`): look up the MR from the pipeline event via the GitLab API, evaluate rules, and approve if Konflux statuses are ready. This handles the case where Konflux posts commit statuses after the initial MR event. ## Prerequisites - AWS CLI configured with appropriate credentials (IAM permissions for Lambda, SNS, CloudFormation, CloudWatch Logs) - Podman or Docker (for containerized SAM build/deploy) - Python 3.11 or later (for development) - GitLab API token(s) with `api` scope and permission to approve MRs on target projects - Target project webhooks must send `merge_requests_events` and `pipeline_events` to the gitlab-event-forwarder endpoint ## Deployment Build and deploy using containerized AWS SAM CLI: ```bash cd mr-auto-approver make build # Build Lambda package make deploy # First deployment (interactive/guided) make redeploy # Subsequent deployments (non-interactive) ``` ### Parameters | Parameter | Description | Default | |------------------|--------------------------------------------|-----------------------| | `ResourcePrefix` | Prefix for resources | `myapp-prod` | | `SnsTopicArn` | SNS topic ARN | (required) | | `GitLabUrl` | GitLab instance URL | `https://gitlab.com` | | `ConfigPath` | Path to YAML config (bundled with Lambda) | `config.yaml` | | `ApprovalTokens` | JSON map of env var names to tokens | `{}` | | `SentryDsn` | Optional Sentry DSN | `` | **Resource naming:** Lambda follows `{ResourcePrefix}-lambda-approver` pattern. **Prerequisites:** Requires an existing SNS topic. Deploy [hummingbird-events-topic][het-docs] first to create the topic, then use its ARN for the `SnsTopicArn` parameter. [het-docs]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/documentation/hummingbird-events-topic.md ## Configuration The Lambda uses a YAML config file bundled at deploy time. The config defines per-project rules: ```yaml settings: gitlab_url: https://gitlab.com projects: org/group/repo: token_env: APPROVAL_GITLAB_TOKEN_REPO rules: - branch_regexes: ["renovate/skip/.*"] action: deny - user_ids: [12345678] branch_regexes: ["renovate/.*"] - user_ids: [87654321] check_konflux: true ``` ### Rule fields | Field | Type | Default | Description | |------------------|-----------|-----------|------------------------------------------| | `user_ids` | list[int] | `[]` | GitLab user IDs (immutable, preferred) | | `user_regexes` | list[str] | `[]` | Fullmatch regexes for username | | `branch_regexes` | list[str] | `[]` | Fullmatch regexes for source branch | | `check_konflux` | bool | `false` | Require Konflux statuses before approval | | `action` | str | `approve` | `approve` or `deny` | **Match logic**: AND across field types, OR within a field. Empty fields match anything. First matching rule wins. `user_ids` and `user_regexes` are both user identity constraints -- if either is set, the user must match at least one entry from across both lists. Prefer `user_ids` over `user_regexes` to prevent username confusion attacks. ### Adding a new project 1. Create a project-scoped GitLab token (Developer, `api` scope) in Vault 2. Add the token to `vars.sh` and the `jq` command that builds `APPROVAL_TOKENS` 3. Add a project entry to `config.yaml` in the infrastructure repo with `token_env` and rules 4. Deploy the Lambda 5. Ensure the project's GitLab webhook sends events to the gitlab-event-forwarder endpoint ### Environment variables | Variable | Description | |--------------------|----------------------------------------------------| | `CONFIG_PATH` | Path to YAML config file | | `GITLAB_URL` | GitLab instance URL | | `APPROVAL_TOKENS` | JSON object mapping env var names to GitLab tokens | | `SENTRY_DSN` | Optional Sentry DSN | ## Security - Fork MRs are unconditionally rejected before any rule evaluation (`source_project_id != target_project_id`) - User matching supports immutable `user.id` (preferred) in addition to `user.username` from the webhook payload, preventing username confusion attacks. Never uses Git commit author/committer metadata - For pipeline events, the Lambda looks up `head_pipeline.user` (both `id` and `username`) via the GitLab API to verify the last pusher - Approval tokens stored as CloudFormation parameters with `NoEcho: true` - SNS subscription filter policy limits events to `merge_request` and `pipeline` - CloudWatch logs capture all approval decisions (7-day retention) ## Development See the main [README][readme] for development workflows. ```bash make setup # Install dependencies make check # Lint code (ruff) make fmt # Format code make test # Run unit tests make coverage # Run tests with coverage ``` ## License This project is licensed under the GNU General Public License v3.0 or later - see the [LICENSE][license] file for details. [readme]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/README.md [license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE -------------------------------------------------------------------------------- # Container Catalog url: https://hummingbird-project.io/docs/background/tools/container-catalog/ Per-distro serverless catalog API for Hummingbird container images. ## Features - **Image Directory** - Browse all container images with metadata - **Tag Browser** - View tags, digests, architectures per image - **Specifications** - Per-architecture OCI config details (env, cmd, user, labels) - **SBOM** - Per-architecture package lists from SPDX attestations - **Vulnerabilities** - CVE scanning results via Grype - **CVE Metrics** - Vulnerability aggregate and structured exposure logs - **Provenance** - Source traceability from SLSA attestations - **Release History** - Timeline of past builds with drill-down - **OpenAPI Spec** - Machine-readable API documentation at /v1/openapi.json - **Swagger UI** - Interactive API explorer at /v1/docs/ ## Architecture Rust serverless stack deployed as two isolated per-distro stacks: - **API Lambda** - DynamoDB pass-through (~10ms response) - **Sync Lambda** - Incremental DynamoDB sync from SNS Release events (~22 registry calls per release) - **Index Lambda** (`index-lambda`) - SNS-triggered Syft JSON generation, uploads native Syft JSON to S3 for scanner consumption - **Scan Lambda** (`scan-lambda`) - SQS-triggered CVE scanning via Grype from native Syft JSON in S3, per-digest processing with `first_seen` tracking and partial batch failure reporting - **Enqueue Lambda** (`enqueue-lambda`) - Hourly EventBridge-triggered fan-out that checks Grype DB for updates and enqueues non-superseded digests to SQS - **Metrics Lambda** (`metrics-lambda`) - DynamoDB Stream-triggered vulnerability aggregate and structured CVE exposure logs - **DynamoDB** - Pre-computed JSON items (single-table PK/SK design, Streams: KEYS_ONLY) - **S3 ScanDataBucket** - Native Syft JSON storage for scanner data (gzipped, keyed by `grype/{image}/{digest_hex}.json.gz`) - **SQS ScanQueue** - Central queue for scan tasks (fed by S3 events and Enqueue Lambda) - **CloudFront** - CDN with per-endpoint cache TTLs - **CloudWatch** - Structured CVE exposure logs - **catalog sync** - Full DynamoDB population from GitLab + Quay.io OCI v2 registry - **catalog scan** - CLI CVE scanning via Grype; reads native Syft JSON from S3 (with `--bucket`) or builds synthetic JSON from DynamoDB SBOMs (fallback) - **catalog index** - CLI Syft JSON generation for backfilling S3 - **Catalog SPA** - Lit 3 web app served from S3 via CloudFront ### Scanner Architecture CVE vulnerability data must match direct `grype ` scans exactly for all images. Synthetic Syft JSON (built from stored SBOM packages) cannot reliably reproduce the native output because Grype relies on metadata fields, artifact relationships, and deduplication logic that are lost in the SPDX-to-API roundtrip. Additionally, raw SPDX from the build system contains package bloat (e.g. Go sub-modules, empty-version entries) that native Syft filters out when scanning a compiled binary directly. To guarantee exact-match results, the scanner chain stores native Syft JSON in S3 rather than DynamoDB: a single image's Syft JSON is typically 1-10 MB (gzipped to 100 KB - 1 MB), which exceeds DynamoDB's 400 KB item limit. S3 has no per-object size constraint, avoids provisioned throughput costs for large blobs, and supports event notifications for triggering downstream scanners. An independent Index Lambda runs `syft ` on each new release and uploads the full output to `s3://{bucket}/grype/{image}/{hex}.json.gz`. The S3 upload fires an event notification to SQS, which triggers the Scan Lambda to read the native JSON, run Grype, and write per-canonical vulnerability data to DynamoDB. The scanner chain is intentionally independent of the catalog data chain: the same SNS Release event triggers both the Sync Lambda (catalog data to DynamoDB) and the Index Lambda (Syft JSON to S3), with no ordering dependency between them. ### CVE Data Flow ```mermaid graph TD Konflux([Konflux Release]) -->|SNS| Sync[SyncFunction] Konflux -->|SNS| Index[IndexFunction] Sync -->|update| Tags[(Tags)] Tags -->|"stream via ESM"| Metrics Tags -->|read| Scan[ScanFunction] Index -->|upload| S3Syft[(S3 Syft JSON)] S3Syft -->|"S3 event via SQS"| Scan Hourly([Schedule]) -->|hourly| Enqueue[EnqueueFunction] Enqueue --> Gate[check Grype DB] Gate -->|"fan out via SQS"| Scan Scan -->|update| Vulns[(Image Vulnerabilities)] Vulns -->|"stream via ESM"| Metrics[MetricsFunction] Metrics -->|"structured logs"| CloudWatch[CloudWatch Logs] Metrics -->|"write aggregate"| CatalogVulns[(Catalog Vulnerabilities)] classDef lambda fill:#d4e6f1,stroke:#2980b9 classDef dynamo fill:#fdebd0,stroke:#e67e22 classDef trigger fill:#d5f5e3,stroke:#27ae60 classDef aws fill:#e8daf1,stroke:#8e44ad classDef gate fill:#e5e7e9,stroke:#7f8c8d class Sync,Scan,Enqueue,Metrics,Index lambda class Vulns,Tags,CatalogVulns,S3Syft dynamo class Konflux,Hourly trigger class CloudWatch aws class Gate gate ``` ## Prerequisites - Rust 1.75+ (for building backend) - Node.js 22+ (for building frontend) - AWS credentials (for DynamoDB access and deployment) - SAM CLI (for deployment) ## API Endpoints | Endpoint | Description | | --------------------------------------------------------- | ------------------------------ | | `GET /v1/images` | Image directory | | `GET /v1/images/{name}` | Image overview (README) | | `GET /v1/images/{name}/tags` | Tags for an image | | `GET /v1/images/{name}/details/{canonical}` | Per-canonical details | | `GET /v1/images/{name}/sbom/{canonical}` | Package list | | `GET /v1/images/{name}/vulnerabilities/{canonical}` | Vulnerability scan | | `GET /v1/images/{name}/history/{stream}/{variant}` | Release timeline | | `GET /v1/images/{name}/releases/details/{digest}` | Release details (immutable) | | `GET /v1/images/{name}/releases/sbom/{digest}` | Release SBOM (immutable) | | `GET /v1/images/{name}/releases/vulnerabilities/{digest}` | Release vulnerabilities | | `GET /v1/vulnerabilities` | Catalog-wide CVE aggregate | | `GET /v1/openapi.json` | OpenAPI 3.1 specification | | `GET /v1/docs/` | Interactive Swagger UI | ## Timestamp Fields The `oldest_created` field on `ImageSummary`, `Tag`, and `HistorySummary` is the earliest OCI `created` timestamp across all architectures in the release. All architectures were built at or after this date, making it useful for conservative staleness detection. The specifications endpoint returns per-architecture data keyed by architecture name. Each architecture's `created` field is the direct OCI config root timestamp for that specific architecture. ## Usage All tools are built as a single `catalog` binary with subcommands (`api`, `sync`, `sync-lambda`, `scan`, `scan-lambda`, `index`, `index-lambda`, `enqueue-lambda`, `metrics`, `metrics-lambda`). The binary is built in the Rust container and CLI subcommands are run in the gitlab-ci container (which provides grype, syft, and other tools). Only `make` and `podman` are required. ### Sync ```bash # Dry run (print items to stdout) make container-catalog/sync ARGS="--distro rawhide --dry-run" # Populate DynamoDB make container-catalog/sync ARGS="--distro rawhide --table-name " ``` ### Sync Lambda The `sync-lambda` subcommand runs as an AWS Lambda function triggered by SNS Release events from [kubernetes-event-forwarder](kubernetes-event-forwarder.md). It incrementally syncs a single image release to DynamoDB (~22 registry API calls per release vs ~104 for a full sync). The Lambda: 1. Decodes gzip+base64 SNS messages 2. Filters for Succeeded releases targeting the configured Quay.io namespace 3. Fetches OCI manifest data for the new digest 4. Writes per-digest items (DETAILS, SBOM, RELEASE_DETAILS, RELEASE_SBOM) 5. Merges into aggregate items (TAGS, HISTORY, OVERVIEW, DIRECTORY) 6. Fetches README from GitLab for OVERVIEW content (uses `README.redhat.md` for hummingbird, `README.md` for rawhide) Registry fetch errors (manifest, SBOM, attestation) propagate as hard failures so the Lambda retries automatically (up to 2 retries with backoff) before sending to the DLQ. GitLab README failures are non-fatal -- the existing README is preserved if the fetch fails. | Environment Variable | Description | | -------------------- | -------------------------------------- | | `TABLE_NAME` | DynamoDB table name | | `DISTRO` | `rawhide` or `hummingbird` | | `SENTRY_DSN` | Optional Sentry DSN for error tracking | ### Index The index subcommand generates native Syft JSON for all non-superseded image digests and uploads them to S3 for scanner consumption. It reads the image directory and TAGS from DynamoDB, checks S3 for existing objects (dedup via HeadObject), runs `syft --platform linux/amd64`, gzips the output, and uploads to `s3://{bucket}/grype/{image}/{hex}.json.gz`. ```bash # Dry run (show what would be uploaded) make container-catalog/index ARGS="--distro hummingbird --table-name
--bucket --dry-run" # Backfill S3 for all images make container-catalog/index ARGS="--distro hummingbird --table-name
--bucket " # Backfill a single image make container-catalog/index ARGS="--distro hummingbird --table-name
--bucket --image caddy" ``` ### Index Lambda The `index-lambda` subcommand runs as an AWS Lambda function triggered by the same SNS Release events as the Sync Lambda. It operates independently of the catalog chain (no DynamoDB access) and generates native Syft JSON for scanner consumption. The Lambda: 1. Decodes gzip+base64 SNS messages (shared with Sync Lambda via `release_event` module) 2. Filters for Succeeded releases targeting the configured Quay.io namespace 3. Checks S3 for existing objects (HeadObject dedup by digest) 4. Runs `syft @ --platform linux/amd64 --output syft-json` 5. Gzips and uploads to `s3://{bucket}/grype/{image}/{hex}.json.gz` Failures propagate for Lambda retry (up to 2 retries) before sending to the IndexDLQ. The `catalog index` CLI backfills any gaps. | Environment Variable | Description | | --------------------- | -------------------------------------- | | `SCAN_DATA_BUCKET` | S3 bucket for scanner data | | `DISTRO` | `rawhide` or `hummingbird` | | `SENTRY_DSN` | Optional Sentry DSN for error tracking | ### Scan The scan subcommand reads image listings, tags, and SBOMs from DynamoDB (no registry access needed) and runs Grype against each image's stored SBOM packages. Results include a `first_seen` timestamp per CVE, tracked at the group+variant+stream level and carried across releases for SLI computation. ```bash # Dry run (print items to stdout) make container-catalog/scan ARGS="--distro hummingbird --table-name
--dry-run" # Scan and write to DynamoDB (purge stale vuln data first, implies --scope=all) make container-catalog/scan ARGS="--distro hummingbird --table-name
--purge" # Scan only non-superseded (current) tags make container-catalog/scan ARGS="--distro hummingbird --table-name
--scope non-superseded" # Scan all releases including historic (tagless) releases make container-catalog/scan ARGS="--distro hummingbird --table-name
--scope all" # Scan a single image make container-catalog/scan ARGS="--distro hummingbird --table-name
--image caddy --dry-run" ``` ### Scan Lambda The `scan-lambda` subcommand runs as an AWS Lambda function triggered by SQS messages. Two paths feed the ScanQueue: 1. **Real-time** (S3 event notification): When the Index Lambda uploads a new Syft JSON to S3, an `s3:ObjectCreated` event (filtered on `grype/` prefix) is sent directly to SQS 2. **Hourly** (Enqueue Lambda): Fans out all non-superseded digests when the Grype vulnerability database has been updated, sending `{"bucket": "...", "key": "grype/{image}/{hex}.json.gz"}` messages The Lambda accepts both S3 event notification JSON and direct `{"bucket", "key"}` messages. For each message it: 1. Downloads and decompresses the native Syft JSON from S3 2. Runs Grype on the raw bytes 3. Looks up all canonical tags matching the digest from the TAGS item 4. Writes `VULNERABILITIES#{canonical}` for each matching tag (with `first_seen` tracking) and `RELEASE_VULNERABILITIES#{hex}` once per digest Processing is per-digest: a single Syft JSON serves all canonicals sharing that digest, avoiding redundant Grype invocations. The Lambda loads the Grype DB on cold start (cached in `/tmp` for warm invocations), processes up to 10 messages per batch, and reports partial batch failures so only failed records return to the queue. | Environment Variable | Description | | -------------------- | -------------------------------------- | | `TABLE_NAME` | DynamoDB table name | | `DISTRO` | `rawhide` or `hummingbird` | | `SENTRY_DSN` | Optional Sentry DSN for error tracking | ### Enqueue Lambda The `enqueue-lambda` subcommand runs hourly via EventBridge Schedule. On each invocation it performs a lightweight HTTP GET of the public Grype DB listing (`latest.json`, ~200 bytes) and compares the `built` timestamp against the stored `CATALOG/LAST_FULL_SCAN_DB` item in DynamoDB. If the DB hasn't changed, it returns early (~23 of 24 hourly invocations short-circuit). When an update is detected, it reads the image directory and all TAGS items, deduplicates by digest, and sends one SQS message per unique digest using `SendMessageBatch`. Messages use the format `{"bucket": "...", "key": "grype/{image}/{hex}.json.gz"}`. | Environment Variable | Description | | ---------------------- | ------------------------------------------ | | `TABLE_NAME` | DynamoDB table name | | `DISTRO` | `rawhide` or `hummingbird` | | `SCAN_QUEUE_URL` | SQS queue URL for scan messages | | `SCAN_DATA_BUCKET` | S3 bucket for scanner data | | `GRYPE_DB_LATEST_URL` | Grype DB listing URL (has sensible default)| | `SENTRY_DSN` | Optional Sentry DSN for error tracking | ### Metrics The `metrics` subcommand performs a one-shot read of all non-superseded vulnerability data from DynamoDB, outputs structured CVE exposure logs, and writes the catalog-wide vulnerability aggregate to DynamoDB (`PK=CATALOG, SK=VULNERABILITIES`). With `--dry-run`, it skips the DynamoDB aggregate write (useful for local inspection). ```bash # Dry run (print structured logs to stdout) make container-catalog/metrics ARGS="--distro hummingbird --table-name
--dry-run" # Write aggregate to DynamoDB and print structured logs make container-catalog/metrics ARGS="--distro hummingbird --table-name
" ``` ### Metrics Lambda The `metrics-lambda` subcommand runs as a DynamoDB Stream-triggered Lambda that writes a catalog-wide vulnerability aggregate to DynamoDB (served by `GET /v1/vulnerabilities`) and emits structured CVE exposure logs to CloudWatch Logs. #### How It Works The Lambda is triggered by DynamoDB Stream events filtered on `VULNERABILITIES#` and `TAGS` changes, with a 60-second batching window and reserved concurrency of 1 (single instance). It maintains an in-memory active CVE table across warm invocations: - **Cold start**: Reads `CATALOG/DIRECTORY`, all `TAGS`, and all non-superseded `VULNERABILITIES#` items from DynamoDB to build the full table (~2-3s at 10000 tags) - **Warm invocations**: Incrementally updates the table from stream event keys (~50-100 GetItem calls per batch) - **After each invocation**: Recomputes aggregate and emits structured logs #### Structured Logs Each invocation emits one JSON log line per active CVE to stdout (captured by CloudWatch Logs). Example query for all active CVEs: ```text filter message = "active_cve" | fields cve, severity, exposure_hours, repository, stream, variant, component | sort exposure_hours desc ``` | Environment Variable | Description | | ---------------------- | -------------------------------- | | `TABLE_NAME` | DynamoDB table name | | `DISTRO` | `rawhide` or `hummingbird` | | `SENTRY_DSN` | Optional Sentry DSN | ### Deployment ```bash make container-catalog/build make container-catalog/deploy ``` ## Configuration ### catalog sync | Argument | Description | | -------------------- | ------------------------------- | | `--distro` | `rawhide` or `hummingbird` | | `--table-name` | DynamoDB table name | | `--purge` | Delete all items before writing | | `--cache-dir` | Cache directory (auto-detected) | | `--image` | Sync only a specific repo | | `--legacy-discovery` | Use GitLab-based repo discovery | ### catalog index | Argument | Description | | ---------------- | ----------------------------------------------- | | `--distro` | `rawhide` or `hummingbird` | | `--table-name` | DynamoDB table name (for image/tag discovery) | | `--bucket` | S3 bucket for scanner data | | `--dry-run` | Print what would be uploaded without uploading | | `--image` | Index only a specific image | | `--parallel` | Number of concurrent operations (default: 2) | ### catalog scan | Argument | Description | | ---------------- | -------------------------------------------------------- | | `--distro` | `rawhide` or `hummingbird` | | `--table-name` | DynamoDB table name (required) | | `--bucket` | S3 bucket with native Syft JSON (uses S3 scan path) | | `--scope` | `non-superseded`, `tags` (default), or `all` | | `--dry-run` | Print items without writing | | `--purge` | Purge vuln data before writing (implies `--scope all`) | | `--cache-dir` | Cache directory (auto-detected) | | `--parallel` | Number of concurrent scans (default: 4) | | `--image` | Scan only a specific image | | `--tag` | Scan only a specific tag (requires `--image`) | ### SAM Parameters | Parameter | Description | | ------------------- | ------------------------------------------------------- | | `Distro` | `rawhide` or `hummingbird` | | `CacheEnabled` | Enable CloudFront caching | | `CatalogDomainName` | Catalog web UI domain | | `ApiDomainName` | API domain | | `HostedZoneId` | Route53 hosted zone | | `CorsOrigins` | Comma-separated CORS origins (default `*`) | | `SnsTopicArn` | SNS topic ARN for Release events (enables sync Lambda) | ## Frontend The catalog web UI is a Lit 3 SPA (Web Components) with Tailwind CSS, built per-distro with Vite. Source is in `container-catalog/frontend/`. Only `make` and `podman` are required (no local Node.js needed). Defaults from `.envrc.defaults` are applied automatically. ```bash # Install dependencies make container-catalog/frontend/setup # Development server at http://localhost:5173 make container-catalog/frontend/dev # Production build make container-catalog/frontend/build ``` Host variants (`*-host`) run without podman (for CI or local Node.js). ### Frontend Build Variables | Variable | Description | | ------------------------------ | ----------------------------------------------------------------- | | `VITE_API_URL` | API base URL for the distro | | `VITE_DISTRO` | `rawhide` or `hummingbird` | | `VITE_DISTRO_LABEL` | Display label for current distro | | `VITE_OTHER_CATALOG_URL` | URL of the other distro's catalog (optional, hides link if unset) | | `VITE_OTHER_DISTRO_LABEL` | Display label for other distro (optional) | | `VITE_VULNERABILITIES_ENABLED` | Show vulnerabilities tab | ## Development ```bash # Backend cargo test # Run tests cargo clippy --all-targets # Lint cargo fmt # Format # Frontend (host variants, requires local Node.js) cd container-catalog/frontend npm run typecheck # Type check npm run build # Production build ``` ## License This project is licensed under the GNU General Public License v3.0 or later - see the [LICENSE][license] file for details. [license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE -------------------------------------------------------------------------------- # Hummingbird Agent url: https://hummingbird-project.io/docs/background/tools/hummingbird-agent/ An event-driven LLM agent that investigates CI/CD failures and posts findings as GitLab merge request notes. The agent executes markdown-defined workflows using tool calling, with all data processing running inside an isolated sandbox container. For the architectural design rationale, module boundaries, security invariants, and design decision registry, see [Agent Design](hummingbird-agent-design.md). For the model loop wire format, see [Agent Model Loop](hummingbird-agent-model-loop.md). ```mermaid flowchart TD Pipeline["Pipeline Fails"] MREvent["MR Created / Updated"] Slash["/hummingbird command"] Pipeline -->|"event"| Agent MREvent -->|"event"| Agent Slash -->|"command"| Agent subgraph Agent ["Hummingbird Agent"] APIs["GitLab + Konflux +
Testing Farm"] Model["LLM
(Gemini / Claude)"] subgraph Sandbox ["Isolated Sandbox (no network)"] Tools["jq / python3 / yq
data processing"] end APIs <-->|"data"| Model Model <-->|"commands"| Tools end Note["MR Note
(analysis or review)"] Agent -->|"posts"| Note Note -->|"reply to continue"| Agent ``` ## Features - **Workflow-driven analysis** - Investigation logic lives in `.md` files, not in code; easy to iterate without redeployment - **Centralized YAML config** - A single config file defines operational settings, workflows, enabled data sources with token env var names, project allowlists, and per-project limits - **Sandboxed execution** - All untrusted commands (`jq`, `python3`, shell) run in an isolated container, never on the host - **Five sandbox backends** - Podman for local development (network-isolated), direct K8s pod creation, Deployment-backed pod pool for low-latency production use (restricted-v2 SCC compliant), KubeVirt VM (direct), or VMIRS-backed VM pool for heavier workloads requiring full OS isolation - **Data source abstraction** - GitLab, Konflux, and Testing Farm are registered as tool-calling functions the model invokes directly - **Auto-spill for large outputs** - Stdout/stderr and data source responses exceeding 4 KB are automatically saved to sandbox files with a compact preview returned to the model, keeping context usage bounded - **Prompt caching** - Gemini uses implicit server-side caching automatically; Claude uses explicit sliding-window cache breakpoints that reduce input token costs by ~80% on multi-turn agent runs - **Token budget management** - Per-call context ceiling and iteration-based soft/hard limits prevent runaway sessions - **Session persistence** - Conversation history, transcript, and sandbox files saved to S3 (production) or local directory (development) for debugging and future session resumption. Sessions are stored in a provider-neutral format, enabling model switching between conversations. ## Architecture ```mermaid flowchart LR subgraph input [Input] SQS["SQS Queue"] CLI["CLI --event"] end subgraph agentLoop [Agent Loop] WF["Workflow .md
(system prompt)"] LLM["LLM<br/>(Gemini / Claude)"] TR["Tool Registry"] end subgraph tools [Tools] SE["sandbox_exec"] FTS["fetch_to_sandbox"] DS["Data Sources"] end subgraph sandbox [Sandbox Container] JQ["jq / python3 / yq"] Files["Spilled files"] end subgraph external [External APIs] GL["GitLab API"] KX["Konflux K8s"] TF["Testing Farm"] end subgraph output [Output] Note["GitLab MR Note"] Session["Session State"] end SQS --> WF CLI --> WF WF --> LLM LLM -->|"tool calls"| TR TR --> SE --> sandbox TR --> FTS --> sandbox TR --> DS DS --> GL DS --> KX DS --> TF DS -->|"auto-spill"| sandbox LLM -->|"final text"| Note LLM --> Session ``` An event (CLI `--event` JSON or SQS message) identifies a GitLab project and MR IID. The config file maps project paths to workflows and provides `action`, `max_iterations`, enabled data sources, and token env var names. Both `run` and `serve` use this config. The markdown body of the prompt file becomes the LLM system prompt. The agent loop iterates until the model produces a final text response or hits the iteration/context limit. Tool calls are dispatched through the `ToolRegistry`: `sandbox_exec` runs shell commands, `fetch_to_sandbox` pipes data source output into the sandbox, and direct data source calls return results inline (or auto-spill large responses to files). For a detailed walkthrough of the model loop -- what gets sent to the model each iteration, how tool calls flow, the exact wire format, and how user replies integrate for session resumption -- see [Agent Model Loop](hummingbird-agent-model-loop.md). ## Event-Driven Triggers In production, the agent consumes events from an SQS queue subscribed to the central SNS topic. The SNS filter policy delivers three event types: - **`gitlab::pipeline`** -- Fires when a GitLab CI pipeline completes. The agent triggers on `status=failed` pipelines from `merge_request_event` sources where the triggering user has at least Developer access on the project. Only workflows with `trigger: pipeline` are executed. Since the pipeline stays open until all Konflux external stages resolve, this naturally waits for all builds and tests to finish before triggering. - **`gitlab::merge_request`** -- Fires on MR open and update events. Only workflows with `trigger: merge_request` are executed. The agent triggers in three cases: a new MR is opened (`action=open`), code is pushed to an existing MR (`action=update` with `oldrev`), or a draft MR is marked as ready (`action=update` with a `changes.draft` transition from `true` to `false`). All other update events (labels, assignees, description changes) are skipped -- they carry no new code. Draft MRs are skipped; SHA-based deduplication prevents reviewing the same code revision twice. Access checks and `trigger_rules` filtering evaluate the **push author**: for open and push events the webhook user is the pusher; for undraft transitions the agent resolves the actual pusher from MR system notes so that the person undrafting cannot bypass the access check on code pushed by an untrusted user. - **`gitlab::note`** -- MR comment events. Two sub-flows: - **Slash command** (`/hummingbird `): triggers a specific workflow. Prefix matching is supported (e.g. `/hummingbird analyze` matches `analyze-failures`). `/hummingbird` or `/hummingbird help` lists available workflows. The note author must have Developer+ access on the project. Optional runtime overrides can be appended: `/hummingbird code-review model=claude-sonnet-4-6 max_iterations=25`. - **Reply to agent note**: when a user replies to an existing agent note (which contains a session marker), the agent loads the previous session from S3 (conversation history + sandbox files) and continues the conversation with the user's reply as input. The system prompt includes a `CONTINUATION_PROMPT` that prevents the model from re-running the full workflow. If the session is not found (expired/deleted), the agent falls back to a cold start. Replies may include overrides on a separate line (e.g. `/hummingbird model=claude-opus-4-6`); override lines are stripped from the user message. Overrides persist in the session until explicitly changed. Notes generated by the agent itself (containing session markers) are skipped to prevent infinite loops. Reply threading respects the `internal_notes` config: if the project requires internal notes but the original thread was public, the reply is posted as a new top-level internal note instead. Both pipeline and merge_request triggers apply per-workflow **trigger rules** filtering. Each workflow may define an ordered `trigger_rules` array; the first matching rule decides whether the event is allowed or denied (implicit deny on fallthrough). Rules can match on pipeline `status`, `user_regex`/`user_regexes`, `branch_regex`/`branch_regexes`, and `title_regex`/`title_regexes`. All conditions within a rule are ANDed; multiple patterns within a regex field are ORed. The `!` prefix negates a pattern. If no `trigger_rules` are specified, `merge_request` triggers allow all events and `pipeline` triggers default to allowing only `failed` status (preserving backward compatibility). The legacy `ignore_users` and `ignore_branches` fields are still accepted and automatically desugared into equivalent `trigger_rules` (deny + catch-all allow). They cannot be mixed with explicit `trigger_rules`. Rate limiting is per-workflow: each workflow's thread count is tracked independently via JSON session markers that embed the workflow name and commit SHA. The `max_runs_per_mr` limit applies separately to each workflow on a given MR. Events flow through a two-stage SQS pipeline. A slim ingress router forwards webhooks from the standard queue to an SQS FIFO queue, grouped by `discussion_id` for note events (ensuring same-discussion ordering) and by SQS `MessageId` for pipeline/MR events (no serialization needed). Phase 1 handlers validate the event and resolve the session, then post a continuation message back to the FIFO grouped by `session_id`. Phase 2 picks up the continuation and runs the workflow. This ensures all work for a given session is serialized even across multiple discussion threads. The SQS infrastructure is defined in `template.yaml` (SAM/CloudFormation): a standard ingress queue (60s visibility timeout for the router hop) and a FIFO work queue (30-minute visibility timeout for workflow execution). ## Model Configuration The agent supports multiple LLM providers via a unified adapter interface. ### Gemini - **API key (local dev)** -- Set `GOOGLE_API_KEY`. Calls `generativelanguage.googleapis.com` directly. No region configuration needed. - **Vertex AI (production)** -- Set `GOOGLE_CLOUD_PROJECT` and configure `model_regions` in the YAML settings. Uses Application Default Credentials (ADC) via `google-auth`. For OpenShift, mount a service account key JSON file and set `GOOGLE_APPLICATION_CREDENTIALS`, or use workload identity. Gemini uses implicit server-side caching -- repeated prefixes are automatically cached by the Vertex AI backend with no opt-in required. Cached input tokens are billed at 25% of the base input rate. The agent tracks cached token counts from API responses for cost estimation. ### Anthropic Claude Claude models are accessed via Vertex AI using the same GCP project (`GOOGLE_CLOUD_PROJECT`) with the region resolved from `model_regions`. Enable the desired model in the GCP Model Garden. Use `model: claude-sonnet-4-20250514` in workflow config. Prompt caching is enabled automatically for Claude models. The agent places explicit cache breakpoints on messages so that the full conversation prefix is served from cache on every turn after the first. Cache writes cost 1.25x the base input rate; cache reads cost 0.10x (90% discount). For a typical 20-iteration agent run, this reduces input token costs by approximately 80%. Ephemeral messages (iteration warnings, nudges) are excluded from cache writes to avoid polluting the cache with transient content. ### Region configuration GCP regions are configured via `settings.model_regions` in the config file -- a map of model name prefixes to GCP regions. The agent resolves the region by longest-prefix match on the effective model name (same algorithm as cost estimation). Example: ```yaml settings: model_regions: gemini-2.5-pro: us-east5 gemini-3.1-pro: global claude: global ``` At least one of `GOOGLE_API_KEY` or `GOOGLE_CLOUD_PROJECT` must be set. If both are set, the API key takes precedence for Gemini models. Claude models always require Vertex AI mode (`GOOGLE_CLOUD_PROJECT` + `model_regions`); `GOOGLE_API_KEY` direct mode is for Gemini only. ## Prerequisites - Python 3.11+ - Podman (for local sandbox) or `kubectl` (for K8s sandbox) - Authentication: `GOOGLE_API_KEY` or `GOOGLE_CLOUD_PROJECT` (see above) - For Claude models: enable the desired model in GCP Model Garden - GitLab tokens referenced in the config file (model tool tokens, orchestrator tokens) - Kubeconfig with access to the Konflux cluster (if using Konflux data sources) ## Installation ```bash cd hummingbird-agent pip install -e . ``` ## Usage ### Local development (`run`) The `run` command uses the config file for workflow lookup, data source registration, and token resolution -- the same code path as `serve`. By default, results are printed to stdout (dry-run). Use `--execute` to post the result as a GitLab MR note. There are two ways to select what to run: **Direct workflow selection** (`--workflow` + `--project` + `--event`): ```bash # Run a specific workflow on an MR, Podman sandbox, print to stdout hummingbird-agent run \ --workflow analyze-failures \ --project org/group/project \ --event '{"iid": 123, "sha": "abc123"}' # Same but with K8s sandbox hummingbird-agent run \ --workflow analyze-failures \ --project org/group/project \ --event '{"iid": 123, "sha": "abc123"}' \ --context my-cluster/my-namespace # Post the result as a GitLab MR note (also saves session to S3 if configured) hummingbird-agent run \ --workflow analyze-failures \ --project org/group/project \ --event '{"iid": 123, "sha": "abc123"}' \ --execute # Save session locally for debugging (context.json, transcript.md, sandbox.tar.gz) hummingbird-agent run \ --workflow analyze-failures \ --project org/group/project \ --event '{"iid": 123, "sha": "abc123"}' \ --save-session /tmp/my-session # Resume from a saved session with a follow-up question hummingbird-agent run \ --workflow analyze-failures \ --project org/group/project \ --event '{"iid": 123, "sha": "abc123"}' \ --resume-session /tmp/my-session \ --message "Can you look at the clair-scan timeout more closely?" # Chain: resume and save the new session for another round hummingbird-agent run \ --event-file event.json \ --resume-session /tmp/my-session \ --message "What layer hash failed?" \ --save-session /tmp/my-session-2 ``` **Event-file replay** (`--event-file`): routes the event through the same project-index lookup as `serve`, but skips rate limiting and status filtering: ```bash # Replay a real webhook event, dry-run with Podman hummingbird-agent run \ --event-file event.json # Replay with K8s sandbox and post notes hummingbird-agent run \ --event-file event.json \ --context my-cluster/my-namespace \ --execute ``` ### Production (`serve`) ```bash # Poll SQS queue for events (pool sandbox by default, posts results as MR notes) CONFIG_PATH=config.yml hummingbird-agent serve ``` Requires `CONFIG_PATH` pointing to a config file with `settings.sqs_queue_url` and `settings.sqs_fifo_queue_url` set. The config file defines which workflows run on which projects, with per-project limits and data source token mappings. Handles SIGTERM/SIGINT for graceful shutdown. A background router thread forwards events from the standard queue to the FIFO; the main thread consumes the FIFO with a semaphore-gated thread pool (controlled by `settings.max_concurrent_agents`) so excess messages stay in SQS for other instances. **Config hot-reload**: In serve mode a background thread polls the config file for changes (every 5 seconds by default). When the file changes, the new config is validated and atomically swapped in -- subsequent event dispatches use the updated config. If the new config is invalid, the previous config is kept and a warning is logged. No restart required for config changes. The `serve` command also accepts `--sandbox`, `--context`, and `--namespace` for local development with a different sandbox backend (e.g. Podman). ### CLI Options **`run` subcommand:** | Option | Description | Default | | --- | --- | --- | | `--event` | Inline event JSON string (mutually exclusive with `--event-file`) | - | | `--event-file` | Read event from a JSON file (mutually exclusive with `--event`) | - | | `--workflow` | Workflow name from config file (requires `--project`) | - | | `--project` | GitLab project path (requires `--workflow`) | - | | `--execute` | Post result as GitLab MR note and save session to S3 | - | | `--save-session` | Save session artifacts to this directory | - | | `--resume-session` | Resume from a saved session directory (requires `--message`) | - | | `--message` | Follow-up message for session resumption (requires `--resume-session`) | - | | `--sandbox` | Sandbox backend (`podman`, `k8s`, `k8spool`, `kubevirt`, or `kubevirtpool`) | `podman` | | `--context` | K8s context (implies K8s backend) | - | | `--namespace` | K8s namespace | - | | `-v`, `--verbose` | Enable debug logging | - | **`serve` subcommand:** | Option | Description | Default | | --- | --- | --- | | `--sandbox` | Sandbox backend (`podman`, `k8s`, `k8spool`, `kubevirt`, or `kubevirtpool`) | `k8spool` | | `--context` | K8s context (implies K8s backend) | - | | `--namespace` | K8s namespace | - | | `-v`, `--verbose` | Enable debug logging | - | ## Configuration ### Config file Both `run` and `serve` use a single YAML config file. Set the path via `CONFIG_PATH` (default: `config.example.yaml`). The config has two sections: `settings` for operational parameters, and `workflows` for workflow definitions. The `settings` section provides defaults that can be omitted for local development (sensible defaults are used). ```yaml settings: gitlab_url: https://gitlab.com # GitLab instance URL sandbox: # sandbox pod configuration image: quay.io/.../gitlab-ci:latest # container image (k8s mode only) namespace: default # K8s namespace (required for K8s/pool/kubevirt) active_deadline_seconds: 1800 # pod hard timeout / reap interval linger_seconds: 300 # keep pod alive after success (pool mode, 0=disable) max_lingering_pods: 2 # max idle lingering pods before eviction (pool mode) vm_image: quay.io/.../vm-disk:latest # containerDisk image (kubevirt mode only) vm_memory: 1Gi # VM guest memory (kubevirt mode only) vm_ssh_private_key: /path/to/key # SSH private key for pool VMs (kubevirtpool mode) metadata: # pod/VMI metadata (k8s/kubevirt mode) labels: app.kubernetes.io/name: hummingbird-agent-sandbox resources: # K8s resource requests/limits (k8s mode) requests: cpu: "100m" memory: "256Mi" ephemeral-storage: "256Mi" limits: cpu: "1" memory: "1Gi" ephemeral-storage: "2Gi" max_concurrent_agents: 4 # max concurrent workflows (serve) sqs_queue_url: "" # standard SQS ingress queue URL (serve) sqs_fifo_queue_url: "" # FIFO work queue URL (serve) s3_session_bucket: "" # S3 bucket for session persistence model: gemini-3.1-pro-preview # or claude-sonnet-4-20250514 model_regions: # model prefix -> GCP region gemini-2.5-pro: us-east5 gemini-3.1-pro: global claude: global max_iterations: 30 # default iteration limit max_runs_per_mr: 5 # default per-MR rate limit internal_notes: true # default note visibility docs_url: https://gitlab.com/org/group/project/-/blob/main/docs/agent.md source_url: https://gitlab.com/org/group/project slack_url: https://slack.example.com/archives/C0123456789 slack_label: "#my-channel" workflows: code-review: trigger: merge_request # auto-trigger on MR events description: Performs AI-powered code review sandbox: k8spool # per-workflow sandbox override (optional) workflow_url: https://gitlab.com/org/group/project/-/blob/main/workflows/code-review.md trigger_rules: # ordered rule chain, first match wins - user_regex: "renovate\\[bot\\]" # deny bot MRs (regex fullmatch) action: deny - branch_regex: "chore/.*" # deny maintenance branches action: deny - action: allow # allow everything else prompt: workflows/code-review.md action: post_gitlab_note model: gemini-3.1-pro-preview # or claude-sonnet-4-20250514 max_iterations: 15 max_inline_size: 200000 # keep full diffs in context context_limit: 500000 # Gemini 3.1 Pro / Claude Sonnet 4 have large context windows data_sources: gitlab: token_env: GITLAB_TOKEN_RO projects: org/group/project: {} analyze-failures: trigger: pipeline # auto-trigger on failed pipelines description: Investigates CI/CD pipeline failures sandbox: kubevirtpool # use VM sandbox for heavier workloads auto_resolve_on_push: true # resolve threads when a new SHA is pushed auto_resolve_on_success: true # resolve threads when pipeline succeeds workflow_url: https://gitlab.com/org/group/project/-/blob/main/workflows/analyze-failures.md prompt: workflows/analyze-failures.md # relative to config file dir action: post_gitlab_note model: gemini-3.1-pro-preview # or claude-sonnet-4-20250514 max_iterations: 50 # per-workflow iteration override data_sources: # model tool tokens (read-only) gitlab: token_env: GITLAB_TOKEN_RO # env var name, not the token konflux: cluster_url: https://example.com:6443/ns/my-tenant kubeconfig_env: KUBECONFIG kubearchive_url: https://kubearchive-api-server-product-kubearchive.apps.example.com testing_farm: {} projects: redhat/hummingbird/containers: tokens: # per-project model token overrides gitlab: GITLAB_TOKEN_CONTAINERS_RO action_tokens: # per-workflow write tokens gitlab: HUMMINGBIRD_AGENT_ACTION_ANALYZE_FAILURES_GITLAB_TOKEN_CONTAINERS ``` With `--workflow`/`--project`, the workflow and project are looked up directly in the config. With `--event-file`, the project is extracted from the event body and matched against the project index to find applicable workflows. ### Discussion threads All workflow results are posted as discussion threads: a placeholder note starts the discussion and the full result is posted as a reply. The placeholder is never edited, so email notifications include the actual result text. For slash commands, the result is posted as a reply in the triggering discussion. ### Auto-resolve Workflows can opt in to automatic resolution of their discussion threads: - **`auto_resolve_on_push`** (default `false`): When a new commit is pushed to the MR (i.e. a `merge_request` event with `action: update`), all agent discussion threads for the workflow whose SHA differs from the new HEAD are resolved. This clears stale failure analyses when the developer pushes a fix. - **`auto_resolve_on_success`** (default `false`): When the head pipeline succeeds, all agent discussion threads for the workflow on that MR are resolved (regardless of SHA). This handles pipeline reruns on the same SHA where a transient failure is now green. Both flags are independent and can be combined. Resolution runs before rate limit checks, so threads are resolved even if the workflow's per-MR run limit has been reached. ### Web search Workflows can enable Anthropic's built-in web search server tool by listing `web_search` as a data source: ```yaml workflows: renovate-babysit: model: claude-sonnet-4-20250514 data_sources: gitlab: {} web_search: {} # ... ``` When `web_search` is present in `data_sources`, the `web_search_20250305` server tool is included in API requests to Claude. The model decides autonomously when to search. Search execution happens server-side (no client-side tool dispatch), and results appear as `server_tool_use` / `web_search_tool_result` content blocks in the response. These blocks are preserved through session save/resume. Unlike other data sources, `web_search` has no configuration options and does not register any orchestrator-side tools -- it is handled entirely by the model provider. If the API returns a `pause_turn` stop reason (server-side search loop hit its iteration limit), the adapter automatically re-sends the conversation to continue, up to 5 continuations per `generate()` call. Web search is only supported with Claude models. The setting is ignored for Gemini. ### Placeholder footer The first agent-authored note (placeholder) includes a footer with links to documentation, source code, the Slack channel, the workflow prompt, and a continuation prompt. These links are configured via global settings: | Setting | Description | | --- | --- | | `settings.docs_url` | Link to agent documentation | | `settings.source_url` | Link to agent source repository | | `settings.slack_url` | Link to support Slack channel | | `settings.slack_label` | Display text for Slack link (default: "Slack") | Per-workflow, set `workflow_url` to link to the workflow's prompt file. The footer stays on the placeholder and the result is a separate reply. ### Token separation Tokens are split into three tiers that never mix: - **Model tool tokens** (in YAML `data_sources` / `tokens`): read-only tokens passed to the LLM's tool calls. Declared in the config file as env var names. These are user-defined and resolved at runtime from the referenced env vars. Create as project access tokens with **Reporter role** and `read_api` scope. Reporter is the minimum role required to see internal (confidential) notes in the discussions tool. - **Workflow action tokens** (in YAML `action_tokens`, per-project): write tokens used by the orchestrator for per-workflow GitLab writes (notes, thread resolution). Each workflow gets a dedicated bot user per project, providing clear audit trails for which agent produced each note. Declared in the project config as env var names. Create as project access tokens with **Developer role** and `api` scope. These tokens are never exposed to the LLM and never enter the ToolRegistry. Naming convention: `HUMMINGBIRD_AGENT_ACTION__GITLAB_TOKEN_`. - **Orchestrator tokens** (`ORCHESTRATOR_*` env vars, NOT in YAML): tokens used by the runner for operational reads (member access checks, push author lookup, head pipeline queries) and infrastructure notes (access-denied replies, rate-limit notices). Create as project access tokens with **Developer role** and `read_api` scope (or `api` for infrastructure notes). Developer role is required because the discussions tool trust-filters notes by author access level (>= Developer); if the orchestrator bot has only Reporter access, its own notes are redacted. Resolved by convention: `ORCHESTRATOR_GITLAB_TOKEN_` (per-project) or `ORCHESTRATOR_GITLAB_TOKEN` (fallback). The `ORCHESTRATOR_` prefix makes these impossible to confuse with model tokens. ### Environment Variables The agent reads only secrets and authentication from environment variables. All operational settings come from the config file's `settings` section. | Variable | Required | Default | Description | | --- | --- | --- | --- | | `CONFIG_PATH` | no | `config.example.yaml` | Path to config YAML | | `GOOGLE_API_KEY` | yes* | - | Gemini API key; Gemini direct mode only (not for Claude) | | `GOOGLE_CLOUD_PROJECT` | yes* | - | GCP project ID (Vertex AI mode); required for Claude models | | `ORCHESTRATOR_GITLAB_TOKEN` | serve | - | Orchestrator GitLab token (global fallback) | | `ORCHESTRATOR_GITLAB_TOKEN_` | no | - | Per-project orchestrator token | | `SENTRY_DSN` | no | - | Sentry DSN for error tracking | *One of `GOOGLE_API_KEY` or `GOOGLE_CLOUD_PROJECT` is required. Claude models require Vertex AI (`GOOGLE_CLOUD_PROJECT`); `GOOGLE_API_KEY` is for Gemini direct mode only. Model tool tokens (e.g. `GITLAB_TOKEN_RO`, `GITLAB_TOKEN_CONTAINERS_RO`) and data source credentials (e.g. `KONFLUX_CLUSTER_URL`, `KUBECONFIG`) are referenced by name in the config file's `data_sources` and `tokens` sections. They are not listed in the table above because their names are user-defined. ## Security and Design Constraints The agent is designed to run in a shared OpenShift cluster without cluster-admin access, processing potentially untrusted merge requests. These constraints shaped the architecture: **Sandbox isolation.** All arbitrary commands executed by the LLM run inside an ephemeral container, never on the host: - **Podman (local)**: `--network=none`, `--user 65532`, no host mounts. Complete network isolation. - **Kubernetes (production)**: Pods comply with OpenShift's `restricted-v2` Security Context Constraint: `runAsNonRoot`, `seccompProfile: RuntimeDefault`, `allowPrivilegeEscalation: false`, `capabilities.drop: ["ALL"]`, `automountServiceAccountToken: false` (no K8s API access from sandbox), `activeDeadlineSeconds` (configurable, default 1800). Security context fields are hardcoded in the pod manifest for portability to vanilla Kubernetes with Pod Security Admission (restricted level). Resource requests/limits, metadata, and `activeDeadlineSeconds` are configurable via `settings.sandbox` in the config file. Network access is denied by a `NetworkPolicy` on the sandbox namespace that blocks all egress from all pods (`podSelector: {}`). **No cluster-admin required.** The agent operates with namespace-scoped permissions only. The orchestrator's ServiceAccount needs only: - `pods: create, get, list, delete, patch` -- sandbox pod lifecycle and pool claims - `pods/exec: create` -- command execution via `kubectl exec` - `virtualmachineinstances.kubevirt.io: create, get, list, delete, patch` -- KubeVirt VMI lifecycle (kubevirt/kubevirtpool modes only) - `secrets: create, get, delete` -- SSH key Secrets for VMI provisioning (kubevirt/kubevirtpool modes only) These permissions are granted via a Role in the sandbox namespace, not the orchestrator's own namespace. No CRDs, no custom runtimes, no cluster-scoped resources. Konflux data is fetched via bearer token from kubeconfig, not from inside the cluster. **Namespace separation.** Sandbox pods are created in a dedicated namespace, separate from the orchestrator. This limits blast radius: even if a sandbox pod is compromised, it has no visibility into the orchestrator's Secrets, Pods, or ServiceAccount tokens. The sandbox namespace is locked down with standard K8s resources: - **RBAC**: Role + RoleBinding scoped to the namespace, granting only the permissions above to the orchestrator's ServiceAccount - **NetworkPolicy**: uses `podSelector: {}` to select all pods in the dedicated sandbox namespace, denying all egress (`egress: []`). The sandbox cannot reach the internet, the K8s API, or other pods. - **activeDeadlineSeconds**: sandbox pods self-terminate after the configured timeout (default 1800s / 30 minutes) even if the orchestrator crashes or is killed, preventing orphaned pods **Credential separation.** Data source credentials (GitLab tokens, kubeconfig) live in the orchestrator process only, injected via K8s Secrets. The sandbox container has no credentials, no SA token (`automountServiceAccountToken: false`), and no network access. Data flows into the sandbox via `stdin` piping through `write_file`. **Command execution via kubectl exec.** The K8s sandbox uses a hybrid approach: the Kubernetes Python client manages pod lifecycle (create, wait, delete), while `kubectl exec` handles command execution. This avoids the complexity and reliability issues of the websocket-based exec API. **KubeVirt VM isolation.** KubeVirt sandbox VMs run as `root` inside the guest OS, but the VM itself is contained by the KubeVirt hypervisor (QEMU/KVM). The VM has no access to the Kubernetes API, no ServiceAccount token, and no credentials. SSH keys are ephemeral (generated per sandbox start for direct mode, or shared per pool for pool mode) and cleaned up with the VMI. ## Sandbox Backends | | Podman (local) | K8s (direct) | K8sPool (production) | KubeVirt (direct) | KubeVirtPool | | --- | --- | --- | --- | --- | --- | | **Start** | `podman run -d --network=none` | `create_namespaced_pod` | Claim standby pod from Deployment | Create VMI + SSH Secret | Claim standby VMI from VMIRS | | **Exec** | `podman exec` | `kubectl exec` | `kubectl exec` | `ssh` | `ssh` | | **Auth** | Local Podman socket | In-cluster SA or kubeconfig | In-cluster SA or kubeconfig | In-cluster SA or kubeconfig | In-cluster SA or kubeconfig | | **Network** | None (`--network=none`) | None (deny-all `NetworkPolicy`) | None (deny-all `NetworkPolicy`) | Cluster pod network (SSH) | Cluster pod network (SSH) | | **User** | `65532` (fixed) | Namespace UID range (SCC) | Namespace UID range (SCC) | `root` (inside VM) | `root` (inside VM) | | **Cleanup** | `podman rm -f` | `delete_namespaced_pod` | `delete_namespaced_pod` | Delete VMI + Secret + temp keys | Delete VMI | All five implement the `Sandbox` protocol: `start()`, `exec()`, `write_file()`, `read_file()`, `cleanup()`, `linger()`. The pod pool backend (`k8spool`) eliminates pod startup latency by claiming pre-warmed pods from a Kubernetes Deployment. Claimed pods are detached from the ReplicaSet and the Deployment automatically creates replacements. After a successful workflow, pool pods linger for `linger_seconds` (default 300, configurable, 0 to disable) so that user replies can reuse the same pod without re-creating it or restoring from S3. The reaper runs once per workflow execution and deletes expired lingering pods. It also evicts excess lingering pods beyond `max_lingering_pods` (default 2), starting with those closest to their deadline. See the design doc (section 8.8) for details. The KubeVirt backends (`kubevirt`, `kubevirtpool`) provide full VM isolation using KubeVirt VirtualMachineInstances. The VM boots from a containerDisk image and SSH keys are injected via a Kubernetes Secret volume (the VM image's `inject-ssh-keys.service` reads from `/dev/disk/by-id/virtio-ssh-pubkeys`). Command execution uses SSH instead of `kubectl exec`. The `kubevirtpool` backend claims pre-warmed VMIs from a VirtualMachineInstanceReplicaSet, with the same claim/reap/linger semantics as the pod pool. The sandbox backend can be set per-workflow via the `sandbox:` field in the workflow config, with resolution order: workflow config > CLI `--sandbox` > default. ## Data Sources Data sources are registered as tool-calling functions. The model invokes them by name; the orchestrator executes them and returns results (or auto-spills large responses to the sandbox). ### GitLab | Tool | Description | | --- | --- | | `gitlab_get_mr_details` | MR metadata (title, author, state, SHA, labels) | | `gitlab_get_mr_unified_diff` | Complete unified diff in patch format | | `gitlab_get_mr_diff` | Per-file structured change data | | `gitlab_get_mr_commits` | List of commits in a merge request | | `gitlab_get_mr_discussions` | Discussion threads with redacted agent transcripts and trust-filtered comments | | `gitlab_get_commit_statuses` | CI/CD pipeline statuses for a commit | | `gitlab_get_file_at_ref` | Raw file content at a git ref | | `gitlab_get_repo_archive` | Repository tar.gz (binary, auto-spilled) | | `gitlab_get_job_log` | CI job trace output (ANSI codes stripped) | ### Konflux Fetches Tekton PipelineRuns and TaskRuns from both the live K8s API and Kubearchive (for completed resources), with deduplication by UID. | Tool | Description | | --- | --- | | `konflux_list_pipelineruns` | All PipelineRuns for a commit SHA | | `konflux_list_taskruns` | All TaskRuns for a commit SHA | | `konflux_list_pods` | All pods for a commit SHA | | `konflux_get_pod` | Full pod resource (spec, status, conditions, container statuses) | | `konflux_get_pod_log` | Pod logs; optional `container` param, fetches all containers when omitted | Response metadata includes `konflux_ui` base URL for building reviewer-facing links. ### Testing Farm | Tool | Description | | --- | --- | | `tf_get_results` | JUnit XML results for a request ID | | `tf_get_test_log` | Individual test log by URL (restricted to Testing Farm artifact URLs) | | `tf_get_request_status` | Request state, queue/run times | Response metadata includes `artifacts_base` URL for building artifact links. ## Workflow System Workflows are `.md` files whose content becomes the LLM system prompt verbatim. Workflow metadata (`action`, `model`, `max_iterations`, enabled data sources, project allowlists) is defined in the config file. The `.md` file is pure system prompt text. Available workflows: - **`analyze-failures.md`** - Investigates CI/CD pipeline failures by fetching MR details, identifying failed pipelines via commit statuses, retrieving PipelineRuns/TaskRuns from Konflux, analyzing test results from Testing Farm, and producing a grouped root-cause report with reviewer-facing URLs. - **`code-review.md`** - Performs AI-powered code review by fetching MR details, the unified diff, and prior discussion threads in parallel, then attempting to load per-project rules from `workflows/repo-rules/` (a mandatory step -- rules take precedence over generic standards when present) and producing structured feedback with severity ratings, code examples, and actionable suggestions. On follow-up reviews (after SHA updates), the agent sees its own previous findings, developer responses, and resolved threads -- avoiding duplicate findings and respecting developer explanations. Uses elevated `max_inline_size` (200 KB) and `context_limit` (500K tokens) to keep the full diff in context. - **`renovate-babysit.md`** - Triages a single Renovate-authored MR triggered by a successful pipeline. Classifies the MR as safe or risky based on diff scope, upstream dependency changes (from MR description, web search), and local codebase impact (via `gitlab_get_repo_archive` + `grep`). Posts a structured note with verdict, upstream change summary, and suggested actions for risky MRs. Requires a Claude model with `web_search` data source. ## Token Budget Management Both Gemini and Claude benefit from prompt caching that reduces the effective cost of full history replay. Cached token counts from both providers feed into the `estimate_cost()` calculation. See [Agent Model Loop -- Prompt caching](hummingbird-agent-model-loop.md#prompt-caching) for details on how caching works per provider. The agent uses a dual-limit approach instead of a cumulative token budget: - **Iteration limit** (`settings.max_iterations`, default 30, overridable per-workflow) - Hard cap on tool-calling rounds. A wrap-up prompt is injected at 80% (`SOFT_ITERATION_RATIO`). - **Context limit** (`CONTEXT_LIMIT`, default 60,000 tokens, overridable per-workflow via `context_limit`) - Per-call input token ceiling. When exceeded, a wrap-up prompt forces the model to finalize. Large outputs are automatically redirected to sandbox files to keep the LLM context small. The spill threshold defaults to 4 KB (`MAX_INLINE_SIZE`) but can be overridden per-workflow via `max_inline_size` in the config: - **`sandbox_exec`** - stdout/stderr exceeding the threshold saved to `/tmp/_out/{N}.txt`; model receives a preview (head + tail) with file path - **Data sources** - text exceeding the threshold saved to `/tmp/_out/{name}_{N}.txt` with preview; binary data saved to `.bin` - **`fetch_to_sandbox`** - always writes to the caller-specified path; returns metadata only All tuning constants are centralized in `config.py`: | Constant | Default | Purpose | | --- | --- | --- | | `DEFAULT_MAX_ITERATIONS` | 30 | Hard iteration cap | | `SOFT_ITERATION_RATIO` | 0.8 | Inject wrap-up at this fraction | | `CONTEXT_LIMIT` | 60,000 | Per-call input token ceiling | | `OUTPUT_PREVIEW_BYTES` | 4,096 | Preview size for spilled outputs | | `OUTPUT_TAIL_BYTES` | 512 | Extra tail appended to previews | | `MAX_INLINE_SIZE` | 4,096 | Max inline size for data source responses | ## Development See the main [README][readme] for development workflows. ```bash make hummingbird-agent/setup # Install dependencies make check # Lint code (ruff) make fmt # Format code make test # Run unit tests make coverage # Run tests with coverage ``` ## License This project is licensed under the GNU General Public License v3.0 or later - see the [LICENSE][license] file for details. [readme]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/README.md [license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE -------------------------------------------------------------------------------- # Hummingbird Agent Model Loop url: https://hummingbird-project.io/docs/background/tools/hummingbird-agent-model-loop/ How the Hummingbird agent communicates with the LLM (Gemini or Claude), calls tools, and how user replies integrate into the conversation. For the architectural design rationale behind the loop (why full history replay, why iteration-based budgets, why nudge via system prompt), see [Agent Design -- Section 6](hummingbird-agent-design.md#6-agent-loop-design). For operational usage documentation, see [Hummingbird Agent](hummingbird-agent.md). ## What gets sent to the model Every call to the model API sends three things: ### 1. systemInstruction A single text string, rebuilt every iteration. Gemini sends it as `systemInstruction.parts[].text`; Anthropic sends it as a top-level `system` string. Composed of layers: ```text BASE_SYSTEM_PROMPT # agent.py: sandbox rules, tool usage tips + tool_notes # per-data-source notes from ToolRegistry + workflow_prompt # full content of e.g. workflows/analyze-failures.md ``` Near the iteration or context limit, a warning suffix is appended to the system prompt for that call. The suffix escalates through three levels: - `ITERATION_WARNING` / `CONTEXT_WARNING` -- soft "start wrapping up" at 80% of the iteration or context limit. Tools remain available. - `FINAL_TURN_WARNING` -- hard stop on the absolute last iteration or after the context limit is exceeded. Combined with tool-calling disabled for that request (`toolConfig.functionCallingConfig.mode: NONE` on Gemini, `tool_choice.type: none` on Anthropic) to mechanically prevent further tool calls. - `EMPTY_RESPONSE_NUDGE` -- appended when the model returns an empty response (no text, no tool calls). Tools remain available. ### 2. contents A list of message dicts -- the **full conversation history**. Grows every iteration. Each entry has a `role` (`user`, `model`, or `tool`) and `parts`. On Anthropic, tool results use `role: "user"` (not a separate role); the adapter merges consecutive user messages so the wire shape matches what each API expects. ```json { "contents": [ {"role": "user", "parts": [{"text": "{\"project\":\"org/repo\",\"iid\":42,...}"}]}, {"role": "model", "parts": [{"functionCall": {"name": "gitlab_get_mr_details", "args": {...}}}]}, {"role": "tool", "parts": [{"functionResponse": {"name": "gitlab_get_mr_details", "response": {...}}}]}, ... ] } ``` ### 3. tools Tool definitions, static across all iterations. Gemini uses `functionDeclarations` (JSON Schema in `parametersJsonSchema`); Anthropic uses `input_schema` per tool. ```json { "tools": [{"functionDeclarations": [ {"name": "sandbox_exec", "description": "...", "parametersJsonSchema": {...}}, {"name": "fetch_to_sandbox", ...}, {"name": "fetch_batch_to_sandbox", ...}, {"name": "gitlab_get_mr_details", ...}, {"name": "konflux_list_pipelineruns", ...} ]}] } ``` ### 4. toolConfig (conditional) On the final turn (last iteration or after context limit exceeded), the request disables tools to mechanically prevent function calls. Gemini: ```json { "toolConfig": {"functionCallingConfig": {"mode": "NONE"}} } ``` Anthropic equivalent: `tool_choice: {"type": "none"}`. This is only sent when `FINAL_TURN_WARNING` is active. All other iterations omit this constraint, allowing the model to choose freely between text and tools. ## The agent loop, turn by turn The loop in `run_agent_loop` runs up to `max_iterations` times. Each iteration is one round-trip to the model. ### Initialization ```text system_prompt = BASE_SYSTEM_PROMPT + tool_notes + workflow_prompt tool_defs = [sandbox_exec, fetch_to_sandbox, fetch_batch_to_sandbox, ] contents = [user: {"project": "org/repo", "iid": 42, "sha": "abc...", "session_id": "uuid"}] ``` ### Iteration 1 ```text -> Send: system_instruction + contents (1 item) + tool_defs <- Response: functionCall(gitlab_get_mr_details, {project: "org/repo", iid: 42}) contents.append(model: response.raw_content) execute tool -> result = {"result": "{\"title\": \"Fix auth\",...}"} contents.append(tool: model.make_tool_responses([(name, result)])) ``` ### Iteration 2 ```text -> Send: system_instruction + contents (3 items) + tool_defs <- Response: functionCall(fetch_batch_to_sandbox, {requests: [...]}) contents.append(model: ...) execute tool -> result = {"results": [{"saved_to": "/tmp/data/pipelineruns.json", "bytes": 85432}]} contents.append(tool: ...) ``` ### Iteration 3 ```text -> Send: system_instruction + contents (5 items) + tool_defs <- Response: functionCall(sandbox_exec, {command: "jq '[...]' /tmp/data/pipelineruns.json"}) contents.append(model: ...) execute tool -> result = {"exit_code": 0, "stdout": "...(preview)...", "stdout_file": "/tmp/_out/0.txt"} contents.append(tool: ...) ``` ### Iteration N (final) ```text -> Send: system_instruction + contents (2N-1 items) + tool_defs <- Response: text("## Konflux Failure Analysis\n\n...") // text + no tool_calls = DONE contents.append(model: {text: "## Konflux Failure Analysis..."}) BREAK -- return (text, usage, transcript, contents) ``` ## Response handling and termination Each iteration, the model's response can contain text, tool calls, both, or neither. The agent handles each case: | Response | Action | | --- | --- | | **Text only** | Final response. Save text, break. (Happy path.) | | **Text + tool calls** | Save text as `last_text` fallback, execute the tool calls, continue loop. The model is thinking aloud while also acting. | | **Tool calls only** | Execute the tool calls, continue loop. | | **Neither** | Empty response -- retry up to `MAX_EMPTY_RETRIES` (2) times. Each retry nudges the model via `EMPTY_RESPONSE_NUDGE` appended to the system prompt (not injected as a user message, to keep the contents list clean). Tools remain available during nudge retries. | The text + tool calls case is worth noting: the model's text is **not** returned to the user immediately. It is stored as `last_text` (a fallback in case the loop terminates later without a clean final text, e.g. by hitting max iterations). The `raw_content` dict -- which includes both the text and functionCall parts -- is appended to `contents` as a single model turn: ```json {"role": "model", "parts": [ {"text": "Let me check the clair-scan logs..."}, {"functionCall": {"name": "sandbox_exec", "args": {"command": "grep timeout ..."}}} ]} ``` The loop terminates on: 1. **Text only** -- the model produced a final response. 2. **Max iterations reached** -- the final iteration uses `FINAL_TURN_WARNING` + `toolConfig NONE` to force text output. Falls back to whatever `last_text` was seen, or `"[Agent did not produce a final response]"`. 3. **Context limit exceeded** -- `input_tokens >= CONTEXT_LIMIT` sets `context_exceeded`, making the next iteration final (same as #2). A soft warning (`CONTEXT_WARNING`) fires earlier at 80% of the limit. 4. **Fatal model error** -- HTTP error, timeout, or connection error after retries exhausted. Transport errors (timeouts, connection failures) are wrapped as retryable `ModelError` and retried with exponential backoff alongside HTTP 5xx/429. 5. **Empty responses exhausted** -- `MAX_EMPTY_RETRIES` (2) nudge retries failed. 6. **Unexpected error** -- any exception not handled by the retry mechanism (e.g. malformed API response) breaks the loop and returns partial results. Accumulated contents, transcript, and sandbox are preserved and the session is saved normally. ### Budget escalation Both iterations and context size use the same two-tier pattern: | | Soft warning (80%) | Hard stop (100%) | | --- | --- | --- | | **Iterations** | `ITERATION_WARNING` at 80% of `max_iterations` | `FINAL_TURN_WARNING` on last iteration | | **Context** | `CONTEXT_WARNING` at 80% of `CONTEXT_LIMIT` | `FINAL_TURN_WARNING` on next iteration after exceeding `CONTEXT_LIMIT` | Soft warnings are advisory ("start wrapping up") and keep tools available. The hard stop uses `toolConfig.mode: NONE` to mechanically prevent further tool calls, ensuring the model produces text. ## The contents list in detail Each entry in `contents` follows the provider's native wire format during execution. ### Canonical format (persistence) During the agent loop, `contents` use the provider's native format. On save, `to_canonical()` converts them to OpenAI Chat Completions-style messages. On load, `from_canonical()` converts back to the current provider's native format. That enables cross-provider session resumption. Anthropic-specific blocks are stored as opaque pass-through fields on the canonical assistant message: `thinking_blocks` for thinking/redacted_thinking, `server_tool_blocks` for server_tool_use/web_search_tool_result. These are restored in correct position order (thinking first, then text/tool_use, then server tool blocks) by `from_canonical()`. ### User turn ```json {"role": "user", "parts": [{"text": "..."}]} ``` Created by `model.make_user_content(text)`. In a cold start there is exactly one user turn at the start: the JSON event. No additional user turns appear during a normal run -- empty-response nudges are delivered via the system prompt, not as user messages. ### Model turn ```json {"role": "model", "parts": [ {"functionCall": {"name": "sandbox_exec", "args": {"command": "jq ..."}}} ]} ``` Or for the final response: ```json {"role": "model", "parts": [{"text": "## Konflux Failure Analysis..."}]} ``` This is `response.raw_content` -- the exact dict from the model response candidate, appended verbatim. Can contain text, tool calls, or both. Parallel tool calls appear as multiple `functionCall` parts in one model turn. ### Tool turn ```json {"role": "tool", "parts": [ {"functionResponse": {"name": "sandbox_exec", "response": {"exit_code": 0, "stdout": "..."}}} ]} ``` Created by `model.make_tool_responses(results)`. One `functionResponse` part per tool call in the preceding model turn. Tool results are always JSON dicts; large outputs are spilled to sandbox files and only a preview is included. ### Typical 8-iteration conversation shape ```text contents[0] = user: {"project":"org/repo","iid":42,...} # initial event contents[1] = model: functionCall(gitlab_get_mr_details) # iter 1 contents[2] = tool: functionResponse(gitlab_get_mr_details) # iter 1 contents[3] = model: functionCall(fetch_batch_to_sandbox) # iter 2 contents[4] = tool: functionResponse(fetch_batch_to_sandbox) # iter 2 contents[5] = model: functionCall(sandbox_exec) # iter 3 contents[6] = tool: functionResponse(sandbox_exec) # iter 3 ... contents[13] = model: functionCall(sandbox_exec) # iter 7 contents[14] = tool: functionResponse(sandbox_exec) # iter 7 contents[15] = model: text("## Konflux Failure Analysis...") # iter 8 (final) ``` This entire list is persisted as `context.json` in the session (see **Canonical format (persistence)** above). It is the full state needed to resume a conversation. ## How tools are called When the model's response contains `functionCall` parts, `_execute_tool_calls` iterates them sequentially: ```text tool_registry.execute(ToolCall(name, args)) match name: "sandbox_exec" -> runs sh -c in container -> {exit_code, stdout, stderr} "fetch_to_sandbox" -> calls data source func -> writes to sandbox file -> {saved_to, bytes} "fetch_batch_to_sandbox" -> multiple fetch_to_sandbox in one call any data source name -> calls func directly -> returns inline or auto-spills large output ``` ### Large output handling When stdout or a data source response exceeds 4KB, it is automatically saved to a sandbox file (`/tmp/_out/N.txt`) and only a preview (head + tail) is returned to the model. The model gets the file path and can use `sandbox_exec` with `jq`/`grep`/`head` to process it. This keeps the context window manageable. ## How a user reply integrates (session resumption) When a user replies to an agent note on GitLab, the orchestrator resumes the conversation by restoring the previous state and appending the reply. ### What gets restored from S3 - **`context.json`** -- the persisted conversation in **canonical** (OpenAI Chat Completions-style) form; `from_canonical()` maps it to the active provider's native `contents` before the loop runs - **`sandbox.tar.gz`** -- `/tmp/_out/` files (PipelineRun JSONs, test logs, jq output, etc.) restored into the new sandbox Because load uses `from_canonical()`, you can switch models across providers (e.g. Gemini to Claude or the reverse) when resuming, as long as the session was saved with canonical contents. ### The resumed contents list ```text // Restored from context.json (previous run) contents[0] = user: {"project":"org/repo","iid":42,...} # original event contents[1] = model: functionCall(gitlab_get_mr_details) # iter 1 contents[2] = tool: functionResponse(gitlab_get_mr_details) # iter 1 ... # all prior turns contents[15] = model: text("## Konflux Failure Analysis...") # previous final text // NEW: user reply appended contents[16] = user: "Can you look at the clair-scan timeout more closely?" ``` ### The agent loop continues ```text ITERATION 1 (resumed): -> Send: system_instruction (+ CONTINUATION_PROMPT) + contents[0..16] + tool_defs <- Response: functionCall(sandbox_exec, {command: "grep -i timeout /tmp/data/..."}) ^ using restored sandbox files contents[17] = model: functionCall(sandbox_exec) contents[18] = tool: functionResponse(sandbox_exec) ITERATION 2 (resumed): <- Response: text("The clair-scan timeout is caused by...") contents[19] = model: text("The clair-scan timeout is caused by...") BREAK ``` ### Cold start vs resumed -- key differences | Aspect | Cold start | Resumed | | --- | --- | --- | | System prompt | BASE + tool_notes + workflow | BASE + tool_notes + workflow + **CONTINUATION_PROMPT** | | First user message | `{"project":..., "iid":...}` | `{"project":..., "iid":...}` (restored) | | Conversation history | Empty (just the event) | Full prior conversation | | Latest user message | -- | `"Can you look at the clair-scan timeout?"` | | Sandbox files | Empty | Restored from archive | | Tool definitions | Same | Same | | Session format | Native (per provider) in memory | Canonical on disk; native after `from_canonical()` | ### The CONTINUATION_PROMPT Without this, the workflow prompt (e.g. `analyze-failures.md`) tells the model to follow a rigid workflow: Data.1, Data.2, Data.3, Analysis.1... The model might try to re-run the entire analysis. The continuation prompt overrides this: ```text ## Continuation This is a follow-up to a previous conversation. The conversation history contains your prior analysis and tool calls. The user is replying with a question or request about your previous work. **Do NOT re-run the full workflow from scratch.** Instead: - Respond directly to the user's question - Use your tools to investigate further if needed (logs, data are still available in the sandbox) - Reference your previous findings where relevant - Keep your response focused on what the user asked ``` The model now understands: "I already did the analysis (it's all in the conversation history). The user has a specific question. Let me answer it." ## Iteration and token budget for resumed sessions The resumed session gets a **full fresh iteration budget**. `run_workflow` sets `max_iterations` from the workflow config (e.g. 50 for analyze-failures), and the loop counter starts at 0 regardless of how many iterations the previous session used. The model can make as many tool calls as it needs to answer the follow-up. The practical constraint is the **context window**, not iterations. The restored contents can be large -- a typical 8-iteration cold start uses ~50-100K input tokens. Since the model re-counts the full history every iteration, the resumed session starts at roughly the token count where the previous session ended, plus the new user message. Each new tool call adds more. Token counts logged by the agent are **billed** tokens (cumulative across API calls; each call re-sends the full history, so these overlap). The existing `CONTEXT_LIMIT` check still applies: at 80% of the limit, a soft `CONTEXT_WARNING` tells the model to start wrapping up. If the limit is exceeded, the next iteration becomes final (`FINAL_TURN_WARNING` + `toolConfig NONE`). For the typical case (user asks one focused follow-up, agent does 1-3 tool calls), this is fine. Multi-turn deep dives will eventually hit the context limit, at which point the agent wraps up -- and the user can start a fresh conversation if needed. ## Prompt caching Since the agent replays the full conversation history on every API call, prompt caching significantly reduces the cost of repeated prefixes. Both providers support caching, but with different mechanisms. ### Gemini (implicit) Gemini's Vertex AI backend automatically caches repeated request prefixes server-side. No request-side annotations are needed. The agent reads `cachedContentTokenCount` from `usageMetadata` in each response and reports it as `cache_read_tokens`. There is no `cache_creation_tokens` for Gemini -- implicit caching has no write surcharge, only a read discount (25% of the base input rate). ### Claude (explicit breakpoints) Vertex AI does not support Anthropic's automatic caching. The agent places explicit `cache_control: {"type": "ephemeral"}` annotations on message content blocks. The API hashes the cumulative prefix -- tools, system prompt, and all messages from the start of the request up to the annotated block -- and caches the result. A breakpoint on a message therefore covers everything before it; separate breakpoints on the system prompt or tools would be redundant. On the wire, an annotated message looks like this: ```json { "role": "user", "content": [ {"type": "text", "text": "Describe the sandbox.", "cache_control": {"type": "ephemeral"}} ] } ``` For messages with multiple content blocks (e.g. `tool_result` arrays), the `cache_control` is placed on the last block in the array. #### Sliding-window breakpoints The agent uses two of the four available breakpoint slots per request. A sliding pair of breakpoints (B1, B2) ensures the full conversation prefix is cached and only the newest turn is processed at full price: - **B2** is placed on the last message (writes the current prefix to cache) - **B1** is placed where B2 was on the previous call (reads the prior prefix from cache) ```text Call 1: [msg0 B2] B2: WRITE entire prefix to cache Call 2: [msg0 B1] [msg1] [msg2 B2] B1: READ (matches call 1's B2 -- same position, same prefix) B2: WRITE (extends cache to include new messages) Call 3: [msg0] [msg1] [msg2 B1] [msg3] [msg4 B2] B1: READ (matches call 2's B2) B2: WRITE Call N: ... [msg(N-2) B1] [msg(N-1)] [msg(N) B2] B1: READ (matches call N-1's B2) B2: WRITE ``` The `cache_control` metadata is not part of the prefix hash. Moving B1 to a position that previously had B2 (and no longer has `cache_control`) does not invalidate the cache -- the hash matches because the content is identical. The cache has a 5-minute TTL, refreshed on each hit. Cache writes cost 1.25x the base input rate (25% surcharge); cache reads cost 0.10x (90% discount). The minimum cacheable prefix length is 1024 tokens for Sonnet/Opus and 4096 tokens for Haiku -- the system prompt alone exceeds these thresholds. #### Ephemeral turns When the agent loop injects a transient message (iteration warning, context warning, empty-response nudge), the `generate()` call receives `ephemeral=True`. The transient message is appended as a user message, merged with the preceding tool-result user message by `_merge_consecutive_user`, and popped from `contents` after the call. - **B1 is still placed** at the previous B2 position, so the cache read for the stable prefix still works - **B2 is placed on the second-to-last** merged message (the last stable message before the merged ephemeral tail), so the cached prefix advances without including transient content - The internal B2 position advances to the second-to-last index, so successive ephemeral turns keep sliding B1 forward This ensures the B1=previous-B2 invariant holds through ephemeral turns: ```text T1: U1·B2 prev=0 T2: U1·B1 M1 U2·B2 prev=2 T3(E): U1 M1 U2·B1 M2·B2 U3+E prev=3 T4(E): U1 M1 U2 M2·B1 U3 M3·B2 U4+E prev=5 T5: U1 M1 U2 M2 U3 M3·B1 U4 M4 U5·B2 prev=8 ``` (U = user message, M = model message, E = ephemeral, +E = merged with preceding user message.) When the merged message list has fewer than two entries during an ephemeral call (e.g. a single merged user+ephemeral on the first turn), no breakpoints are placed and the B2 position is not updated. #### Thinking blocks Extended thinking blocks in assistant responses are part of the cached prefix. They do not break cache hits when the following user message contains only `tool_result` blocks (the normal case in the agent loop). #### Server tool blocks (web search) When `web_search` is listed in a workflow's data sources, the Anthropic adapter adds the `web_search_20250305` server tool to the request. The API executes searches server-side and returns `server_tool_use` and `web_search_tool_result` content blocks in the assistant response alongside regular text/tool_use blocks. These blocks are: - **Transparent to the agent loop** -- they don't generate `ToolCall` entries, so the agent loop doesn't attempt to dispatch them. - **Preserved in raw_content** -- the full content block array (including server tool blocks) is stored in `raw_content` and persisted via `to_canonical()`. - **Restored on session resume** -- `from_canonical()` reconstructs them as opaque pass-through blocks, similar to thinking blocks. If the API returns `pause_turn` (server-side search loop hit its iteration limit), the adapter automatically re-sends the partial response to continue, merging content blocks and accumulating usage tokens across continuations. ### Caching and session resumption After session resumption, the model instance is fresh and has no record of previous breakpoint positions. The first call has no B1 (no cache read), only B2 (cache write). From call 2 onward, caching works normally. This is the same behavior as a cold start. -------------------------------------------------------------------------------- # Red Hat Catalog url: https://hummingbird-project.io/docs/background/tools/redhat-catalog/ Web UI for browsing Hummingbird container images, security advisories, and API documentation. ## Features - **Image Catalog** - Browse container images with tags, architectures, and specifications - **Security Feed** - Security advisory feed with CVE details - **API Documentation** - Interactive API documentation browser - **Image Details** - SBOM, provenance, and release history per image ## Architecture React 18 + [PatternFly v6][pf] SPA. Data fetching via [TanStack Query][tsq] against the [container-catalog](container-catalog.md) API (proxied through webpack-dev-server in development, direct in production). Webpack 5 build. Hosted on CloudFront + S3 (production) with MR previews on GitLab Pages. [pf]: https://www.patternfly.org/ [tsq]: https://tanstack.com/query/latest ## Prerequisites - Node.js 22+ (for host targets) - `make` + `podman` (for container targets) ## Usage Only `make` and `podman` are required for container targets. Host variants (`*-host`) run without podman. | Target | Description | | --------------------------- | ------------------------------------- | | `redhat-catalog/setup` | Install dependencies (container) | | `redhat-catalog/check` | Type-check, lint, and test (container)| | `redhat-catalog/build` | Production build (container) | | `redhat-catalog/dev` | Dev server on port 9000 (container) | | `redhat-catalog/setup-host` | Install dependencies (host) | | `redhat-catalog/check-host` | Type-check, lint, and test (host) | | `redhat-catalog/build-host` | Production build (host) | | `redhat-catalog/dev-host` | Dev server (host) | Set `ASSET_PATH` to control the base path for non-root deployments (used by webpack `output.publicPath` and React Router `basename`). ## Configuration ### Build Variables | Variable | Description | | ------------ | -------------------------------------------------- | | `ASSET_PATH` | Webpack public path / React Router basename | API base URLs are configured in [`app-config.ts`][app-config]. [app-config]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/redhat-catalog/src/app/app-config.ts ### SAM Parameters | Parameter | Description | | ------------------- | ------------------------------------------------------------ | | `ResourcePrefix` | Prefix for all resource names | | `CatalogDomainName` | Custom domain (optional, leave empty for CloudFront default) | ## Deployment Production deployment uses a CloudFront + S3 stack defined in [`template.yaml`][template]. The infrastructure post-deploy script builds the SPA in a Node.js container, syncs it to S3, and creates a CloudFront cache invalidation. Custom domain support is optional via the `CatalogDomainName` SAM parameter. When set, an ACM certificate is created in-template with DNS validation -- the validation CNAME must already exist in the external DNS zone before deployment. [template]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/redhat-catalog/template.yaml MR previews are deployed to GitLab Pages with `path_prefix` per MR, auto-cleaned on merge/close with a 1-week expiry. ## Environments and CI promotion Experimental, staging, and production URLs, MR vs `main` pipelines, deploy triggers, and UAT sign-off are documented in [Red Hat Catalog Environment Promotion](redhat-catalog-environments.md) (monorepo `documentation/`) and [Red Hat Catalog UAT Program](redhat-catalog-uat.md). ## Development Detailed contributor documentation lives in-tree: - [Developer Guide][dev-guide] -- component patterns, state management, data fetching - [Testing Guide][test-guide] -- Vitest + RTL patterns and commands - [UAT Checklist][uat] -- human acceptance testing before prod promotion (HUM-2179) - [Architecture][arch] -- system design, data flow, React Query patterns - [API Responses][api-resp] -- observed API response shapes - [Settings Architecture][settings] -- settings/theme system [dev-guide]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/redhat-catalog/docs/developer-guide.md [test-guide]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/redhat-catalog/docs/testing-guide.md [uat]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/redhat-catalog/docs/uat-checklist.md [arch]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/redhat-catalog/docs/architecture.md [api-resp]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/redhat-catalog/docs/api-actual-responses.md [settings]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/redhat-catalog/docs/global-settings-architecture.md ## License This project is licensed under the GNU General Public License v3.0 or later - see the [LICENSE][license] file for details. [license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE -------------------------------------------------------------------------------- # Hummingbird Agent Design url: https://hummingbird-project.io/docs/background/tools/hummingbird-agent-design/ Content not included (137958 characters, exceeds 50000 limit). Fetch the full page: https://hummingbird-project.io/docs/background/tools/hummingbird-agent-design/index.md -------------------------------------------------------------------------------- # Hummingbird Data Flow url: https://hummingbird-project.io/docs/background/tools/data-flow/ Overview of all data sources, what is stored where, and how data moves between systems. --- ## Data sources | Source | What it provides | | ------ | ---------------- | | **Jira REST API** | CVE tracker ticket fields (status, timestamps, labels, custom fields) | | **Jira whiteboard** (`customfield_10841`) | Per-ticket timestamp cache written back by the analysis tool | | **Red Hat Pulp** (`packages.redhat.com`) | SRPM/RPM publish timestamps for Hummingbird repos | | **Hummingbird container catalog API** | Image rebuild history (when a package first appeared in a rebuilt image) | | **Red Hat CSAF VEX feed** (`security.access.redhat.com`) | Advisory fix/not-affected status per CVE | | **OSIDB** (`osidb.prodsec.redhat.com`) | Subpackage-level affectedness data | | **NVD / CVE list** | CVE publish dates, product/version data | | **GitHub / GitLab** | Upstream fix commit / PR / release dates | | **Fedora updates** | Fedora update availability timestamps | | **Konflux SNS/SQS events** | Pipeline run, snapshot, release, MR, push events | --- ## PostgreSQL databases There are two Postgres databases. ### 1. `hummingbird-status` — Konflux pipeline events Fed by SNS/SQS events via the hummingbird-status ingestor. | Table | Primary key | What is stored | | ----- | ----------- | -------------- | | `gitlab_pushes` | `(sha, repo, ref)` | Git push events — sha, branch, commit list | | `components` | `name` | Konflux component definitions and git context | | `pipelineruns` | `name` | Build/test PLR outcomes, status, start/completion times | | `snapshots` | `name` | Multi-component snapshots, source PLR, sha | | `releases` | `name` | Release outcomes, images, LLM analysis text | | `gitlab_merge_requests` | `(project, iid)` | Current MR state, merge commit sha | | `gitlab_mr_versions` | `(project, iid, sha)` | Per-commit history of each MR head | ### 2. `hummingbird-dashboard` — CVE lifecycle + dashboard overlays | Table | Primary key / unique | What is stored | | ----- | -------------------- | -------------- | | `cve_ticket_events` | `(ticket_key, event_type)` | **One row per lifecycle milestone per ticket.** `occurred_at` = timestamp; `metadata` = JSONB (see below). The source of all R-Time computations. | | `cve_analysis_log` | `id` | Each analysis tool run: timestamp, ticket count, log output | | `cve_ticket_claims` | `ticket_key` | Claim/lock for deduplicating concurrent analysis runs | | `dashboard_settings` | `key` | Runtime flags: auto-rerun enabled, analysis enabled, etc. | | `auto_rerun_log` | `id` | History of automatic retrigger attempts | | `blocked_error_patterns` | `id` | Regex patterns that suppress auto-rerun | | `blocked_snapshots` | `snapshot_name` | Manually blocked snapshots | | `analysis_log` / `analysis_costs` | `id` | LLM failure analysis runs and token costs | | `blocked_push_builds` / `push_build_rerun_log` | `id` | Push build blocking and retry history | | `package_lifecycle` | `(package, event_type)` | Package-scoped lifecycle milestones (e.g. `rpm_first_published`). Distinct from `cve_ticket_events` — one row per package/milestone, not per ticket. | #### `cve_ticket_events` lifecycle milestone `event_type` values These are the canonical names after the HUM-5918 migration: | `event_type` | Meaning | | ------------ | ------- | | `cve_published` | CVE published date (NVD / CVE list) | | `hum_ticket_created` | HUM Jira ticket creation date | | `hum_ticket_closed` | HUM Jira ticket resolution date | | `upstream_fix_merged` | Upstream fix commit / release merged | | `fedora_update_available` | Fedora update containing the fix became available | | `rpm_fix_published_to_pulp` | Fix RPM published to Hummingbird Pulp repo | | `image_rebuilt_on_quay` | Hummingbird container image rebuilt with the fix | | `vex_resolved` | Red Hat CSAF VEX advisory confirmed the resolution (HUM-5843) | `rpm_fix_published_to_pulp` and `image_rebuilt_on_quay` are **mutable** — they are updated when a newer delivery event supersedes the previous one. All other event types are **write-once**: once set, `occurred_at` is never overwritten. #### `cve_ticket_events.metadata` JSONB fields Each row carries a `metadata` blob that reflects the ticket's state at the time of the most recent analysis run: | Field | Source | Description | | ----- | ------ | ----------- | | `computed_resolution` | analysis tool | Hummingbird's computed fix status | | `jira_status` | Jira `status` field | Current Jira issue status | | `labels` | Jira `labels` field | All labels on the ticket | | `fixed_in_build` | Jira `customfield_10578` | SRPM set manually in the "Fixed in Build" field | | `detected_fixed_build` | Pulp / SRPM detection | SRPM filename auto-detected from Pulp repodata | | `vex_status` | Red Hat VEX feed | `fixed` / `known_not_affected` / … | | `vex_match_state` | reconciliation logic | `matched` / `pending` / `mismatch` | | `vex_resolved` | VEX reconciliation | Timestamp when VEX first agreed with Jira resolution | | `catalog_image_source` | collector catalog map | `true` when the package is a catalog image SBOM source (R-Time delivery is image publish); `false` means RPM publish. Missing on old rows: dashboard still requires image | --- ## Jira whiteboard (`customfield_10841`) The CVE analysis tool uses the Jira whiteboard field as a **per-ticket timestamp cache**. This is being phased out in favour of Postgres as the durable store (HUM-5917), but is still the source for a subset of fields. The whiteboard holds compact JSON. The relevant sub-key is `cve_cycle`: | `cve_cycle` key | Meaning | Mutable? | | --------------- | ------- | -------- | | `cve_published` | CVE publish date | No — write-once | | `jira_created` | HUM ticket creation date | No — write-once | | `jira_closed` | HUM ticket close date | No — disabled (255-char limit) | | `upstream_fix` | Upstream fix timestamp | No — write-once | | `fedora_fix` | Fedora update timestamp | No — write-once | | `hb_rpm_fix` | RPM published to Pulp | **Yes** — re-evaluated each run | | `hb_image_fix` | Image rebuilt on Quay | **Yes** — re-evaluated each run | The whiteboard is read and written only by the **CVE analysis tool**. The dashboard never reads it directly; its authoritative source is always `cve_ticket_events`. --- ## Jira fields read by the CVE analysis tool On each run the analysis tool fetches every open HUM CVE tracker ticket and reads the following fields from the Jira REST API: | Jira field / custom field | Purpose | | ------------------------- | ------- | | `summary` | Ticket title / package detection | | `status`, `resolution`, `resolutiondate` | Ticket lifecycle state | | `created` | Jira ticket creation date → `hum_ticket_created` | | `description`, `comment` | CVE ID extraction, fix evidence | | `labels` | `cve-next-release`, fix-in-progress, etc. | | `security` | Embargo level | | `assignee` | Ticket owner | | `customfield_10578` (Fixed in Build) | Manually set SRPM identifying the fix build | | `customfield_10841` (Whiteboard) | Cached `cve_cycle` timestamps (read + write) | | `customfield_10667` (CVE ID) | Structured CVE identifiers | | `customfield_10860` (Embargo Status) | Embargo flag | | `customfield_10020` (Sprint) | Sprint membership | | `customfield_10014` (Epic Link) | Parent epic | --- ## Data flow ```text ┌─────────────────────────────────────────────────────────────────┐ │ External data sources │ │ │ │ Jira REST API ─────────────────────────────────────┐ │ │ NVD / CVE list ────────────────────────────────────┤ │ │ GitHub / GitLab (upstream fix commits) ────────────┤ │ │ Fedora updates ────────────────────────────────────┤ │ │ Red Hat Pulp (RPM publish times) ─────────────────►│ │ │ Hummingbird catalog API (image rebuild times) ─────┤ │ │ Red Hat CSAF VEX feed ─────────────────────────────┤ │ │ OSIDB (subpackage affectedness) ───────────────────┘ │ └──────────────────────────────┬──────────────────────────────────┘ │ ▼ ┌──────────────────────────────────────────────────────────────────┐ │ CVE analysis tool (cron, hummingbird-cve-analysis) │ │ │ │ • Reads Jira ticket fields + whiteboard cache │ │ • Resolves fix SRPM from Pulp repodata │ │ • Resolves image rebuild timestamp from catalog API │ │ • Checks upstream PR/release dates (GitHub/GitLab) │ │ • Fetches CVE publish date (NVD/CVE list) │ │ • Reconciles VEX status (CSAF feed) │ │ • Checks subpackage affectedness (OSIDB) │ │ │ │ Writes back to: │ │ ├── Jira whiteboard (cve_cycle timestamp cache) │ │ ├── Jira labels / Fixed-in-Build field │ │ ├── Jira comments │ │ └── dashboard DB → cve_ticket_events (one row per milestone) │ │ cve_analysis_log (run record) │ └──────────────────────────────┬───────────────────────────────────┘ │ ┌───────────────────┴───────────────────┐ │ │ ▼ ▼ ┌──────────────────────────┐ ┌───────────────────────────────┐ │ Jira whiteboard │ │ dashboard DB │ │ (per-ticket JSON cache) │ │ cve_ticket_events table │ │ cve_cycle timestamps │ │ one row per (ticket, │ │ ← being phased out │ │ event_type) milestone │ │ (HUM-5917) │ └───────────────┬───────────────┘ └──────────────────────────┘ │ ▼ ┌──────────────────────────────────┐ │ hummingbird-dashboard API │ │ │ │ Pivots cve_ticket_events rows │ │ into per-ticket dicts, computes:│ │ • entry-level R-Time fields │ │ (HUM-5920) │ │ • duration legs inside stages │ │ (HUM-5921) │ │ │ │ Serves JSON + Jinja templates │ └──────────────────────────────────┘ ┌─────────────────────────────────────────────────────────┐ │ Konflux SNS/SQS event stream │ │ │ │ GitLab pushes, PipelineRuns, Snapshots, Releases, MRs │ │ → hummingbird-status ingestor │ │ → hummingbird-status DB (pipeline/release tables) │ │ → hummingbird-dashboard API (build/release views) │ └─────────────────────────────────────────────────────────┘ ``` --- ## API field names The dashboard API exposes these computed fields per R-Time entry: ### Entry-level fields | API field | Meaning | | --------- | ------- | | `cve_published_at` | Start of R-Time: earlier of CVE published and HUM created (Include CVE-HUM), or HUM created (Exclude) | | `fix_delivered_at` | Done timestamp: latest of delivery and Jira close (Exclude VEX), plus VEX when Include VEX is on; unset until the required gates exist. Delivery is image publish for catalog image sources, RPM publish otherwise | | `cve_to_delivery_hours` | Completed R-Time only (`cve_published_at` → `fix_delivered_at`); `None` while still accumulating | | `cve_to_hum_created_hours` | Filing lag after NVD: `max(HUM created − NVD, 0)`; `0.0` when HUM is first | | `advisory_to_vex_hours` | Completed ADV-VEX only (`hum_ticket_closed` → `vex_resolved`); `None` until VEX exists | | `display_advisory_to_vex_hours` | ADV-VEX hours, or elapsed-to-now when the VEX feed has not updated yet | | `fix_before_cve_published` | True when the delivery timestamp predates notification (informational) | | `hum_ticket_open` | True until the selected done gates exist | | `deferred_to_next_release` | True for `cve-next-release` tickets | | `display_duration_hours` | R-Time hours, or elapsed-to-now when the ticket is not yet done | ### Duration legs (`stages` dict) | API field | Interval | | --------- | -------- | | `cve_published_to_hum_created` | CVE publish → HUM ticket opened | | `cve_published_to_upstream_fix` | CVE publish → upstream fix merged | | `hum_created_to_upstream_fix` | HUM ticket → upstream fix merged | | `upstream_fix_to_fedora_update` | Upstream fix → Fedora update available | | `fedora_update_to_rpm_fix_published` | Fedora update → fix RPM in Pulp | | `upstream_fix_to_rpm_fix_published` | Upstream fix → fix RPM in Pulp | | `hum_created_to_rpm_fix_published` | HUM ticket → fix RPM in Pulp | | `rpm_fix_published_to_image_rebuilt` | Fix RPM in Pulp → image rebuilt on Quay | | `image_rebuilt_to_hum_closed` | Image rebuild → HUM ticket closed | | `hum_closed_to_vex_resolved` | HUM ticket closed (advisory MR merge) → VEX feed update | ### Aggregate stats | API field | Meaning | | --------- | ------- | | `avg_hum_created_to_rpm_fix_hours` | Mean HUM ticket → RPM fix (skips missing legs) | | `avg_rpm_fix_to_image_rebuilt_hours` | Mean RPM publish → image rebuild | | `avg_advisory_to_vex_hours` | Mean Done-Errata close → VEX feed (completed ADV-VEX only) | --- ## Package lifecycle data `rpm_first_published` is the earliest timestamp at which any SRPM for a given package appeared in the Hummingbird Pulp repo. It is **package-scoped** — one row per package — as opposed to `cve_ticket_events` which is ticket-scoped. ### Collection ```bash scripts/collect_rpm_first_published --prod --rpms-repo /path/to/rpms ``` Flow: 1. Loads the package list from `rpms_repo.load_package_map_from_metadata()`. 2. For each package: calls `pulp.fetch_earliest_srpm_time()`, which browses `packages.redhat.com/.../source/Packages/{letter}/`, parses the HTML listing already used by the analysis tool, and returns the earliest upload timestamp across all SRPMs for that package. 3. POSTs all results in one request to `/api/cve-import` (same endpoint used for `cve_ticket_events` sync). The dashboard upserts with `COALESCE(LEAST(existing, incoming), existing, incoming)` — re-runs only update `occurred_at` if the incoming timestamp is earlier, and NULLs are never stored over a real value. ### Surfaced in `dump_lifecycle` ```bash scripts/dump_lifecycle --prod HUM-1234 ``` After fetching ticket milestones from `cve_ticket_events`, the script resolves the package name from those rows and makes a second call to `/api/cve-export?section=package_lifecycle` to fetch `rpm_first_published` for that package. It is shown at the bottom of the text output and under `rpm_first_published` in the JSON output. -------------------------------------------------------------------------------- # Kubernetes Event Forwarder url: https://hummingbird-project.io/docs/background/tools/kubernetes-event-forwarder/ A Kubernetes deployment that watches resource changes (ADDED/MODIFIED/DELETED) across multiple clusters and forwards them to an SNS topic with structured metadata for filtering. Uses kubeconfig contexts as the source of truth for which clusters and namespaces to watch. The full Kubernetes object JSON is forwarded as the SNS message body, compressed with gzip and base64-encoded. ## Features - **Multi-Cluster Support**: Watch resources across multiple Kubernetes clusters using kubeconfig contexts - **Dynamic Resource Watching**: Configure any namespaced resource type using standard Kubernetes `apiVersion` and `kind` - **SNS Integration**: Publish events with structured message attributes for precise filtering - **Compression**: Events are compressed (gzip+base64) to reduce SNS message size - **Automatic Reconnection**: Handles watch connection failures and reconnects automatically ## Architecture ### Threading Model The forwarder uses a multi-threaded architecture: - **Watcher threads** (one per context + resource type): Each runs an independent LIST+Watch loop. Isolation ensures one slow/failing cluster doesn't affect others. - **Publisher thread** (single, shared): Reads events from a queue and publishes to SNS. Decouples K8s API interaction from SNS latency (~150ms per publish), preventing watch loop stalls that could cause resourceVersion staleness. **Memory optimization:** When `SPOOL_DIR` is set, messages are written to temporary files instead of being held in memory. This reduces memory pressure during event bursts (e.g., LIST operations or many resources created at once) and enables recovery of unsent messages after restarts (e.g., after OOM kills). ### LIST + Watch Pattern The forwarder uses explicit LIST followed by Watch rather than `resource_version="0"`: 1. **LIST** retrieves all current objects and a snapshot `resourceVersion` 2. **Watch** starts from the LIST's `resourceVersion` (not object RVs) 3. On graceful watch timeout (300s), restart watch from last known RV (no re-list) 4. On errors, fresh LIST to ensure current state This is necessary because synthetic ADDED events from `resource_version="0"` have unsorted, potentially-stale resourceVersions (they reflect when objects were last modified). If the watch disconnects mid-stream, the tracked RV could be arbitrarily old and may already be compacted by etcd, causing a 410 error cascade. ## Prerequisites - **Target cluster**: ServiceAccount with watch permissions on the resources you want to monitor - **Deployment cluster**: Kubernetes cluster to run the forwarder - **AWS credentials**: SNS publish permissions (optional - logs events if not configured) ## Deployment The container image is built via Konflux CI/CD and published to `quay.io/hummingbird-ci/kubernetes-event-forwarder:latest`. Example Kubernetes manifests are provided in the [`kubernetes/`][k8s-dir] directory: - `rbac.yaml` - ServiceAccount, Role, RoleBinding for the **target** cluster - `secret.yaml` - Kubeconfig and AWS credentials - `configmap.yaml` - Resource watch configuration - `deployment.yaml` - Forwarder deployment ### Quick Start 1. On the **target** cluster (the one you want to watch), apply RBAC and create a token: ```bash kubectl apply -f kubernetes/rbac.yaml kubectl create token kubernetes-event-forwarder --duration=8760h ``` 2. Update the example manifests with your values: - `secret.yaml`: cluster URL, token, AWS credentials - `configmap.yaml`: resources to watch - `deployment.yaml`: SNS topic ARN, AWS region 3. Apply the manifests to your deployment cluster: ```bash kubectl apply -f kubernetes/secret.yaml kubectl apply -f kubernetes/configmap.yaml kubectl apply -f kubernetes/deployment.yaml ``` **Prerequisites:** Deploy [hummingbird-events-topic][het-docs] first to create the SNS topic, then deploy the AWS resources (see below). [k8s-dir]: https://gitlab.com/redhat/hummingbird/tools/-/tree/main/kubernetes-event-forwarder/kubernetes [het-docs]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/documentation/hummingbird-events-topic.md ### AWS Resources The SAM template (`template.yaml`) provisions IAM resources for SNS publishing: - **IAM User** (`${ResourcePrefix}-user`) - Service account for the forwarder - **IAM Policy** (`${ResourcePrefix}-policy`) - Grants `sns:Publish` to the SNS topic Deploy using containerized AWS SAM CLI: ```bash cd kubernetes-event-forwarder sam build sam deploy --guided # First deployment (interactive) sam deploy # Subsequent deployments ``` After deployment, create access keys for the IAM user and store them securely. ### SAM Parameters | Parameter | Description | Default | |------------------|--------------------------|--------------| | `ResourcePrefix` | Prefix for resources | `myapp-prod` | | `SnsTopicArn` | SNS topic ARN | (required) | **Resource naming:** IAM resources follow `{ResourcePrefix}-{type}` pattern (e.g., `myapp-prod-user`, `myapp-prod-policy`). ## Usage Configure resources to watch in `config.yaml`: ```yaml resources: - apiVersion: v1 kind: Pod - apiVersion: apps/v1 kind: Deployment - apiVersion: v1 kind: ConfigMap ``` **SNS Subscription Filter Examples:** Pod events in a specific namespace: ```json { "source": ["kubernetes"], "kind": ["Pod"], "namespace": ["production"] } ``` All deployment changes: ```json { "source": ["kubernetes"], "kind": ["Deployment"] } ``` Deleted resources across all clusters: ```json { "source": ["kubernetes"], "event_type": ["DELETED"] } ``` ## Development See the main [README][readme] for development workflows. ```bash make kubernetes-event-forwarder/setup # Install dependencies make check # Lint code (ruff) make fmt # Format code make test # Run unit tests make coverage # Run tests with coverage ``` ## Configuration The deployment is configured via environment variables: | Variable | Description | |---------------------------|-------------------------------------------------------| | `CONFIG_PATH` | Path to config YAML file | | `CONFIG` | Inline config YAML (alternative) | | `SNS_TOPIC_ARN` | SNS topic ARN (optional - logs if unset) | | `SPOOL_DIR` | Optional spool directory for file-based message queue | | `AWS_ACCESS_KEY_ID` | AWS access key ID | | `AWS_SECRET_ACCESS_KEY` | AWS secret access key | | `AWS_DEFAULT_REGION` | AWS region | | `METRICS_PORT` | Prometheus metrics port (default: `9090`) | | `SENTRY_DSN` | Optional Sentry DSN | ## Event Metadata The forwarder extracts metadata from Kubernetes events and adds them as SNS message attributes: | Attribute | Description | Example | |--------------------|-----------------------|--------------------------------| | `source` | Always `"kubernetes"` | `kubernetes` | | `cluster` | API server host | `https://api.cluster:6443` | | `namespace` | Object namespace | `production` | | `api_version` | Resource API version | `v1`, `apps/v1` | | `kind` | Resource kind | `Pod`, `Deployment` | | `event_type` | Event type | `ADDED`, `MODIFIED`, `DELETED` | | `object_name` | Name of the object | `nginx-7d8c4c9d6f` | | `content_encoding` | Message encoding | `gzip+base64` | ## Security & Limitations **Security:** - AWS credentials stored in Kubernetes Secret - Kubeconfig credentials stored in Kubernetes Secret - SNS topic follows least privilege principle (publish-only) - Sentry integration for error tracking **Limitations:** - Only namespaced resources supported - One thread per context + resource type combination - SNS message size limit: 256 KB (after compression) ## License This project is licensed under the GNU General Public License v3.0 or later - see the [LICENSE][license] file for details. [readme]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/README.md [license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE -------------------------------------------------------------------------------- # Jira Image Requests url: https://hummingbird-project.io/docs/background/tools/jira-image-requests/ AWS Lambda proxy that serves Hummingbird Image Request issues from Jira (with DynamoDB caching) and creates Stories/Epics from the catalog request form. ## Features - **Public list**: GET `/image-requests` returns Ready Image Request issues - **Form submit**: POST `/image-requests` creates a version Story. An Epic is created only when a second request uses the same image name; later requests reuse that Epic. A first-time name is Story-only. - **Customer scoring**: Writes the Customer criterion on the Epic from paid email domains (placeholder until subscriber login) - **Total scoring**: Maintained by Jira Automation from manual triage fields ## Prerequisites - AWS CLI with permissions for Lambda, API Gateway, DynamoDB, SSM, CloudFormation - Podman or Docker for SAM build - Jira bot token in SSM (SecureString) - HUM project custom fields for scoring (see below) ## Deployment ```bash make jira-image-requests/build make jira-image-requests/deploy ``` ## Configuration | Parameter / env | Description | |----------------------------|-----------------------------------------------------------------------------| | `ResourcePrefix` | Prefix for AWS resource names | | `JiraTokenParameter` | SSM parameter name for the Jira API token | | `CacheTtlSeconds` | Cache TTL for the public list (default 300) | | `CorsAllowOrigin` | Allowed CORS origin | | `ScoringCustomerField` | Jira custom field id for Customer (`customfield_…`); empty skips write | | `PaidCustomerDomains` | Comma-separated email domains scored as paid (e.g. `acme.com,contoso.com`) | | `SentryDsn` | Optional Sentry DSN | ## Image request scoring Triage scores live on the **Image Request Epic**, not on customer-facing Stories. ### Custom fields (HUM Epics) Create (or map) these fields, then record each `customfield_XXXXX` id: | Field | Type / values | Who sets it | |--------------------------|---------------------------------------|--------------------------------------| | IR Legal | Select: Y / N | Manual | | IR Competitive | Number (typically 0–1) | Manual | | IR Marketable | Number | Manual | | IR Supportability | Number | Manual | | IR Upstream Health | Number | Manual | | IR Level of Effort | Number: 3 easy, 2 medium, 1 hard | Manual | | IR Technical Feasibility | Number: 0 blocking, 1 feasible | Manual | | IR Portfolio Conflicts | Number: 0 conflict, 1 good | Manual | | IR Customer | Number: 0 or 1 | Lambda (`ScoringCustomerField`) | | IR Total | Number | Jira Automation only | **Gate:** Legal must be **Y** to continue scoring. Competitive, Marketable, and Supportability / Maturity are also expected to be **1** before a meaningful Total (team convention from the triage spreadsheet). ### Lambda: Customer criterion When a POST creates or reuses an Epic (second or later request for a name), the Lambda: 1. Loads Stories under the Epic 2. Sets Customer to **1** if any submitter email domain is in `PaidCustomerDomains`, otherwise **0** 3. Writes the value to `ScoringCustomerField` when configured A first request for a name does not create an Epic, so Customer is not written until a second request for that name arrives. When subscriber login lands, replace the domain list with real subscription status. ### Jira Automation: Total and Legal flag Configure two rules in Jira (Project settings → Automation). Use the real field names/ids from your instance. #### Rule 1 — Sum Total when Legal is Y - **Trigger:** Field value changed for any of: Competitive, Marketable, Supportability, Upstream Health, Level of Effort, Technical Feasibility, Portfolio Conflicts, Customer (and optionally Legal) - **Condition:** Issue type = Epic AND component = Image Request AND IR Legal = Y - **Action:** Edit issue → set IR Total to the sum of the numeric criteria (exclude Legal). Example smart value shape: ```text {{#=}} {{issue.IR Competitive}} + {{issue.IR Marketable}} + {{issue.IR Supportability}} + {{issue.IR Upstream Health}} + {{issue.IR Level of Effort}} + {{issue.IR Technical Feasibility}} + {{issue.IR Portfolio Conflicts}} + {{issue.IR Customer}} {{/}} ``` Use your instance’s smart-value field keys (often `customfield_XXXXX`). - **Optional action:** Remove label `ir-legal-blocked` when Legal is Y #### Rule 2 — Flag when Legal is not Y - **Trigger:** IR Legal changed - **Condition:** Issue type = Epic AND component = Image Request AND IR Legal is empty OR IR Legal = N - **Actions:** 1. Clear IR Total (or set to empty) 2. Add label `ir-legal-blocked` Jira Automation recalculates Total whenever triage fields change — no webhook or Lambda rescoring path is required. ## Development ```bash make jira-image-requests/setup make test ``` ## License GPL-3.0-or-later -------------------------------------------------------------------------------- # PAC Trigger url: https://hummingbird-project.io/docs/background/tools/pac-trigger/ A CLI tool to manually trigger Konflux PAC (Pipelines-as-Code) pipelines for a specific component and branch. Useful for debugging pipeline issues or re-running builds without pushing a new commit. ## Usage ```bash # Trigger pipeline for a component (uses branch HEAD) pac-trigger --gitlab-project-url https://gitlab.com/org/group/project \ --component myimage--default--main \ --cluster-url https://konflux-ui.apps.cluster.example.com # Trigger with specific commit pac-trigger --gitlab-project-url https://gitlab.com/org/group/project \ --component myimage--default--main \ --commit abc123def456 \ --cluster-url https://konflux-ui.apps.cluster.example.com # Dry run (preview without creating) pac-trigger --gitlab-project-url https://gitlab.com/org/group/project \ --component myimage--default--main \ --cluster-url https://konflux-ui.apps.cluster.example.com \ --dry-run ``` ### Options | Option | Description | Default | |-------------------------|------------------------------------------------|---------| | `--gitlab-project-url` | GitLab project URL (required) | - | | `--component` | Component name (required) | - | | `--branch` | Branch name | `main` | | `--commit` | Specific commit SHA | HEAD | | `--cluster-url` | Konflux cluster URL (required) | - | | `--dry-run` | Preview PipelineRun without creating | false | | `-v`, `--verbose` | Enable debug logging | false | ## Installation ```bash cd pac-trigger pip install -e . ``` ## Prerequisites - **Kubeconfig**: Context with access to the target Konflux cluster/namespace - **Repository resource**: PAC Repository must exist for the GitLab project - **Push template**: Component must have a push template in `.tekton/*.yaml` ## How It Works 1. **Fetch commit**: Gets HEAD commit SHA from GitLab (or uses provided commit) 2. **Fetch template**: Downloads `.tekton/*.yaml` files via GitLab API, finds push template matching the component 3. **Get namespace**: Extracts namespace from template metadata 4. **Find credentials**: Looks up PAC Repository resource to find git secret 5. **Create git-auth secret**: Creates ephemeral `pac-trigger-gitauth-*` secret with git credentials 6. **Create PipelineRun**: Substitutes template variables and creates the PipelineRun 7. **Link secret**: Sets ownerReference on secret for garbage collection 8. **Print URL**: Outputs Konflux UI URL for the PipelineRun ### Template Variables The following PAC template variables are supported: | Variable | Substituted with | |-----------------------|---------------------| | `{{revision}}` | Commit SHA | | `{{target_branch}}` | Branch name | | `{{repo_url}}` | GitLab project URL | | `{{git_auth_secret}}` | Created secret name | Unsupported template variables will cause an error. ### Labels Added | Label | Value | Purpose | |--------------------------|--------|-----------------------------| | `pac-trigger/manual` | `true` | Identifies manual triggers | ## Features - **Template-based**: Fetches PipelineRun templates from `.tekton/*.yaml` files via anonymous GitLab API - **Push-only**: Only triggers push templates (filters by CEL expression) - **Automatic credentials**: Creates ephemeral git-auth secrets from existing PAC Repository secrets - **Garbage collection**: Secrets are linked to PipelineRun via ownerReference - **Kubeconfig-based**: Uses local kubeconfig for cluster authentication ## Development See the main [README][readme] for development workflows. ```bash make pac-trigger/setup # Install dependencies make check # Lint code (ruff) make fmt # Format code make test # Run unit tests make coverage # Run tests with coverage ``` ## Comparison with PAC-triggered Runs Manually triggered PipelineRuns differ from PAC-triggered ones: **Present in both:** - `appstudio.openshift.io/application` - `appstudio.openshift.io/component` - `pipelines.appstudio.openshift.io/type: build` - `build.appstudio.redhat.com/commit_sha` - `build.appstudio.redhat.com/target_branch` **Unique to pac-trigger:** - `pac-trigger/manual: true` label **Missing (PAC internal metadata):** - `pipelinesascode.tekton.dev/event-type` - `pipelinesascode.tekton.dev/sha` - GitLab status reporting annotations These differences don't affect pipeline execution—they're used for PAC's internal tracking and GitLab commit status updates. ## Limitations - Only push templates supported (not pull-request) - Only GitLab repositories supported - Requires existing PAC Repository resource - No GitLab commit status reporting ## License This project is licensed under the GNU General Public License v3.0 or later - see the [LICENSE][license] file for details. [readme]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/README.md [license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE -------------------------------------------------------------------------------- # Hummingbird Status url: https://hummingbird-project.io/docs/background/tools/hummingbird-status/ SQS worker and database management for ingesting Konflux pipeline events. ## Features - **SQS Worker** - Real-time database updates from SNS pipeline events - **GitLab Event Ingestion** - Push events and merge request events with version tracking - **Database Init** - Populate from pg_dump, S3 archive, or local filesystem mirror - **Prometheus Metrics** - Built-in metrics endpoint for monitoring - **MR Reconciliation** - Fix stale MR state by checking GitLab API - **Schema Management** - Idempotent database schema creation ## Prerequisites - Python 3.11+ - PostgreSQL 16 - AWS credentials (for SQS worker and S3 initialization) ## Installation ```bash cd hummingbird-status pip install -e . ``` ## Usage ### Local Development ```bash cd hummingbird-status # Database management ./dev.sh db-start # Start PostgreSQL container ./dev.sh db-init # Initialize from available source ./dev.sh db-shell # PostgreSQL interactive shell ./dev.sh db-dump # Create pg_dump file ./dev.sh db-stop # Stop PostgreSQL ./dev.sh db-reset # Stop and delete volume # With data source SNS_MIRROR=/path/to/mirror ./dev.sh db-init # From filesystem S3_BUCKET=bucket-name ./dev.sh db-init # From S3 # SQS worker (requires credentials) SQS_QUEUE_URL=https://... ./dev.sh worker ``` ### Container Deployment The container runs the SQS worker by default: ```bash podman build -f Containerfile -t hummingbird-status . podman run -e DATABASE_URL=... -e SQS_QUEUE_URL=... hummingbird-status ``` For database initialization: ```bash podman run -e DATABASE_URL=... -e S3_BUCKET=... \ hummingbird-status python3 -m hummingbird_status.worker.init_db ``` ### MR Reconciliation MR state is updated via GitLab webhooks. If a webhook is lost, an MR can appear as "opened" in the dashboard when it is actually merged or closed. The reconcile command checks all open MRs against the GitLab API and corrects stale entries: ```bash # Using environment variables GITLAB_TOKEN=glpat-... python -m hummingbird_status.reconcile # Using CLI arguments python -m hummingbird_status.reconcile \ --database-url postgresql://... \ --gitlab-token glpat-... ``` In a container: ```bash podman run -e DATABASE_URL=... -e GITLAB_TOKEN=... \ hummingbird-status python3 -m hummingbird_status.reconcile ``` ## Configuration ### Environment Variables | Variable | Default | Description | | ------------------- | ---------------------- | --------------------------------- | | `DATABASE_URL` | - | PostgreSQL connection URL | | `SQS_QUEUE_URL` | - | SQS queue URL (worker) | | `S3_BUCKET` | - | S3 bucket for init | | `S3_PREFIX` | `sns/` | S3 key prefix | | `LOCAL_MIRROR_PATH` | `/data/sns-mirror` | Local S3 mirror path | | `INIT_DUMP_PATH` | `/data/init/dump.sql` | pg_dump file path | | `GITLAB_URL` | `https://gitlab.com` | GitLab instance URL (reconcile) | | `GITLAB_TOKEN` | - | GitLab API token (reconcile) | | `METRICS_PORT` | `9090` | Prometheus metrics port | ### Database Initialization Priority 1. **pg_dump file** - `$INIT_DUMP_PATH` if exists 2. **Local S3 mirror** - `$LOCAL_MIRROR_PATH/sns/*.json.gz` if exists 3. **S3 bucket** - `$S3_BUCKET/$S3_PREFIX` with AWS credentials ## Database Schema The database stores Konflux pipeline events and GitLab notifications (pushes and MRs). ```mermaid flowchart TD push[["gitlab_pushes
commit sha, changed files"]] mr[["gitlab_merge_requests
current MR state"]] mrv[["gitlab_mr_versions
head commit history"]] comp[["components
git_context → component mapping"]] build[["pipelineruns (type=build)
one per affected component"]] snap[["snapshots
image digest, links via source_plr"]] test[["pipelineruns (type=test)
integration tests per snapshot"]] rel[["releases
publish to registry, links to snapshot"]] relplr[["pipelineruns (type=release)
executes release, linked via release_plr"]] push -- "+ affected" --> build mr -- "sha" --> mrv mrv -- "sha joins" --> build comp -- "components" --> build build -- "success
creates" --> snap snap -- "triggers" --> test test -- "success
creates" --> rel rel -- "managed
by" --> relplr ``` ### Tables | Table | Primary Key | Description | |-------------------------|-------------------|--------------------------------------------| | `gitlab_pushes` | sha,repo,ref | GitLab push events to main branch | | `gitlab_merge_requests` | project,iid | Current state of merge requests | | `gitlab_mr_versions` | project,iid,sha | Head commit history for each MR | | `components` | name | Konflux Component resources | | `pipelineruns` | name | Build, test, and release pipelines | | `snapshots` | name | Image snapshots after successful builds | | `releases` | name | Published releases to target registry | ### Merge Request Tracking The MR tables enable tracking build status across MR versions: - **`gitlab_merge_requests`** - Stores current MR metadata (title, state, branches, author, latest head SHA). Updated via `ON CONFLICT ... WHERE updated_at <` to keep the most recent state. - **`gitlab_mr_versions`** - Records each unique head commit SHA for an MR. When force-pushing the same SHA, `created_at` is updated to the latest event timestamp via `GREATEST()`, ensuring correct ordering even after force pushes. ## Development ### Running Tests ```bash cd hummingbird-status pip install -e ".[dev]" pytest ``` ### Project Structure ```text hummingbird-status/ ├── Containerfile ├── dev.sh ├── template.yaml # SAM template for AWS resources ├── hummingbird_status/ │ ├── db.py # Database schema and utilities │ ├── ingest.py # SNS event parsing and ingestion │ ├── reconcile.py # MR state reconciliation with GitLab │ └── worker/ │ ├── sqs.py # SQS consumer │ └── init_db.py # Database initializer └── tests/ ``` ## AWS Resources Deploy the SQS queue using SAM: ```bash cd hummingbird-status make build # Build SAM application make deploy # First deployment (guided) make redeploy # Subsequent deployments ``` ### Parameters | Parameter | Description | Default | | ---------------- | --------------------------------- | ------------ | | `ResourcePrefix` | Prefix for all resource names | `myapp-prod` | | `SnsTopicArn` | ARN of the SNS topic to subscribe | (required) | **Prerequisites:** Requires an existing SNS topic. Deploy [hummingbird-events-topic][het-docs] first. See the main [README][readme] for development workflows. [het-docs]: hummingbird-events-topic.md ## License This project is licensed under the GNU General Public License v3.0 or later - see the [LICENSE][license] file for details. [readme]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/README.md [license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE -------------------------------------------------------------------------------- # Lambda S3 Cache url: https://hummingbird-project.io/docs/background/tools/lambda-s3-cache/ An AWS Lambda setup that caches files from public URLs in S3. When a URL is requested, the service returns a 302 redirect to either the cached S3 copy (via presigned URL) or the original source. Cache misses trigger asynchronous downloads to S3, ensuring future requests are served from the cache. Caching uses the original URL's host and path as the S3 key - each unique URL maps to a single cache entry that persists until expiration. Frequently accessed content automatically extends its cache lifetime on each access. **Note:** Cached content behavior depends on file type: - **Immutable content** (matching immutable extension filter, e.g., `.rpm`): Changes at origin won't be reflected until cache expires - **Mutable content** (non-matching extension, e.g., `repomd.xml`): Cache is revalidated on each request via HEAD check; stale behavior is configurable **Cache Revalidation:** Mutable content uses HTTP ETags for efficient validation. The uploader stores the origin's ETag in S3 metadata when caching content. On subsequent requests, the handler sends a HEAD request with `If-None-Match` header containing the stored ETag. If the origin responds with `304 Not Modified`, the cache is fresh and served directly. If the ETag differs, the cache is stale and an async refresh is triggered; response behavior depends on `StaleCacheBehavior`. **Note:** If the origin doesn't provide ETags, the cache cannot validate freshness. In this case, mutable content always redirects to origin without triggering cache updates (caching would be ineffective since every request would redirect anyway). ## Features - **Streaming Upload**: Handles files efficiently by streaming directly from source to S3 without loading into memory - **Presigned S3 URLs**: Returns short-lived signed URLs for cached content - **Automatic Expiration**: Cached content expires after a certain time, with lifetime extended on each access - **URL Prefix Allowlist**: Only caches content matching explicitly allowed host+path prefixes - **Immutable Extension Filter**: Identifies immutable content that doesn't require revalidation - **Cache Revalidation**: Mutable content (non-matching extensions) is validated on each request; stale cache behavior is configurable (origin or cache) - **Custom Domain**: Optional custom domain with automatic TLS certificate management via ACM and Route53 ## Architecture Three Lambda functions handle the caching workflow: 1. **Handler** - API Gateway endpoint that checks cache, returns 302 redirects, and triggers async operations. Implements touch cooldown to prevent S3 throttling by only touching objects after a configurable time period has elapsed since last modification. 2. **Uploader** - Downloads from origin and streams to S3 on cache misses (invoked asynchronously) 3. **Touch** - Updates S3 object timestamps to extend cache lifetime on cache hits (invoked asynchronously only when cooldown period has elapsed) ### Request Flow ```mermaid flowchart TD Start([Request]) --> CheckURL{URL matches
AllowedPrefixes?} CheckURL -->|No| RedirectOrigin CheckURL -->|Yes| HeadS3[HEAD S3]:::network HeadS3 --> CheckCache{Cache exists?} CheckCache -->|No: Cache Miss| InvokeUploaderMiss[Invoke Uploader async]:::async CheckCache -->|Yes: Cache Hit| CheckImmutable{Immutable extension?} CheckImmutable -->|No: Mutable| HeadOrigin[HEAD Origin
If-None-Match: stored ETag]:::network HeadOrigin --> CheckChanged{Origin changed?} CheckChanged -->|No ETag from origin| RedirectOrigin CheckChanged -->|ETag differs| InvokeUploaderStale[Invoke Uploader async]:::async CheckChanged -->|ETag matches| CheckCooldown CheckChanged -->|304 Not Modified| CheckCooldown CheckChanged -->|"Error/timeout"| CheckCooldown CheckImmutable -->|Yes| CheckCooldown CheckCooldown{Touch cooldown
elapsed?} -->|Yes| InvokeTouch[Invoke Touch async]:::async InvokeUploaderMiss --> RedirectOrigin InvokeUploaderStale -->|"STALE_CACHE_BEHAVIOR=origin"| RedirectOrigin InvokeUploaderStale -->|"STALE_CACHE_BEHAVIOR=cache"| RedirectCache CheckCooldown -->|No| RedirectCache InvokeTouch --> RedirectCache subgraph redirectGroup [ ] RedirectOrigin[302 to Origin URL]:::redirect RedirectCache[302 to S3 Presigned URL]:::redirect end style redirectGroup fill:none,stroke:none classDef network fill:#10b981,color:#000 classDef async fill:#ff9900,color:#000 classDef redirect fill:#3b82f6,color:#fff ``` **Legend:** Green = network call, Orange = async Lambda invocation, Blue = 302 redirect response ## Prerequisites - AWS CLI configured with appropriate credentials (IAM permissions for Lambda, API Gateway, S3, CloudFormation, CloudWatch Logs, and optionally Route53/ACM for custom domain) - Podman or Docker (for containerized SAM build/deploy) - Python 3.11 or later (for development) ## Deployment Build and deploy using containerized AWS SAM CLI: ```bash make lambda-s3-cache/build # Build Lambda package make lambda-s3-cache/deploy # First deployment (interactive/guided) make lambda-s3-cache/redeploy # Subsequent deployments (non-interactive) ``` **Deployment output:** `ApiEndpoint` - the API Gateway URL to use for requests ### Custom Domain Optional custom domain with automatic TLS certificate management (ACM + Route53). Requires a Route53 hosted zone. Deploy with `CustomDomainName` and `HostedZoneId` parameters - CloudFormation handles certificate creation, DNS validation, and configuration. Certificate validation takes 5-30 minutes; allow up to 1 hour for DNS propagation. ### Parameters | Parameter | Description | Default | |--------------------------|--------------------------------------------------------------|---------------------| | `ResourcePrefix` | Prefix for all resource names | `myapp-prod` | | `PresignedUrlExpiration` | Presigned URL expiration (seconds) | `3600` | | `AllowedPrefixes` | Whitespace-separated list of allowed URL prefixes | `example.com/path/` | | `ImmutableExtensions` | File extensions for immutable content (no revalidation) | `.rpm` | | `CacheExpirationDays` | Days to keep cached content (minimum: 1) | `14` | | `TouchCooldownMinutes` | Minimum minutes between touch operations | `60` | | `StaleCacheBehavior` | When cache is stale: `origin` or `cache` | `origin` | | `CustomDomainName` | Optional custom domain name | `` | | `HostedZoneId` | Route53 hosted zone ID (required if custom domain) | `` | | `SentryDsn` | Optional Sentry DSN for error tracking | `` | **Resource naming:** All AWS resources follow `{ResourcePrefix}-{type}-{name}` pattern (e.g., `myapp-prod-bucket`, `myapp-prod-lambda-handler`). ## Usage Make GET requests to the API endpoint (or custom domain if configured). The URL to cache is encoded in the path (without the `https://` prefix): ```bash curl -L "https:///example.com/path/to/file.rpm" ``` **Behavior:** - **First request** (cache miss): Redirects to original URL while triggering async S3 upload - **Subsequent requests** (cache hit): - **Immutable content** (matching immutable extension filter): Redirects to presigned S3 URL and resets cache expiration - **Mutable content** (non-matching extension): HEAD request validates cache freshness (10s timeout). If stale, behavior depends on `StaleCacheBehavior`; if fresh, serves from cache - **Disallowed prefixes**: URLs not matching allowed prefixes are transparently redirected to the original URL (302 pass-through) ### Immutable Extension Filter Behavior The `ImmutableExtensions` parameter determines caching behavior: - **Matching extension** (e.g., `.rpm`): Treated as **immutable content**. Cached without revalidation - changes at origin won't be reflected until cache expires. Served directly from cache on all requests. - **Non-matching extension** (e.g., `.xml`, `.gz`): Treated as **mutable content**. Cached with revalidation - HEAD request on each cache hit validates freshness using ETags. If stale, behavior depends on `StaleCacheBehavior`: - `origin` (default): Redirects to origin for fresh content, async refresh - `cache`: Serves stale cache immediately (faster), async refresh in background **Important:** All files matching `AllowedPrefixes` are cached, regardless of extension. The extension filter only determines whether revalidation is performed. ### Usage as Repository Proxy The cache can be used as a DNF/yum `baseurl` for RPM repositories. Configure `ImmutableExtensions` to include `.rpm`: - **`.rpm` files** (matching extension): Cached as immutable - fast, no revalidation - **Metadata files** (non-matching extension, e.g., `repomd.xml`, `primary.xml.gz`): Cached with revalidation - ensures fresh metadata while benefiting from cache ```ini [myrepo] name=My Repository baseurl=https://koji-s3-cache.example.com/download.example.org/pub/repo/$basearch/ enabled=1 ``` This provides caching benefits for RPM downloads while ensuring repository metadata stays fresh. ## Development See the main [README][readme] for development workflows. ```bash make lambda-s3-cache/setup # Install dependencies make check # Lint code make fmt # Format code make test # Run unit tests make coverage # Run tests with coverage ``` ## Configuration Lambda functions receive configuration via environment variables (automatically set by CloudFormation): | Variable | Handler | Uploader | Touch | Description | |----------------------------|---------|----------|-------|----------------------------------| | `S3_BUCKET_NAME` | ✅ | ✅ | ✅ | S3 bucket name | | `PRESIGNED_URL_EXPIRATION` | ✅ | | | Presigned URL expiration (sec) | | `UPLOADER_LAMBDA_ARN` | ✅ | | | Uploader Lambda ARN | | `TOUCH_LAMBDA_ARN` | ✅ | | | Touch Lambda ARN | | `TOUCH_COOLDOWN_MINUTES` | ✅ | | | Min minutes between touch ops | | `ALLOWED_PREFIXES` | ✅ | | | Allowed URL prefixes | | `IMMUTABLE_EXTENSIONS` | ✅ | | | Extensions for immutable content | | `STALE_CACHE_BEHAVIOR` | ✅ | | | Stale behavior: origin or cache | | `SENTRY_DSN` | ✅ | ✅ | ✅ | Optional Sentry DSN | ## Security & Limitations **Security:** - S3 bucket has public access blocked; all objects encrypted at rest (AES256) - Presigned URLs expire after a certain time - IAM policies follow least privilege principle - URL prefix allowlist prevents caching arbitrary URLs - Immutable extension filter distinguishes immutable vs mutable content for revalidation **Limitations:** - Lambda timeout: 15 min (uploader), 30 sec (handler, touch) - Lambda memory: 1024 MB (uploader), 256 MB (handler, touch) - S3 object size: Up to 5 TB (AWS limit) ## License This project is licensed under the GNU General Public License v3.0 or later - see the [LICENSE][license] file for details. [readme]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/README.md [license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE -------------------------------------------------------------------------------- # RPM CVE Count url: https://hummingbird-project.io/docs/background/tools/rpm-cve-count/ A CLI tool to count the number of known CVEs for a given list of RPM packages by querying the Red Hat OSIDB database. ## Features - **Batch processing**: Query multiple packages from a file - **Impact filtering**: Filter by CVE severity (CRITICAL, IMPORTANT, MODERATE, LOW) - **Date filtering**: Count only CVEs created after a specific date - **CSV output**: Easy to import into spreadsheets or process with other tools ## Prerequisites - **Red Hat VPN**: Must be connected to access the OSIDB database - **Go 1.21+**: For building from source ## Installation Install the latest version: ```bash go install gitlab.com/redhat/hummingbird/tools/rpm-cve-count@latest ``` Or build from source: ```bash git clone https://gitlab.com/redhat/hummingbird/tools.git cd tools/rpm-cve-count go build ``` ## Usage ```bash rpm-cve-count -file [-after ] [-impact ] ``` ### Options | Option | Description | Required | |-----------|----------------------------------------------------------|----------| | `-file` | Read packages from file (one per line) | Yes | | `-after` | Count CVEs created after date (YYYY-MM-DD) | No | | `-impact` | Filter by impact: CRITICAL, IMPORTANT, MODERATE, LOW | No | ### Examples Create a file with package names (one per line): ```bash # packages.txt kernel systemd openssl glibc ``` Count all CVEs for the packages: ```bash $ rpm-cve-count -file packages.txt kernel,342 systemd,87 openssl,156 glibc,234 ``` Count only CRITICAL CVEs: ```bash $ rpm-cve-count -file packages.txt -impact CRITICAL kernel,23 systemd,5 openssl,18 glibc,12 ``` Count CVEs created after a specific date: ```bash $ rpm-cve-count -file packages.txt -after 2024-01-01 kernel,45 systemd,12 openssl,28 glibc,31 ``` Combine filters to count CRITICAL CVEs from the last year: ```bash $ rpm-cve-count -file packages.txt -impact CRITICAL -after 2024-01-01 kernel,8 systemd,2 openssl,5 glibc,3 ``` Save results to CSV: ```bash rpm-cve-count -file packages.txt > results.csv ``` ### Output Format CSV format with two columns: - Package name - CVE count ## Development ```bash # Build go build # Run tests go test ./... # Install locally go install ``` ## License This project is licensed under the GNU General Public License v3.0 or later - see the [LICENSE][license] file for details. [license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE -------------------------------------------------------------------------------- # Hummingbird Dashboard url: https://hummingbird-project.io/docs/background/tools/hummingbird-dashboard/ Web dashboard and CLI for monitoring Konflux build pipeline status. ## Features - **Web Dashboard** - Real-time view of build, test, and release status - **Commits View** - Detailed pipeline status per commit with expandable releases - **Components View** - Latest status per component grouped by state - **Merge Requests View** - Track build status across MR versions - **CLI Tool** - Command-line access to pipeline data in YAML/JSON/table formats - **Failed Releases** - View and manage failed releases with LLM-powered analysis - **Failed Push Builds** - Detect and auto-retry failed on-push Konflux builds - **Auto-Rerun** - Automatically retry transient release and push build failures - **Blocked Snapshots** - Block/unblock snapshots from auto-rerun with error pattern matching - **Smart Triggering** - Application-aware rules for determining affected components ## Prerequisites - Python 3.11+ - PostgreSQL database (via [hummingbird-status][hs-docs] or port-forward) ## Installation ```bash cd hummingbird-dashboard pip install -e . ``` ## Usage ### Local Development ```bash cd hummingbird-dashboard # Option 1: Port-forward to production database ./dev.sh port-forward # In terminal 1 DATABASE_URL=postgresql://postgres@localhost:15432/events ./dev.sh start # In terminal 2 # Option 2: Use local database (via hummingbird-status) cd ../hummingbird-status && ./dev.sh db-start && ./dev.sh db-init cd ../hummingbird-dashboard DATABASE_URL=postgresql://postgres:dev@localhost:5432/events ./dev.sh start ``` The dashboard runs at `http://localhost:8080` with live reload. ### Web UI | Endpoint | Description | | ------------------------ | ------------------------------------- | | `/` | Dashboard with all applications | | `/apps/{app}/commits` | Commits view for an application | | `/apps/{app}/components` | Components view for an application | | `/mrs` | Merge requests overview (filterable) | | `/mr/{project}/{iid}` | MR detail with build status per version | | `/releases/failed` | Failed releases with analysis | | `/releases/blocked` | Blocked snapshots management | | `/releases/analysis-log` | LLM analysis run history | | `/cve/status` | CVE ticket status dashboard | | `/cve/run-log` | CVE analysis run history | | `/cve/open` | Open CVE trackers with SLO status | | `/cve/closed` | Closed tickets with VEX reconciliation | | `/cve/r-time` | CVE R-Time (notified to done) metrics | | `/health` | Health check endpoint | | `/metrics` | Prometheus metrics | Each CVE tracker page (`/cve/status`, `/cve/open`, `/cve/closed`, `/cve/r-time`) has a "Download CSV" button at the bottom of the page that downloads the current view (same filters as the page) as a CSV file, via a matching `/cve/{tab}/csv` route (for example `/cve/open/csv?slo=24`). A "Show CVE ID" toggle in the header reveals the CVE ID next to each HUM ticket key (`HUM-1234/CVE-2026-56789`); the preference persists via `localStorage`. ### CVE Search API `GET /api/cve-runs/search` — search CVE analysis run history. | Parameter | Type | Default | Description | |-----------|--------|---------|------------------------------------------| | `package` | string | — | Filter by package name (substring match) | | `ticket` | string | — | Filter by ticket key (substring match) | | `q` | string | — | Free-text search across ticket details | | `log` | string | — | Search CronJob log output | | `days` | int | 30 | Time window in days (1–365) | | `limit` | int | 500 | Max results (1–2000) | At least one of `package`, `ticket`, `q`, or `log` is required. The response includes two result sets: - `results` — per-ticket matches from the JSONB details column - `log_matches` — per-run matches from CronJob log output, each with a `log_excerpt` showing the matching lines in context Example: ```bash # Find runs whose logs mention "merge train" — show excerpts curl -s '/api/cve-runs/search?log=merge+train' | jq '.log_matches[] | {run_at, log_excerpt}' # Find tickets for a package, also searching logs curl -s '/api/cve-runs/search?package=openssl&log=advisory+failed&days=7' \ | jq '{tickets: [.results[].detail.key], log_hits: .log_matches | length}' # List all ticket keys matching a free-text query curl -s '/api/cve-runs/search?q=needs-attention&days=14' | jq '[.results[].detail.key] | unique' ``` ### CLI ```bash # View latest commit status hummingbird-dashboard --application myapp --format table commit # View specific commit hummingbird-dashboard --application myapp commit abc1234 # View multiple commits hummingbird-dashboard --application myapp --format table commit --limit 10 # Component status overview hummingbird-dashboard --application myapp component # JSON output hummingbird-dashboard --format json commit | jq . # Auto-rerun failed releases hummingbird-dashboard auto-rerun # Analyze failed releases via LLM hummingbird-dashboard analyze-failures ``` ## REST API Reference All endpoints return JSON. Read-only GET endpoints are unauthenticated. Interactive OpenAPI/Swagger documentation is available at `/docs`. ### Tier 1 — Core Pipeline Status #### `GET /api/apps/{app_name}/commits` Pipeline status for commits in an application. | Parameter | Type | Default | Description | |-------------|--------|---------|-----------------------------------| | `sha` | string | — | Filter by commit SHA (prefix) | | `component` | string | — | Filter by component name | | `limit` | int | 10 | Max commits to return (max 100) | ```bash curl -s 'http://localhost:8080/api/apps/rpms/commits?limit=5' | jq . curl -s 'http://localhost:8080/api/apps/rpms/commits?sha=abc123&component=openssl' | jq . ``` #### `GET /api/apps/{app_name}/components` Latest build status per component, grouped by state. | Parameter | Type | Default | Description | |-----------|--------|---------|--------------------------------------| | `search` | string | — | Filter components by name (substring)| ```bash curl -s 'http://localhost:8080/api/apps/rpms/components' | jq . curl -s 'http://localhost:8080/api/apps/rpms/components?search=openssl' | jq . ``` ### Tier 2 — Operational Visibility #### `GET /api/dashboard` Full overview across all applications: aggregate build counts, failed releases, component staleness, and recent activity. ```bash curl -s 'http://localhost:8080/api/dashboard' | jq . ``` #### `GET /api/releases/failed` List of currently failed releases with analysis results. | Parameter | Type | Default | Description | |---------------|--------|---------|----------------------------------| | `application` | string | — | Filter by application name | ```bash curl -s 'http://localhost:8080/api/releases/failed' | jq . curl -s 'http://localhost:8080/api/releases/failed?application=rpms' | jq . ``` #### `GET /api/releases/{name}/analysis` LLM failure analysis for a specific release (root cause, classification, recommendation). ```bash curl -s 'http://localhost:8080/api/releases/my-release-abc/analysis' | jq . ``` #### `GET /api/releases/auto-rerun-log` History of automatic rerun attempts. | Parameter | Type | Default | Description | |-----------|------|---------|--------------------| | `limit` | int | 50 | Max entries | ```bash curl -s 'http://localhost:8080/api/releases/auto-rerun-log?limit=10' | jq . ``` #### `GET /api/releases/analysis-log` History of LLM analysis runs. | Parameter | Type | Default | Description | |-----------|------|---------|--------------------| | `limit` | int | 50 | Max entries | ```bash curl -s 'http://localhost:8080/api/releases/analysis-log?limit=10' | jq . ``` #### `GET /api/releases/analysis-costs` Cumulative token usage and estimated cost for LLM analysis runs. ```bash curl -s 'http://localhost:8080/api/releases/analysis-costs' | jq . ``` #### `GET /api/mrs` Merge requests across monitored repositories. | Parameter | Type | Default | Description | |-----------|--------|----------|------------------------------------------------| | `state` | string | `opened` | MR state: `opened`, `merged`, `closed`, `all` | | `project` | string | — | Filter by project (repository) name | ```bash curl -s 'http://localhost:8080/api/mrs' | jq . curl -s 'http://localhost:8080/api/mrs?state=merged&project=rpms' | jq . ``` #### `GET /api/mrs/projects` List of all projects (repositories) with tracked merge requests. ```bash curl -s 'http://localhost:8080/api/mrs/projects' | jq . ``` #### `GET /api/mr/{project}/{iid}` Detail for a single merge request, including all versions and per-version build status. ```bash curl -s 'http://localhost:8080/api/mr/rpms/42' | jq . ``` #### `GET /api/cve/status` Current CVE ticket status across all tracked packages. ```bash curl -s 'http://localhost:8080/api/cve/status' | jq . ``` #### `GET /api/cve/open` Open HUM CVE tracker tickets with SLO status. Notified is the later of HUM ticket created and fix available (upstream or Fedora). SLO is green PASS unless a fix is available and elapsed time is within 8h of the selected SLO (yellow AT RISK) or past it (red FAIL). | Parameter | Type | Default | Description | |------------------------|-------|---------|--------------------------------------------| | `slo` | float | 24 | SLO threshold hours (`24`, `72`, or `168`) | | `include_next_release` | bool | true | Include `cve-next-release` tickets | ```bash curl -s 'http://localhost:8080/api/cve/open?slo=24' | jq . ``` #### `GET /api/cve/closed` Closed HUM CVE tracker tickets with Red Hat CSAF VEX reconciliation (HUM-5843). Analysis stores package-scoped `vex_status` and Jira resolution; the dashboard computes MATCH from those facts (`Done-Errata` → `fixed`, `Not a Bug` → `known_not_affected` or `package_not_listed`). Shows Hummingbird `vex_status`, match state (`matched` / `pending` / `mismatch`), and `vex_resolved` (first scan time where VEX agreed with the Jira resolution). Event metadata is merged across a ticket's rows in `occurred_at` order (HUM-6091): later non-empty fields overlay earlier ones, so Close/VEX facts are not stuck on the first `cve_published` row. The Resolution column uses Jira `Closed / {resolution}` when stored analysis text is not already Closed. The Closed tab shows the VEX timestamp as a link to the CVE's Red Hat CSAF VEX document when the package is named in CSAF. `package_not_listed` (Not a Bug, package absent from the document) shows N/A with no link: this ticket did not produce a VEX change. | Parameter | Type | Default | Description | |-----------|------|---------|-------------------------------------| | `days` | int | 30 | Time window (`0` = all time) | ```bash curl -s 'http://localhost:8080/api/cve/closed?days=30' | jq . ``` #### `GET /api/cve/r-time` CVE R-Time (notified to done) for Done-Errata tickets. Include CVE-HUM starts at the earlier of CVE publication (NVD `datePublished` / `cve_published`) and HUM ticket creation. Exclude CVE-HUM starts at HUM created. Delivery is the catalog image publish when the package is a catalog image source, otherwise the RPM publish. Include VEX requires delivery, VEX updated, and Closed / Done-Errata; Exclude VEX requires delivery and close only. The done timestamp is the latest of the required gates, and only when all of them exist. Incomplete tickets stay on the table with elapsed time until now; the `days` window filters by Done-Errata close (fallback: notified), same as completed tickets use done. CVE-HUM is filing lag after NVD (`max(HUM created − NVD, 0)`); HUM-first tickets show `0.0h`. ADV-VEX is Done-Errata close to VEX feed update (`max(VEX − close, 0)`); close is recorded when the advisory MR merges. Upstream/Fedora are not start or end. Pre-built (delivery before notification) is informational only. Rows without `catalog_image_source` in metadata still require an image until the collector restamps them. Each entry includes package onboarding timestamps and delay analysis (`rpm_first_published_at`, `pkg_lag_hours`, and `our_delay_hours`). Default excludes cve-next-release, CVE-HUM time, and VEX time. | Parameter | Type | Default | Description | |------------------------|-------|---------|--------------------------------------------| | `days` | int | 30 | Time window (`0` = all time) | | `slo` | float | 168 | SLO threshold hours (`24`, `72`, or `168`) | | `include_next_release` | bool | false | Include `cve-next-release` tickets | | `include_hum_cve_time` | bool | false | Include NVD-to-HUM filing lag in R-Time | | `include_vex_time` | bool | false | Require VEX feed update before R-Time ends | ```bash curl -s 'http://localhost:8080/api/cve/r-time?days=10&slo=168' | jq . ``` #### `GET /api/cve/run-log` History of CVE analysis CronJob runs. | Parameter | Type | Default | Description | |-----------|------|---------|--------------------| | `limit` | int | 100 | Max entries | ```bash curl -s 'http://localhost:8080/api/cve/run-log?limit=10' | jq . ``` #### `GET /api/cve-export` Export CVE sync data as JSON for prod-to-preprod replication. Valid `section` values are `cve_analysis_log`, `cve_ticket_events`, and `package_lifecycle`. Optional pagination parameters: | Parameter | Type | Default | Description | |-----------|--------|---------|-------------------| | `section` | string | — | Section to export | | `limit` | int | 2000 | Rows per page | | `offset` | int | 0 | Pagination offset | ```bash curl -s -H "Authorization: Bearer $CVE_REPORT_TOKEN" \ 'http://localhost:8080/api/cve-export' | jq '.cve_ticket_events | length' curl -s -H "Authorization: Bearer $CVE_REPORT_TOKEN" \ 'http://localhost:8080/api/cve-export?section=cve_ticket_events&limit=1000&offset=0' \ | jq '.count,.has_more' # Package-scoped lifecycle milestones (rpm_first_published, etc.) curl -s -H "Authorization: Bearer $CVE_REPORT_TOKEN" \ 'http://localhost:8080/api/cve-export?section=package_lifecycle&limit=10000&offset=0' \ | jq '.rows[] | select(.event_type == "rpm_first_published")' ``` **Event type validation.** The import path (`POST /api/cve-import`) validates every `event_type` value against a canonical allowlist defined in `hummingbird_dashboard/sources.py`: - `CANONICAL_TICKET_EVENT_TYPES` — values accepted for `cve_ticket_events`: `cve_published`, `hum_ticket_created`, `hum_ticket_closed`, `upstream_fix_merged`, `fedora_update_available`, `rpm_fix_published_to_pulp`, `image_rebuilt_on_quay`, `vex_resolved`. - `CANONICAL_PACKAGE_EVENT_TYPES` — values accepted for `package_lifecycle`: `rpm_first_published`. Legacy names (e.g. `jira_created`, `delivered_in_rpm`, ...) are mapped to their canonical equivalents via `_LEGACY_EVENT_KEY_MAP`. Unknown names are rejected with HTTP 400. #### `POST /api/cve-import` Replace local CVE sync data from a payload previously returned by `/api/cve-export`. ```bash curl -s -X POST \ -H "Authorization: Bearer $CVE_REPORT_TOKEN" \ -H "Content-Type: application/json" \ --data @cve-export.json \ 'http://localhost:8080/api/cve-import' | jq . curl -s -X POST \ -H "Authorization: Bearer $CVE_REPORT_TOKEN" \ -H "Content-Type: application/json" \ --data '{"section":"cve_ticket_events","rows":[],"reset":true}' \ 'http://localhost:8080/api/cve-import' | jq . ``` To copy prod CVE sync rows onto preprod (both sections; first page of each section uses `reset=true`): ```bash CVE_REPORT_TOKEN=... ./hummingbird-cve-analysis/scripts/copy_prod_to_preprod.sh CVE_REPORT_TOKEN=... ./hummingbird-cve-analysis/scripts/copy_prod_to_preprod.sh --dry-run ``` The script prefers paginated `/api/cve-export`. If that endpoint returns HTTP 500, it retries unpaged `/api/cve-export`. Analysis logs may fall back to `/api/cve/run-log` (latest 100 runs). Ticket events are not reconstructed from Open/Closed/R-Time: those views omit ticket-event metadata. Before import, ticket events collapse pre-HUM-5918 names (`jira_closed`, `delivered_in_image`, …) onto canonical types so a null legacy row cannot wipe a timestamp or VEX label. Prod ticket-event rows often omit `catalog_image_source`; the copy fills that flag from the catalog source map (same rule as the collector) and keeps prod's True/False when present. The copy fails if the catalog map cannot be built. `cve_analysis_log` is paged 5 rows at a time by default. Each row includes full `log_output` (up to 512 KiB). Ticket events default to 1000 rows per page. ### Tier 3 — Analytics & SRE Metrics #### `GET /api/stats/build-throughput` Build counts over time (successful, failed, total). | Parameter | Type | Default | Description | |-----------|------|---------|--------------------------| | `days` | int | 30 | Lookback window in days | ```bash curl -s 'http://localhost:8080/api/stats/build-throughput?days=7' | jq . ``` #### `GET /api/stats/release-success-rate` Release success/failure ratios over time. | Parameter | Type | Default | Description | |---------------|--------|---------|--------------------------| | `days` | int | 30 | Lookback window in days | | `application` | string | — | Filter by application | ```bash curl -s 'http://localhost:8080/api/stats/release-success-rate?days=14&application=rpms' | jq . ``` #### `GET /api/stats/auto-rerun-effectiveness` Success rate and time-to-resolution for automatic reruns. | Parameter | Type | Default | Description | |-----------|------|---------|--------------------------| | `days` | int | 30 | Lookback window in days | ```bash curl -s 'http://localhost:8080/api/stats/auto-rerun-effectiveness?days=7' | jq . ``` #### `GET /api/components/staleness` Components ranked by time since last successful build. ```bash curl -s 'http://localhost:8080/api/components/staleness' | jq . ``` #### `GET /api/releases/similar-failures` Find releases with error messages similar to a given string. | Parameter | Type | Default | Description | |-----------|--------|---------|----------------------------------| | `error` | string | — | Error text to match against | ```bash curl -s 'http://localhost:8080/api/releases/similar-failures?error=timeout+connecting' | jq . ``` #### `GET /api/apps/{app_name}/last-successful` Timestamp and SHA of the last fully successful pipeline per component. ```bash curl -s 'http://localhost:8080/api/apps/rpms/last-successful' | jq . ``` ### Tier 4 — System #### `GET /api/service-status/json` Health of upstream services the dashboard depends on (database, Konflux API, KubeArchive, etc.). ```bash curl -s 'http://localhost:8080/api/service-status/json' | jq . ``` #### `GET /api/settings` Current runtime settings (auto-rerun enabled, analysis enabled, blocked patterns, etc.). ```bash curl -s 'http://localhost:8080/api/settings' | jq . ``` #### `GET /health` Enriched health check returning service version, uptime, database connectivity, and dependency status. ```bash curl -s 'http://localhost:8080/health' | jq . ``` ## Configuration ### Environment Variables | Variable | Default | Description | | -------------- | --------------------------------- | ------------------------- | | `DATABASE_URL` | `postgresql://...localhost:5432/` | PostgreSQL connection URL | | `PORT` | `8080` | Web server port | ### Authentication Variables These control the retrigger functionality (requires OAuth proxy in production): | Variable | Default | Description | | -------------------- | ---------------------------------- | --------------------------------------- | | `TRIGGER_AUTH_MODE` | `oauth` | `oauth` (production) or `local` | | `TRIGGER_AUTH_GROUP` | `konflux-hummingbird-admin-access` | OpenShift group required to retrigger | | `TRIGGER_LOCAL_USER` | `local-dev` | Username when `TRIGGER_AUTH_MODE=local` | | `CVE_REPORT_TOKEN` | | Bearer token for CVE API auth | For local development with retrigger enabled: ```bash TRIGGER_AUTH_MODE=local DATABASE_URL=... ./dev.sh start ``` ### Auto-Rerun Variables These control the auto-rerun cronjob and failure analysis: | Variable | Default | Description | | -------------------------------- | ---------------------------------------- | ----------------------------------------------------------------------------- | | `AUTO_RERUN_MIN_AGE_MINUTES` | `30` | Minimum failure age before retrying | | `AUTO_RERUN_MAX_RETRIES` | `3` | Max rerun attempts per snapshot+plan | | `KONFLUX_KUBECONFIG_PATH` | | Path to Konflux kubeconfig file | | `RELEASE_NAMESPACE` | | Namespace where releases are created | | `MANAGED_NAMESPACE` | | Namespace for fetching release PLRs | | `KUBEARCHIVE_URL` | | KubeArchive API URL for archived resources | | `GOOGLE_APPLICATION_CREDENTIALS` | | Path to GCP SA key for Vertex AI | | `GOOGLE_CLOUD_PROJECT` | | GCP project ID for Vertex AI | | `ANALYSIS_MODEL_API_KEY` | | Gemini API key (fallback if no GCP creds) | | `ANALYSIS_MODEL` | `gemini-2.5-flash` | LLM model for failure analysis | | `ANALYSIS_MODEL_REGION` | `global` | Vertex AI region | | `MAX_ANALYSES_PER_CYCLE` | `5` | Max releases to analyze per cycle | | `SLACK_WEBHOOK_URL` | | Slack webhook for rerun/analysis notifications | | `DASHBOARD_URL` | | Dashboard base URL for Slack links | | `GITLAB_SLACK_MAP_PATH` | `/etc/hummingbird/gitlab-slack-map.yaml` | GitLab→Slack map for author @-mentions (infra-mounted; missing omits mention) | | `PUSH_RERUN_MIN_AGE_MINUTES` | `10` | Min push build failure age before retrying | | `PUSH_RERUN_MAX_RETRIES` | `3` | Max retry attempts per component+sha | ## Failures The failures section is accessible via the "Failures" nav item and provides two views selectable by tab: Releases and Push Builds. ### Failed Releases The `/failures/releases` page shows all failed releases with: - **LLM Analysis** — Each failure is analyzed by Gemini with root cause, classification, and recommendation - **Failure Classification** — Transient, Configuration, Code, External Service, or Unknown - **Rerun History** — Past rerun attempts and outcomes per snapshot - **Error Pattern Matching** — Auto-block snapshots matching known error patterns - **Auto-Rerun** — Automatically retry transient failures (configurable via dashboard toggle) - **Analysis Toggle** — Enable/disable LLM analysis from the dashboard ### CLI Subcommands The `auto-rerun` subcommand retries eligible failed releases: ```bash hummingbird-dashboard auto-rerun --application myapp ``` The `auto-rerun-push` subcommand retries eligible failed push builds: ```bash hummingbird-dashboard auto-rerun-push --application myapp ``` The `analyze-failures` subcommand runs LLM analysis on unanalyzed failures: ```bash hummingbird-dashboard analyze-failures --managed-namespace rhtap-releng-tenant ``` All three are designed to run as Kubernetes CronJobs. Slack messages for auto-rerun, push-retry, analysis, and manual UI reruns include a GitLab MR link (and author @-mention when mapped) when the failure SHA resolves to an MR. ## Merge Requests The `/mrs` page shows merge requests across all monitored repositories with: - **State filter** - Open, merged, closed, or all MRs - **Project filter** - Filter by specific repository - **Build status** - Aggregate status across all components The `/mr/{project}/{iid}` detail page shows: - **All MR versions** - Each head commit SHA that was pushed to the MR - **Build status per version** - Full pipeline status (build, snapshot, test, release) - **Links to Konflux UI** - Direct links to PipelineRuns and Snapshots Versions are ordered by latest event timestamp, so force-pushed commits appear in the correct position even if they reuse an earlier SHA. ## Component Status The dashboard tracks component build status with these states: | Status | Icon | Description | |------------|------|------------------------------------------------| | Success | ✅ | Build passed for expected commit | | Superseded | 🔄 | Expected commit not built, but newer succeeded | | Failed | ❌ | Build failed for expected commit | | Stale | ⚠️ | Build not triggered for expected commit | | Running | ⏳ | Build in progress | | Missing | ❓ | No build found | Components are grouped by status: Failed/Stale → Running → OK → Missing. ## Trigger Rules The dashboard uses application-specific rules to determine which components are affected by a push: - **containers**: Changes in `images/{component}` trigger builds - **rpms**: Changes in `rpms/{component}` or `mock/mock.cfg` trigger builds - **tools**: Changes in `{component}` directory trigger builds Certain files are excluded from triggering (README, templates, etc.). ## Development See the main [README][readme] for development workflows. ### Running Tests ```bash cd hummingbird-dashboard pip install -e ".[dev]" pytest ``` ### Project Structure ```text hummingbird-dashboard/ ├── Containerfile ├── dev.sh ├── hummingbird_dashboard/ │ ├── analysis.py # LLM failure analysis (Gemini) │ ├── cli.py # CLI entry point │ ├── db/ # SQLAlchemy engine/session + ORM models │ │ ├── engine.py │ │ └── models.py │ ├── konflux.py # Konflux API client │ ├── models.py # Data models (Component, PushEvent) │ ├── sources.py # PostgreSQL queries │ ├── table.py # CLI table formatting │ ├── triggers.py # Application-specific trigger rules │ ├── views.py # Status computation and aggregation │ └── web/ │ ├── app.py # FastAPI application │ └── templates/ # Jinja2 templates └── tests/ ``` ### Building Container Image ```bash cd hummingbird-dashboard podman build -f Containerfile -t hummingbird-dashboard . ``` ## License This project is licensed under the GNU General Public License v3.0 or later - see the [LICENSE][license] file for details. [hs-docs]: hummingbird-status.md [readme]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/README.md [license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE -------------------------------------------------------------------------------- # Service metrics url: https://hummingbird-project.io/docs/background/tools/service-metrics/ Long-running exporter for Hummingbird AWS Cost Explorer spend, Kubernetes Metrics API CPU/memory gauges, and OpenShift cluster resource quota usage. ## Features - **Explicit collectors** - nothing runs unless listed in `METRICS_CONFIG` - **Kubernetes Metrics API** - per-container CPU and memory gauges - **Cluster resource quotas** - AppliedClusterResourceQuota usage and limits - **AWS Cost Explorer** - yesterday's `NetAmortizedCost` by team and service - **Prometheus** - HTTP metrics on port 9090 ## Configuration Collectors are off unless listed in `METRICS_CONFIG`: ```yaml enabled: [kubernetes] ``` Unknown names fail startup. | Variable | Purpose | | -------- | ------- | | `METRICS_CONFIG` | YAML with `enabled` collector list (required) | | `KUBERNETES_CONFIG` | YAML with `namespaces` for the kubernetes collector | | `METRICS_PORT` | HTTP port (default `9090`) | | `SENTRY_DSN` | Optional Sentry DSN | | `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` | Cost Explorer IAM user (hub) | | `AWS_DEFAULT_REGION` | Must be `us-east-1` for Cost Explorer | ## Kubernetes Metrics API `hummingbird_k8s_resource_usage{namespace,pod,container,resource,app}` with `resource=cpu|memory` and `app` sourced from the pod's `app.kubernetes.io/name` label (empty string if absent). Instant gauges from `/apis/metrics.k8s.io/v1beta1/namespaces//pods`. This is not cAdvisor: managed OpenShift does not grant `nodes/metrics` or `cluster-monitoring-view`, so kubelet `/metrics/cadvisor` and platform Prometheus federation are not available. If `KUBERNETES_CONFIG.namespaces` is unset, the collector uses the pod's ServiceAccount namespace. ## Cluster resource quotas The kubernetes collector also reads namespace-scoped `AppliedClusterResourceQuota` objects from the `quota.openshift.io/v1` API and exposes two gauges that mirror the [openshift-state-metrics](https://github.com/openshift/openshift-state-metrics) schema with a `hummingbird_` prefix: - `hummingbird_clusterresourcequota_usage{name,resource,type}` -- cluster-wide totals from `status.total` (`type=hard|used`) - `hummingbird_clusterresourcequota_namespace_usage{name,namespace,resource,type}` -- per-namespace breakdown from `status.namespaces` (`type=hard|used`) `resource` values match Kubernetes quantity names: `cpu`, `memory`, `requests.cpu`, `limits.memory`, `pods`, etc. Quantities are converted to base units (cores, bytes, count). The same ACRQ may appear in multiple namespaces; the global metric is idempotent across duplicates. Stale series are removed when ACRQs disappear. Requires `list` on `appliedclusterresourcequotas` in `quota.openshift.io` (namespace-scoped Role -- no ClusterRole needed). ## AWS costs `hummingbird_aws_cost{group,type,key}` is yesterday's `NetAmortizedCost`. Queries run at process start and every 24 hours. `group=all` is the shared account by `app-code` and SERVICE. `group=hummingbird` filters `app-code=RPRM-001` and groups by SERVICE, OPERATION, USAGE_TYPE, Name tag, and SERVICE+OPERATION. Untagged Cost Explorer keys such as `app-code$` become `key=untagged`. Well-known SERVICE names are shortened (`S3`, `EC2`). Dual grouping uses `S3|PutObject`. The SAM template creates an IAM user with `ce:GetCostAndUsage` only. Access keys are created after deploy and stored in Vault. Deployment lives in the [infrastructure repo][infra-repo]. ## License This project is licensed under the GNU General Public License v3.0 or later - see the [LICENSE][license] file for details. [infra-repo]: https://gitlab.com/redhat/hummingbird/infrastructure [license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE -------------------------------------------------------------------------------- # VEX Checker url: https://hummingbird-project.io/docs/background/tools/vex-checker/ A CLI tool that works with the [Red Hat CSAF VEX feed] and the Hummingbird catalog API to check CVE statuses and track coverage across Jira, VEX advisories, and container image scan results. ## Subcommands | Subcommand | Purpose | |-------------|----------------------------------------------------------------| | `check` | Look up a single CVE and show Hummingbird product statuses | | `reconcile` | Cross-reference Jira CVE tickets with the VEX feed | | `catalog` | Cross-reference catalog API CVEs with Jira tickets | ## Prerequisites - **Python 3.11+** - **Internet access** to `security.access.redhat.com` (no VPN required) for `check`/`reconcile`; to `api-hummingbird.hummingbird-project.io` (no VPN required) for `catalog` - **`JIRA_EMAIL`** Atlassian account email — required for `reconcile` and `catalog` - **`JIRA_TOKEN`** Jira API token — required for `reconcile` and `catalog` ## `check` — Single CVE lookup Fetches the CSAF VEX document for a given CVE and prints the status of all Hummingbird products, filtering out all other RHEL/OpenShift/etc. entries. ### check: usage ```bash ./check_vex.py check CVE-YYYY-NNNNN [--json] ``` ### check: options | Option | Description | |--------------|------------------------| | `CVE-ID` | CVE to look up | | `--json, -j` | Output results as JSON | ### check: examples ```bash $ ./check_vex.py check CVE-2014-8090 CVE-2014-8090 — CVE-2014-8090 ruby: REXML billion laughs attack (Moderate) hummingbird-1:ruby.src known_affected none_available hummingbird-1:ruby3.3.src known_affected none_available hummingbird-1:ruby3.4.src known_affected none_available hummingbird-1:ruby4.0.src known_affected none_available ``` ```bash $ ./check_vex.py check CVE-2014-8090 --json { "cve": "CVE-2014-8090", "title": "...", "severity": "Moderate", "hummingbird": [ { "product_id": "hummingbird-1:ruby.src", "status": "known_affected", "remediation": "none_available" }, ... ] } ``` ### check: exit codes | Code | Meaning | |------|---------------------------------------------------------------------------------------| | `0` | No VEX document found, no Hummingbird products listed, or no `known_affected` status | | `1` | One or more Hummingbird products are `known_affected` | | `2` | Invalid CVE ID format | ## `reconcile` — Jira/VEX sync check Queries the Jira HUM project (Security component) for all CVE tickets, then checks each CVE in the VEX feed. Reports two types of mismatches: - **Open in Jira but not `known_affected` in VEX** — VEX says the issue is resolved but the Jira ticket is still open. - **Closed in Jira but `known_affected` in VEX** — the Jira ticket was closed but Red Hat's advisory still marks Hummingbird as affected. Continuous closed-ticket VEX reconciliation (Done-Errata → `fixed`, Not a Bug → `known_not_affected` or `package_not_listed`, scoped to the ticket's package, with `awaiting-vex` work queue and dashboard persistence) lives in `hummingbird-cve-analysis` / the dashboard Closed tab (HUM-5843). `reconcile` remains the one-shot CLI audit. Jira tickets are considered closed when their status is one of: `Done`, `Closed`, `Won't Fix`, `Not a Bug`. VEX documents are fetched in parallel (`--workers`, default 20) so the command is fast even with hundreds of CVE tickets. ### reconcile: usage ```bash export JIRA_EMAIL=you@redhat.com export JIRA_TOKEN= ./check_vex.py reconcile [--workers N] [--json] ``` Credentials are read exclusively from environment variables to avoid exposing them in process listings. ### reconcile: options | Option | Description | |--------------|---------------------------------------------------------| | `--jira-url` | Jira base URL (default: `https://redhat.atlassian.net`) | | `--workers` | Parallel VEX fetch workers (default: 20) | | `--json, -j` | Output results as JSON | ### reconcile: example ```bash $ ./check_vex.py reconcile Fetching Jira tickets... 142 ticket(s) Fetching VEX statuses for 89 unique CVE(s)... 89/89 2 mismatch(es) found: HUM-1234 CVE-2025-1234 closed → known_affected (VEX still open) Jira: https://redhat.atlassian.net/browse/HUM-1234 VEX: https://security.access.redhat.com/data/csaf/v2/vex-feed/2025/cve-2025-1234.json HUM-5678 CVE-2024-5678 open → fixed (VEX not affected) Jira: https://redhat.atlassian.net/browse/HUM-5678 VEX: https://security.access.redhat.com/data/csaf/v2/vex-feed/2024/cve-2024-5678.json ``` ### reconcile: exit codes | Code | Meaning | |------|--------------------------------------------| | `0` | No mismatches found | | `1` | One or more mismatches detected | | `2` | Missing `JIRA_EMAIL` or `JIRA_TOKEN` | ## `catalog` — Catalog API / Jira coverage check Fetches all CVEs currently detected in Hummingbird container images (via Grype scan results stored in the catalog API) and cross-references them with Jira HUM Security tickets. Reports CVEs that have no corresponding Jira ticket. CVEs with a closed Jira ticket are reported separately as expected propagation delay — the typical flow is CVE fix → RPM → image rebuild → Grype DB update, and Jira tickets are closed early while the catalog API reflects the latest Grype scan (updated ~once daily). Non-CVE vulnerability identifiers (e.g. `GHSA-*`) from Grype are filtered out since they are not tracked in Jira. ### catalog: usage ```bash export JIRA_EMAIL=you@redhat.com export JIRA_TOKEN= ./check_vex.py catalog [--api-url URL] [--workers N] [--json] ``` Each CVE line shows severity, age since first detection, affected images, and for tracked CVEs the Jira ticket key. Components with known fix versions are shown in brackets when available. Closed Jira tickets are cross-referenced with the VEX feed to distinguish between different root causes: - **Done-Errata, VEX still affected** -- fix shipped in RPM but not yet propagated to container images; needs rebuild or lockfile refresh. - **Done-Errata, VEX resolved** -- fix fully propagated, Grype DB just needs its daily update to stop flagging. - **Won't Do / Not a Bug, VEX resolved** -- VEX already marks CVE as not affected; Grype DB delay, will self-resolve. - **Won't Do / Not a Bug, VEX still affected** -- genuine known issue that will persist in scans (accepted risk, upstream won't fix, etc.). ### catalog: options | Option | Description | |--------------|------------------------------------------------------------------------------------------| | `--api-url` | Catalog API base URL (default: `https://api-hummingbird.hummingbird-project.io/v1`) | | `--jira-url` | Jira base URL (default: `https://redhat.atlassian.net`) | | `--workers` | Parallel VEX fetch workers (default: 20) | | `--json, -j` | Output results as JSON | ### catalog: example ```bash $ ./check_vex.py catalog Catalog: 72 CVEs across 29 images (scanned 2026-04-28T10:45:38Z) Jira: 748 tickets (437 unique CVEs) 11 untracked CVE(s) (no Jira ticket): CVE-2008-2662 High 8d ruby (12 tags) [ruby3.3@3.3.10, +36 more] CVE-2026-2950 Medium 8d aspnet-runtime, ... (30 tags) [lodash@4.17.21] ... 45 tracked CVE(s) open in Jira: CVE-2025-68114 High 8d php (3 tags) [capstone@5.0.6] HUM-925 ... 8 closed in Jira (Done-Errata) but VEX still affected -- verify fix propagated: CVE-2026-27143 Critical 8d caddy, go-fdo-client, ... (10 tags) HUM-969 4 closed in Jira (Done-Errata), VEX resolved -- Grype DB delay: CVE-2025-61732 High 8d go, xcaddy (6 tags) HUM-1119 3 closed in Jira (Won't Do / Not a Bug), VEX resolved -- Grype DB delay: CVE-2026-27140 High 8d caddy, go, ... (10 tags) HUM-967 1 closed in Jira (Won't Do / Not a Bug), VEX still affected -- genuine: CVE-2025-12781 Medium 8d postgresql, python, ... (17 tags) HUM-1387 ``` ### catalog: exit codes | Code | Meaning | |------|------------------------------------------------------------| | `0` | All catalog CVEs are tracked (have a Jira ticket) | | `1` | One or more untracked CVEs found | | `2` | Missing `JIRA_EMAIL` or `JIRA_TOKEN` | ## Development ```bash # Run tests cd vex-checker && python3 -m unittest discover tests -v # Run linter (from repo root) ruff check vex-checker ``` ## License This project is licensed under the GNU General Public License v3.0 or later - see the [LICENSE][license] file for details. [Red Hat CSAF VEX feed]: https://security.access.redhat.com/data/csaf/v2/vex-feed/ [license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE -------------------------------------------------------------------------------- # Red Hat Catalog Environment Promotion url: https://hummingbird-project.io/docs/background/tools/redhat-catalog-environments/ ## Environments | Environment | URL | Source | Deploy trigger | |--------------|------------------------------------------------------------|-----------------------|-----------------------------------| | MR Preview | GitLab Pages (`/mr-{IID}`) | MR branch | Auto on MR pipeline | | Experimental | `images.experimental.hummingbird-project.io/` | `experiment/*` branch | Auto on push to experiment branch | | Staging | `images.staging.hummingbird-project.io` | `main` | Auto on merge to main | | Production | `images.redhat.com` | `main` | Manual after staging deploy | > **Disambiguation:** The **infra experimental host** (`images.experimental.hummingbird-project.io`, > `experiment/*` branches) is an off-main deploy sandbox. The former in-app `/experimental` > product route was removed in HUM-2171 — use `/api` instead. See > [Archived surfaces](../redhat-catalog/docs/archived-surfaces.md). ## Promotion Paths ### Routine: staging to production 1. MR merged to `main` 2. `redhat-catalog-visual-staging-reference` captures live visuals on current staging 3. `deploy_redhat_catalog_staging` runs automatically 4. `redhat-catalog-e2e-smoke` (smoke) and `visual_redhat_catalog_staging` (presentation diff) validate staging 5. **UAT sign-off** on staging — product, design, and engineering release approvers complete the [UAT checklist][uat-checklist] and file a record ([UAT program runbook][uat-program]) 6. Approval owner clicks `deploy_redhat_catalog` (manual gate) after automated gates pass **and** UAT is approved 7. Production deploys from same commit ### Exploration: experimental to staging to production 1. Create branch `experiment/` from `main` (must use exactly `experiment/`, not `experimental/` or other prefixes — without it, no deploy triggers) 2. Push triggers `deploy_redhat_catalog_experimental` automatically 3. Branch deploys to `images.experimental.hummingbird-project.io//` 4. Iterate on experimental host for days or weeks 5. **Light UAT** (optional spot-check on experimental URL — see [UAT program — experimental path][uat-program]) 6. When ready: open MR from `experiment/` to `main` 7. Normal MR review, merge, then routine promotion path (full UAT on staging) ### Retiring an experiment 1. Delete the `experiment/` branch 2. Remove the branch prefix from the experimental S3 bucket: ```bash aws s3 rm s3://redhat-catalog-experimental-spa// --recursive ``` 3. Invalidate the CloudFront cache: ```bash aws cloudfront create-invalidation \ --distribution-id \ --paths "//*" ``` ## Build Configuration Per Environment The infrastructure pipeline sets these environment variables at build time: | Variable | Experimental | Staging | Production | |------------------------|----------------------------------------------|-----------------------------------------|---------------------| | `ASSET_PATH` | `//` | `/` | `/` | | `DPAL_USE_STAGING` | `true` | `true` | (unset, uses prod) | | `TRUSTARC_DATA_DOMAIN` | `images.experimental.hummingbird-project.io` | `images.staging.hummingbird-project.io` | `images.redhat.com` | | `CATALOG_API_BASE_URL` | (unset, uses app default) | (unset, uses app default) | (unset, uses prod) | | `CATALOG_BASE_URL` | (unset, uses app default) | (unset, uses app default) | (unset, uses prod) | | `HUMMINGBIRD_API_URL` | (unset, uses app default) | (unset, uses app default) | (unset, uses prod) | | `JIRA_API_URL` | Jira API URL | Jira API URL | Jira API URL | ## Infrastructure ### SAM Stacks | Stack | Template | ResourcePrefix | CatalogDomainName | |-------------------------------|------------------------------|-------------------------------|-----------------------------------------------| | `redhat-catalog-prod` | `template.yaml` | `redhat-catalog` | `images.redhat.com` | | `redhat-catalog-staging` | `template.yaml` | `redhat-catalog-staging` | `images.staging.hummingbird-project.io` | | `redhat-catalog-experimental` | `template-experimental.yaml` | `redhat-catalog-experimental` | `images.experimental.hummingbird-project.io` | Staging uses the same `template.yaml` as production (single-site CloudFront distribution). Experimental uses `template-experimental.yaml` which adds a CloudFront Function for path-prefix SPA routing across multiple branches. ### DNS ALIAS records (`images.staging/experimental.hummingbird-project.io` → CloudFront) are created automatically by `post_deploy.sh` after each deploy. **First-time bootstrap:** The SAM template creates an ACM certificate with DNS validation. The ACM validation CNAME must exist in the `hummingbird-project.io` Route 53 zone **before** the first `sam deploy`, or CloudFormation will wait 30 minutes and roll back. To bootstrap: 1. Deploy with `CatalogDomainName=""` (uses CloudFront default domain, no cert) 2. Create the ACM validation CNAME in Route 53 (get the value from the ACM console or `aws acm describe-certificate`) 3. Redeploy with `CatalogDomainName` set to the real domain After the first successful deploy, no manual DNS steps are needed. ### CI/CD Variables Set in GitLab project settings: | Variable | Value | |-------------------------------|--------------------------------------------------| | `REDHAT_CATALOG_STAGING_URL` | `https://images.staging.hummingbird-project.io` | ## Pipeline Flow ```text experiment/* push: redhat-catalog-experimental-check -> deploy_redhat_catalog_experimental MR (target main): redhat-catalog (build + test) -> redhat-catalog-browser-tests -> redhat-catalog-visual-mock (Playwright mock API; MR vs merge-base diff) -> pages (preview) main push: redhat-catalog-visual-staging-reference (pre-deploy live capture) -> deploy_redhat_catalog_staging (auto) -> redhat-catalog-e2e-smoke (smoke) -> visual_redhat_catalog_staging (post-deploy vs pre-deploy diff) -> deploy_redhat_catalog (manual; needs staging deploy + smoke + visual + UAT sign-off) ``` [uat-checklist]: ../redhat-catalog/docs/uat-checklist.md [uat-program]: redhat-catalog-uat.md Visual regression details: [redhat-catalog testing guide — Visual regression][testing-guide] and [e2e/visual/README.md][visual-readme]. [testing-guide]: ../redhat-catalog/docs/testing-guide.md#visual-regression [visual-readme]: ../redhat-catalog/e2e/visual/README.md -------------------------------------------------------------------------------- # S3 Lookaside Cache url: https://hummingbird-project.io/docs/background/tools/s3-lookaside-cache/ Infrastructure for caching dist-git lookaside content, providing access to source artifacts for the Hummingbird build pipeline. ## Overview This stack provides S3-based infrastructure for dist-git artifacts with CloudFront CDN. It caches content from upstream sources like Git tarballs. A companion stack (`s3-lookaside-cache-upload-role`) creates a GitLab OIDC identity provider and an IAM role that GitLab CI jobs can assume using short-lived JWT tokens, eliminating the need for long-lived access keys. ## Architecture ### Cache Infrastructure ```mermaid flowchart LR clients["Build Clients"] cf["CloudFront\nDistribution"] s3["S3 Bucket\n(dist-git-cache)"] logs["S3 Logs\nBucket"] backup["AWS Backup\nVault"] headers["Security\nHeaders"] clients --> cf --> s3 cf --> headers s3 --> logs --> backup ``` ### GitLab CI Upload Role ```mermaid flowchart LR gitlab["GitLab CI\n(.gitlab-ci.yml)"] sts["AWS STS\n(validates JWT\nvia OIDC/JWKS)"] role["IAM Role\n(scoped to\nS3 PutObject)"] s3["S3 Bucket\n(dist-git cache)"] gitlab -- "id_token" --> sts sts -- "AssumeRoleWithWebIdentity" --> role role -- "temporary credentials" --> sts sts -- "credentials" --> gitlab gitlab -- "s3:PutObject" --> s3 ``` ## Components ### Cache Stack (`s3-lookaside-cache`) | Resource | Type | Description | |-------------------------------------|--------------------|----------------------------------------------------| | `DistGitCacheBucket` | S3 Bucket | Main cache storage with versioning and object lock | | `DistGitLogBucket` | S3 Bucket | Access logs with tiered storage lifecycle | | `DistGitCacheDistribution` | CloudFront | CDN with HTTP/2+3, IPv6, TLS 1.2+ | | `DistGitCacheOAC` | Origin Access Ctrl | Secure S3 access from CloudFront | | `DistGitCacheCachePolicy` | Cache Policy | 1-day default TTL, 1-year max, Gzip/Brotli | | `DistGitCacheResponseHeadersPolicy` | Response Headers | HSTS, X-Frame-Options, XSS protection | | `DistGitBackupVault` | Backup Vault | AWS Backup vault for data protection | | `DistGitBackupPlan` | Backup Plan | Daily (35-day) and weekly (365-day) backups | | `DistGitUploadPolicy` | IAM Managed Policy | Grants `s3:PutObject` to the cache bucket | ### Upload Role Stack (`s3-lookaside-cache-upload-role`) | Resource | Type | Description | |----------------------|-------------------|--------------------------------------------------------------| | `GitLabOIDCProvider` | IAM OIDC Provider | Registers `gitlab.com` as a trusted identity provider | | `DistGitUploadRole` | IAM Role | Web identity role assumable by the configured GitLab project | ## Parameters ### Cache Stack Parameters | Parameter | Description | |------------------|-------------------------------------------------------------------------| | `ResourcePrefix` | Prefix for resource names (e.g., `arr-hummingbird-prod-dist-git-cache`) | | `BucketName` | Globally unique name for the cache bucket (logs bucket appends `-logs`) | ### Upload Role Parameters | Parameter | Description | |---------------------|----------------------------------------------------------------------------------| | `ResourcePrefix` | Prefix for resource names (e.g., `arr-hummingbird-prod-dist-git-upload`) | | `CacheStackName` | Name of the deployed `s3-lookaside-cache` CloudFormation stack | | `GitLabProjectPath` | GitLab project path allowed to assume the role (e.g., `redhat/hummingbird/rpms`) | ## S3 Key Structure Files are stored using the dist-git lookaside path convention: ```text {namespace}/{package}/{filename}/{hashType}/{hash}/{filename} ``` Example: ```text rpms/tar/tar-1.35.tar.xz/sha512/abc123.../tar-1.35.tar.xz ``` ## Trust Policy The upload role's trust policy restricts access using three conditions (all `StringEquals`): - **Audience** (`gitlab.com:aud`): Must be `https://gitlab.com` - **Subject** (`gitlab.com:sub`): Must match `project_path::ref_type:branch:ref:main` - **Protected ref** (`gitlab.com:ref_protected`): Must be `"true"` Only the `main` branch of the configured project can assume the role. ## GitLab CI Usage Configure your `.gitlab-ci.yml` to assume the role using [`id_tokens`](https://docs.gitlab.com/ci/cloud_services/aws/). The AWS CLI automatically calls `AssumeRoleWithWebIdentity` when `AWS_WEB_IDENTITY_TOKEN_FILE` and `AWS_ROLE_ARN` are set: ```yaml upload to cache: image: name: amazon/aws-cli:latest entrypoint: [""] id_tokens: GITLAB_OIDC_TOKEN: aud: https://gitlab.com script: - set +x - printenv GITLAB_OIDC_TOKEN > /tmp/oidc-token - export AWS_WEB_IDENTITY_TOKEN_FILE=/tmp/oidc-token - export AWS_ROLE_ARN="arn:aws:iam::${AWS_ACCOUNT_ID}:role/${ROLE_NAME}" - export AWS_ROLE_SESSION_NAME="gitlab-ci-${CI_JOB_ID}" - aws s3 cp "$FILE" "s3://${BUCKET}/${S3_KEY}" - rm -f /tmp/oidc-token ``` The `aud` value in `id_tokens` must match the audience configured in the OIDC identity provider (`https://gitlab.com`). The token is written to a file because the AWS SDK reads it from `AWS_WEB_IDENTITY_TOKEN_FILE` rather than accepting it inline. ## Security Features - **Encryption**: AES256 server-side encryption with S3 bucket keys - **Public Access**: All public access blocked on both buckets - **Transport Security**: HTTPS enforced via bucket policy - **Object Lock**: GOVERNANCE mode with 1-day default retention (will be increased in the future) - **CloudFront OAC**: Modern Origin Access Control (not legacy OAI) - **Security Headers**: HSTS, X-Frame-Options (DENY), X-Content-Type-Options, X-XSS-Protection, strict referrer policy - **TLS 1.2 Minimum**: For all CloudFront connections - **OIDC Federation**: Short-lived tokens instead of long-lived access keys ## Backup Strategy | Schedule | Retention | Cold Storage | |-----------------------------|-----------|---------------| | Daily (5:00 AM UTC) | 35 days | - | | Weekly (Sunday 5:00 AM UTC) | 365 days | After 30 days | Continuous backup is enabled for point-in-time recovery. ## Log Lifecycle Access logs transition through storage tiers: | Age | Storage Class | |------------|---------------| | 0-30 days | Standard | | 30-60 days | Standard-IA | | 60-90 days | Glacier IR | | 90+ days | Expired | ## Outputs ### Cache Stack Outputs | Output | Description | |--------------------------|--------------------------------------------------| | `BucketName` | Name of the cache S3 bucket | | `BucketArn` | ARN of the cache S3 bucket | | `LogBucketName` | Name of the logs S3 bucket | | `LogBucketArn` | ARN of the logs S3 bucket | | `DistributionDomainName` | CloudFront domain name | | `DistributionId` | CloudFront distribution ID | | `DistributionArn` | CloudFront distribution ARN | | `BackupVaultName` | AWS Backup vault name | | `BackupVaultArn` | AWS Backup vault ARN | | `BackupPlanId` | AWS Backup plan ID | | `UploadPolicyArn` | ARN of the managed policy granting upload access | ### Upload Role Outputs | Output | Description | |-------------------|----------------------------------------------------------| | `RoleArn` | ARN of the upload role (pass to GitLab CI as `ROLE_ARN`) | | `RoleName` | Name of the upload role | | `OIDCProviderArn` | ARN of the GitLab OIDC identity provider | ## References - [GitLab: Configure OIDC in AWS](https://docs.gitlab.com/ci/cloud_services/aws/) - [AWS: Setting up OIDC with GitLab CI/CD](https://aws.amazon.com/blogs/apn/setting-up-openid-connect-with-gitlab-ci-cd-to-provide-secure-access-to-environments-in-aws-accounts/) - [AWS: Obtain OIDC provider thumbprint](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html) -------------------------------------------------------------------------------- # Red Hat Catalog UAT Program url: https://hummingbird-project.io/docs/background/tools/redhat-catalog-uat/ User Acceptance Testing (UAT) for the Red Hat Catalog SPA — human sign-off on staging before production promotion. **Epic:** [HUM-2179](https://redhat.atlassian.net/browse/HUM-2179) · **Story:** [HUM-2183](https://redhat.atlassian.net/browse/HUM-2183) · **Depends on:** [HUM-2060](https://redhat.atlassian.net/browse/HUM-2060) (staging host + promotion gates) ## Purpose UAT complements automated validation (smoke, E2E, visual regression, security scans). It catches product, content, and UX issues that automation does not cover. UAT does **not** replace automated gates. | Layer | Owner | Blocks prod? | | ----- | ----- | ------------ | | Unit / lint / MR security | CI | MR merge | | Staging smoke + visual | CI | Manual prod job availability | | **UAT sign-off** | Product + design + eng | **Manual prod deploy** | | Post-prod smoke | CI / on-call | Rollback decision | ## Environments (disambiguation) | Term | Meaning | URL pattern | | ---- | ------- | ----------- | | **Staging** | Pre-prod validation target for UAT | `https://images.staging.hummingbird-project.io` | | **Experimental host** | Infra sandbox for `experiment/*` branches | `https://images.experimental.hummingbird-project.io//` | | **Former `/experimental` route** | Removed in-app product route (HUM-2171) | Returns **404** — use `/api` instead. See [archived-surfaces.md](../redhat-catalog/docs/archived-surfaces.md) | See [Red Hat Catalog Environment Promotion](redhat-catalog-environments.md) for full promotion paths. ## Promotion gates ### Staging → production (full UAT) All of the following are required before clicking `deploy_redhat_catalog`: 1. `deploy_redhat_catalog_staging` succeeded on `main` 2. `redhat-catalog-e2e-smoke` passed against staging 3. `visual_redhat_catalog_staging` passed (pre/post deploy diff) 4. MR-blocking security jobs clean (or approved exceptions) — see [security-scanning.md](../redhat-catalog/docs/security-scanning.md) 5. **UAT sign-off** recorded (checklist + Jira comment; git record in `redhat-catalog/docs/uat-records/`) The prod deploy job is manual in GitLab; the release owner must verify UAT before triggering it. ### Light UAT on experimental host Experimental hosts are for iteration, not customer release. When opening an MR from `experiment/` to `main`, use a **light UAT** pass on the experimental URL — not the full checklist: | Check | Required? | | ----- | --------- | | Feature under test works on experimental host | Yes | | No console errors on primary journey | Yes | | Masthead/footer not broken | Yes | | Full staging UAT checklist | No — runs on staging after merge | | Named approver sign-off | Eng owner only (no prod gate yet) | After merge, staging receives the **full UAT** before prod. ## Named approvers (staging → prod) Confirm names with the product owner during checklist review (HUM-2060 open question on approval owners). Until confirmed, use these roles: | Role | Primary | Backup | Responsibility | | ---- | ------- | ------ | -------------- | | **Product** | _TBD_ | — | Catalog journeys, copy, feature completeness | | **Design** | _TBD_ | — | Visual polish, layout, theme, responsive UX | | **Engineering release** | _TBD_ | _TBD_ | Pipeline green, staging commit matches intent, prod deploy authorization | Any **Block** from product or design stops prod promotion until defects are fixed on staging and UAT re-run. ## Runbook — request and run UAT ### 1. Request UAT (release owner) When `main` pipeline completes staging deploy + automated gates: 1. Create or reuse a **Release** ticket in Jira (label `redhat-catalog`, link HUM-2179 epic). 2. Comment with pipeline URL, commit SHA, and summary of user-facing changes. 3. Label ticket `uat-requested`. 4. Notify approvers (#hummingbird or team channel — use team default). ### 2. Execute UAT (approvers / delegates) 1. Open [uat-checklist.md](../redhat-catalog/docs/uat-checklist.md) against staging. 2. File results in `redhat-catalog/docs/uat-records/YYYY-MM-DD-.md`. 3. Log defects as Jira bugs linked to the release ticket. **Expected turnaround:** 2 business days from `uat-requested` for routine releases; same-day for hotfixes when approvers are available. ### 3. Sign off or block - **Pass:** All checklist items pass or have accepted exceptions; all three approver rows signed; Jira comment posted (template below); label `uat-pass`. - **Fail:** Label `uat-fail`; do **not** trigger `deploy_redhat_catalog` until staging is fixed and UAT re-run. ### 4. Promote to production (engineering release) 1. Verify Jira release ticket has `uat-pass` and linked sign-off record. 2. In GitLab `main` pipeline, run manual job `deploy_redhat_catalog`. 3. Comment on release ticket with prod pipeline URL. ## What blocks release | Blocker | Who resolves | | ------- | ------------ | | Staging smoke or visual CI failed | Engineering | | Open Critical/High security finding without exception | Engineering + security | | UAT checklist failure (product/design) | Engineering fixes → re-UAT | | Missing approver sign-off | Approver | | Staging commit ≠ intended release SHA | Release owner | ## Jira sign-off template Post on the release ticket when UAT completes: ```text h3. UAT sign-off — staging → prod *Staging URL:* https://images.staging.hummingbird-project.io *Commit:* {full SHA} *Pipeline:* {GitLab pipeline URL} *Checklist record:* redhat-catalog/docs/uat-records/{filename}.md ||Role||Name||Decision|| |Product|{name}|Approve / Block| |Design|{name}|Approve / Block| |Engineering release|{name}|Approve / Block| *Result:* Approved for prod / Blocked *Defects:* {HUM-xxx links or "none"} *Tester:* {name}, {date} ``` ### Suggested Jira labels | Label | Meaning | | ----- | ------- | | `uat-requested` | Staging ready; waiting for human validation | | `uat-pass` | Full sign-off complete | | `uat-fail` | Blocked; defects logged | | `uat-light` | Experimental → staging spot-check only | Optional custom field (if added in Jira project settings): **UAT status** (`Not started` / `In progress` / `Passed` / `Failed`). ## Release ticket template Create a Jira **Task** or **Release** issue per prod promotion: ```markdown ## Release summary - **Target:** staging → production - **Commit:** - **User-facing changes:** ## Automated gates - [ ] Staging deploy green - [ ] Smoke passed - [ ] Visual regression passed - [ ] Security scans clean (or exceptions documented) ## UAT - [ ] Checklist completed ([uat-checklist.md](link)) - [ ] Sign-off record filed ([uat-records/](link)) - [ ] Product approve - [ ] Design approve - [ ] Engineering release approve ## Prod promote - [ ] deploy_redhat_catalog triggered - [ ] Post-deploy verification ``` ## Related documentation - [UAT checklist](../redhat-catalog/docs/uat-checklist.md) - [UAT records](../redhat-catalog/docs/uat-records/) - [Environment promotion](redhat-catalog-environments.md) - [Testing guide — promotion gates](../redhat-catalog/docs/testing-guide.md#promotion-gates-hum-2060) ## Out of scope - Automating UAT (Playwright covers regression; UAT stays human) - Legal/compliance sign-off (separate security program) -------------------------------------------------------------------------------- # Hummingbird MR Collaboration url: https://hummingbird-project.io/docs/background/tools/hummingbird-mr-collaboration/ How to open merge requests and help on someone else's MR in the `redhat/hummingbird/tools` monorepo. Applies to all components (including `redhat-catalog/`). ## Features - **Same-repo branches** — contributors work in the shared GitLab project, not personal forks for day-to-day changes - **One MR per change** — seniors push fixes to the author's source branch instead of opening a nested MR - **Automatic CI** — each push to the MR branch re-runs the pipeline, GitLab Pages preview, and dashboard status links ## Prerequisites - GitLab account with **Developer** (or higher) access on `redhat/hummingbird/tools` - Git clone of the monorepo - Optional: [glab](https://gitlab.com/gitlab-org/cli) CLI authenticated to GitLab ## Git remotes Many developers use two remotes: | Remote | Typical URL | Use for | |------------|-----------------------------------------------|------------------------------------| | `upstream` | `git@gitlab.com:redhat/hummingbird/tools.git` | Fetch MRs, push to shared branches | | `origin` | Personal fork (if configured) | Optional; not used for team MRs | Add `upstream` if missing: ```bash git remote add upstream git@gitlab.com:redhat/hummingbird/tools.git git fetch upstream ``` > **Do not open fork MRs** for routine team work. [mr-auto-approver](mr-auto-approver.md) > unconditionally rejects MRs where `source_project_id != target_project_id`. ## Open an MR (author) ```bash git clone git@gitlab.com:redhat/hummingbird/tools.git cd tools git checkout -b hum-XXXX-short-description # edit, commit git push -u upstream hum-XXXX-short-description ``` Open an MR targeting `main` in the GitLab UI. Branch naming: prefer `hum-XXXX-*` or a descriptive `feat/*`, `fix/*`, or `experiment/*` prefix (see [Environment Promotion](redhat-catalog-environments.md) for `experiment/*` deploy behavior). ## Help on someone else's MR (reviewer) Use the MR **source branch name** from the GitLab MR page (not necessarily your local checkout name). ### 1. Check out the MR locally **Option A — fetch by MR number** (works without glab): ```bash git fetch upstream merge-requests//head:mr- git checkout mr- ``` **Option B — checkout the source branch**: ```bash git fetch upstream git checkout -B upstream/ ``` **Option C — glab**: ```bash glab mr checkout --repo redhat/hummingbird/tools ``` ### 2. Edit and verify locally Component-specific checks (example for Red Hat Catalog): ```bash cd redhat-catalog npm run ci-checks npm run start:dev # optional UI smoke test ``` ### 3. Push to the author's branch If your **local branch name matches** the MR source branch: ```bash git push upstream ``` If you checked out via `mr-` (local name differs from remote), push explicitly: ```bash git push upstream HEAD: ``` Example: local branch `mr-748`, remote source branch `feat/image-request-form`: ```bash git push upstream HEAD:feat/image-request-form ``` Optional — rename locally so future pushes are simpler: ```bash git branch -m mr- git push -u upstream ``` The existing MR updates; CI re-runs; no second MR is needed. ## What runs on each MR push For `redhat-catalog/**` changes, see [Red Hat Catalog Environment Promotion — Pipeline Flow](redhat-catalog-environments.md#pipeline-flow). Summary: | Stage | Result | |---------|------------------------------------------------------------------------| | Test | Build, lint, type-check, unit tests, audit, SAST | | Visual | Mock visual diff (MR head vs merge-base) | | Preview | GitLab Pages at `/mr-{IID}` | | Status | Internal MR note with Hummingbird dashboard link | | Agent | Code review on open/update; `/hummingbird analyze-failures` on CI fail | After merge to `main`, staging deploys automatically; production requires manual gate and UAT sign-off. ## Troubleshooting ### `error: src refspec does not match any` Git has no **local** branch with that name. You are probably on `mr-` while pushing `git push upstream feat/...`. **Fix:** push the current branch to the remote source branch: ```bash git push upstream HEAD: ``` ### Push rejected / permission denied - Confirm **Developer+** access on `redhat/hummingbird/tools` - Confirm you are pushing to `upstream`, not a personal fork - Check whether the MR author enabled "Prevent pushing to source branch" (uncommon) ### `fatal: couldn't find remote ref merge-requests/.../head` Fetch from `upstream`, not `origin`: ```bash git fetch upstream merge-requests//head:mr- ``` ## Team tips | Tip | Why | |----------------------------------------------|--------------------------------------| | Assign yourself when taking over pushes | Clear ownership on the MR | | Use MR Pages preview | Review UI without local dev | | Use dashboard link in MR internal note | Per-commit Konflux/build status | | Use GitLab suggestions | Small fixes without a local checkout | | Use `experiment/` for long exploration | Live host before opening an MR | ## Related documentation - [Red Hat Catalog Environment Promotion](redhat-catalog-environments.md) - [MR Auto-Approver](mr-auto-approver.md) — fork rejection and approval rules - [Hummingbird Dashboard — Merge Requests](hummingbird-dashboard.md#merge-requests) - [Hummingbird Agent](hummingbird-agent.md) — `/hummingbird` MR commands -------------------------------------------------------------------------------- # K8s Integration Tests url: https://hummingbird-project.io/docs/background/tools/k8s-integration-tests/ Smoke tests that verify tools container images work in a real Kubernetes cluster. Each test creates short-lived pods, checks expected behavior, and cleans up. Tests run automatically on every MR via the Konflux [K8s test pipeline](https://hummingbird-project.io/l/k8s-test-pipeline-design). ## Component Inventory | Component | Test file | What it tests | | ---------------------------- | --------------- | ----------------------------------------------- | | `container-catalog` | `tests-k8s.yml` | Bootstrap `--help`, grype/syft present | | `gitlab-ci` | `tests-k8s.yml` | 13 tools present (shellcheck, go, kubectl, ...) | | `gorget` | `tests-k8s.yml` | Version output, go/node/rust/composer present | | `hummingbird-agent` | `tests-k8s.yml` | CLI `--help`, Python imports, kubectl present | | `hummingbird-cve-analysis` | `tests-k8s.yml` | Python imports, git/rpm present | | `hummingbird-dashboard` | `tests-k8s.yml` | Starts and serves on port 8080 | | `hummingbird-status` | `tests-k8s.yml` | Python import | | `hummingbird-tools` | `tests-k8s.yml` | Python imports, psql/dnf present | | `service-metrics` | `tests-k8s.yml` | Python import | | `kubernetes-event-forwarder` | `tests-k8s.yml` | Python import | | `playwright-test` | `tests-k8s.yml` | Node/npm/xvfb-run present | Components without tests: `hummingbird-agent-vm-sandbox` (bootc VM image, not a container workload), `hummingbird-agent-vm-sandbox-disk` (qcow2 extraction build, not a runtime image). ## Test Runner Tests are executed by [`ci/run_tests_k8s.sh`](../ci/run_tests_k8s.sh). For each component it: 1. Reads `{component}/tests-k8s.yml` 2. Parses each test entry (YAML -> JSON via Python) 3. Runs the `command` block with `kubectl` wired to the target cluster 4. Cleans up labeled resources (`hum-k8s-test=`) after each test ## Running Locally Test against a local cluster (kind, minikube, or remote): ```bash # Single component with a published image IMAGE_URL=quay.io/hummingbird-ci/gitlab-ci:latest \ IMAGE_NAME=gitlab-ci--tools \ ci/run_tests_k8s.sh --context kind-kind gitlab-ci--tools # Single component with a locally-built image podman build -t localhost/hummingbird-dashboard:dev hummingbird-dashboard/ kind load docker-image localhost/hummingbird-dashboard:dev IMAGE_URL=localhost/hummingbird-dashboard:dev \ IMAGE_NAME=hummingbird-dashboard--tools \ ci/run_tests_k8s.sh --context kind-kind hummingbird-dashboard--tools ``` ## CI Integration In Konflux, the `tools-k8s-test` [IntegrationTestScenario](https://hummingbird-project.io/l/k8s-test-pipeline-design) triggers on every MR. The pipeline: 1. Checks whether `tests-k8s.yml` exists for the changed component 2. Provisions an ephemeral [EaaS namespace](https://hummingbird-project.io/l/k8s-test-eaas) 3. Runs `ci/run_tests_k8s.sh` with the built image Components without `tests-k8s.yml` skip the test (the pipeline exits early after the check step). ## Adding Tests for a New Component 1. Create `{component}/tests-k8s.yml` with one or more named tests: ```yaml --- smoke-test: command: | name="${TEST_GROUP}-smoke-${TEST_RUN_ID}" kubectl run "${name}" --image="${TEST_IMAGE:?}" --restart=Never \ --labels="${TEST_RUN_LABEL}" \ --command -- echo "hello" kubectl wait --for=jsonpath='{.status.phase}'=Succeeded \ "pod/${name}" --timeout=120s || test_fail "Pod did not succeed" kubectl logs "${name}" ``` 2. Use `${TEST_IMAGE}` for the image under test, `${TEST_RUN_ID}` and `${TEST_RUN_LABEL}` for unique naming and cleanup. 3. Call `test_fail "message"` to fail a test with a clear message. 4. Keep tests fast (under 2 minutes) — these are smoke tests, not integration suites. ### Common pitfalls - All lines in `command: |` block scalars must be indented relative to the `command` key. Unindented lines break YAML parsing. - Use `command -v` instead of `which` to check for commands — `which` is not available in all container images. - For multi-statement Python one-liners, use semicolons on a single line: `python3 -c 'import foo; import bar; print("ok")'` ## Debugging Failures See the [EaaS and Debugging](https://hummingbird-project.io/l/k8s-test-eaas) guide for accessing ephemeral namespaces and using Kubearchive for historical PipelineRun data. -------------------------------------------------------------------------------- # Agentic SDLC url: https://hummingbird-project.io/docs/agentic-sdlc/ description: How Project Hummingbird uses agentic AI to drive velocity across the software development lifecycle. Project Hummingbird's core mission is velocity: shipping RPM updates, CVE fixes, and new container images as fast as possible. The pipeline already delivers at a scale unprecedented within Red Hat — consistently over 1,000 automated commits per week across the [`containers`][containers-repo] and [`rpms`][rpms-repo] repositories, a throughput that would be impossible to sustain manually — work that in most Linux distributions is still performed manually by dedicated teams. Managing 400+ packages and 1,750+ image builds with a 24-hour security SLO, that pipeline is not a nice-to-have — it is what makes the project viable with a small team. The pipeline solves throughput. What it does not solve is the full software development lifecycle: assessing what to build, implementing it, documenting it, testing it, reviewing UX, coordinating releases. For a small team maintaining hundreds of containers and building an entire OS, staffing all of those functions manually does not scale — and agentic AI is what changes that equation. Where rule-based automation handles the deterministic, agents take on everything that has historically required a human to pick up, think through, and act on. Hummingbird is building toward becoming an open source reference implementation for responsible agentic AI in software engineering: demonstrating through production use, not promises, that this kind of automation can be adopted incrementally and grounded in engineering discipline. ## The Agentic SDLC Vision The target state is a development pipeline driven by agents at every stage — with humans engaged where their judgment is needed, not where their time is consumed by routine work. Every change flows through the same lifecycle: **1. Request.** A customer, product team, or automated trigger submits a change: a new container image, an RPM update, a feature request, a bug fix. **2. Analysis.** A **product management agent** assesses the request — evaluating feasibility, priority, and fit against existing components and conventions. For straightforward requests it proceeds automatically; for complex or ambiguous ones it surfaces the relevant context and flags the case for human input before continuing. This is the work that normally sits in a product manager's queue. **3. Implementation.** The request splits based on its nature: - *Deterministic changes* — dependency updates, lockfile syncs, rebuilds triggered by upstream changes — are implemented by rule-based automation and **merged automatically** once CI passes. No ticket, no approval, no delay. - *Everything else* is handled by an **engineering agent**: a developer co-programming with an agent, or an agent working autonomously to write the code or configuration, run validation, and open a merge request. Agentic AI is the default answer for everything that has historically required a human to pick it up, think through it, and act on it. - Specialist agents contribute alongside: a **tech writer agent** drafts or updates the documentation and long-form Red Hat product content; a **testing agent** generates or extends tests; and a **user design agent** applies UX patterns and practices to any interface-related changes. **4. Review.** Automated agents check for policy compliance and code quality, including the production [code review agent][code-review-workflow]. The same agent capabilities that contribute during implementation also apply here: a **tech writer agent** reviews documentation changes for clarity and correctness; a **testing agent** reviews test coverage and flags gaps; and a **user design agent** reviews UX-related changes against established patterns. A human engineer makes the final call on all of it. No agent-produced work is merged without human approval — the human is not eliminated, but elevated to the role where judgment actually matters. **5. Merge.** The change ships. The goal is maximum automation, pursued pragmatically rather than dogmatically — the right tool for each kind of work. Where agentic AI excels is in the space just before human judgment is required: gathering context, surfacing options, resolving the routine, so that when a human does engage, the decision in front of them is sharp, informed, and fast. ## Responsible AI Agent-produced code and decisions require human approval before merge. This is not a temporary constraint — it is a design principle. Humans may stay in the loop to steer or closely supervise complex work when that helps, but the standing requirement is approval — not constant supervision of every agent run. The feedback between human judgment and agent behavior is what keeps the system trustworthy over time. The high-velocity automated pipeline — over 1,000 commits per week — operates through rule-based automation and does not involve agents; no human review is required there. The failure analysis agent does run at high volume — triggering on every pipeline, build, or test failure — but it produces analysis and comments, not code changes requiring approval. That alone saves hundreds of minutes per week of manual log triage. Agent work that requires human approval is scoped to SDLC tasks: feature requests, documentation, code review — a much lower volume where the requirement is entirely manageable. ### Security Agents run in containers — locally via Podman, in production on OpenShift. This is not a novel security model: it is the same approach the industry has long used for untrusted code, applied consistently. The Hummingbird agent platform is architecturally designed to ensure agents have no access to secrets or tokens. That single constraint effectively neutralizes prompt injection as a threat: an agent that cannot access credentials or act on external systems is, in practice, just another contributor — and the review process treats it as such. The same gatekeeping that applies to any third-party contribution applies to agent-produced work. The worst-case outcome is a low-quality merge request — and that is precisely what the review process, including the review agents themselves, is designed to catch. ### Learning from Feedback Agents learn from feedback — and this is already working in production. When a reviewer comments on an agent-produced merge request, the agent reads those conversations, incorporates the feedback, and updates its work. It avoids re-raising issues already addressed and respects when a developer has explained an intentional design choice. That feedback is also preserved — stored so that future agent runs can draw on it, improving with each iteration rather than repeating the same mistakes — an approach grounded in the frameworks established in Pascal Bornet's [*Agentic Artificial Intelligence*][bornet-book]. ## From Pair Programmer to Autonomous Agent Deploying production agents requires a foundation. The approach that works is not to automate first and document later — it is the reverse. **Phase 1 — Agent as pair programmer.** Agents assist individual developers: drafting code, explaining unfamiliar systems, proposing changes for review. This phase builds understanding of what agents need: context, conventions, constraints. **Phase 2 — Documentation as fuel.** Agent output quality is directly proportional to documentation quality. Every repository carries an `AGENTS.md` — a file with instructions targeting agents specifically, but one that mostly points to other documentation rather than duplicating it. That underlying documentation is written for humans and agents alike: clear, specific, and rich with rationale. The `AGENTS.md` is the entry point; the broader docs are the substance. An agent with good documentation reasons from the same foundation as the team; an agent without it guesses. **Phase 3 — Agents in production.** With the foundation in place, agents move into the critical path: first as reviewers proposing concrete changes, then as autonomous contributors on well-defined tasks. The next section describes what that looks like in practice for Hummingbird today. An agent running autonomously in production is not fundamentally different from a developer running an agent during pair programming. The same agent, the same context, the same documentation — the difference is who initiates the run and how much human supervision follows. If an agent consistently does good work when a developer is in the loop, that is a strong signal the project is ready to let it operate more independently. The quality of pair programming performance is the leading indicator for autonomous deployment readiness. Trust is built incrementally, one well-handled task at a time. See [From Pair Programmer to Autonomous Agent][vrothberg-article-linkedin] by Valentin Rothberg for the full treatment. ## Documentation as Infrastructure ### Writing Documentation with Agents In an agentic system, documentation is infrastructure — the primary interface between the team and its agents, and the quality of what an agent produces is bounded by the quality of what it can read. That makes writing documentation with agent assistance not a separate task but a natural extension of the work: the same agent that helps an engineer write and review code can help write the documentation for it — capturing decisions, explaining rationale, drafting contribution guides. This is not a shortcut; it is the same pair-programming model applied to a different kind of output. This is how Hummingbird built its documentation from the start: engineers worked alongside agents to draft and improve it in the same way they wrote code — iteratively, collaboratively, with humans in the lead. The result is a virtuous cycle: better docs make agents more capable, and more capable agents help produce better docs. Tech writers are not a prerequisite — engineers writing clear, intentional documentation with agent assistance is sufficient to start. But their skills become more valuable in this model, not less: precision over generality, rationale alongside rules, explicit anti-patterns. A tech writer contributing to an agentic codebase is directly shaping the behavior of the agents that work within it. In Hummingbird, this is not theoretical. The [`containers`][containers-repo] repository is developed in active collaboration with Red Hat tech writers, whose customer-facing product documentation lives in the same repository as the code — keeping internal conventions, agent context, and customer-facing content in one place, shared across engineers, tech writers, and agents alike. ### Knowledge Transfer Across Teams The same principle — knowledge encoded in markdown, readable by humans and agents alike — also answers a broader question: how does one team's expertise become available to agents maintained by another? The answer is the same as everything else in this system — markdown. A UX team encodes their patterns and design practices in a skills file; the user design agent picks it up. The documentation team maintains their style guide, standards, and best practices — including Red Hat's documentation guidelines — encoded once and available to every agent that produces or reviews written content. No code changes, no integrations, no handoff meetings. Any team can contribute their domain knowledge to the agentic system simply by writing it down in a form that both humans and agents can read. The markdown file is the interface between teams. ## Agents in Production Today Two agents are running in production today, built on the [Hummingbird Agent][hummingbird-agent-docs] platform. They operate on every merge request across the [`containers`][containers-repo] and [`rpms`][rpms-repo] repositories, which together see over 1,000 automated commits per week — a throughput that would be impossible to review and maintain manually without agent assistance. More agents are under active development in parallel, with production deployment planned after Red Hat Summit 2026 — likely by end of Q2. One example is the **merge conflict agent** — which will automatically resolve merge conflicts that arise during automated RPM updates. The full roadmap is tracked in the [HUM-687 epic][hum-687]. ### Failure Analysis Agent When a merge request pipeline fails, this agent automatically investigates and posts its findings directly on the merge request. It pulls data from multiple sources — GitLab CI job logs, Konflux build and test pipeline runs, Testing Farm test results — and correlates them to identify root causes. Where the same underlying issue causes multiple failures across different images or architectures, it groups them so engineers see patterns rather than an overwhelming list of individual failures. Every finding includes links to the specific logs and artifacts so engineers can dig deeper without having to hunt. What previously required clicking through multiple levels of web UI, downloading logs, and working through thousands of lines of output to find a single root cause now happens automatically. It is not rocket science — but it is some of the most time-consuming and draining work an engineer can be asked to do repeatedly. *Workflow definition: [`analyze-failures.md`][failure-analysis-workflow]* ### Code Review Agent When a merge request is opened, this agent performs a technical code review with the priorities of a senior engineer: security first, then correctness, then performance, then maintainability. It provides specific line references and concrete code examples for every issue it raises — not vague suggestions, but actual fixes. It reads prior review discussions before posting, avoids re-raising issues already addressed, and respects when a developer has explained an intentional design choice. The result is consistent, high-quality review coverage on every merge request, with human reviewers free to focus on the issues that genuinely need their judgment. The team is already pleased with the quality of the reviews — and this is only the beginning. A false positive costs nothing: a human reviewer reads it, disagrees, and moves on. The design ensures that bad agent output has no blast radius. And as human reviewers comment on agent-produced work, the agent learns and incorporates that feedback — improving with every iteration. *Workflow definition: [`code-review.md`][code-review-workflow]* The near-term impact is measured in cycle time. A feature request that previously took weeks to move from intake through product assessment, implementation, documentation, and review can be compressed to days, sometimes hours. That changes what a small team can realistically take on. The longer-term picture is equally straightforward: the system already delivers value today. If agents continue to improve, Hummingbird benefits more. If they plateau, the team continues to operate at a level that would otherwise require far more people. The infrastructure built now is not a bet on a technology that might not materialize — it is an investment that is already paying off, with optionality for more. [hum-687]: https://redhat.atlassian.net/browse/HUM-687 [containers-repo]: https://gitlab.com/redhat/hummingbird/containers [rpms-repo]: https://gitlab.com/redhat/hummingbird/rpms [hummingbird-agent-docs]: https://hummingbird-project.io/docs/background/tools/hummingbird-agent-design/ [failure-analysis-workflow]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/hummingbird-agent/workflows/analyze-failures.md [code-review-workflow]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/hummingbird-agent/workflows/code-review.md [vrothberg-article-linkedin]: https://www.linkedin.com/pulse/from-pair-programmer-autonomous-agent-how-actually-adopt-rothberg-qdyqe/ [bornet-book]: https://pascalbornet.com/#bestseller -------------------------------------------------------------------------------- # Architecture Decision Records url: https://hummingbird-project.io/docs/adrs/ Architecture Decision Records (ADRs) capture significant technical decisions made across the Hummingbird project, along with their context and consequences. ## Records | ID | Title | Status | Date | |----------|--------------------------------------------------------------------|-------------|------------| | ADR-0001 | [Unified Observability Stack](0001-unified-observability-stack.md) | Implemented | 2026-07-19 | | ADR-0002 | [AI-Assisted SDLC Workqueue](0002-ai-assisted-sdlc-workqueue.md) | Proposed | 2026-08-24 | -------------------------------------------------------------------------------- # ADR-0001: Unified Observability Stack url: https://hummingbird-project.io/docs/adrs/0001-unified-observability-stack/ - **Status:** Implemented - **Date:** 2026-07-19 (updated 2026-08-19) - **Author:** Robert Sturla - **Jira:** [HUM-4790][hum-4790] (Epic), [HUM-4791][hum-4791] (Design task) ## Context Hummingbird had no single place to view the health of builds, infrastructure, and services. Multiple disconnected monitoring and alerting implementations existed — CloudWatch dashboards, a custom `cloudwatch-log-forwarder` Lambda, an SNS `AlertsTopic` for consumer CloudWatch alarms, and per-service custom dashboards. Workloads span multiple clusters with no cross-cluster visibility. This created blind spots, duplicated effort, and inconsistent alerting behaviour. One standardised observability stack is needed. No duplication. One toolchain, used everywhere, federated across clusters from day one. Operational runbooks live in `documentation/monitoring.md` in the [infrastructure repository][infrastructure]. This ADR records the architecture decision. ### Cluster Constraints Hummingbird has no cluster-admin access on any cluster. All resources must be deployed within tenant-provisioned namespaces. Operator CRDs and RBAC grants are controlled by the cluster platform team — what is available differs between clusters. This constraint drives a **hub-spoke split architecture**: the hub (MPP) runs the backends and Grafana, while spoke clusters run lightweight collectors that forward all telemetry to the hub. ## Decision ### Stack **Hub (MPP, `hummingbird--monitoring-hub`):** | Signal | Backend | Storage | |---------------|---------------------------------|----------------------------------| | Metrics | Mimir (monolithic) | S3 via ObjectBucketClaim | | Logs | Loki (monolithic) | S3 via ObjectBucketClaim | | Collection | Alloy Hub (YACE + textfile) | N/A | | Visualisation | Grafana | N/A | | Alerting | Mimir ruler + Alertmanager | N/A | **Spoke (every cluster with `hummingbird--monitoring`):** | Signal | Backend | Storage | |---------------|----------------------------------------------------|---------| | Metrics | Alloy `prometheus.scrape` → hub Mimir remote-write | N/A | | Logs | Alloy `loki.source.kubernetes` → hub Loki | N/A | | Collection | Alloy + kube-state-metrics | N/A | | Visualisation | None (use hub Grafana) | N/A | Spoke clusters have no local backends. All storage, alerting, and visualisation is centralised on the hub. Each spoke forwards to both the staging and production hubs. Distributed tracing (Tempo) is not deployed. See [Future Work](#future-work). ### Alerting Architecture Mimir evaluates PromQL alerting rules (ruler) and delivers notifications through its built-in Alertmanager. Grafana is not in the evaluation path. ```text Mimir ruler (PromQL) → Mimir Alertmanager → Slack / email ``` - Rules are Git-provisioned in the infrastructure repository and mounted into Mimir as a ConfigMap. - All alerting runs on the hub. Spoke clusters do not run Alertmanager. Spoke alerting depends on network connectivity to the hub — if a spoke loses connectivity, alerts for that cluster are delayed until the connection is restored. This is an accepted tradeoff given the operator constraints on spoke clusters (see [Cluster Constraints](#cluster-constraints)). - Staging Alertmanager receivers are empty (a blackhole) so staging does not page Slack or email. ### Alert Routing | Severity | Channel | |-----------------|---------------| | `critical` | Slack + email | | `info` | Slack | | everything else | Slack + email | Production only; staging is a blackhole. ### Deployment Model The hub runs the backends, Grafana, and Alloy Hub. Spoke clusters run Alloy and kube-state-metrics. **Hub (MPP):** - **Namespace:** dedicated `hummingbird--monitoring-hub` namespace (tenant-provisioned) - **Tenant-scoped:** all resources deployed in namespaces owned by Hummingbird, no cluster-admin dependency - **Mimir (monolithic):** accepts Prometheus remote-write from spoke Alloy and Alloy Hub, stores TSDB blocks in S3, serves PromQL, and runs the ruler and Alertmanager. Grafana queries Mimir in-cluster (no nginx basic auth on the Service port). - **Loki (monolithic):** receives logs from spoke Alloy via authenticated Routes. Grafana queries Loki in-cluster. - **Alloy Hub:** a dedicated Alloy instance in the hub namespace. CloudWatch metrics are ingested with `prometheus.exporter.cloudwatch` (YACE), filtered to Hummingbird resources (`app-code=RPRM-001`). Credential-expiry metrics are ingested with the textfile collector from a CI-generated ConfigMap. Alloy Hub remote-writes to in-cluster Mimir (Service DNS; no Route or CA bundle). - **Grafana:** exposed via an internal Route. Authentication is Generic OAuth against gitlab.com (`redhat/hummingbird` group). GitLab Owner and Maintainer map to Grafana Admin, Developer to Editor, everyone else in the allowed group to Viewer. Datasources and dashboards live in `grafana_data/` and are synced with `grafana_data/deploy.sh`. Grafana also has a CloudWatch datasource for ad-hoc browsing of AWS logs and metrics; that path is query-only and is not used for alerting or pod-log forwarding. - **External access:** Loki and Mimir Routes sit behind an nginx sidecar with HTTP basic auth. In-cluster Service access (Grafana, Alloy Hub) does not require auth. **Spoke (every cluster):** - **Namespace:** `hummingbird--monitoring` - **Alloy:** collects container logs via `loki.source.kubernetes` and scrapes `prometheus.io/scrape` Services, including kube-state-metrics. Forwards logs to hub Loki and metrics to hub Mimir. Dual-writes to staging and production hubs. - **kube-state-metrics:** Kubernetes object metrics (deployments, pods, jobs, PVCs, resource quotas) with a curated allowlist. - **No local Mimir, Loki, Grafana, or Alertmanager.** ### AWS Lambda Services Some Hummingbird services (e.g. container-catalog) run as AWS Lambda functions. Observability for those services uses the existing AWS signals rather than an OpenTelemetry Lambda extension: - **Metrics:** Alloy Hub YACE scrapes curated `AWS/Lambda` metrics (Invocations, Errors, Duration, Throttles) into Mimir. Mimir alerting rules cover error rate and throttling. - **Logs:** Grafana queries CloudWatch Logs directly via the CloudWatch datasource. Pod logs are not involved; Lambda logs stay in CloudWatch. - Low-traffic utility Lambdas are covered by the same YACE allowlist when tagged `app-code=RPRM-001`. ### Log Collection Container stdout/stderr is tailed by spoke Alloy using `loki.source.kubernetes` (Kubernetes API; no DaemonSet, no sidecar, no ClusterLogForwarder). Logs are labeled with cluster, namespace, pod, container, and app, then dual-written to hub Loki. This requires no application code changes. ClusterLogForwarder is available on some clusters but is not used. ### High Availability **Hub:** | Component | Replicas | Rationale | |---------------|----------|----------------------------------------------------| | Mimir | 1 | Monolithic; brief downtime acceptable | | Loki | 1 | Monolithic; brief downtime acceptable | | Grafana | 1 | View-only; brief downtime acceptable | | Alloy Hub | 1 | Stateless scrape; gap until next scrape is OK | **Spoke:** | Component | Replicas | Rationale | |---------------------|----------|-----------------------------------------| | Alloy | 1 | Stateless forwarder; brief gap OK | | kube-state-metrics | 1 | Stateless; brief gap OK | Spoke clusters have no stateful observability components. HA concerns are concentrated on the hub. Single-replica monolithic Mimir and Loki are an accepted tradeoff against the proposed Prometheus + Thanos HA pair: there is no existing Prometheus fleet to retrofit, and Alloy already remote-writes. ### Retention | Environment | Mimir | Loki | |-------------|-------|------| | Production | 90d | 90d | | Staging | 14d | 14d | S3 is provisioned with ObjectBucketClaim. There is no separate warm/cold tier or multi-year archive. ### Operator Availability CRD availability was audited on 2026-08-09. Hummingbird has no cluster-admin access — only operator CRDs explicitly granted to tenant namespaces are usable. That audit is why this stack does **not** use Cluster Observability Operator (MonitoringStack / Thanos), LokiStack, TempoStack, ClusterLogForwarder, or an OpenTelemetry Collector as the live path: - **LokiStack and TempoStack** are not available to tenant namespaces on any cluster. Loki is a standalone Deployment. Tempo is deferred. - **COO MonitoringStack** is available on MPP but not on spoke, and would have required Prometheus + Thanos for a fleet we do not have. Mimir accepts remote-write directly from Alloy. - **ClusterLogForwarder** is available, but Alloy `loki.source.kubernetes` collects the same pod logs without a second operator. - **OpenTelemetry Collector** was deployed earlier and removed once Alloy wrote directly to Loki and Mimir. ## Relationship to Existing Dashboard The existing `hummingbird-dashboard` app serves two distinct roles: 1. **Data visualisation** — CVE status charts, failed release counts, build metrics 2. **Operational workflows** — click to rerun a failed release, interactive triage, actionable controls This observability stack takes over **data collection, storage, and visualisation** (role 1). Grafana is the primary place for charts, time-series dashboards, and alert-driven visualisations. The dashboard app retains **operational workflow features** (role 2) — interactive actions that Grafana cannot provide. The two systems complement each other: - The dashboard app can embed Grafana panels for richer visualisation without reimplementing charting - The dashboard app can query the same backends (Mimir, Loki) directly for data it needs to drive workflows - Over time, the dashboard app consumes data from the unified backends instead of maintaining parallel collection Alerting (e.g. failed release Slack notifications) moves to Alertmanager. The dashboard app no longer needs its own alerting path. ## What This Replaces | Previous | Replaced by | |------------------------------------------------|-------------------------------------------| | `cloudwatch-log-forwarder` Lambda | Spoke Alloy → hub Loki | | SNS `AlertsTopic` + consumer CloudWatch alarms | Mimir ruler + Alertmanager | | Per-service custom dashboards | Grafana dashboards | | Dashboard app data collection | Unified backends (Mimir, Loki) | | Dashboard app alerting (Slack) | Alertmanager | | Dashboard app visualisations | Grafana (embeddable in the dashboard app) | The dashboard app itself is **not decommissioned** — its operational workflow features (release reruns, interactive triage) remain. Only its data collection and visualisation responsibilities shift to the unified stack. `hummingbird-events-topic` is the live event bus. It was not an alerting channel and is **not** replaced by this stack. Consumer CloudWatch alarms and the log-forwarder were removed after the Mimir/Loki path was in place. Remaining CloudWatch use is intentional: - Grafana CloudWatch datasource for ad-hoc AWS log and metric browsing - CVE error-budget SLO dashboards and `hummingbird-slo-alerts` (no Grafana port yet) ## Operations Day-to-day endpoints, datasources, query examples, and onboarding a new cluster are documented in `documentation/monitoring.md` in the [infrastructure repository][infrastructure]. ## Future Work - **Distributed tracing:** Tempo (or equivalent) for request and pipeline traces. Not deployed; traces are not a current signal. - **CVE lifecycle tracing:** instrument CVE analysis, RPM build, and container image pipelines with spans. Trace a CVE from detection → RPM fix → image rebuild → publish. Requires a trace backend and a schema (separate story). - **Spoke local alerting:** if operator access on spoke clusters changes, a local evaluation path could remove the hub dependency for spoke alerts. Not required today. ## Alternatives Considered ### Alerting: Grafana-only vs Alertmanager-only vs hybrid - **Grafana-only:** less mature routing, dedup, and silencing. - **Hybrid Grafana + Alertmanager (original proposal):** Grafana would evaluate rules across metrics, logs, and traces, then forward to Alertmanager. That is not what shipped: there is no Tempo, and log alerting is not required yet. - **Mimir ruler + Alertmanager (chosen):** PromQL rules and delivery in one component. Matches the metrics-first rollout. Applications can still POST to the Alertmanager API later if needed. ### Metrics backend: Prometheus + Thanos vs Mimir Prometheus + Thanos (MonitoringStack) was the original hub design. It fits retrofitting an existing Prometheus fleet. Hummingbird had no such fleet — Alloy already remote-writes. Mimir is one component for receive, storage, PromQL, ruler, and Alertmanager, instead of Prometheus + Thanos Sidecar + Store Gateway + Querier + a separate Alertmanager. ### Logs: Sumo Logic vs Loki Red Hat has a company-wide Sumo Logic license. However, the [Sumo Logic Grafana plugin][sumo-plugin] requires Grafana Enterprise or Grafana Cloud — it is not available in open-source Grafana. Loki integrates natively with open-source Grafana and keeps logs on-cluster. ### Log collection: ClusterLogForwarder vs Alloy ClusterLogForwarder is available on hub and spoke. Alloy `loki.source.kubernetes` collects the same pod logs and already runs on every spoke for metrics, so a second operator was not justified. ### Spoke architecture: full stack vs collector-only - **Full stack per cluster:** each cluster runs its own backends and Grafana. Local alerting survives network partitions. Requires operator CRDs Hummingbird does not have on spoke. - **Collector-only on spoke (chosen):** Alloy + kube-state-metrics forwarding to the hub. Simpler spoke footprint. Tradeoff: spoke alerting depends on hub connectivity. ### Lambda: OTel extension vs CloudWatch ingest An OpenTelemetry Collector Lambda extension would export OTLP to the hub. That needs a network path from Lambda to MPP and a collector that no longer exists. YACE metrics into Mimir plus Grafana's CloudWatch datasource for logs reuse AWS signals already produced by the functions. ### HA: all components vs selective vs single replica Full HA doubles cost with little benefit for view-only tools. The original proposal kept HA only on Prometheus and Alertmanager. The as-built stack uses single-replica monolithic Mimir and Loki; Alertmanager is in-process with Mimir. Brief downtime is accepted. ## Consequences - Team must learn Grafana dashboarding and PromQL/LogQL basics - New hub services should expose Prometheus metrics (scrape annotation) and may add a Grafana dashboard under `grafana_data/` - New spoke services must be scraped by Alloy (scrape annotation) and emit logs to stdout/stderr - Spoke alerting depends on hub connectivity — no local alerting on spoke - Deploying a new hub cluster requires Mimir, Loki, Grafana, and Alloy Hub, plus ObjectBucketClaims - Deploying a new spoke cluster requires a `hummingbird--monitoring` namespace, Alloy, and kube-state-metrics pointing at both hubs - CloudWatch remains for Lambda logs (Grafana datasource) and CVE SLO error-budget dashboards; it is not used for pod-log forwarding or consumer alarms [hum-4790]: https://redhat.atlassian.net/browse/HUM-4790 [hum-4791]: https://redhat.atlassian.net/browse/HUM-4791 [sumo-plugin]: https://grafana.com/grafana/plugins/grafana-sumologic-datasource/ [infrastructure]: https://gitlab.com/redhat/hummingbird/infrastructure -------------------------------------------------------------------------------- # ADR-0002: AI-Assisted SDLC Workqueue url: https://hummingbird-project.io/docs/adrs/0002-ai-assisted-sdlc-workqueue/ - **Status:** Proposed - **Date:** 2026-08-24 - **Author:** Michael Hofmann - **Jira:** [HUM-6342] (spike) ## Context The [`containers`][containers-repo] and [`rpms`][rpms-repo] repositories produce over 1,000 automated commits per week. The automation that drives this throughput is spread across multiple independent mechanisms: GitLab CI scheduled jobs running shell scripts, Renovate CronJobs in the infrastructure repo, AWS Lambda functions for MR approval, and manual developer workflows run from dev machines. Each mechanism creates and manages MRs with its own lifecycle, credentials, and error handling. There is no unified system that tracks an MR from creation through approval to merge. The [Hummingbird Agent][hummingbird-agent-docs] already runs AI-powered failure analysis and code review on every MR, but cannot create or modify MRs. The security foundation for agent-authored MRs is designed in [HUM-851] but not yet implemented. This ADR documents the current state of MR automation across all repositories and entrypoints as discovery for a spike to design a unified workqueue-based system. ### Scope This document inventories what exists today. It does not prescribe the target architecture — that is the output of the spike. Design questions and patterns are included to inform the spike investigation. ## Current State ### Repositories **rpms** (`redhat/hummingbird/rpms`): RPM package definitions. Each package lives in its own directory with a `.spec` file and `metadata/.json` for upstream repo URL, CVE product mapping, and modification status. Fedora dist-git imports and upstream version bumps are the primary automated flows. **containers** (`redhat/hummingbird/containers`): Container image definitions. Each image lives in `images//` with Containerfiles and RPM lockfiles (`rpms.lock.yaml`). Lockfile refresh and Renovate dependency updates are the primary automated flows. **tools** (`redhat/hummingbird/tools`): Lambdas, CLI tools, and the Hummingbird Agent framework. Contains `mr-auto-approver`, `gitlab-event-forwarder`, `hummingbird-agent`, and the CVE analysis pipeline. **infrastructure** (`redhat/hummingbird/infrastructure`): Kubernetes manifests, AWS SAM templates, and deployment configuration. Hosts Renovate CronJob definitions and agent deployment manifests. ### MR Creation Entrypoints | Entrypoint | Repo | Trigger | What it creates | | ------------------------------------------------------ | -------------------- | --------------------------- | --------------------------------------------------------------- | | [`dist_git_update_multi_mr.sh`][rpms-dist-git] | rpms | GitLab schedule (every 4h) | One MR per package needing Fedora dist-git sync | | [`upstream_update_multi_mr.sh`][rpms-upstream] | rpms | GitLab schedule | One MR per package with a newer upstream release | | [`rebuild_multi_mr.sh`][rpms-rebuild] | rpms | Manual (operator-triggered) | One MR per package needing rebuild | | [`create_lockfile_update_mrs.sh`][containers-lockfile] | containers | GitLab schedule | One MR per image with changed RPM lockfiles | | [`ci/create_mr.sh`][containers-create-mr] (metrics) | containers | GitLab schedule (daily) | Single chore MR for metrics report | | Renovate (rpms) | rpms | K8s CronJob (hourly) | Tekton pipeline digest and Konflux release-catalog updates only | | Renovate (containers) | containers | K8s CronJob (hourly) | Dependency updates, lockfile maintenance, pipeline migration | | Renovate (tools/infra/k8s-test) | tools/infra/k8s-test | K8s CronJob (hourly) | Dependency updates | | CVE analysis (`advisory_handler`) | CEE advisories | K8s CronJob (30 min) | Advisory MRs; handles rebase on conflicts | | `/cve` skill | rpms | Manual (Cursor/Claude) | Fix MRs from staged CVE analysis data in Jira | | `gitlab_sync` | tools | Various | File sync MRs with poll, rebase, and best-effort approve | | Manual dev workflows | rpms/containers | Human on dev machine | Package updates, CVE fixes, spec changes | ### MR Creation Mechanism: Git Push Options All shell-script-based MR creation uses **git push options**, not the GitLab Merge Requests REST API. The push options control MR creation, title, description, labels, draft status, and merge-when-pipeline-succeeds (MWPS): ```text git push \ --push-option merge_request.create \ --push-option "merge_request.title=chore(rpms): Update foo to 1.2.3" \ --push-option merge_request.remove_source_branch \ --push-option merge_request.merge_when_pipeline_succeeds \ origin HEAD:refs/heads/chore/upstream-update-foo ``` Approval is always a separate step — never part of the creation scripts. ### MR Lifecycle and Approval Flows #### GitLab Project Settings (containers and rpms) | Setting | Value | | ------------------------------ | ----------------------------------------------------------------- | | Approvals required | 1 (`any_approver`); author cannot self-approve | | Pipeline must succeed | Yes | | Discussions resolved | Required | | Jira required | Yes (`prevent_merge_without_jira_issue`) | | Merge trains / merge pipelines | Off (Konflux external statuses must stay blocking) | | Merge method | Merge commit | | Auto-merge | MWPS via `merge_request.merge_when_pipeline_succeeds` push option | #### Bot Identities | Role | Identity | Token type | | ------------------------- | -------------------------- | ------------------------------ | | Create chore MRs | `chore-mr` | Project access token | | Approve chore MRs (CI) | `chore-mr-approval` | Separate project access token | | Create lockfile MRs | `lockfile-update` | Project access token | | Approve lockfile MRs (CI) | `lockfile-update-approval` | Separate project access token | | Renovate + Konflux PAC | `pipelines-as-code` | MintMaker token | | Lambda approvals | Per-project tokens | Separate project access tokens | #### Approval Paths Three parallel approval mechanisms exist, with overlapping scope: **1. CI approval jobs** — delayed jobs in each repo's `.gitlab-ci.yml`: - rpms `chore_mr_approval`: gates on `check_konflux_statuses.py` (polls every 60s up to 2 hours for Konflux statuses to appear and succeed), then approves with `CHORE_MR_APPROVAL_GITLAB_TOKEN`. Triggered for author `chore-mr` or `pipelines-as-code` on `chore/*` branches. - containers `chore_mr_approval`: same author/branch gate, approves with `CHORE_MR_APPROVAL_GITLAB_TOKEN`. No Konflux polling. `allow_failure: true`. - containers `lockfile_update_approval`: triggered for `pipelines-as-code` on `lock-file-maintenance(-vulnerability)?` branches. Real `when: delayed` / `start_in: 10 minutes`. No Konflux polling. **2. `mr-auto-approver` Lambda** — event-driven via GitLab webhooks: ```text GitLab webhook → gitlab-event-forwarder → SNS → mr-auto-approver Lambda ``` Approves only (does not set MWPS). Rejects forks. Checks Konflux statuses where configured. | Project | Allowed authors | Branch patterns | Konflux check | | -------------- | --------------- | ------------------------------------------------- | ------------- | | containers | PAC + chore-mr | `renovate/.*`, `chore/.*` | Yes | | rpms | PAC + chore-mr | `chore/.*` only | Yes | | tools | PAC | `renovate/.*` (deny `renovate/redhat-catalog/.*`) | No | | infrastructure | renovate token | `renovate/.*` | No | The Lambda is intended to replace CI approval jobs; both currently run in parallel for chore/lockfile paths. **3. Renovate automerge** — Renovate sets `automerge: true` in its MR configuration, but GitLab still requires a non-author approval before merge. For rpms Renovate MRs, neither CI approval jobs nor the Lambda cover `renovate/.*` branches — these MRs are stuck until a human approves. #### Konflux Status Gate External commit statuses prefixed `Konflux kflux-prd-rh03 /` are set by Konflux PipelineRuns triggered via Pipelines-as-Code. The `check_konflux_statuses.py` script (rpms) polls these statuses: - Missing → wait (poll continues) - Failed → do not approve - Invoked/running/pending/success → approve (MWPS then waits for final green, including Testing Farm via IntegrationTestScenarios) #### Lifecycle by MR Type **RPMs dist-git clean** (`chore/dist-git-update-`): schedule creates → MWPS at creation → GitLab CI + Konflux builds + Testing Farm → CI approval job (poll Konflux, up to 2h) + Lambda → merge. Typical: tens of minutes to hours. **RPMs dist-git conflict**: same branch, but MR is created as draft with `no-test` label and no MWPS. Requires human conflict resolution (or `claude_resolve_conflict.sh`), then remove draft/`no-test` and re-trigger. **RPMs upstream** (`chore/upstream-update-`): same as dist-git clean. Uses OIDC AWS lookaside upload for tarballs. **RPMs rebuild** (`chore/rebuild-`): same approval path. Typically operator-triggered, not scheduled. **RPMs Renovate** (`renovate/…`): Renovate sets automerge, but neither CI approval job nor Lambda covers `renovate/.*` branches on rpms → stuck until human approves. The `renovate-babysit` agent workflow posts a triage note but does not approve or merge. **Containers Renovate lockfile** (`lock-file-maintenance`): `lockfile_update_approval` (10 min delay) + Lambda (`renovate/.*` with Konflux check) → Renovate automerge / MWPS. Typical: 30–90+ minutes. **Containers other Renovate**: Lambda covers `renovate/.*`. No `lockfile_update_approval` unless branch matches lockfile pattern. **Containers metrics report** (`chore/metrics-report`): daily schedule → MWPS → `chore_mr_approval` + Lambda. **Containers CI lockfile** (`rpm-lockfile-updates/`): schedule is currently `active: false`. Approval path is incomplete — CI rule matches `lock-file-maintenance`, Lambda does not match `rpm-lockfile-updates/.*`. ### Shell Script Automation Detail #### rpms `create_mr.sh` (shared helper) Used by all rpms multi-MR scripts. Expects commits already on HEAD. Inputs: `--branch NAME` (required), `--title TEXT` (required), optional `--description`, `--auto-merge`, `--draft`, `--label NAME` (repeatable). Algorithm: 1. Count commits ahead of target branch; if 0 → exit 0. 1. Dedup: `git ls-remote --heads` for branch name. If present → exit 2 ("MR already exists"). 1. Create branch, push with MR push options. 1. Parse push stdout for MR URL; if missing → exit 1. Exit code contract: 0 = MR created, 2 = remote branch exists (skip), 1 = failure. Limitation: stale remote branch from a closed MR still triggers exit 2. Does not update existing MR/branch content. No `--force-with-lease`. #### rpms `dist_git_update_multi_mr.sh` Creates one MR per package needing Fedora dist-git sync. Modes: production (cwd = repo, always create MRs) vs clone dry-run vs clone + `--create-mrs`. Algorithm: 1. Select `metadata/*.json` files; apply `--clean-only` / `--modified-only` filter; skip native packages. 1. Record `START_COMMIT` on target branch. 1. For each package: run `./ci/dist_git.py update PACKAGE`. Exit 0 + new commit = clean update. Exit 2 + new commit = conflict update. 1. Collect all new commits, hard-reset target to `START_COMMIT`. 1. Per commit: parse subject (`^(Update|Sync) ([^ ]+) from ... to ...`), create branch `chore/dist-git-update-`, cherry-pick. Conflict → draft + `no-test`, no auto-merge. Clean → `--auto-merge`. 1. Call `create_mr.sh`. Branch convention: `chore/dist-git-update-`. Title: `chore(rpms): ` or `CONFLICT: chore(rpms): `. #### rpms `upstream_update_multi_mr.sh` Same multi-MR / cherry-pick pattern as dist-git, for upstream version bumps. Algorithm: runs `./ci/check_upstream_versions.py check --update --sign-off`, collects commits, splits into per-package MRs. Always `--auto-merge`. Branch: `chore/upstream-update-`. #### rpms `rebuild_multi_mr.sh` Does not generate rebuilds. Expects local commits from `dist_git.py rebuild` or `rebuild-rev-deps`, then splits them into per-package MRs. Unique feature: `--force` flag deletes existing remote branch before create, working around stale branches from closed MRs. Branch: `chore/rebuild-`. #### containers `create_lockfile_update_mrs.sh` Creates or updates one MR per image group when RPM lockfiles changed. Prerequisite: lockfiles already refreshed (`make all-host FORCE_REFRESH=true`). Algorithm (`process_single_image`): 1. Branch: `rpm-lockfile-updates/`. 1. For each distro/variant lockfile: compare to HEAD after stripping `.arches[].packages[].url` and `.arches[].source[].url` via `yq` (URL churn ignored). 1. If no meaningful changes → skip. 1. If `push-mode` ≠ `force` and `origin/` exists: compare to remote; if identical → skip. 1. `git switch --force-create` branch, commit lockfiles. 1. Push `--force-with-lease` + MR push options. Key difference from rpms: uses `--force-with-lease` to update existing branches in place. Dedup compares lockfile content, not just branch existence. #### containers `create_mr.sh` (simpler helper) Used by metrics report. Creates from working tree (not existing commits). `git add --all`, commit, `--force-with-lease` push. No dedup — overwrites existing branch. #### Deduplication Summary | Script | Existing branch | Content refresh | | ------------------------- | ------------------------------ | -------------------------------- | | rpms `create_mr.sh` | Skip (exit 2) | Never | | rpms multi-MR scripts | Via `create_mr.sh` | Never (unless `rebuild --force`) | | containers lockfile | Skip if remote content matches | Yes (`--force-with-lease`) | | containers `create_mr.sh` | Overwrites | Yes | ### Hummingbird Agent Framework The [Hummingbird Agent][hummingbird-agent-docs] is an event-driven LLM agent deployed on OpenShift that processes GitLab events via SQS. #### Architecture - **Event-driven**: consumes SQS events (pipeline failures, MR events, slash commands). No REST API — triggered only by SQS or CLI. - **Workflow-driven**: investigation logic defined in markdown files as LLM system prompts. New behaviors do not require code changes. - **Sandboxed execution**: all LLM commands run in isolated containers (Podman/K8s/KubeVirt) with no network access and no credentials. - **Multi-model**: supports Gemini and Claude via Vertex AI. - **Token separation**: model tokens (read-only, Reporter-level) are distinct from orchestrator tokens (write-capable, used only by deterministic code outside the sandbox). - **Single output action**: `post_gitlab_note` is the only way the agent affects the outside world. It cannot create, modify, or merge MRs. #### Current Workflows **`analyze-failures`**: triggered on pipeline failure. Bulk-fetches GitLab CI job logs, Konflux PipelineRun/TaskRun metadata and logs, and Testing Farm results. Groups failures by root cause. Posts a structured MR note with collapsible forensic evidence and links. - Tools: `gitlab_get_mr_details`, `gitlab_get_mr_diff`, `gitlab_get_commit_statuses`, `konflux_list_pipelineruns`, `konflux_get_pod_log`, `tf_get_results`, `tf_get_test_log`, `sandbox_exec` - Pattern: bulk metadata fetch → selective deep log pulls → grouped analysis → structured note - Flexibility: medium-low (tightly coupled to Konflux/Tekton/TF pipelines) **`code-review`**: triggered on MR open/update. Reads the unified diff, prior review discussions, and per-repo rules. Posts review findings with severity, category, and concrete code fix examples. - Tools: `gitlab_get_mr_details`, `gitlab_get_mr_unified_diff`, `gitlab_get_mr_discussions`, `gitlab_get_file_at_ref`, `sandbox_exec` - Pattern: diff-first analysis, discussion-aware follow-up (avoids re-raising resolved issues), per-project rules overlay - Per-repo rules: `workflows/repo-rules/-.md` files provide project-specific review guidance (e.g., `hummingbird-rpms.md` documents package metadata conventions) - Flexibility: high (generic for any GitLab MR; domain knowledge plugs in via `repo-rules/`) **`renovate-babysit`**: triggered on successful Renovate MR pipeline. Classifies the update as safe or risky based on diff scope, upstream changelogs (via web search), and local API usage. Posts a triage verdict. - Tools: `gitlab_get_mr_details`, `gitlab_get_mr_unified_diff`, `web_search`, `gitlab_get_repo_archive`, `sandbox_exec` - Pattern: diff scope classification, embedded changelog first then web search, archive + grep for call-site impact - Flexibility: medium-high (ecosystem-agnostic risk matrix; does not approve/merge — triage note only) #### What the Agent Has That Maps to a Workqueue - SQS FIFO queue with message grouping (serializes per session) - Session persistence (S3 `context.json`) - Config hot-reload in serve mode - Per-project token separation (model / action / orchestrator) - Rate limiting (per-workflow per-MR) - Data source abstraction (GitLab, Konflux, Testing Farm) - Auto-resolve on push/success #### What the Agent Lacks for MR Lifecycle Management - No MR creation capability (no git write operations, no push access) - Read-only GitLab tokens for the model - No workqueue for MR lifecycle state (events are fire-and-forget) - Single output action (`post_gitlab_note` only) - No safe-outputs pattern (no validation of proposed writes) - No deterministic fallback paths for well-understood operations - No request/response API (SQS consumption and CLI only; external services cannot invoke a workflow and retrieve structured results) ### Event Bus The existing event pipeline could serve as the backbone for a workqueue: ```text GitLab MR / note / pipeline webhooks → gitlab-event-forwarder (API Gateway → Lambda) → SNS hummingbird-events-topic → mr-auto-approver Lambda (approve safe bot MRs) → hummingbird-agent SQS (analyze / review / babysit) → hummingbird-status SQS (pipeline tracking) ``` `mr-auto-approver` is a pure rule-based approver — no LLM involvement. `hummingbird-mr-human-tracker` detects human fixes on Renovate/bot MRs and tracks them in Jira epics. ### Security Foundation: HUM-851 [HUM-851] ("Enable Secure Agent MR Creation") is an existing epic (status: New) that designs the security posture for agents creating MRs. #### Core Principle Instead of protecting CI secrets from agent-authored MR pipelines, eliminate all secrets from containers/rpms/tools CI entirely. Move all secret-dependent jobs to the infrastructure repo's CI. With zero secrets, a compromised agent MR pipeline can only access `CI_JOB_TOKEN` (single-project-scoped for bot users) and the runner's network. #### What Moves to Infrastructure Repo CI | Job | Current project | Current trigger | | ------------------------- | ----------------------- | ------------------ | | `deploy_hugo_tag` | containers, rpms, tools | On merge to main | | `lockfile_update` | containers | Scheduled | | Lockfile/chore approval | containers, rpms | After MR CI passes | | Chore MR creation | containers, rpms | Scheduled | | `dist_git_update` | rpms | Scheduled | | `upstream_version_update` | rpms | Scheduled | | `metrics_report` | containers | Scheduled | | `update_quay_description` | containers | Scheduled/on merge | This migration directly enables centralized MR creation. #### Target Security Posture - Zero CI secrets in target projects - Restricted runners with egress filtered to package registries only - Agent never holds write tokens — writes happen in deterministic orchestrator code - Diff safety validation — blocklist prevents changes to CI config, `.tekton/`, dependency files - Human review required for every agent MR, auto-labeled `agent-generated` - Rate limiting per-project and global; auto-close stale MRs - Monitoring: every agent MR logged and Slack-notified #### Task Breakdown | Task | Status | What it does | | --------- | ------ | ---------------------------------- | | [HUM-852] | New | Threat model and security design | | [HUM-853] | New | Move ops jobs to infra CI | | [HUM-854] | Closed | K8s CronJob/Job manifests (done) | | [HUM-855] | New | Validate and remove CI variables | | [HUM-856] | New | Restrict runner egress | | [HUM-857] | New | MR creation orchestrator action | | [HUM-858] | New | Rate limiting and abuse prevention | | [HUM-859] | New | Monitoring and incident response | ### Renovate Deployment Deployed as Kubernetes CronJobs in the infrastructure repo ([`kubernetes/renovate/20-cronjobs.yml.j2`][infra-renovate-cronjobs]): - 5 repos: containers, infrastructure, k8s-test-pipeline, rpms, tools - Each gets its own CronJob running hourly with `concurrencyPolicy: Replace` - Uses MintMaker image (`MINTMAKER_IMAGE`) - containers/rpms use bash wrappers and custom config files - Custom env variables for RPM lockfile cache, Rust/Cargo paths ### CVE Analysis Pipeline The CVE analysis pipeline is a complex, mature system. The relevant interface points for a workqueue are: 1. **`hummingbird-cve-analysis` CronJob** (K8s, every 30 min prod) — runs `cve_analysis --resolve` which triages Jira Security tickets, transitions statuses, creates advisory MRs, manages labels. 1. **`collect_cve_dashboard`** — runs after analysis, posts lifecycle data to the dashboard API. 1. **`/cve` skill** — reads staged CVE analysis data from Jira tickets and creates fix MRs in the rpms repo. The workqueue integration point: when `/cve` or CVE analysis determines an MR should be created, that request becomes a work item. ### RPM-to-Container Pipeline Flow A single RPM spec change triggers a cascade that spans both repos and multiple build systems. The full flow is documented in the [RPM Pipeline][rpm-pipeline] and [Image Pipeline][image-pipeline] docs. ```text RPM spec change (rpms repo) → MR validation (GitLab CI + Konflux build + Testing Farm) → Merge to main → Konflux RPM build (Tekton PipelineRun per package, mock hermetic) → RPM signing (Kerberos-based) → Publish to Pulp (packages.redhat.com) → Lockfile update detects new RPM versions (containers repo) → Lockfile MR created (Renovate or create_lockfile_update_mrs.sh) → Container build (Konflux, multi-arch) → Container testing (Testing Farm + K8s tests) → Enterprise Contract validation (Conforma) → Release to Quay.io ``` Key observations for the workqueue: - The cascade is implicit — there is no explicit trigger from RPM publish to container lockfile update. Lockfile updates discover new RPMs by polling Pulp (via `make all-host FORCE_REFRESH=true` or Renovate's RPM lockfile maintenance). - A single RPM change can fan out to many container images (any image that includes that package in its lockfile). - The RPM pipeline has 5 stages (spec change, MR validation, build, signing, publishing). The container pipeline has 6 stages (source templates, generation, build, testing, Enterprise Contract, release). - Testing Farm runs integration tests on merge requests only, not on main branch builds. Container tests include reverse dependency testing (rebuilding dependent images locally in the test environment). - Enterprise Contract (Conforma) validates supply chain security, hermetic builds, and policy compliance before release. ## Decision *To be determined by the spike investigation. This section will be updated with the chosen architecture.* The spike should investigate a unified workqueue-based system where: - Queue feeders replace or wrap the current scheduled scripts and Renovate outputs - Queue processors handle MR creation, approval, rebasing, and merge through deterministic code - Agentic capabilities (review, patch generation, triage) are invoked as composable services with structured output - The security model follows HUM-851's design: agents reason read-only, orchestrators execute writes ## Design Patterns and Prior Art ### GitHub Agentic Workflows (gh-aw) [gh-aw] provides a reference architecture for AI-assisted repository automation with [security guardrails][gh-aw-arch]: - **Read-only agent execution**: agents run with no write permissions, no secrets - **[Safe Outputs][gh-aw-safe-outputs]**: agents buffer intended write operations as structured JSON artifacts; separate permission-controlled jobs validate and execute them - **Threat detection**: a separate AI agent scans agent outputs before write execution - **Plan-level trust**: trust is embedded in the workflow definition (what the agent is allowed to do), not in the agent's runtime decisions Key pattern: separate reasoning (agentic, sandboxed, read-only) from execution (deterministic, permission-controlled, auditable). ### Existing Hummingbird Patterns - **SQS FIFO queue with message grouping**: already serializes work per session; can be extended to serialize per-MR - **Workflow-as-prompt**: markdown files define behavior; new behaviors do not need code changes - **Data source abstraction**: GitLab API access is modular - **Token tier separation**: model read tokens, action write tokens, orchestrator tokens — maps directly to the safe-outputs pattern - **Sandbox isolation**: reusable for agentic review/analysis tasks ### Agent as Composable AI Service One approach to bridge the agent framework and a workqueue is to evolve the agent from a self-contained event processor into a building block that non-AI services can call: - **Request/response API**: HTTP endpoint accepts workflow requests, returns a UUID; caller polls for structured JSON results. The caller (e.g., a workqueue herder) applies deterministic logic to decide what to do with the output. - **Structured output**: agent produces typed JSON (not just a GitLab note), enabling callers to process results programmatically. - **Safe-output actions**: new action types beyond `post_gitlab_note` (e.g., `create_mr`, `update_mr`) with per-workflow path allowlists, diff size limits, and branch naming constraints. The orchestrator validates each action against the workflow's configured constraints before execution. ## Open Questions 1. **Queue technology**: extend SQS FIFO (already in use by the agent) or use a different queue (K8s-native, Redis, PostgreSQL)? 1. **Processor deployment**: K8s Deployments polling SQS, GitLab CI jobs triggered by webhooks, Lambda functions, or a combination? 1. **Renovate strategy**: keep Renovate as a separate MR creator and add a herder, or replace Renovate's MR creation with the workqueue? 1. **Write credentials**: per-repo project access tokens (current model) or a single bot account? How does HUM-851's "eliminate CI secrets" model interact with workqueue processors that need write access? 1. **Safe outputs validation**: simple schema validation, deterministic allowlist checks, or full threat detection (as in gh-aw)? 1. **Review scope**: which MRs need AI review before merging vs which can be auto-merged with deterministic checks only? 1. **State tracking**: queue, GitLab labels, a database, or the existing `hummingbird-status` PostgreSQL? 1. **Agent evolution**: request/response API, safe-output actions, or hybrid? How does this relate to HUM-857? 1. **HUM-851 relationship**: is the workqueue spike a superset of HUM-851, or a separate epic that depends on HUM-851 Phase 1? ## Consequences *To be updated after the spike investigation determines the architecture.* Known consequences of any workqueue approach: - Centralizing MR creation provides a single point for audit, rate limiting, and monitoring - Moving secret-dependent jobs to the infrastructure repo (per HUM-851) is a prerequisite regardless of workqueue design - The existing CI approval jobs, Lambda approver, and Renovate automerge have overlapping scope that must be reconciled - RPMs Renovate MRs currently have no automated approval path — any design must address this gap - The agent's current workflows (failure analysis, code review, Renovate babysit) complement the workqueue and should not be disrupted ## File and System Inventory ### Infrastructure repo | Path | Contents | | --------------------------------------------------- | -------------------------------------- | | [`kubernetes/renovate/`][infra-renovate] | Renovate CronJob definitions (5 repos) | | [`kubernetes/hummingbird-agent/`][infra-agent] | Agent deployment manifests (8 files) | | [`kubernetes/hummingbird-cve-analysis/`][infra-cve] | CVE analysis CronJob | | [`aws/hummingbird-agent/`][infra-agent-aws] | Agent SQS/SNS SAM templates | ### Tools repo | Path | Contents | | ------------------------------------------------------------- | ---------------------------------------------------------------------------- | | [`hummingbird-agent/`][tools-agent] | Agent framework (loop, sandbox, models, data sources, workflows) | | [`hummingbird-agent/workflows/`][tools-workflows] | Workflow definitions (`analyze-failures`, `code-review`, `renovate-babysit`) | | [`hummingbird-agent/workflows/repo-rules/`][tools-repo-rules] | Per-project review rules (`hummingbird-rpms.md`) | | [`hummingbird-cve-analysis/`][tools-cve] | CVE analysis pipeline (preserve) | | [`hummingbird-dashboard/`][tools-dashboard] | Dashboard web app (preserve) | | [`mr-auto-approver/`][tools-approver] | Lambda: rule-based MR approval | | [`hummingbird-mr-human-tracker/`][tools-tracker] | Lambda: tracks human fixes on bot MRs | | [`gitlab-ci/gitlab_sync/`][tools-sync] | File sync MR creation with rebase | | [`gitlab-event-forwarder/`][tools-forwarder] | Webhook → SNS bridge | | [`hummingbird-events-topic/`][tools-events] | SNS topic infrastructure | ### RPMs repo | Path | Contents | | ---------------------------------------------------- | ----------------------------------------------------------- | | [`ci/create_mr.sh`][rpms-create-mr] | Shared MR creation helper (push options, dedup by branch) | | [`ci/dist_git_update_multi_mr.sh`][rpms-dist-git] | Dist-git sync: commit-all → reset → cherry-pick per package | | [`ci/upstream_update_multi_mr.sh`][rpms-upstream] | Upstream version bumps: same cherry-pick pattern | | [`ci/rebuild_multi_mr.sh`][rpms-rebuild] | Rebuild MR splitter (with `--force` for stale branches) | | [`ci/dist_git.py`][rpms-dist-git-py] | CLI for import/update/sync/rebuild operations | | [`ci/check_konflux_statuses.py`][rpms-konflux-check] | Polls Konflux external statuses before approval | | [`ci/claude_resolve_conflict.sh`][rpms-conflict] | AI-assisted conflict resolution for dist-git MRs | | `metadata/.json` | Per-package metadata (upstream URL, CVE mapping, status) | ### Containers repo | Path | Contents | | --------------------------------------------------------- | ---------------------------------------------------------------- | | [`ci/create_lockfile_update_mrs.sh`][containers-lockfile] | Per-image lockfile MR creation (force-with-lease, content dedup) | | [`ci/create_mr.sh`][containers-create-mr] | Generic chore MR helper (force-with-lease, no dedup) | | [`ci/internal/shared_lib.sh`][containers-shared-lib] | Shared utilities (`get_distro_variants`, build helpers) | | `images//` | Per-image Containerfiles and RPM lockfiles | [containers-create-mr]: https://gitlab.com/redhat/hummingbird/containers/-/blob/main/ci/create_mr.sh [containers-lockfile]: https://gitlab.com/redhat/hummingbird/containers/-/blob/main/ci/create_lockfile_update_mrs.sh [containers-repo]: https://gitlab.com/redhat/hummingbird/containers [containers-shared-lib]: https://gitlab.com/redhat/hummingbird/containers/-/blob/main/ci/internal/shared_lib.sh [gh-aw]: https://github.github.com/gh-aw/ [gh-aw-arch]: https://github.github.com/gh-aw/introduction/architecture/ [gh-aw-safe-outputs]: https://github.github.com/gh-aw/reference/safe-outputs/ [hum-6342]: https://redhat.atlassian.net/browse/HUM-6342 [hum-851]: https://redhat.atlassian.net/browse/HUM-851 [hum-852]: https://redhat.atlassian.net/browse/HUM-852 [hum-853]: https://redhat.atlassian.net/browse/HUM-853 [hum-854]: https://redhat.atlassian.net/browse/HUM-854 [hum-855]: https://redhat.atlassian.net/browse/HUM-855 [hum-856]: https://redhat.atlassian.net/browse/HUM-856 [hum-857]: https://redhat.atlassian.net/browse/HUM-857 [hum-858]: https://redhat.atlassian.net/browse/HUM-858 [hum-859]: https://redhat.atlassian.net/browse/HUM-859 [hummingbird-agent-docs]: https://hummingbird-project.io/docs/background/tools/hummingbird-agent-design/ [image-pipeline]: https://gitlab.com/redhat/hummingbird/containers/-/blob/main/documentation/background/image-pipeline.md [infra-agent]: https://gitlab.com/redhat/hummingbird/infrastructure/-/tree/main/kubernetes/hummingbird-agent [infra-agent-aws]: https://gitlab.com/redhat/hummingbird/infrastructure/-/tree/main/aws/hummingbird-agent [infra-cve]: https://gitlab.com/redhat/hummingbird/infrastructure/-/tree/main/kubernetes/hummingbird-cve-analysis [infra-renovate]: https://gitlab.com/redhat/hummingbird/infrastructure/-/tree/main/kubernetes/renovate [infra-renovate-cronjobs]: https://gitlab.com/redhat/hummingbird/infrastructure/-/blob/main/kubernetes/renovate/20-cronjobs.yml.j2 [rpm-pipeline]: https://gitlab.com/redhat/hummingbird/rpms/-/blob/main/documentation/background/rpm-pipeline.md [rpms-conflict]: https://gitlab.com/redhat/hummingbird/rpms/-/blob/main/ci/claude_resolve_conflict.sh [rpms-create-mr]: https://gitlab.com/redhat/hummingbird/rpms/-/blob/main/ci/create_mr.sh [rpms-dist-git]: https://gitlab.com/redhat/hummingbird/rpms/-/blob/main/ci/dist_git_update_multi_mr.sh [rpms-dist-git-py]: https://gitlab.com/redhat/hummingbird/rpms/-/blob/main/ci/dist_git.py [rpms-konflux-check]: https://gitlab.com/redhat/hummingbird/rpms/-/blob/main/ci/check_konflux_statuses.py [rpms-rebuild]: https://gitlab.com/redhat/hummingbird/rpms/-/blob/main/ci/rebuild_multi_mr.sh [rpms-repo]: https://gitlab.com/redhat/hummingbird/rpms [rpms-upstream]: https://gitlab.com/redhat/hummingbird/rpms/-/blob/main/ci/upstream_update_multi_mr.sh [tools-agent]: https://gitlab.com/redhat/hummingbird/tools/-/tree/main/hummingbird-agent [tools-approver]: https://gitlab.com/redhat/hummingbird/tools/-/tree/main/mr-auto-approver [tools-cve]: https://gitlab.com/redhat/hummingbird/tools/-/tree/main/hummingbird-cve-analysis [tools-dashboard]: https://gitlab.com/redhat/hummingbird/tools/-/tree/main/hummingbird-dashboard [tools-events]: https://gitlab.com/redhat/hummingbird/tools/-/tree/main/hummingbird-events-topic [tools-forwarder]: https://gitlab.com/redhat/hummingbird/tools/-/tree/main/gitlab-event-forwarder [tools-repo-rules]: https://gitlab.com/redhat/hummingbird/tools/-/tree/main/hummingbird-agent/workflows/repo-rules [tools-sync]: https://gitlab.com/redhat/hummingbird/tools/-/tree/main/gitlab-ci/gitlab_sync [tools-tracker]: https://gitlab.com/redhat/hummingbird/tools/-/tree/main/hummingbird-mr-human-tracker [tools-workflows]: https://gitlab.com/redhat/hummingbird/tools/-/tree/main/hummingbird-agent/workflows