This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Contributing to a Project Hummingbird container image

1: Quickstart Guide
2: Development Workflow
3: Resolving Merge Conflicts in Generated Files
4: Adding New Images
5: Adding FIPS Variants
6: Testing Guide
7: Adding Kubernetes Tests
8: Image Configuration Reference
9: Test Configuration Reference

Welcome to the Project Hummingbird contributor documentation. This section provides guides and references for contributing to container images.

Getting Started

New to contributing? Start here:

Quickstart Guide - Get your first contribution done in 5 minutes
Development Workflow - Detailed development environment and workflow

Guides

Step-by-step guides for common tasks:

Adding Images - How to add a new container image
Testing - How to run and write tests locally

Reference

Detailed reference documentation:

Image Configuration Reference - Complete properties.yml reference
Test Configuration Reference - Complete test definition reference

1 - Quickstart Guide

Get your first contribution done in 5 minutes

Welcome! This guide will help you make your first contribution to Project Hummingbird container images.

For more detailed information, see the full contributor documentation.

Prerequisites

Container tools: Podman or Docker
Build tools: buildah, make, git
Python: Python 3.11+ with dependencies from requirements.txt

Install on Fedora/RHEL:

sudo dnf install podman buildah make git python3 python3-pip
pip install -r requirements.txt

Quick Start Workflow

1. Fork and Clone

Fork the repository on GitLab
Clone your fork:

git clone --recurse-submodules https://gitlab.com/<your-username>/containers.git
cd containers
git remote add upstream https://gitlab.com/redhat/hummingbird/containers.git

2. Make Your Changes

Edit files in the images/<image-name>/ directory:

properties.yml - Image configuration
Containerfile.j2 - Container build template
tests-container.yml - Integration tests

3. Generate and Build

# Update dependent files
make

# Build the image
ci/build_images.sh <image-name>

4. Test Your Changes

# Run integration tests
ci/run_tests_container.sh <image-name>

# Run linters and checks
make check

5. Submit a Merge Request

Create a branch: git checkout -b fix-nginx-config
Commit your changes: git commit -am "Fix nginx configuration"
Push to your fork: git push origin fix-nginx-config
Open a merge request from your fork to the upstream repository on GitLab

Note for external contributors: If you’re not a project member, the CI pipeline won’t run automatically. Please ping one of the project maintainers in your merge request to trigger the build and test pipeline.

License

By contributing to this project, you agree that your contributions will be licensed under the MIT License. See LICENSE.txt for the full license text.

Common Tasks

Adding a New Image

# Create image directory
mkdir images/myapp

# Copy templates
cp images/Containerfile.j2 images/myapp/Containerfile.j2
cp images/properties.yml images/myapp/properties.yml

# Edit the template, configuration, and tests
vim images/myapp/properties.yml        # Configure packages and variants
vim images/myapp/Containerfile.j2      # Customize the build template
vim images/myapp/tests-container.yml   # Add integration tests

# Update dependent files
make

# Build and test
ci/build_images.sh myapp
ci/run_tests_container.sh myapp

Testing with Docker

# Automatic Docker-in-Docker setup
ci/build_images.sh --engine docker --setup <image-name>
ci/run_tests_container.sh --engine docker --setup <image-name>

Building Specific Variants

# Build only the rawhide builder variant
ci/build_images.sh nginx/rawhide/builder

# Test only the rawhide default variant
ci/run_tests_container.sh nginx/rawhide/default

# Build/test all variants for a specific distro
ci/build_images.sh nginx/rawhide
ci/run_tests_container.sh nginx/rawhide

Next Steps

Development Workflow - Detailed development environment and workflow
Adding Images - Step-by-step guide for adding new container images
Testing - How to run and write tests locally

Getting Help

Documentation: hummingbird-project.io/l/contributing-images/
Code: gitlab.com/redhat/hummingbird

2 - Development Workflow

Detailed development environment setup and contribution workflow

Prerequisites

Ensure the required tools are installed:

Container tools: Podman or Docker
Build tools: buildah, make, git
Python: Python 3 with PyYAML package

Install on Fedora/RHEL:

sudo dnf install podman buildah make git python3-pyyaml

macOS Setup

macOS requires bash 5+ and GNU command-line tools. Install via Homebrew:

brew install podman bash grep coreutils
podman machine init && podman machine start

# Add to ~/.zshrc for persistence
export PATH="/opt/homebrew/bin:/opt/homebrew/opt/grep/libexec/gnubin:/opt/homebrew/opt/coreutils/libexec/gnubin:$PATH"

Note: On Intel Macs, use /usr/local/ instead of /opt/homebrew/.

Repository Setup

Fork and Clone

Fork the containers repository on GitLab
Clone your fork with submodules:

git clone --recurse-submodules https://gitlab.com/<your-username>/containers.git
cd containers

Add the upstream remote:

git remote add upstream https://gitlab.com/redhat/hummingbird/containers.git

Development Workflow

1. Create a Branch

Create a descriptive branch starting from upstream main:

git fetch upstream
git checkout -b feature-add-redis-image upstream/main

2. Make Changes

Edit the relevant files in images/<image-name>/:

properties.yml - Image configuration (packages, variants, tags)
Containerfile.j2 - Container build template
tests-container.yml - Container tests
tests-k8s.yml - K8s tests
README.md.j2 - Image documentation template

3. Generate Files

Regenerate derived files:

make

4. Build Locally

Build the image to verify changes:

ci/build_images.sh <image-name>

# Build specific variant
ci/build_images.sh <image-name>/rawhide/builder

# Build with verbose output
ci/build_images.sh --verbose <image-name>

5. Test Locally

Container tests:

ci/run_tests_container.sh <image-name>

# Test specific variant
ci/run_tests_container.sh <image-name>/rawhide/default

# Test with verbose output
ci/run_tests_container.sh --verbose <image-name>

By default, tests use Podman. To test with Docker:

# Automatic Docker-in-Docker setup
ci/run_tests_container.sh --engine docker --setup <image-name>

K8s tests:

ci/run_tests_k8s.sh --context <context> <image-name>

# Test specific variant
ci/run_tests_k8s.sh --context <context> <image-name>/rawhide/default

# Test with verbose output
ci/run_tests_k8s.sh --verbose --context <context> <image-name>

See the Testing Guide for K8s test prerequisites and local development workflow.

Testing base images:

When modifying base images (like core-runtime), test dependent images:

ci/build_images.sh core-runtime
ci/build_images.sh --build-deps core-runtime
ci/run_tests_container.sh --include-reverse-deps core-runtime
ci/run_tests_k8s.sh --include-reverse-deps --context <context> core-runtime

Troubleshooting test failures:

Use --pause flag to inspect resources before cleanup:

ci/run_tests_container.sh --pause <image-name>
ci/run_tests_k8s.sh --pause --context <context> <image-name>

6. Run Linters

Ensure code quality:

make check

7. Commit and Push

Commit changes with a descriptive message and push to the fork:

git add .
git commit -m "feat: add Redis container image

- Add Redis 7.x image with default and builder variants
- Include basic health check tests
- Add compatibility notes for official Redis image"

git push origin feature-add-redis-image

8. Open a Merge Request

Open a merge request on GitLab:

Go to the containers repository
Select your fork and branch as the source, main as the target
Fill in the description with what changed, why, and what tests were added

9. CI Pipeline

The CI pipeline automatically:

Builds images for multiple architectures
Runs integration tests
Checks for linting issues

Note for external contributors: If not a project member, the CI pipeline won’t run automatically. Ping one of the project maintainers in the merge request to trigger the pipeline.

Next Steps

Adding Images - Step-by-step guide for adding new images
Testing Guide - Comprehensive testing documentation
Image Pipeline - How the complete pipeline works

3 - Resolving Merge Conflicts in Generated Files

How to resolve merge conflicts in generated files during rebases

Overview

During rebases of branches in this repository, you can commonly trigger merge conflicts in files that are generated by the various toolings and Makefile targets.

Rules

This documentation is only for generated files.
If there is not a section below for the conflicted file, you may need to check make help or search the Makefile for how to regenerate it (e.g., grep -r "target_file" Makefile).

Procedures for generated files

The following sections describe how to rebase given merge conflicts in specific paths.

Regenerating rpms.lock.yaml

When rebasing a branch, you may encounter conflicts in rpms.lock.yaml files. You should first delete the conflicting file, then run make container. Once in the container, rerun make and pass the relative path of the target. The file will be regenerated.

Below shows an interactive example. If attempting to do this non-interactively, remove the file as described and then run something like:

podman run --pull=newer --rm -v "$PWD:$PWD:z" -w "$PWD" \
    quay.io/hummingbird-ci/gitlab-ci:latest \
    make <target_file>

# 1. Delete the conflicted lockfile (replace with your actual conflicting path)
rm <path/to/conflicting/rpms.lock.yaml>

# 2. Enter the CI container
make container

# 3. Inside container, regenerate the lockfile
make <path/to/conflicting/rpms.lock.yaml>

# 4. Exit container
exit

# 5. Stage the regenerated file and continue the rebase
git add <path/to/conflicting/rpms.lock.yaml>
git rebase --continue

# 6. After rebase completes, validate the result
make -j$(nproc) check

Regenerating Containerfile

If the merge conflict is in one or more of the generated Containerfiles, then always accept the upstream (rebasing-onto) branch as the source of truth (i.e., git checkout --theirs -- <Containerfile> and git add <Containerfile>). When you have committed the upstream versions and are done with the rebase, you must then regenerate the Containerfiles so they pick up your changes. Containerfiles can be generated in two ways:

To rebuild Containerfiles for all images that are changed by your PR:

make -j$(nproc)

To rebuild a specific Containerfile, you can also use make and pass the path of the generated Containerfile as an argument.

make images/nginx/rawhide/default/Containerfile

After regenerating Containerfiles:

# Stage regenerated Containerfiles
git add <path/to/regenerated/Containerfile>

# If still in rebase, continue
git rebase --continue

# After rebase completes, validate the result
make -j$(nproc) check

4 - Adding New Images

Step-by-step guide for adding new container images to the project

This guide covers two scenarios:

Adding a new image - Creating an entirely new image (e.g., adding nginx for the first time)
Adding a versioned image - Adding a new version of an existing image (e.g., go-1-25 when go already exists)

Choose the appropriate section below based on your use case.

Adding Completely New Images

Follow these steps to add an entirely new container image to the project.

1. Create Image Directory

Create a directory for the new image:

mkdir images/your-service

2. Copy Base Templates

Copy the template files:

cp images/properties.yml images/your-service/properties.yml
cp images/Containerfile.j2 images/your-service/Containerfile.j2

3. Configure Image Properties

Edit images/your-service/properties.yml to configure:

Variants: Define variants (default: [default, builder])
Packages: List RPM packages needed in rpm_packages.all
Main package: Set main_package for version labeling
Repository and stream: Set repository and stream (both required). Add a YAML comment above stream documenting the reasoning (see Choosing a Stream Value)
Tags: Configure image tags
Release metadata: Set application_category, summary, description, and url
Support level: New images start with support_level: community

Example minimal configuration:

---
rpm_packages:
  all:
    - coreutils-single
    - your-main-package
  default:
    - ca-certificates-bundle  # smaller than ca-certificates; sufficient for non-builder variants
  builder:
    - ca-certificates         # full package needed for builder toolchains

main_package: your-main-package
repository: your-service
# <reasoning for stream choice> -- see "Choosing a Stream Value" section
stream: "latest"
# deprecated: true  # Optional: mark the image stream as deprecated
support_level: community
application_category: "Other"
summary: "One-line description of the image"
description: >-
  Description of the software, its purpose, and its main capabilities.
url: "https://upstream-project.example.com/"

tags:
  - value: latest

Prefer ca-certificates-bundle over ca-certificates in non-builder variants — it is smaller and sufficient for TLS trust roots. Builder variants need the full ca-certificates package for toolchain compatibility.

See the Image Configuration Reference for complete properties.yml options.

4. Customize Containerfile Template

Edit images/your-service/Containerfile.j2:

Add service-specific configuration (entrypoint, exposed ports, volumes, environment variables)
The template automatically includes packages from properties.yml via the {{ main_packages_arg() }} macro

The base template macros handle the standard container setup automatically:

Macro	What it provides
`{{ final_stage() }}`	`ENV HOME=/tmp`, `WORKDIR /tmp`, labels, OCI archive extraction
`{{ set_user() }}`	`USER 65532` (when `user: default` in `properties.yml`)

The user field in properties.yml defaults to default (UID 65532) if not specified. No passwd entry is needed — Linux allows running as any UID. See the Container Configuration reference for per-variant and custom user options.

When to override the defaults

The macros provide sensible defaults, but the image should match what the upstream image does. If the upstream image uses a custom user, WORKDIR, or HOME, the Hummingbird image should match for compatibility. Check the upstream image’s Dockerfile or inspect it with podman inspect before deciding.

When the defaults are sufficient, avoid duplicating what the macros already provide:

{# The macros already handle HOME=/tmp, WORKDIR=/tmp, USER 65532.     #}
{# Only add useradd/groupadd, WORKDIR, HOME, or USER when the         #}
{# upstream image does something different that users depend on.       #}

When the upstream image requires a specific user or working directory, set user: in properties.yml and add WORKDIR in Containerfile.j2 as needed. Use {{ default_user }} instead of hardcoding 65532.

Docker/registry auth in README templates

For images that interact with container registries, the auth config must be mounted to $HOME/.docker/config.json inside the container. Check what HOME is set to — for default-user images it is /tmp:

-v ${XDG_RUNTIME_DIR}/containers/auth.json:/tmp/.docker/config.json:ro,Z

Multi-Architecture Support

When Containerfiles or build scripts need to reference the target architecture (e.g., downloading architecture-specific binaries), use the correct naming convention for each context.

	Docker	Linux
Intel	`amd64`	`x86_64`
ARM	`arm64`	`aarch64`
Containerfile	`$TARGETARCH` ARG variable	`$(arch)` output

In Containerfiles or Go builds: Use the $TARGETARCH ARG variable provided by buildah (values: amd64, arm64)
In shell scripts or RPM builds: Use the arch or uname -m commands (values: x86_64, aarch64)

5. Add Comparison Report

Create the required images/your-service/report.yml. Its first comparison supplies the upstream image and documentation URL used by compatibility reports and generated README sections:

---
compare:
  - pull: docker.io/upstream/your-service:latest
    url: https://upstream-project.example.com/docs/

Add ignore entries only for intentional differences from the upstream image. See existing images with similar entrypoints and runtime behavior for examples.

6. Add Integration Tests

Create test files in images/your-service/:

tests-container.yml - Required container tests (Podman/Docker)
tests-k8s.yml - Optional Kubernetes tests for workload or cluster integration behavior

Container test example:

---
version-check:
  command: |
    test_engine_run --rm "${TEST_IMAGE}" your-service --version

K8s test example:

---
run-as-pod:
  command: |
    name="test-pod-${TEST_RUN_ID}"
    kubectl run "${name}" --image="${TEST_IMAGE}" --restart=Never --labels="${TEST_RUN_LABEL}"
    kubectl wait --for=jsonpath='{.status.phase}'=Succeeded "pod/${name}" --timeout=60s

See the Testing Guide for how to write and run tests.

7. Add Image Documentation

Create images/your-service/README.md.j2 with a high-level description, usage instructions, and compatibility notes. The template generates multiple README files for different target audiences (defined in images/variables.yml under readme_targets).

Important notes:

The readme_heading() macro automatically generates the heading with the correct product name
The readme_description() macro renders the image description from description in properties.yml; use it after the heading instead of hardcoding introductory text
Use {{ readme_targets[target].registry }} for registry references (not hardcoded quay.io/hummingbird)
The readme_available_tags() macro renders the variants and tags table
The readme_trailing_sections() macro appends compatibility, verification, vulnerability, and about sections

Example template:

{{ readme_heading() }}

{{ readme_description() }}

{{ readme_available_tags(variants, tags) }}

## Usage

podman run -d -p 8080:8080 {{ readme_targets[target].registry }}/your-service:latest

{{ readme_trailing_sections() }}

For more details, see README Generation.

8. Generate, Validate, and Submit

Follow the standard contribution workflow from the Development Workflow guide:

Generate files (make); commit all generated changes
Build locally (ci/build_images.sh your-service)
Run container tests (ci/run_tests_container.sh your-service)
Run Kubernetes tests when tests-k8s.yml exists (ci/run_tests_k8s.sh your-service)
Run linters (make check)
Commit and push changes
Open a merge request
Wait for CI pipeline

9. Add Konflux Components and Trigger Testing

When generated konflux-templates/rendered.yml changes, the merge request pipeline runs add_components_to_konflux automatically. This adds only the new components. After the job completes, add a /retest comment to the merge request so Konflux starts the new image pipelines. The manual deploy_components_to_konflux job redeploys all components and is not needed for normal image onboarding.

Adding Versioned Images

When adding a new version of an existing image (e.g., go-1-25, helm-4, or rabbitmq-4-3), follow this streamlined workflow.

1. Check Package Availability

Versioned packages may be available in different repositories:

Hummingbird repository - Custom packages built by the Hummingbird project (e.g., golang1.25, nodejs24, dotnet-sdk-10.0)
Fedora repositories - Standard Fedora packages (e.g., golang, python3)

The project uses two distros. Their current repository files are defined in images/variables.yml:

rawhide - Uses only Fedora Rawhide packages (fedora-44.repo)
hummingbird - Uses Fedora 44 + Hummingbird packages (fedora-44.repo + hummingbird.repo)

To check if a package is available in Hummingbird:

# Check the Hummingbird RPMs repository
ls ../rpms/rpms/ | grep <package-name>

Important: If the package is only available in the Hummingbird repository, add distros: [hummingbird] to properties.yml (see step 3).

2. Create Image Directory and Copy Files

Follow the naming pattern <language>-<version> with dashes (e.g., go-1-25, nodejs-24, dotnet-sdk-10-0):

# Create directory
mkdir images/go-1-25

# Copy templates from the closest existing version, not the generic templates
cp images/go-1-26/Containerfile.j2 images/go-1-25/
cp -P images/go-1-26/tests-container.yml images/go-1-25/
cp images/go-1-26/report.yml images/go-1-25/

Tests shared by a version family may instead live under ci/shared-tests/, with each image’s tests-container.yml and tests-k8s.yml symlinked to the shared files. Preserve the sibling image’s arrangement.

Only the version that publishes latest owns README.md.j2 and the generated README files for the shared repository. When a new version becomes the latest, move or copy the template to that version and remove it from the previous version. Older versions without latest do not need a README template. README files are generated from a template in the same image directory; they are not shared automatically.

3. Configure properties.yml

Create images/go-1-25/properties.yml with the versioned package:

---
main_package: golang1.25
repository: go
# endoflife.date: parallel at major.minor (Docker Hub + Wolfi)
stream: "1.25"
application_category: "Programming Languages & Runtimes"
summary: "Statically-compiled language with concurrency and a large standard library"
description: >-
  Statically typed, compiled programming language with garbage collection,
  memory safety, and a large standard library. It includes the complete Go
  toolchain.
url: "https://go.dev"
distros:
  - hummingbird  # Only if package is unavailable in Fedora
user: root
version_constraints: '1.25.*'
tags:
  # NOTE: Do NOT include 'latest' or major-version tags for older versions
  # Only the newest version (e.g., go-1-26) should have those tags
  - value: '{{ package_major_minor_version("golang1.25") }}'
    label: io.hummingbird-project.major-minor-version
  - value: '{{ package_version("golang1.25") }}'
    label: org.opencontainers.image.version
rpm_packages:
  all:
    - coreutils-single
    - git-core
    - glibc-devel
    - golang1.25
    - make

Key differences from a new image:

Set main_package to the versioned package name
Set repository to match the base image family (for registry organization)
Set stream to the version this image tracks (e.g., "1.25" for go-1-25), with a YAML comment above it documenting the reasoning (copy from an existing sibling image)
Set version_constraints so lockfile updates cannot cross the image’s version boundary
Copy application_category, summary, description, and url from the sibling image, then update any version-specific text
Add distros: [hummingbird] if the package is only in the Hummingbird repository
Preserve sibling settings such as variants, FIPS packages, user, and support level unless the new upstream version requires a deliberate change
Tag strategy for multiple versions:
- Only the newest version includes latest
- Include a major-version tag on each stream when it is unique (e.g., Helm 3 and Helm 4)
- If several streams share a major version (e.g., Go 1.25 and 1.26), only the newest includes that shared major-version tag
- Every version includes its non-conflicting stream tag and full version tag

4. Update Containerfile Template

Edit images/go-1-25/Containerfile.j2 to reference the versioned package in any environment variables or version-specific commands:

 ENV GOPATH=/go \
     GOTOOLCHAIN=local \
-    GOLANG_VERSION={{ package_version('golang') }} \
+    GOLANG_VERSION={{ package_version('golang1.25') }} \
     PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

5. Update Comparison Images

Edit images/go-1-25/report.yml to compare against version-specific upstream images:

compare:
  - pull: docker.io/golang:1.25
    url: https://hub.docker.com/_/golang

  - pull: cgr.dev/chainguard/go:latest-dev
    url: https://images.chainguard.dev/directory/image/go/overview

6. Tag Strategy for Multiple Versions

When multiple versions share a repository, coordinate tags to avoid conflicts. The following Go versions both have major version 1, so only the newest can publish the 1 tag.

Newest version (go-1-26):

tags:
  - value: latest                                        # Points to newest
  - value: '{{ package_major_version("golang1.26") }}'   # e.g., "1"
    label: io.hummingbird-project.major-version
  - value: '{{ package_major_minor_version("golang1.26") }}'  # e.g., "1.26"
    label: io.hummingbird-project.major-minor-version
  - value: '{{ package_version("golang1.26") }}'         # Full version
    label: org.opencontainers.image.version

Older versions (go-1-25):

tags:
  # NO 'latest' tag
  # NO major-version tag (would conflict with go-1-26's "1" tag)
  - value: '{{ package_major_minor_version("golang1.25") }}'  # e.g., "1.25"
    label: io.hummingbird-project.major-minor-version
  - value: '{{ package_version("golang1.25") }}'         # Full version
    label: org.opencontainers.image.version

Why: This ensures users can reference:

quay.io/hummingbird/go:latest → go-1-26
quay.io/hummingbird/go:1 → go-1-26 (latest 1.x)
quay.io/hummingbird/go:1.25 → go-1-25 (specific minor version)
quay.io/hummingbird/go:1.26 → go-1-26 (specific minor version)

When to update: If you add go-1-27, update go-1-26 to remove its latest and major-version tags, and add them to go-1-27.

7. Generate Files and Test

# Generate distro directories, lockfiles, VERSION, TAGS, etc.
make

# Build the image
ci/build_images.sh go-1-25/hummingbird/default

# Run tests
ci/run_tests_container.sh go-1-25/hummingbird/default

# Run linters
make check

Common Issues

Error: No match for argument: golang1.25

The package is not available in the configured distro repositories. If it is available only in the Hummingbird repository, add distros: [hummingbird] to properties.yml, then regenerate files:

make

Generation removes stale distro directories. Do not edit or remove generated files individually.

Choosing a Stream Value

Every image must have a stream field in properties.yml. The stream is a version series identity (e.g., "24", "3.11", "10.1", "latest") that is emitted as the io.hummingbird-project.stream label on the container image. It enables the image catalog to provide consistent structured data about multi-stream repositories.

Why Stream Matters

The image catalog must provide consistent structured data across time, including for images published before a second stream exists. Labels on published images cannot be changed retroactively. If a repository gains a new stream, all previously published images must already carry the correct version-based stream value — otherwise the catalog has inconsistent data that requires fragile repository-specific fixups.

Decision Process

flowchart TD
    Start["New image: choose stream value"] --> Q1{"Does endoflife.date show\nmultiple concurrently\nsupported branches?"}
    Q1 -->|Yes| Assess["Assess ecosystem granularity:\nupstream project, Docker Hub,\nDebian/Ubuntu, Wolfi/Chainguard,\nFedora, endoflife.date"]
    Assess --> Version["Use version-based stream\n(match ecosystem granularity)"]
    Q1 -->|No| Q2{"Does the software have a\nmeaningful version series\nidentity?"}
    Q2 -->|Yes| Current["Use version-based stream\n(current major version)"]
    Q2 -->|No| Latest["stream: latest"]
    Version --> Doc["Document reasoning\nas YAML comment"]
    Current --> Doc
    Latest --> Doc

Document the reasoning for the stream choice as a YAML comment above the stream field in properties.yml. This ensures future maintainers understand why a particular granularity was chosen:

# endoflife.date: parallel at major.minor (2.8, 3.0, 3.2)
stream: "3.0"

# rolling release with date-based versions, no version branches
stream: "latest"

Step 1: Does endoflife.date show multiple concurrently supported branches?

Check endoflife.date for the upstream project. It provides structured data on upstream release branches, security support timelines, and versioning schemes for most projects. If it shows multiple concurrently supported branches, use a version-based stream at the granularity of those branches. If the product is not on endoflife.date, that is itself evidence that parallel version lifecycle management is not a concern for that software — proceed to step 2.

The stream granularity must match the level at which upstream maintains parallel security-supported branches. Check:

Upstream project: At what version granularity are parallel branches maintained?
Docker Hub official images: Are there parallel version tags? At what granularity?
Debian/Ubuntu: Are there parallel versioned packages?
Wolfi/Chainguard: Are there versioned package variants?
Fedora: Are there parallel versioned packages? Single-version availability is not evidence against versioning, but multi-version availability is strong evidence for it.

Important: the image’s directory name is not evidence for granularity. The directory may use a coarser version or no version at all. The stream must reflect upstream’s actual branch structure.

Similarly, the purpose of upstream’s branches (LTS vs stable, stable vs mainline, STS vs LTS) is irrelevant to granularity. If they are parallel branches receiving security updates, the stream granularity must match.

Match the upstream project’s version naming convention. If the project and ecosystem consistently identify branches using a format that includes a minor component (e.g., .NET uses 8.0, 9.0 — never just 8 or 9), use that format for the stream value. This applies even when endoflife.date abbreviates to just the major number.

Step 2: Does the software have a meaningful version series identity?

If the software uses numbered versioning where the major (or major.minor) number identifies a version series, use a version-based stream at the current major version. The risk is asymmetric: using a version-based stream when a new major version never ships is harmless (the label is still correct), while using "latest" when a new version does ship breaks catalog consistency.

Note: if endoflife.date lists minor versions (e.g., memcached 1.4, 1.5, 1.6) but each immediately supersedes the previous with no overlap in support periods, the minor number is a sequential release counter, not a branch identity. Use the major version as the stream.

Reserve "latest" only for images where no meaningful version series identity exists. This is a narrow criterion with two cases:

Rolling/date-based releases with no version series (e.g., minio uses RELEASE.2024-01-18...)
Base images that track the distro rather than independent software (e.g., core-runtime)

Step 3: For unversioned images, track the current version

For images without a version in their directory name or version constraints, the stream value reflects the version currently shipped and must be updated when the upstream version changes. This is analogous to version_constraints — an explicit declaration that is updated periodically.

Field Reference

See the Image Configuration Reference for the stream field definition and examples.

Next Steps

Testing Guide - How to run and write tests
Image Configuration Reference - Complete properties.yml reference
FIPS Variant Guide - How to add FIPS variants to images
Image Pipeline - How the complete pipeline works

5 - Adding FIPS Variants

How FIPS variants work and step-by-step guide for adding them

Overview

FIPS variants provide container images that use only FIPS 140-3 validated cryptographic modules. FIPS is implemented as a cross-cutting modifier variant (like builder), not a separate image — the FIPS variant uses the same base image structure but layers FIPS configuration on top via additional RPM packages.

FIPS variants are restricted to the Hummingbird distro only, because FIPS-validated packages are not available in Fedora Rawhide.

FIPS images must not contain any non-validated cryptographic libraries. The presence of an unvalidated crypto library (such as libgcrypt or gnutls) would undermine the FIPS compliance guarantee, even if the application itself does not use that library. Global tests enforce this constraint.

FIPS Validation Scope

The cryptographic modules shipped in FIPS images are FIPS 140-3 validated through the NIST Cryptographic Module Validation Program (CMVP). This validation applies only when running on RHEL systems installed in FIPS mode. Running FIPS images outside this environment (e.g., on Fedora, on RHEL not in FIPS mode, or on other Linux distributions) is outside the validated configuration.

As a best-effort goal, FIPS images aim to behave similarly on both FIPS and non-FIPS hosts — restricting cryptographic operations to FIPS-approved algorithms regardless of host configuration. This is not a guarantee; some images (particularly NSS-based ones like OpenJDK) may behave differently depending on host FIPS mode.

Validated Cryptographic Modules

All validated modules in Hummingbird are pre-built binaries from RHEL 9.2, submitted to NIST for FIPS 140-3 validation:

Module	Package	Verified by global test
OpenSSL FIPS provider	`openssl-config-fips`	`fips-provider-matches-ubi9`
NSS softokn	`nss-softokn-fips`	`fips-nss-modules-match-rhel92`
NSS freebl	`nss-softokn-freebl-fips`	`fips-nss-modules-match-rhel92`

Global tests validate each module’s checksum against the known-good RHEL 9.2 validated binaries. Checksums are architecture-specific because the binaries differ per architecture.

All FIPS variants also install crypto-policies-config-fips, which sets the system-wide crypto policy to FIPS.

Two FIPS Stacks

Two FIPS stacks exist depending on which crypto library the image’s software uses:

OpenSSL stack — crypto-policies-config-fips + openssl-config-fips. The OpenSSL FIPS provider is enabled via a drop-in configuration file at /etc/pki/tls/openssl.d/fips-provider-enable.cnf. Enforcement is container-side: non-approved algorithms are rejected even on non-FIPS hosts. Used by: Nginx, Node.js, Python, Ruby. Go images also include these packages for system tools (e.g., git-core), but Go itself uses its own native FIPS 140-3 module.

NSS stack — crypto-policies-config-fips + nss-softokn-fips + nss-softokn-freebl-fips. NSS checks the host kernel’s FIPS mode (/proc/sys/crypto/fips_enabled), so enforcement behavior differs between FIPS and non-FIPS hosts. The primary Java security provider switches to SunPKCS11-NSS-FIPS. Used by: OpenJDK.

Both stacks share crypto-policies-config-fips for the system crypto policy, but differ in which cryptographic library provides the validated implementation.

Adding a FIPS Variant

1. Determine the FIPS Stack

Identify which crypto library the image’s software uses:

OpenSSL — most languages and servers (Python, Node.js, Ruby, Nginx, Go)
NSS — Java/OpenJDK (uses NSS via PKCS#11)

This determines which FIPS packages to install.

2. Update properties.yml

Add fips to additional_variants with a Hummingbird distro restriction, and add the FIPS packages to rpm_packages.fips.

OpenSSL-based image:

additional_variants:
  - name: fips
    distros: [hummingbird]
rpm_packages:
  fips:
    - crypto-policies-config-fips
    - openssl-config-fips

NSS-based image (OpenJDK):

OpenJDK has existing runtime and runtime-builder base variants. FIPS is added as both fips (full JDK) and runtime-fips (headless JRE):

additional_variants:
  - name: fips
    distros: [hummingbird]
  - name: runtime-fips
    distros: [hummingbird]
rpm_packages:
  fips:
    - java-21-openjdk-devel              # variant-specific packages
    - crypto-policies-config-fips
    - nss-softokn-fips
    - nss-softokn-freebl-fips
  runtime-fips:
    - crypto-policies-config-fips
    - nss-softokn-fips
    - nss-softokn-freebl-fips

See the Image Configuration Reference for complete properties.yml options, including additional_variants and rpm_packages.

3. Generate Files

Run make to generate the hummingbird/fips/ directory structure (Containerfile, RPM lockfiles, VERSION, TAGS):

make

4. Write FIPS Tests

Add FIPS-specific tests to tests-container.yml using filters.variants to target FIPS variants. A standard set of FIPS tests validates:

Non-approved algorithm rejected — e.g., MD5 (for security use), Blowfish, RC2
Approved algorithm works — e.g., SHA-256
Non-approved cipher rejected — e.g., 3DES, CHACHA20-POLY1305
Approved cipher works — e.g., AES-256-GCM (encrypt/decrypt roundtrip)

Example (Python):

fips-rejects-md5:
  filters:
    variants: ["*fips*"]
  command: |
    test_engine_run --rm "${TEST_IMAGE:?}" python3 -c "
    import hashlib
    try:
        hashlib.new('md5')
        print('MD5_ALLOWED')
    except ValueError:
        print('MD5_REJECTED')
    " | grep -q MD5_REJECTED \
        || test_fail "FIPS mode should reject MD5"

fips-allows-sha256:
  filters:
    variants: ["*fips*"]
  command: |
    test_engine_run --rm "${TEST_IMAGE:?}" python3 -c "
    import hashlib
    print(hashlib.sha256(b'test').hexdigest())
    " | grep -q 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 \
        || test_fail "FIPS mode should allow SHA-256"

Host FIPS mode considerations:

Where possible, write tests without filters.fips so they validate behavior on both FIPS and non-FIPS hosts. This supports the best-effort goal of consistent behavior across host types. Use filters.fips: true only for tests that inherently require a FIPS-enabled host kernel — primarily NSS-based images where enforcement depends on the host:

# NSS-based: rejection only works on FIPS hosts
fips-rejects-blowfish:
  filters:
    variants: ["*fips*"]
    fips: true                    # requires FIPS-enabled host kernel
  command: |
    ...

See the Test Configuration Reference for variant filters and FIPS mode selection.

5. Build and Test

# Build the FIPS variant
ci/build_images.sh <image>/hummingbird/fips

# Run tests
ci/run_tests_container.sh <image>/hummingbird/fips

# Run linters
make check

Go-Specific FIPS Requirements

Go 1.24+ includes a native FIPS 140-3 cryptographic module that has been CMVP validated (CMVP Certificate #5247). Unlike the previous golang-fips approach (which replaced the compiler with a fork using OpenSSL as the crypto backend), native FIPS uses the same golang package with build-time and runtime environment variables:

GOFIPS140=v1.0.0 (build-time): Tells go build to include the CMVP-validated FIPS module (v1.0.0) in the resulting binary. Set in the image environment so all builds in the container use it.
GODEBUG=fips140=on (runtime): Enables FIPS 140-3 mode — uses NIST DRBG for randomness, negotiates only FIPS-approved TLS, and runs mandatory self-tests. Set in the image environment as the default for binaries run in the container.
Same package: The FIPS variant installs the standard golang1.25 package (no version_package override needed).
No cgo dependency: Native FIPS works with both CGO_ENABLED=0 and CGO_ENABLED=1 binaries.

main_package: golang1.25
rpm_packages:
  fips:
    - crypto-policies-config-fips
    - golang1.25
    - openssl-config-fips

The openssl-config-fips package is retained for system tools in the image (e.g., git-core) that use OpenSSL for TLS connections.

Go-specific FIPS tests verify:

crypto/fips140.Enabled() returns true (native FIPS module is active)
go version -m output includes build GOFIPS140=v1.0.0-c2097c7c (CMVP-validated module, per the security policy section 11.1)
FIPS works with CGO_ENABLED=0 static binaries (a key advantage over the old golang-fips approach)

OpenJDK-Specific FIPS Requirements

OpenJDK uses the NSS stack instead of OpenSSL, which has several implications:

No openssl-config-fips — uses nss-softokn-fips + nss-softokn-freebl-fips instead
Host FIPS mode dependency — NSS checks /proc/sys/crypto/fips_enabled, so FIPS enforcement behavior differs between FIPS and non-FIPS hosts. Crypto rejection tests must use filters.fips: true.
Compound variants — OpenJDK introduces runtime-fips alongside fips, combining the runtime base variant with the FIPS modifier. Both share the same NSS FIPS packages; fips additionally includes java-*-openjdk-devel.
Security provider — in FIPS mode, the primary Java security provider becomes SunPKCS11-NSS-FIPS instead of the default SUN/SunJCE providers.

Global FIPS Tests

The following tests run automatically for all FIPS images via ci/global-tests/tests-container.yml. Each test skips gracefully for images using the other FIPS stack.

Test	Validates	Applies to
`fips-provider-matches-ubi9`	OpenSSL FIPS module checksum matches RHEL 9.2 binaries	OpenSSL stack
`fips-nss-modules-match-rhel92`	NSS module checksums match RHEL 9.2 validated binaries	NSS stack
`fips-no-libgcrypt`	`libgcrypt` is not present (not a validated module)	All FIPS
`openssl-fips-config-installed`	OpenSSL FIPS drop-in config exists	OpenSSL stack
`crypto-policy-is-fips`	System crypto policy is set to `FIPS`	All FIPS

Image-specific FIPS tests (algorithm rejection/acceptance) are defined in each image’s tests-container.yml.

Testing on FIPS Hosts

The test runner detects host FIPS mode from /proc/sys/crypto/fips_enabled and exports the TEST_FIPS environment variable (true or false). Tests can filter on this using filters.fips:

# Runs only on FIPS-enabled hosts
fips-host-required:
  filters:
    fips: true
  command: |
    ...

# Runs only on non-FIPS hosts
non-fips-only:
  filters:
    fips: false
  command: |
    ...

# Runs regardless of host FIPS mode (default when filters.fips is omitted)
always-runs:
  command: |
    ...

OpenSSL-based images: Most FIPS tests run regardless of host FIPS mode, because the OpenSSL FIPS provider enforces algorithm restrictions at the container level.

NSS-based images: Crypto rejection tests require filters.fips: true, because NSS enforcement depends on the host kernel’s FIPS mode.

Next Steps

Image Configuration Reference — additional_variants, rpm_packages, version_package
Test Configuration Reference — variant filters, FIPS mode selection
Testing Guide — how to run and write tests

6 - Testing Guide

How to run and write integration tests locally and work with CI

Container tests validate image functionality with Docker and Podman. K8s tests validate images in real Kubernetes environments.

Running Container Tests Locally

Prerequisites

Container Engine: Podman or Docker
Python: PyYAML package (pip install PyYAML or dnf install python3-pyyaml)

With Podman (Recommended)

Tests work directly with Podman:

ci/run_tests_container.sh <image-name>

# Test specific distro
ci/run_tests_container.sh <image-name>/rawhide

# Test specific distro/variant
ci/run_tests_container.sh <image-name>/rawhide/default

# Verbose output (shows passing test output and bash trace)
ci/run_tests_container.sh --verbose <image-name>

With Docker

Use --setup to automatically configure Docker-in-Docker:

ci/run_tests_container.sh --engine docker --setup <image-name>

The --setup flag:

Starts hummingbird-docker-dind container with mirror.gcr.io/docker:dind
Configures environment variables (DOCKER_HOST, DOCKER_CERT_PATH, etc.)
Waits for Docker daemon to be ready
Reuses existing container on subsequent runs

Prerequisites:

Docker CLI (dnf install docker-cli)
Podman (to run the dind container)

Building Images

Build images before testing:

# With Podman
ci/build_images.sh <image-name>

# With Docker
ci/build_images.sh --engine docker --setup <image-name>

Testing Base Images

When modifying base images, test dependent images:

ci/build_images.sh core-runtime
ci/build_images.sh --build-deps core-runtime
ci/run_tests_container.sh --include-reverse-deps core-runtime

Reproducing CI Failures

Reproduce Testing Farm failures locally:

# Single distro/variant tests
ci/run_tests_container.sh --include-reverse-deps --engine podman <image-name>/rawhide/default
ci/run_tests_container.sh --include-reverse-deps --engine docker --setup <image-name>/rawhide/default

# Group tests (all distros/variants)
ci/run_tests_container.sh --engine podman <image-name>/rawhide/group
ci/run_tests_container.sh --engine docker --setup <image-name>/rawhide/group

Troubleshooting Container Tests

Inspecting Failed Tests

Use --pause to inspect containers before cleanup:

ci/run_tests_container.sh --pause <image-name>

Viewing Test Output

By default, only failing tests show output. Use --verbose to see passing test output:

ci/run_tests_container.sh --verbose <image-name>

This also enables bash trace mode (set -x), showing each command as it executes.

Running K8s Tests Locally

K8s tests require kubectl and access to a Kubernetes cluster (via kubeconfig or context).

Basic Usage

ci/run_tests_k8s.sh --context <context> <image-name>

# Test specific distro/variant
ci/run_tests_k8s.sh --context <context> <image-name>/rawhide/default

# Verbose output
ci/run_tests_k8s.sh --verbose --context <context> <image-name>

Safety: The script requires explicit --context or --kubeconfig to prevent accidental operations on production clusters.

Local Development Workflow

Build images locally and push to the internal registry:

# Build the image
podman build -t my-nginx:dev images/nginx/hummingbird/default/

# Push to internal registry and test
ci/run_tests_k8s.sh --context mpp-preprod --push-image my-nginx:dev nginx/hummingbird/default

This requires oc login to the target cluster and oc project <namespace> to set the target namespace.

Testing Published Images

Test published images without building locally:

IMAGE_URL=quay.io/hummingbird/nginx:latest \
IMAGE_NAME=nginx--hummingbird--default \
    ci/run_tests_k8s.sh --context mpp-preprod nginx/hummingbird/default

Troubleshooting K8s Tests

Use --pause to inspect resources before cleanup, or --verbose to see passing test output:

ci/run_tests_k8s.sh --pause --context <context> <image-name>
ci/run_tests_k8s.sh --verbose --context <context> <image-name>

Tests that encounter recognized transient infrastructure failures run up to three times. The runner cleans up test resources and waits 60 seconds before each retry. Application assertion failures are reported immediately. Recognized failures include transient registry and container daemon errors, package-download timeouts from the Hummingbird repository, and transient Go crypto/rand entropy-read failures on FIPS-enabled hosts.

Writing Tests

Test File Locations

Container tests: images/<name>/tests-container.yml
K8s tests: images/<name>/tests-k8s.yml

Both use the same YAML format with different available environment variables.

Basic Tests

Create a test file with named tests:

---
version-check:
  command: |
    test_engine_run --rm "${TEST_IMAGE}" your-service --version

basic-functionality:
  command: |
    test_engine_run --rm "${TEST_IMAGE}" your-service --help

Container tests must use test_engine_run to start containers. When a detached container has both a name and a network, the helper waits for Podman network DNS registration automatically before returning. Fresh lookup processes avoid negative DNS caching in a single client process:

test_engine_run -d --network "${NETWORK_NAME}" --name "${CONTAINER_NAME}" "${TEST_IMAGE}"

For K8s tests:

---
cluster-access:
  command: |
    kubectl auth can-i get pods || test_fail "No pod access"

run-as-pod:
  command: |
    name="test-pod-${TEST_RUN_ID}"
    kubectl run "${name}" --image="${TEST_IMAGE}" --restart=Never --labels="${TEST_RUN_LABEL}"
    kubectl wait --for=jsonpath='{.status.phase}'=Succeeded "pod/${name}" --timeout=60s

External Test Scripts

For complex tests, use a separate shell script:

---
complex-test:
  command: ./test-complex-scenario.sh

Create images/<name>/test-complex-scenario.sh:

#!/bin/bash
set -euo pipefail

# Load TEST_IMAGES array for cross-variant testing
# shellcheck disable=SC1090
source "${TEST_IMAGES_PATH:?}"

# Run test
test_engine_run --rm "${TEST_IMAGE:?}" your-service test

Variant-Specific Tests

Limit tests to specific variants:

build-tools-test:
  variants: [builder]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" make --version

Cross-Variant Tests

Test interactions between variants:

cross-variant-test:
  variants: [group]
  command: |
    builder="${TEST_IMAGES[nginx/builder]:?}"
    default="${TEST_IMAGES[nginx/default]:?}"

    test_engine_run --rm "${builder}" nginx -version
    test_engine_run --rm "${default}" nginx -version

Known Issues (Container Tests Only)

Mark container tests with known failures:

test-with-known-issue:
  command: |
    result=$(test_engine_run --rm "${TEST_IMAGE}" some-command)
    [[ "$result" == "expected" ]] || test_fail "Custom error message"
  known_issues:
    - description: "Known configuration issue"
      issue: "https://issues.redhat.com/browse/PROJ-1234"
      pattern: "Custom error message"
      fails: "sometimes"

See the Test Configuration Reference for complete configuration options.

Working with CI Tests

Container Tests (Testing Farm)

Container tests run on Testing Farm with RHEL-9-Nightly systems for both x86_64 and aarch64 architectures.

Viewing CI Test Results

Open the merge request in GitLab
Navigate to the Jobs page for the pipeline
Identify the Testing Farm job (e.g., containers-rawhide-testing-farm-x86-64)
Click on the job name to go to the Konflux PipelineRun page
If wait-for-results is already finished, select it in the pipeline details, and then switch to the Testing Farm job via the ARTIFACTS_URL link in the right side pane; otherwise, select the scheduler job, switch to the Testing Farm API details via the tf-request link in the right side pane, and then follow the run.artifacts link in the JSON data

K8s Tests (Konflux Ephemeral Namespace)

K8s tests run in Konflux ephemeral namespaces provisioned via EaaS (Environment as a Service).

Viewing K8s Test Results

Open the merge request in GitLab
Navigate to the Jobs page for the pipeline
Identify the K8s test job (e.g., containers-rawhide-k8s-test)
Click on the job name to go to the Konflux PipelineRun page
View the run-tests task logs for test output

Rerunning CI Tests

Retrigger a specific pipeline using slash commands in merge request comments:

/retest nginx--hummingbird--default-on-pull-request

For automated retriggers of only failed checks, see Retrying Konflux Checks.

Next Steps

Test Configuration Reference - Complete test configuration reference
Adding Images - How to add new images with tests

7 - Adding Kubernetes Tests

How to add Kubernetes and OpenShift integration tests to container images

This guide describes how to add Kubernetes tests to container images.

Overview

Kubernetes tests validate that container images work correctly in Kubernetes and OpenShift environments, particularly:

Directory permissions (important for OpenShift arbitrary UIDs)
Version checks
Application functionality (web servers, databases, etc.)
Multi-container scenarios (sidecar patterns)

Confirm Test Design

Before implementing:

Read images/{image_name}/tests-container.yml to understand existing tests
Identify which tests need K8s validation:
- Directory permissions (important for OpenShift arbitrary UIDs)
- Version check
- Application functionality (web server, database, etc.)
- Multi-container scenarios (sidecar patterns)
Confirm test approach before creating files

Rules

Script naming: test-k8s-*.sh prefix (e.g., test-k8s-version.sh)
Script location: images/{image_name}/test-scripts/
YAML location: images/{image_name}/tests-k8s.yml
Keep YAML lean: complex logic goes in scripts, not inline YAML
No external downloads: avoid network flakes (exception: container images)
OpenShift compatibility: writable dirs must use gid=0 + g+w permissions

Validation

If images/{image_name}/ doesn’t exist → error “Image not found”
If no tests-container.yml → error “Add container tests first”

Create Test Scripts

If directory doesn’t exist → create images/{image_name}/test-scripts/

Test Script Template

#!/bin/bash
set -euo pipefail

# Test logic
test_result=$(command)

if [[ ! "${test_result}" =~ "expected" ]]; then
    echo "FAIL: Description"
    echo "Got: ${test_result}"
    exit 1
fi

echo "SUCCESS: Test passed"
exit 0

Directory Permissions Test

If testing OpenShift compatibility → create test-k8s-directory-permissions.sh:

Check writable directories exist and are writable
Verify running as non-root (UID != 0)
Test actual write with touch/rm

Version Check Test

If testing version → create test-k8s-version.sh:

Run version command
Check major version only (not major.minor.patch)
Avoids breaking on patch updates

Application-Specific Tests

If web server → deployment test with sidecar (curl container)
If database → connection/persistence test
If tool → primary functionality test

Create tests-k8s.yml

Pattern for each test:

test-name:
  variants: [default]
  command: |
    # Create ConfigMap with script
    kubectl create configmap "${TEST_GROUP}"-scripts-"${TEST_RUN_ID}" \
      --from-file=test-k8s-name.sh=test-scripts/test-k8s-name.sh \
      --dry-run=client -o yaml | kubectl label -f - --local -o yaml hum-k8s-test="${TEST_RUN_ID}" | kubectl apply -f -

    # Create Pod
    kubectl apply -f - <<EOF
    apiVersion: v1
    kind: Pod
    metadata:
      name: ${TEST_GROUP}-name-${TEST_RUN_ID}
      labels:
        hum-k8s-test: "${TEST_RUN_ID}"
    spec:
      restartPolicy: Never
      containers:
      - name: test
        image: ${TEST_IMAGE:?}
        command: ["sh", "/scripts/test-k8s-name.sh"]
        volumeMounts:
        - name: test-scripts
          mountPath: /scripts
      volumes:
      - name: test-scripts
        configMap:
          name: ${TEST_GROUP}-scripts-${TEST_RUN_ID}
          defaultMode: 0755
    EOF

    # Wait for completion
    kubectl wait --for=jsonpath='{.status.containerStatuses[?(@.name=="test")].state.terminated.reason}'=Completed \
      pod/"${TEST_GROUP}"-name-"${TEST_RUN_ID}" --timeout=120s || TEST_FAIL "Test did not complete"

    # Check logs
    out=$(kubectl logs "${TEST_GROUP}"-name-"${TEST_RUN_ID}" -c test)
    [[ "${out}" =~ "SUCCESS" ]] || TEST_FAIL "Test failed: ${out}"

Multi-Container Tests

If test needs multiple containers (app + client):

Use emptyDir volume mounted in both containers
emptyDir has 777 permissions, works with arbitrary UIDs

Readiness Probes

If testing app functionality:

Add readinessProbe to app container
Set timeoutSeconds: 5 for OpenShift network latency

OpenShift Compatibility

Read images/{image_name}/Containerfile.j2:

If writable dirs use user:user ownership → change to user:0
If writable dirs lack g+w → add chmod -R g+w
Why gid=0? OpenShift runs containers with arbitrary UIDs (for security) but always assigns gid=0 (root group). Directories must be group-owned by 0 with group-write permissions so the arbitrary UID can write to them (since it’s in group 0)
IMPORTANT: Use {{ default_user }} template variable, never hardcode UIDs:
- ✅ Correct: chown -R {{ default_user }}:0 ${NEWROOT}/dir
- ❌ Wrong: chown -R 65532:0 ${NEWROOT}/dir
- Template variables follow project conventions and allow centralized config changes

If Containerfile changed:

touch images/{image_name}/Containerfile.j2
make images/{image_name}/{distro}/default/Containerfile
ci/build_images.sh {image_name}/{distro}/default

Lint and Test

Run make -j$(nproc) check → if fails, fix errors before proceeding

Test in minikube:

minikube start
ci/run_tests_k8s.sh --context minikube {image_name}

Monitor pods immediately: kubectl get pods -w

If Containerfile changed → test in OpenShift:

ci/build_images.sh {image_name}/{distro}/default
# Tag with simple localhost name for --push-image
podman tag quay.io/hummingbird/{image_name}:latest localhost/{image_name}:test
ci/run_tests_k8s.sh --context <ocp> --push-image localhost/{image_name}:test {image_name}/{distro}/default

Note: Push images to localhost/{image_name}:test, not to quay.io. Tests should never push to the official registry.

Monitor pods: oc get pods -w

Common Issues

If tests fail OpenShift with “Permission denied” → check Containerfile uses gid=0 and g+w
If wrong image used in OpenShift → verify ci/run_tests_k8s.sh doesn’t override TEST_IMAGE when --push-image set
If multi-container test can’t see files → add emptyDir volume
If readiness probe times out in OpenShift → add timeoutSeconds: 5

Automated Skill

For AI-assisted implementation, use the /add-k8s-tests <image-name> command which follows these patterns automatically.

8 - Image Configuration Reference

Complete reference for properties.yml configuration and image settings

Complete reference for configuring container images via properties.yml.

properties.yml Overview

Each image can have custom settings via a properties.yml file. This reference documents all available configuration options.

Minimal example:

---
rpm_packages:
  all:
    - nginx
main_package: nginx
tags:
  - value: latest

Complete structure:

---
# Distro configuration (see Image Variants section)
# distros: [hummingbird]  # Override default distros (use when package unavailable in some distros)

# Image variants (see Image Variants section)
# Use additional_variants to extend defaults, or variants to override completely
additional_variants: [fpm, fpm-builder]  # Adds to default variants
# variants: [default, builder]  # Alternative: completely replaces defaults

# Container runtime configuration (see Container Configuration section)
user: default  # Optional: 'default' for default_user (65532), or literal user ID/name

# Package management (see Package Management section)
rpm_packages:
  build-deps: [...]
  all:
    - package-name                   # All architectures
    - name: arch-specific-package    # Arch-specific (see Package Management)
      arches:
        only: x86_64
  <distro>: [...]
  <variant>: [...]
additional_repos: [...]
allow_fedora_repos: true  # Optional: allow Fedora repos for hummingbird images (temporary workaround)

# Build configuration (see Build Configuration section)
hermetic: true
# Exceptional legacy source builds only; do not use for new images:
# build_from_source: true
# prefetch_gomod_path: upstream-submodule

# Versioning (see Versioning section)
main_package: package-name
# version_package:            # Optional: override package for version lookup per variant/distro
#   fips: package-fips-name   # Example: variant-specific package name

# Testing configuration (see Testing Configuration section)
reverse_dependency_tests: false  # Optional: disable for images like curl

# Release configuration (see Release Configuration section)
repository: image-name           # Required: Quay.io repository name
# <reasoning for stream choice>
stream: "latest"                 # Required: version series identity for the image catalog
application_category: "Storage"  # Required: Pyxis application category
# support_level: community        # Optional: community or experimental (default: Red Hat supported)
# deprecated: true                # Optional: mark the image stream as deprecated

# Image metadata (see Image Metadata section)
summary: "Short one-liner description"   # Required when adding labels
description: >-                          # Required when adding labels
  Short paragraph describing the image.
  1-2 sentences about key features.
url: "https://upstream-project.org"      # Required when adding labels

tags:
  - value: latest
  - value: '{{ package_major_version("package-name") }}'
    label: io.hummingbird-project.major-version

# Compliance scanning (enabled by default, see Compliance Scanning section)
# oscap:
#   enabled: false         # Optional: set to false to opt out
#   profiles:              # Optional: override which profiles run per variant
#     cis: true
#     stig:
#       variants: ["*fips*"]
#   exclude_rules:         # Optional: exclude specific rules (concatenated with globals)
#     - id: xccdf_org.ssgproject.content_rule_example
#       reason: "Reason for exclusion"
#       variants: ["*fips*"]   # Optional: glob patterns
#       profiles: [cis]        # Optional: limit to specific profiles

Image Metadata

Images can declare metadata fields for container image labels:

summary

Type: String
Required: When adding image labels (currently caddy only, expanding to all)
Length: ~40-80 characters (one-liner)

Short phrase describing what the image is. Used in table/list views and catalog cards.

Do not start with the image name – the name is always visible from context (UI headings, name label, inspect output).

Example: "Web server with automatic HTTPS"

description

Type: String (use >- YAML scalar for multi-line readability)
Required: When adding image labels
Length: ~100-250 characters (1-2 sentences)

Short paragraph for card views, podman inspect, and Kubernetes UIs (io.k8s.description). Describes what the software is and its key distinguishing features.

Use >- (folded scalar) in YAML to keep the value readable across multiple lines – it folds to a single-line string. Do not use | (literal block) as embedded newlines break label syntax.

Avoid embedded double quotes and backslashes – values are emitted as-is in Containerfile LABEL instructions with no escaping.

The same description value is used when generating the image README: the readme_description() macro in README.md.j2 templates renders it as the introductory paragraph. Do not duplicate this text in the template; use {{ readme_description() }}.

Example:

description: >-
  Extensible server platform with automatic HTTPS by default.
  Provides HTTP/3, reverse proxying, load balancing, and static
  file serving with minimal configuration.

url

Type: String (URL)
Required: When adding image labels

URL for the upstream project homepage. Used in org.opencontainers.image.url and the Conforma url label.

Example: "https://caddyserver.com"

deprecated

Type: Boolean
Default: Omitted, which means non-deprecated
Description: Marks the image stream as deprecated. The generated image carries the io.hummingbird-project.deprecated: true label so the image catalog can mark the stream as deprecated after the final release.
Example: deprecated: true

The field applies to the stream represented by the image directory. When all streams sharing a repository are deprecated, the catalog can mark the repository as deprecated. When only some streams are deprecated, the repository remains visible as a mixed-status repository.

Image Variants

Images can have multiple variants (e.g., default and builder):

default: Minimal runtime environment
builder: Includes development tools and build dependencies
custom: Define custom variants in properties.yml

Each variant gets its own:

Containerfile in images/<name>/<variant>/Containerfile
RPM lockfiles in images/<name>/<variant>/rpms
Build pipeline and release tags (non-default variants get -<variant> suffix)

variants

Type: Array of strings
Default: [default, builder]
Description: List of image variants to generate. Completely overrides the default variants. Each variant gets its own Containerfile, lockfiles, build pipeline, and release tags. Non-default variants get a -<variant> suffix in their image tags.
Example: [default] (CI images that only need the default variant)
Note: Prefer additional_variants when extending defaults; use variants only when you need to exclude default variants.

additional_variants

Type: Array of strings or objects
Default: None
Description: Additional variants to add to the base variants. The base is variants if specified, otherwise the default variants. Use this when you want to extend without repeating.

Simple format (string array):

additional_variants: [fpm, fpm-builder]  # Adds FPM variants to base variants for all distros

Object format with distro restrictions:

Each variant can be an object with name and optional distros fields to restrict which distros the variant is built for:

additional_variants:
  - name: fips
    distros: [hummingbird]  # Only build fips variant for hummingbird distro
  - name: fpm               # Simple string still works in array
  - name: fpm-builder
    distros: [hummingbird, rawhide]  # Build for specific distros

The distros field supports glob patterns:

additional_variants:
  - name: fips
    distros: ["*bird"]      # Matches hummingbird
  - name: special
    distros: ["raw*"]       # Matches rawhide

When to use distro restrictions:

Use distro restrictions when a variant requires packages or features only available in specific distros. For example, FIPS variants may only be available for Hummingbird where FIPS-validated packages are present

variant_descriptions

Type: Object (string-to-string mapping)
Required: When image has description and defines additional_variants with non-default base specializations
Description: Maps base variant names to human-readable descriptions. These descriptions are emitted as the io.hummingbird-project.variant.description label on each container image. The global description for default is defined in images/variables.yml; image-specific entries add descriptions for bases like fpm, runtime, or openssl. Modifier display (builder, fips) is handled by catalog consumers using the boolean variant labels, not by these descriptions.

Writing guidelines:

Describe the variant holistically — what the image variant is, not just what packages it adds on top of default. Someone reading the description with no surrounding context (e.g., via podman inspect) should understand the variant’s identity.
Use a noun phrase (2-6 words), not a full sentence.
Do not start with the image name (the image context is already provided by the image name label).
Avoid the word “runtime” in the default description (the OpenJDK runtime variant is a separate base, and OpenJDK’s default includes the full JDK).
Avoid expanding abbreviations redundantly (e.g., “JRE runtime” is redundant because JRE already means “Java Runtime Environment”).
Prefer functional descriptions (what capability the variant provides) over repeating the variant name as a product name.

Example (PHP image with FPM base):

variant_descriptions:
  fpm: "PHP FastCGI process manager"

Example (OpenJDK image with runtime base):

variant_descriptions:
  runtime: "Headless Java runtime"

distros

Type: Array of strings
Default: [rawhide, hummingbird] (from images/variables.yml)
Description: List of distros to build for this image. Overrides the global default_distros. Use this to disable a distro for images where a required package is not available in that distro.
Example: [hummingbird] disables Rawhide builds for this image
Note: Each enabled distro creates a full set of variants, so the final combinations are distros × variants.

Container Configuration

user

Type: String, integer, or object (per-variant dict)
Required: No (defaults to default)
Description: Specifies which user the container runs as. The USER directive is rendered via the {{ set_user() }} macro at the end of Containerfile templates. The special (and default) value default maps to the default_user variable (UID 65532). For user: default, the {{ final_stage() }} macro sets HOME=/tmp and WORKDIR /tmp early in the image to ensure unprivileged users have a writable default environment. Images can override WORKDIR later in their Containerfile.j2. For builder variants with user: default, it additionally sets ENV CONTAINER_DEFAULT_USER=65532.

Same user for all variants:

user: default

user: root  # Root user

user: postgres  # Literal username

Per-variant configuration:

Use when different variants need different users:

user:
  default: root
  builder: root
  fpm: default
  fpm-builder: default

Package Management

Packages are defined in properties.yml under rpm_packages:

rpm_packages:
  build-deps:             # Build-time dependencies (installed in builder layer, not in final
                          # image). Included in lockfiles for hermetic builds.
    - golang              # Example: Go compiler for building from source
    - rpm-build           # Example: Tools for building RPMs during image construction

  all:                    # Included in all variants (Containerfiles and lockfiles)
    - coreutils-single
    - your-main-package
    - name: grub2-efi-x64           # Arch-specific: only on x86_64
      arches:
        only: x86_64
    - name: grub2-efi-aa64          # Arch-specific: only on aarch64
      arches:
        only: aarch64

  hummingbird:            # Included in all Hummingbird variants (distro-level key)
    - versioned-package-1.2

  rawhide:                # Included in all Rawhide variants (distro-level key)
    - unversioned-package

  builder:                # Included only in "builder" variant
    - compiler-packages
    - debug-tools

  custom-variant:         # Included only in "custom-variant" variant
    - variant-specific-packages

Package entry format

Each entry in an rpm_packages list can be either a string (installed on all architectures) or an object with architecture constraints:

String entry (all architectures):

- coreutils-single

Object entry (arch-specific):

- name: grub2-efi-x64
  arches:
    only: x86_64            # Install only on x86_64

- name: some-package
  arches:
    not: aarch64             # Install on all architectures except aarch64

The arches.only and arches.not fields accept either a single string or a list of strings. Supported architectures: aarch64, x86_64.

How arch-specific packages work:

Lockfiles: Arch-specific entries are passed to rpm-lockfile-prototype using its native per-arch package format. The lockfile resolver only includes matching packages for each architecture.
Containerfiles: Arch-specific packages are installed via a case statement on TARGETARCH, so the correct packages are installed during multi-arch builds.

rpm_packages.build-deps

Type: Array of strings or arch-specific objects
Description: Build-time dependencies needed during image construction. Automatically installed in the builder layer (not in the final image) and included in lockfiles for hermetic builds. Use for compilers, build tools, and other packages needed only during the build process.

rpm_packages.all

Type: Array of strings or arch-specific objects
Description: Packages included in all variants, in both Containerfiles and lockfiles

rpm_packages.<distro>

Type: Array of strings or arch-specific objects
Description: Packages included in all variants for the specified distro. Useful when different distros ship different package names (e.g. versioned vs unversioned) and the packages should apply to every variant within that distro.

rpm_packages.<variant>

Type: Array of strings or arch-specific objects
Description: Packages included only in the specified variant, regardless of distro

additional_repos

Type: Array of strings (repository filenames)
Default: None
Description: Additional yum repository files from yum-repos/ to include in the RPM lockfile generation. These repos are added to the variant-specific repos (see [Global Variables Reference
- default_variant_repos]default-variant-repos). Specify only the filename, not the full path.
Example: [konflux-ci-rpm-lockfile-prototype-main-fedora-rawhide.repo]
Usage: Only needed for packages not available in standard Fedora repos or variant-specific repos

allow_fedora_repos

Type: Boolean
Default: false
Description: Allow Fedora repositories during lockfile generation for Hummingbird production images. By default, production Hummingbird images exclude Fedora repos to ensure packages come only from the Hummingbird repository. Set to true to temporarily allow Fedora repos when required packages are not yet available in the Hummingbird repository.
Example: true
Usage: Typically used temporarily for packages not yet built for Hummingbird (e.g., .NET runtime on x86_64, OpenJDK packages). Should be removed once packages are available in the Hummingbird repository.
Note: Toolchain and similar images under images/ (such as hummingbird-builder) use Fedora repositories for signature validation regardless of this setting.

Hermetic Builds

For hermetic builds (enabled by default), all packages from:

rpm_packages.build-deps - included in lockfiles and Containerfiles (builder layer only)
rpm_packages.all - included in lockfiles and Containerfiles (final image)
rpm_packages.<distro> - included in lockfiles and Containerfiles for all variants of a distro
rpm_packages.<variant> - included in lockfiles and Containerfiles for matching variants
default_rpm_packages.builder - automatically added for builder variants (see Global Variables Reference - default_rpm_packages)

are added to rpms.lock.yaml and prefetched using cachi2 for offline builds.

Build-time vs Runtime Packages:

rpm_packages.build-deps: Installed in the builder layer only (not in final image)
rpm_packages.all: Installed in ${NEWROOT} (the final image)
rpm_packages.<distro>: Installed in ${NEWROOT} for all variants of the specified distro
rpm_packages.<variant>: Installed in ${NEWROOT} for the specified variant

Build Configuration

hermetic

Type: Boolean
Default: true
Description: Controls whether the Konflux build pipeline passes HERMETIC=true to the buildah task, enabling hermetic (offline, network-isolated) builds with prefetched dependencies.
Important: Images with hermetic: false are blocked from release by the Conforma hermetic_task policy. hermetic: false should not be used for any image in the containers repository.

build_from_source

Type: Boolean
Default: false
Description: Controls version extraction in the package_version() macro. When true, extracts version from .gitmodules for images built from git submodules. When false, extracts version from RPM package metadata. Does not control whether source code is compiled in the Containerfile.
Warning: Do not use for new images. Software built directly from source is not represented by RPM package metadata and cannot be scanned for CVEs by the project’s package-based scanner. This setting exists only for exceptional legacy images.

prefetch_gomod_path

Type: String (path relative to image directory)
Default: None
Description: Path to a Golang project directory (typically a git submodule) for prefetching Go modules via cachi2. Enables hermetic builds for Go projects without a vendor tree. The path is relative to the image directory.
Usage: Existing exceptional source-built images only. Do not use for new images because source-built software cannot be scanned for CVEs by the project’s package-based scanner.

Versioning

main_package

Type: String
Description: Specifies the main package for version labeling

For RPM-based images:

Set to the package name. Version is extracted from RPM package metadata.

main_package: nginx

For source-built images:

Existing exceptional source-built images set this to the submodule path. Version is extracted from the branch field in .gitmodules. Do not use this pattern for new images: source-built software cannot be scanned for CVEs by the project’s package-based scanner.

main_package: images/minio/minio-upstream
build_from_source: true

version_constraints

Type: String (glob pattern)
Default: None (no constraints)
Description: Defines a version constraint pattern to prevent unwanted major or minor version updates. The constraint applies to all distros for this image. Constraints are validated during make check, and builds fail if VERSION files violate the constraint. Primarily used for multi-version images (e.g., python-3-11, nodejs-20) to ensure they stay on their intended version family.

Constraint patterns:

Patterns use glob-style matching:

3.11.* - Match any 3.11.X version (recommended)
20.* - Match any 20.X version
8.* - Match any 8.X version
1.25.* - Match any 1.25.X version (for 3-part versioning like Go)

The pattern matches against the content of VERSION files generated during the build process.

Example:

main_package: python3.11
version_constraints: '3.11.*'  # Ensure python-3-11 stays on 3.11.X across all distros

main_package: dotnet-sdk-8.0
version_constraints: '8.*'  # Applies to rawhide, hummingbird, and any other distros

When to use:

Multi-version images where each image targets a specific version family
Preventing automatic major/minor version bumps during Renovate updates
Ensuring version tags remain accurate (e.g., python:3.11 doesn’t become python:3.12)

Violation handling:

When a constraint is violated, make check fails with a detailed error message showing:

The VERSION file path
The actual version found
The constraint that was violated
The properties.yml file where the constraint is defined

Options for resolution:

Create a new image for the new version (recommended for major/minor bumps)
Exclude the violating version via yum-repos/*.repo excludepkgs
Update the constraint to allow the new version

See Version Constraints for detailed documentation.

version_package

Type: Object (string-to-string mapping)
Default: None
Description: Override the package name used for version extraction on a per-distro, per-variant, or per-distro/variant basis. When a variant or distro installs a different package that provides the same software (e.g., a FIPS-enabled build), this field tells the version/tag macros which package name to look up in the lockfile. The lookup order is: distro/variant, then variant, then distro, then main_package.

Per-distro override:

Use when different distros ship the same software under different package names:

main_package: ruby
version_package:
  hummingbird: ruby4.0  # hummingbird ships ruby4.0, rawhide ships ruby

Per-variant override:

Use when a variant installs a differently-named package:

main_package: ruby
version_package:
  minimal: ruby-minimal  # minimal variant resolves version from ruby-minimal

Per-distro/variant override:

Use when both distro and variant affect the package name:

main_package: ruby
version_package:
  hummingbird/fips: ruby4.0-fips

Usage in templates:

The resolved package name is available as package_name_for_version in Jinja2 templates:

tags:
  - value: '{{ package_version(package_name_for_version) }}'
    label: org.opencontainers.image.version

ENV GOLANG_VERSION={{ package_version(package_name_for_version) }}

Testing Configuration

reverse_dependency_tests

Type: Boolean
Default: true
Description: Controls whether this image participates in reverse dependency workflows (both building and testing). When true, changes to this image trigger rebuilds and tests of all images that depend on it. Dependencies are detected by searching for TEST_IMAGES[...] references in test scripts. Set to false for images that should not trigger reverse dependency workflows when changed.
Example: Set to false for curl to avoid triggering unnecessary reverse dependency tests

How it works in CI:

When an image is tested, the CI pipeline performs two phases:

Build Phase: Builds all dependencies needed for testing
- Uses ci/build_images.sh --build-deps to build:
  - Forward dependencies (images the main image’s tests depend on)
  - Reverse dependencies (images that depend on the main image, filtered by reverse_dependency_tests: true)
  - Forward dependencies of those reverse dependencies
- Automatically deduplicates to ensure each image is built exactly once
Test Phase: Runs tests on the main image and its reverse dependencies
- Uses ci/run_tests_container.sh and ci/run_tests_k8s.sh with --include-reverse-deps to test:
  - The main image
  - All reverse dependencies (filtered by reverse_dependency_tests: true)

Why set to false?

Set reverse_dependency_tests: false for images like curl that are:

Used pervasively across many images (too many reverse dependencies)
Have a small, stable API that can be fully tested in their own tests
Would cause excessive CI load if every dependent image were tested on each change

Release Configuration

The repository and stream fields together define the release identity for an image. Both are required and validated during CI. The repository specifies the Quay.io repository name, and the stream specifies the version series within that repository.

repository

Type: String
Required: Yes
Description: The Quay.io repository name for this image. For images where the directory name matches the desired repository name (e.g., images/nginx/ → repository: nginx), this is redundant but still required for explicitness. For versioned image directories, this groups multiple versions under one repository (e.g., nodejs-20 and nodejs-24 both use repository: nodejs).
Example: nodejs (used by nodejs-20 and nodejs-24 directories)

stream

Type: String
Required: Yes
Description: The version series identity for this image within its repository. Emitted as the io.hummingbird-project.stream label on the container image, enabling the image catalog to provide consistent structured data about multi-stream repositories.

The stream value must reflect upstream’s actual branch structure at the granularity where parallel security-maintained branches exist. Use endoflife.date to determine the correct granularity.

Key rules:

The stream is never derived from the image directory name. The directory name is not evidence for stream granularity (e.g., a directory named foo/ for an upstream with branches 2.8, 3.0, 3.2 still requires a major.minor stream like "3.0").
The purpose of upstream branches is irrelevant to granularity. Whether upstream calls them “LTS”, “stable”, “mainline”, or “STS” does not matter. If they are parallel branches receiving security updates, the stream granularity must match.
Match the upstream project’s version naming convention. If the project identifies branches as 8.0, 9.0 (not 8, 9), use that format even if endoflife.date abbreviates it.
Sequential minor versions are not parallel branches. If endoflife.date lists minor versions that each immediately supersede the previous with no support overlap, the minor is a release counter. Use the major version as the stream.
Use "latest" only for images where no meaningful version series identity exists: rolling-release tools with date-based versioning (e.g., minio) or base images that track the distro rather than independent software (e.g., core-runtime).
For unversioned images (no version in directory name, no version constraints), the stream value reflects the version currently shipped and must be updated when the upstream version changes.

See the stream assignment guidelines for the full decision process and rationale.

Document the reasoning for the stream choice as a YAML comment above the stream field. This ensures future maintainers understand why a particular granularity was chosen.

Examples:

# endoflife.date: parallel at major (Docker Hub + Debian + Wolfi)
stream: "24"
# endoflife.date: parallel at major.minor (Docker Hub + Debian + Wolfi)
stream: "3.11"
# endoflife.date: branches are 10.1, 11.0, not 10, 11
stream: "10.1"
# rolling release with date-based versions, no version branches
stream: "latest"

application_category

Type: String
Required: Yes
Description: The Pyxis application category for the image’s delivery repository. Emitted as the io.hummingbird-project.application-category label on the container image and used in the generated Pyxis repo config. All images sharing the same repository must use the same value. Validated during make check against the list in ci/application-categories.txt.

Common values for Hummingbird images:

"Programming Languages & Runtimes" – language runtimes and SDKs (Go, Python, Node.js, .NET, Ruby, etc.)
"Web Services" – web servers, reverse proxies, and load balancers (Nginx, httpd, Caddy, HAProxy, Tomcat)
"Database & Data Management" – databases and caches (PostgreSQL, MariaDB, Valkey, Memcached)
"Developer Tools" – CLI utilities and build tools (git, curl, jq)
"Storage" – object storage (MinIO, MinIO Client)
"Operating System" – base runtime images (core-runtime)

Example:

application_category: "Web Services"

tag_suffix_aliases

Type: Mapping of variant names to arrays of suffixes
Description: Add aliases for every configured tag without creating another image variant. The suffix is appended directly to each base tag. Existing canonical tags remain unchanged.
Example:

tag_suffix_aliases:
  default:
    - jdk25
  builder:
    - jdk25-builder

support_level

Type: String
Default: Omitted (image is Red Hat supported)
Values: community, experimental
Description: Controls the support level, Quay organization, and Konflux application assignment for hummingbird-distro images. The mapping from support level to Quay org is defined in images/variables.yml under support_levels; all other names (Konflux app, service account, RPA suffix) are derived from the Quay org name. When omitted, the image is Red Hat supported and assigned to containers-hummingbird. Rawhide images are always assigned to containers-rawhide regardless of this property.
Example: support_level: community

Mapping:

Value	Quay org	Konflux app	CPE	Pyxis/RPA
(omitted)	`hummingbird`	`containers-hummingbird`	Yes	Included
`community`	`hummingbird-community`	`containers-community-hummingbird`	No	Excluded
`experimental`	`hummingbird-ci`	`containers-ci-hummingbird`	No	Excluded

Usage: Use community for images maintained by the community (e.g., minio, bootc-os). Use experimental for internal build infrastructure (e.g., hummingbird-builder). These images are built and tested normally but published to their designated Quay organization, not the official Red Hat registry.

How Labels Work:

Template is rendered and written to Containerfile: LABEL org.opencontainers.image.version=1.27.3-1.fc42
Build system extracts the actual value from the image label
Image is tagged with the extracted value: quay.io/hummingbird/nginx:1.27.3-1.fc42

This ensures tags always match the actual package versions built into the image.

Compliance Scanning

OpenSCAP compliance scanning verifies CIS and STIG compliance during the build. Scanning runs against the container rootfs using oscap-chroot and fails the build if any enabled rules are violated.

Scanning is enabled by default for all images. It only runs for the Hummingbird distro and is skipped for the builder base image.

oscap.enabled

Type: Boolean
Default: true (from images/variables.yml)
Description: Controls whether OpenSCAP compliance scanning runs for this image. When true, a verify-compliance step is added to the generated Containerfile for all Hummingbird variants. Set to false to opt out.

oscap:
  enabled: false

oscap.profiles

Type: Object mapping profile names to enablement rules
Default: {cis: true, stig: {variants: ["*fips*"]}} (from images/variables.yml)
Description: Controls which compliance profiles run for each variant. Each profile can be set to true (all variants), false (disabled), or an object with a variants list of glob patterns.

Default behavior (CIS for all variants, STIG only for FIPS variants):

oscap:
  profiles:
    cis: true
    stig:
      variants:
        - "*fips*"

Enable STIG for all variants:

oscap:
  profiles:
    stig: true

Disable CIS for a specific image:

oscap:
  profiles:
    cis: false

The variants field supports glob patterns using fnmatch syntax (e.g., *fips* matches fips, fips-builder, openssl-fips).

oscap.exclude_rules

Type: Array of rule exclusion objects
Default: Global rule exclusions from images/variables.yml
Description: Rules to exclude from compliance scanning. Image-level exclusions are concatenated with global exclusions (not replaced). Each exclusion object has the following fields:

Field	Type	Required	Description
`id`	String	Yes	XCCDF rule ID (e.g., `xccdf_org.ssgproject.content_rule_configure_crypto_policy`)
`reason`	String	Yes	Explanation for excluding this rule
`variants`	Array of strings	No	Glob patterns for variants this exclusion applies to. Omit for all variants.
`profiles`	Array of strings	No	Profile names (`cis`, `stig`) this exclusion applies to. Omit for all profiles.

Exclude a rule for all variants and profiles:

oscap:
  exclude_rules:
    - id: xccdf_org.ssgproject.content_rule_configure_crypto_policy
      reason: "Caddy uses Go's built-in TLS stack, not system crypto-policies"

Exclude a rule only for specific variants:

oscap:
  exclude_rules:
    - id: xccdf_org.ssgproject.content_rule_configure_crypto_policy
      reason: "crypto-policies state files not generated in installroot"
      variants:
        - "*fips*"

Exclude a rule only from a specific profile:

oscap:
  exclude_rules:
    - id: xccdf_org.ssgproject.content_rule_some_rule
      reason: "Not applicable to CIS for this image"
      profiles:
        - stig

Combine variant and profile filters:

oscap:
  exclude_rules:
    - id: xccdf_org.ssgproject.content_rule_configure_crypto_policy
      reason: "crypto-policies state files not generated in installroot for FIPS variants"
      variants: ["*fips*"]
      profiles: [cis]

How compliance scanning works

Profile resolution: generate_jinja2.py determines which profiles (CIS, STIG) are active for the current variant based on oscap.profiles.
Rule filtering: Exclusion rules are filtered by variant (glob matching) and profile, producing per-profile exclusion lists.
Tailoring file: If any active profile has exclusions, an XCCDF tailoring file (oscap-tailoring.xml) is generated in the variant directory with custom profiles extending the upstream CIS/STIG profiles.
Containerfile: The verify-compliance command is added with explicit --cis and/or --stig flags based on active profiles, plus a --tailoring-file argument if exclusions exist.
Build-time scan: oscap-chroot evaluates the rootfs against each active profile. The build fails if any non-excluded rule fails.

Next Steps

Adding Images - Step-by-step guide for adding new images
Image Pipeline - How the complete pipeline works

9 - Test Configuration Reference

Complete reference for test definition format and configuration options

Complete reference for configuring image tests via tests-container.yml and tests-k8s.yml.

Test Definition Format

Each image can have tests defined in:

images/<name>/tests-container.yml - Container tests (run with Podman/Docker)
images/<name>/tests-k8s.yml - K8s tests (run in Kubernetes cluster)

Both files use the same YAML format:

---
script:
  command: ./test.sh

inline:
  command: |
    image=${TEST_IMAGE:?}
    test_engine_run --rm "${image}" --version

Environment Variables

The test runners provide environment variables to test commands. Some are common to both test types, others are specific to container or K8s tests.

Common Variables

Variable	Description
`TEST_IMAGE`	The container image being tested
`TEST_IMAGES`	Associative array with all `group/variant` image URLs
`TEST_IMAGES_PATH`	Path to file containing serialized `TEST_IMAGES` array
`TEST_VERBOSE`	Show test command output (`true` or `false`)
`TEST_GROUP`	The image group being tested (`nginx`, `python`, etc.)
`TEST_DISTRO`	The distro being tested (`hummingbird`, `rawhide`)
`TEST_VARIANT`	The variant being tested (`default`, `builder`, etc.)

Container Test Variables

Variable	Description
`TEST_ENGINE`	Container engine to use (`podman` or `docker`)
`TEST_USER_ID`	Current user ID (for permission handling)
`TEST_FIPS`	Whether host is in FIPS mode (`true` or `false`)

K8s Test Variables

Variable	Description
`TEST_RUN_ID`	Unique ID for this test run (for resource naming)
`TEST_RUN_LABEL`	Label selector for cleanup (`hum-k8s-test=<id>`)

Helper Functions

Common to All Tests

Both test types provide the test_fail function:

Function	Description
`test_fail`	Fail the test with a custom error message

In K8s tests, kubectl is pre-configured with the context/kubeconfig from CLI args.

Container Tests Only

Container tests provide helper functions that wrap the container engine with appropriate flags. Always use these functions instead of calling ${TEST_ENGINE} directly for create and run commands. This ensures tests respect globally set options such as the hermetic mode setting (--pull=never).

Function	Description
`test_engine_create`	Run `${TEST_ENGINE} create` with appropriate pull flags
`test_engine_run`	Run `${TEST_ENGINE} run` with appropriate pull flags

Note: For other container engine commands (inspect, network, volume, etc.), use ${TEST_ENGINE:?} directly.

Variant-Aware Testing

The testing system automatically includes variant-specific default tests:

Global tests from ci/{variant}-tests/tests-container.yml
Image-specific tests from images/<name>/tests-container.yml

Variant Selection

Tests can specify which variants they apply to using the filters.variants field. The field supports both exact matches and glob patterns.

# Test that only runs for builder variants
build-tools-test:
  filters:
    variants: [builder]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" make --version

# Test that runs for multiple specific variants
multi-variant-test:
  filters:
    variants: [default, builder]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" echo "Hello"

# Test that runs for all variants (no filters.variants field)
universal-test:
  command: |
    test_engine_run --rm "${TEST_IMAGE}" echo "Always runs"

Glob Patterns in Variant Filters

Use glob patterns to match multiple variants dynamically:

# Test that runs for any FIPS variant (fips, fips-builder, etc.)
fips-test:
  filters:
    variants: ["*fips*"]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" check-fips-mode

# Test that runs for all builder variants (builder, fpm-builder, etc.)
builder-test:
  filters:
    variants: ["*-builder", "builder"]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" dnf --version

# Mix exact matches and patterns
mixed-filter:
  filters:
    variants: [default, "*fips*"]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" echo "Runs on default and all fips variants"

Supported glob syntax:

* - matches any sequence of characters
? - matches any single character

Distro Selection

Tests can specify which distros they apply to using the filters.distros field. Like variant filters, this field supports both exact matches and glob patterns.

# Test that only runs for hummingbird distro
hummingbird-only-test:
  filters:
    distros: [hummingbird]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" some-hummingbird-specific-check

# Test that runs for multiple distros
multi-distro-test:
  filters:
    distros: [hummingbird, rawhide]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" echo "Runs on both"

# Test that runs for all distros (no filters.distros field)
universal-test:
  command: |
    test_engine_run --rm "${TEST_IMAGE}" echo "Always runs"

This is useful for tests that check distro-specific features. For example, CPE labels are only present on hummingbird images (Fedora/rawhide has no official CPE).

Glob Patterns in Distro Filters

Use glob patterns to match multiple distros dynamically:

# Test that runs for any distro ending in "bird"
bird-distro-test:
  filters:
    distros: ["*bird"]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" check-something

# Test that runs for any distro starting with "raw"
raw-distro-test:
  filters:
    distros: ["raw*"]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" check-rawhide-feature

Supported glob syntax:

* - matches any sequence of characters
? - matches any single character

FIPS Mode Selection

Tests can specify whether they should run based on the host’s FIPS mode using filters.fips:

# Test that only runs on FIPS-enabled hosts
fips-required-test:
  filters:
    fips: true
  command: |
    test_engine_run --rm "${TEST_IMAGE}" openssl list -providers

# Test that only runs on non-FIPS hosts
non-fips-test:
  filters:
    fips: false
  command: |
    test_engine_run --rm "${TEST_IMAGE}" some-non-fips-check

# Test that runs regardless of FIPS mode (no filters.fips field)
universal-test:
  command: |
    test_engine_run --rm "${TEST_IMAGE}" echo "Always runs"

The TEST_FIPS environment variable is set to true when running on a FIPS-enabled host (detected from /proc/sys/crypto/fips_enabled), and false otherwise.

Group Test Mode

When running tests for all variants (without specifying a specific variant), the system supports a special “group” test mode for tests that need to work across multiple variants simultaneously:

# Test that runs only in group mode
cross-variant-compatibility-test:
  filters:
    variants: [group]
  command: |
    # Access specific group/variant combinations using TEST_GROUP
    builder_image=${TEST_IMAGES[${TEST_GROUP}/builder]:?}
    default_image=${TEST_IMAGES[${TEST_GROUP}/default]:?}

    echo "Builder variant: ${builder_image}"
    echo "Default variant: ${default_image}"

    # Test that both variants have compatible APIs
    test_engine_run --rm "${builder_image}" nginx -version
    test_engine_run --rm "${default_image}" nginx -version

Using TEST_IMAGES in External Scripts

External shell scripts that need to reference other image variants must source the TEST_IMAGES array file at the beginning:

#!/bin/bash
set -euo pipefail

# Load TEST_IMAGES array
# shellcheck disable=SC1090
source "${TEST_IMAGES_PATH:?}"

# Now TEST_IMAGES is available - always use :? for proper error checking
test_engine_run --rm "${TEST_IMAGES[curl/default]:?}" ...
test_engine_run --rm "${TEST_IMAGES[httpd/default]:?}" ...

Note: Inline test commands in YAML files automatically have access to TEST_IMAGES and do not need this sourcing pattern.

Using TEST_GROUP for Dynamic References

The TEST_GROUP variable contains the current image group name (e.g., nginx, python, aspnet-runtime-8-0). Use it with TEST_IMAGES to dynamically reference the current image’s variants without hardcoding the group name:

multi-stage-build:
  filters:
    variants: [builder]
  command: |
    # Build in builder variant, run in default variant
    "${TEST_ENGINE}" build -t localhost/myapp -f - . <<EOF
    FROM ${TEST_IMAGES[${TEST_GROUP}/builder]:?}
    # ... build steps ...

    FROM ${TEST_IMAGES[${TEST_GROUP}/default]:?}
    COPY --from=0 /app /app
    EOF

    test_engine_run --rm localhost/myapp /app/myapp

Test Helper Functions

The test runner provides helper functions available in test commands:

test_fail(message)

Immediately fails the test with a custom error message sent to stderr:

test-name:
  command: |
    result=$(test_engine_run --rm "${TEST_IMAGE}" some-command)
    [[ "$result" == "expected" ]] || test_fail "Expected 'expected', got '$result'"

Known Issues (Container Tests Only)

Container tests can specify known log patterns that should not cause the test to report an error (for which support is still unimplemented) or failure. This helps distinguish between expected failures (tracked issues) and unexpected failures (new regressions), and detects when expected failures stop occurring.

Add known_issues to any test to specify patterns matching expected failures:

test-name:
  command: |
    result=$(test_engine_run --rm "${TEST_IMAGE}" some-command)
    [[ "$result" == "expected" ]] || test_fail "Custom error message"
  known_issues:
    - description: "Known configuration issue"
      issue: "https://issues.redhat.com/browse/PROJ-1234"
      pattern: "Custom error message"
      fails: "sometimes"  # optional, defaults to "always"

Fields

description: Human-readable explanation of the issue (required)
issue: Full URL to the issue tracking this problem (required)
pattern: Regular expression(s) to match against test output (required)
fails: Frequency of failure (optional, defaults to “always”)

Pattern Support

The pattern field supports both single patterns and arrays of patterns:

known_issues:
  # Single pattern
  - description: "Simple failure case"
    issue: "https://issues.redhat.com/browse/PROJ-1"
    pattern: "Connection failed"

  # Multiple patterns (any match triggers)
  - description: "Network connectivity issues"
    issue: "https://issues.redhat.com/browse/PROJ-2"
    pattern:
      - "Connection timeout"
      - "Network unreachable"
      - "curl: \\(28\\)"
    fails: "sometimes"

Failure Frequency

always: Known failures that consistently fail every time (default)
sometimes: Intermittent failures that may pass on retry (flaky tests)

Unexpected Pass Detection

The system automatically detects when tests with known issues labeled as fails: "always" suddenly start passing. This will not prevent missing known issues being masked by other (still occurring) known issues.

Automatic Retries

The container and K8s test runners automatically retry tests that fail with certain transient infrastructure errors. This helps avoid false test failures caused by temporary registry, network, package-download, image-pull, container-startup, or K8s scheduling problems.

Retry Behavior:

Failed tests are checked against a list of retriable error patterns
If a match is found, the test is automatically retried
If the test still fails after all attempts, it’s reported as a normal failure

K8s tests run up to three times, with cleanup and a 60-second delay between attempts. Retriable conditions include image-pull and container-creation errors, as well as kubectl wait timeouts. Application assertion failures are not retried.

Next Steps

Testing Guide - How to run and write tests
Adding Images - How to add new images with tests