This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

CI Scripts

1: build_images.sh
2: run_tests_container.sh
3: run_tests_k8s.sh
4: retrigger_failed_checks.py
5: gitlab_sync.py

This section documents the CI scripts used in the Project Hummingbird container build and test pipeline.

These scripts can be run locally for development and testing, and are also used by the automated CI/CD pipeline.

1 - build_images.sh

Build container images using buildah with support for multiple architectures and container engines

Purpose

Build container images using buildah with support for multiple architectures and container engines.

Usage

Usage: ci/build_images.sh [OPTIONS] [GROUP_NAMES...] [-- BUILDAH_ARGS...]

OPTIONS:
    --verbose, -v        Enable verbose output during build
    --arch ARCHITECTURE  Specify target architecture (e.g., amd64, arm64, arm/v7)
    --engine ENGINE      Specify runtime engine for testing/export (podman or docker)
    --setup              Set up Docker-in-Docker environment before exporting
    --component-name NAME
                         Parse component name (format: group--distro--variant)
                         Example: curl--rawhide--default → curl/rawhide/default
    --local-rpms-dir DIR Directory containing custom RPMs to use during build.
                         These have a higher priority and thus override the
                         standard repositories.
    --build-deps         Build all dependencies of specified groups, but NOT the
                         specified groups themselves. Collects forward, reverse,
                         and transitive dependencies automatically.
    --pull               Pull images from registry instead of building locally.
                         Fast alternative for local development. Uses published
                         images from quay.io.
    --dryrun             Show what would be built (or pulled with --pull) without
                         actually building/pulling.
    --help, -h           Show this help message

Examples

# Build single image group (all distro/variants)
ci/build_images.sh nginx
ci/build_images.sh nodejs-20

# Build specific distro/variant only
ci/build_images.sh nginx/rawhide/builder
ci/build_images.sh nodejs-20/hummingbird/default

# Build multiple image groups at once
ci/build_images.sh nginx curl git
ci/build_images.sh nginx/rawhide/default curl/hummingbird/builder

# Build with options
ci/build_images.sh --verbose curl
ci/build_images.sh --arch arm64 nginx
ci/build_images.sh --engine docker nginx
ci/build_images.sh --engine podman --verbose nginx

# Build multiple image groups with options
ci/build_images.sh --verbose nginx curl git
ci/build_images.sh --arch arm64 nginx postgresql

# Build all dependencies of dotnet-runtime-10-0 (forward + reverse + transitive)
ci/build_images.sh --build-deps dotnet-runtime-10-0

# Pull dependencies from registry instead of building (faster for local dev)
ci/build_images.sh --build-deps --pull dotnet-runtime-10-0

# Pull a specific image from registry
ci/build_images.sh --pull nginx/rawhide/default

# Build from CI component name format (used in CI environments)
ci/build_images.sh --component-name curl--rawhide--default

# Build with custom RPMs (for testing modified packages)
ci/build_images.sh --local-rpms-dir ../rpms/builds/hostname/RPMS git/rawhide/builder

Note: Some images require git submodules initialized (use git init --recurse-submodules when cloning or git submodule update --init if already cloned). When building for foreign architectures, make sure qemu-user-static is available.

Dependency Building

The --build-deps flag builds all dependencies needed for testing an image, but not the image itself. This is used in CI to ensure all required images are available before running tests.

Pulling vs Building Dependencies

For local test development, use --pull with --build-deps to pull pre-built images from the registry instead of building them locally. This is much faster and uses the same dependency collection logic:

# Slow: Build all dependencies locally
ci/build_images.sh --build-deps nginx

# Fast: Pull all dependencies from registry
ci/build_images.sh --build-deps --pull nginx

The --pull flag works without --build-deps too:

# Pull a specific image instead of building it
ci/build_images.sh --pull caddy/rawhide/default

How Dependency Collection Works

When you run ci/build_images.sh --build-deps <image>, it performs a 2-level expansion:

Level 1: Collects forward and reverse dependencies of the specified image
- Forward dependencies: Images that the specified image’s tests depend on (detected by TEST_IMAGES[...] references in test files)
- Reverse dependencies: Images that depend on the specified image (filtered by reverse_dependency_tests: true in properties.yml)
Level 2: Collects forward dependencies of the reverse dependencies
Stops: No further expansion (avoids infinite graph traversal)

All dependencies are automatically deduplicated to ensure each image is built exactly once.

Relationship to Testing

The reverse_dependency_tests property in properties.yml affects both building and testing:

Build phase (--build-deps): Filters which reverse dependencies to build
Test phase (ci/run_tests_container.sh and ci/run_tests_k8s.sh with --include-reverse-deps): Filters which reverse dependencies to test

Set reverse_dependency_tests: false for images like curl that are used pervasively but don’t need reverse dependency workflows.

Dependency Building Examples

# Build dependencies for dotnet-runtime-10-0
# Builds: dotnet-sdk-10-0 (forward dependency)
ci/build_images.sh --build-deps dotnet-runtime-10-0

# Pull dependencies for dotnet-runtime-10-0 (faster alternative)
# Pulls: dotnet-sdk-10-0 from quay.io/hummingbird-rawhide
ci/build_images.sh --build-deps --pull dotnet-runtime-10-0

# Build dependencies for core-runtime
# Builds: xcaddy, go, rust (reverse deps with reverse_dependency_tests: true)
# Plus their forward dependencies
ci/build_images.sh --build-deps core-runtime

# Use --dryrun to see what would be built/pulled without actually doing it
ci/build_images.sh --dryrun --build-deps core-runtime
ci/build_images.sh --dryrun --build-deps --pull core-runtime

Building with custom RPMs

The --local-rpms-dir option enables testing container images with custom-built RPM packages, for iterating on package changes before committing to the RPM repository.

Workflow

Build custom RPMs in the rpms repository (see its documentation for the complete workflow):

cd ../rpms
# modify a package
ci/build_rpms.sh packagename
# Built RPMs will be in builds/packagename/RPMS/

Build container image using the custom RPMs:

cd ../containers
ci/build_images.sh --local-rpms-dir ../rpms/builds/packagename/RPMS imagename/builder

Verify the custom package was installed:

podman run --rm --entrypoint '' quay.io/hummingbird/imagename:latest-builder rpm -qa

2 - run_tests_container.sh

Run tests for image groups using containerized test environment with support for both Docker and Podman engines

Purpose

Run tests for image groups using containerized test environment with support for both Docker and Podman engines.

Usage

Run ci/run_tests_container.sh --help for full usage information.

Usage: ci/run_tests_container.sh [OPTIONS] [GROUP_NAMES...]

OPTIONS:
    --verbose, -v        Enable verbose output during testing
    --engine ENGINE      Specify container engine (podman or docker)
    --setup              Set up Docker-in-Docker environment before running tests
    --pause, -p          Pause failed tests before cleanup to allow debugging
                         Prints a message and waits for Enter before cleaning up containers
    --hermetic           Use hermetic builds (--pull=never, only use prefetched/built images)
                         Default for CI. Without this, missing images are pulled on demand.
    --component-name NAME
                         Parse component name (format: group--distro--variant)
                         Example: curl--rawhide--default → curl/rawhide/default
    --group-component-name NAME
                         Parse component name and add /group variant
                         Example: curl--rawhide--default → curl/rawhide/group
    --include-reverse-deps
                         Also test groups that depend on specified groups
    --help, -h           Show this help message

Note: If no distro/variant is specified, all combinations will be tested. To test only a specific variant, use <group_name>/<distro>/<variant>. To test only a specific test, use <group_name>/<distro>/<variant>/<test>.

Engine Selection: Use --engine to specify the container engine (podman or docker). When using Docker, add --setup for automatic Docker-in-Docker environment setup.

Hermetic vs Non-Hermetic Testing:

Local development (default): Without --hermetic, the test runner allows pulling missing images from the registry. This is convenient for testing individual images without building all dependencies first.
CI environment: Use --hermetic to enforce that tests only use prefetched or locally-built images (via --pull=never). This ensures reproducible builds and prevents accidentally using images from the registry that differ from what Konflux built.

Building and Testing

For local development, you’ll typically want to build the images first, then test them:

# Build the images
ci/build_images.sh <group_name>[/distro/variant]
ci/build_images.sh <group_name1> <group_name2>  # Build multiple image groups

# Test the images
ci/run_tests_container.sh <group_name>[/distro/variant]
ci/run_tests_container.sh <group_name1> <group_name2>  # Test multiple image groups

The build script uses the same syntax as the test script, making it easy to build and test the same image/variant combination.

When working on base images (like core-runtime) that other images depend on, build the base image and its dependencies before running reverse-dependency tests:

# Build core-runtime, then its forward and reverse dependencies
ci/build_images.sh core-runtime
ci/build_images.sh --build-deps core-runtime

# Then test them all
ci/run_tests_container.sh --include-reverse-deps core-runtime

The -p/--pause option pauses the script on failed tests before cleaning up to allow interactive and efficient debugging.

Testing with Specific Image Builds

To reproduce CI failures, you can run the test with mapping the image under test to the Konflux build. Set IMAGE_URL_<GROUP>__<DISTRO>__<VARIANT> environment variables (uppercase, hyphens/dots → underscores, __ separates parts):

# Test with a specific CI build
IMAGE_URL_TOMCAT_10__HUMMINGBIRD__BUILDER='quay.io/redhat-user-workloads/.../tomcat-10--hummingbird--builder@sha256:...' \
  ci/run_tests_container.sh tomcat-10/hummingbird/builder

Automatic Retries for Transient Infrastructure Failures

The test runner automatically retries tests that fail with certain transient infrastructure errors. This helps avoid false test failures caused by temporary issues with external services like container registries or network problems.

Retry Behavior:

Failed tests are checked against a list of retriable error patterns
If a match is found, the test is automatically retried
If the test still fails after all attempts, it’s reported as a normal failure

Examples

# Test single group (all distro/variants)
ci/run_tests_container.sh curl
ci/run_tests_container.sh nginx

# Test single group (specific distro/variant)
ci/run_tests_container.sh curl/rawhide/default
ci/run_tests_container.sh nodejs-20/hummingbird/builder

# Test multiple groups (all distro/variants)
ci/run_tests_container.sh curl nginx mariadb

# Test multiple groups (mixed variants)
ci/run_tests_container.sh curl/rawhide/default nginx mariadb/hummingbird/builder

# Test with different engines
ci/run_tests_container.sh --engine docker --setup git
ci/run_tests_container.sh --engine podman --verbose nginx

# Test with verbose output for debugging
ci/run_tests_container.sh --verbose nginx mariadb dotnet-runtime-8-0

# Test specific tests
ci/run_tests_container.sh curl/rawhide/default/version
ci/run_tests_container.sh nginx/rawhide/default/port
ci/run_tests_container.sh curl/rawhide/default/version nginx/hummingbird/default/port

# Also test groups with reverse dependencies
ci/run_tests_container.sh --include-reverse-deps core-runtime

# Test from CI component name format (used in CI environments)
ci/run_tests_container.sh --component-name curl--rawhide--default

# Test across variants from CI component name format
ci/run_tests_container.sh --group-component-name curl--rawhide--default

Reverse dependency testing is enabled by default for all images. When enabled, changes to an image will automatically:

Find images that depend on it by scanning for TEST_IMAGES[group/distro/variant] references
Run tests for both the changed image and all its dependents

This catches breaking changes in base images early. Images can opt-out by setting reverse_dependency_tests: false in properties.yml (e.g., curl, which is widely used but typically doesn’t need reverse dependency testing).

3 - run_tests_k8s.sh

Run K8s tests for image groups using kubectl with support for local development and CI environments

Purpose

Run K8s tests for image groups using kubectl. Tests execute in a real Kubernetes environment, validating that images work correctly in Kubernetes.

Usage

Run ci/run_tests_k8s.sh --help for full usage information.

Usage: ci/run_tests_k8s.sh [OPTIONS] [GROUP_NAMES...]

OPTIONS:
    --context CONTEXT    K8s context to use (REQUIRED unless --kubeconfig set)
    --kubeconfig FILE    Path to kubeconfig file (REQUIRED unless --context set)
    --push-image IMAGE   Push local image to OpenShift internal registry before testing
                         Sets TEST_IMAGE to the internal registry reference
    --verbose, -v        Show full output for passing tests
    --pause, -p          Pause on test failure before cleanup to allow debugging
    --component-name NAME
                         Parse component name (format: group--distro--variant)
                         Example: curl--rawhide--default → curl/rawhide/default
    --group-component-name NAME
                         Parse component name and add /group variant
                         Example: curl--rawhide--default → curl/rawhide/group
    --include-reverse-deps
                         Also test groups that depend on specified groups
    --output FILE        Write JSON test results to FILE (for CI integration)
                         When set, exits 0 after writing results regardless of test outcome
    --help, -h           Show this help message

Safety: The script requires explicit --context or --kubeconfig to prevent accidental operations on production clusters.

Local Development Workflow

For local development, build images and push them to the internal registry:

# Build the image locally
podman build -t my-nginx:dev images/nginx/hummingbird/default/

# Push to internal registry and test
ci/run_tests_k8s.sh --context mpp-preprod --push-image my-nginx:dev nginx/hummingbird/default

The --push-image flag:

Compares local image digest with remote
Skips push if image already exists with same digest
Uses port-forward to registry-proxy for pushing
Sets TEST_IMAGE to the internal registry reference

Prerequisites:

oc login to the target cluster
oc project <namespace> to set the target namespace
Port-forward access to registry-proxy in hummingbird--internal

Testing with Published Images

Test published images without building locally:

# Test using published images (resolved via IMAGE_URL/IMAGE_NAME)
IMAGE_URL=quay.io/hummingbird/nginx:latest \
IMAGE_NAME=nginx--hummingbird--default \
    ci/run_tests_k8s.sh --context mpp-preprod nginx/hummingbird/default

Environment Variables

Tests have access to these environment variables:

Variable	Description
`TEST_IMAGE`	Container image under test (with digest)
`TEST_IMAGES`	Associative array with `group/variant` image URLs (uses TEST_DISTRO context)
`TEST_IMAGES_PATH`	Path to file containing serialized `TEST_IMAGES` array
`TEST_GROUP`	The image group being tested
`TEST_DISTRO`	The distro being tested (e.g., `rawhide`)
`TEST_VARIANT`	The variant being tested (e.g., `default`)
`TEST_VERBOSE`	Show test command output (`true` or `false`)
`TEST_RUN_ID`	Unique ID for this test run (for resource naming)
`TEST_RUN_LABEL`	Label selector for cleanup (`hum-k8s-test=<id>`)

Cluster Access

The kubectl command is pre-configured with the context/kubeconfig from CLI args, so tests can use it directly without additional configuration.

Helper Function

Function	Description
`test_fail`	Fail the test with a custom error message

Examples

# Test single group with explicit context (all distro/variants)
ci/run_tests_k8s.sh --context mpp-preprod nginx

# Test specific distro/variant
ci/run_tests_k8s.sh --context mpp-preprod nginx/hummingbird/default

# Test specific test
ci/run_tests_k8s.sh --context mpp-preprod nginx/hummingbird/default/readiness-probe

# Test with verbose output for debugging
ci/run_tests_k8s.sh --verbose --context mpp-preprod nginx

# Test with kubeconfig file (CI environments)
ci/run_tests_k8s.sh --kubeconfig /workspace/kubeconfig nginx/hummingbird/default

# Build locally and push to internal registry
podman build -t my-nginx:dev images/nginx/hummingbird/default/
ci/run_tests_k8s.sh --context mpp-preprod --push-image my-nginx:dev nginx/hummingbird/default

# Test from CI component name format
ci/run_tests_k8s.sh --kubeconfig /workspace/kubeconfig --component-name nginx--hummingbird--default

# Write JSON results for CI integration
ci/run_tests_k8s.sh --output /tmp/results.json --kubeconfig /workspace/kubeconfig nginx/hummingbird/default

Resource Cleanup

Tests should label resources with TEST_RUN_LABEL for automatic cleanup:

kubectl create configmap my-config --from-literal=key=value
kubectl label configmap my-config "${TEST_RUN_LABEL}"

The test runner automatically cleans up all resources with the test run label after each test and at script exit.

CI Integration

In CI environments (Konflux), the script receives:

--kubeconfig pointing to the ephemeral namespace kubeconfig
--component-name or --group-component-name for component identification
--output for structured JSON results

The --output flag writes results in Konflux-compatible format and exits 0, allowing the pipeline to read results without relying on exit codes.

4 - retrigger_failed_checks.py

Retrigger failed Konflux CI checks for a given GitLab merge request

Purpose

Retrigger failed Konflux CI checks by posting retest commands and waiting for the checks to start running.

Usage

Run ci/retrigger_failed_checks.py --help for full usage information.

Usage: ci/retrigger_failed_checks.py <MR_URL> [OPTIONS]

OPTIONS:
    -h, --help              show this help message and exit
    --dry-run               just print what comments would be posted
    --token-path, -t PATH   path to file containing GitLab API token

Authentication (in order of precedence):
  1. --token-path: Path to file containing GitLab API token (highest priority)
  2. GITLAB_TOKEN_PATH: Environment variable with path to token file
  3. GITLAB_TOKEN: Environment variable with GitLab API token (fallback)

Behavior

The script will:

Extract failed pipeline runs from the commit statuses on the MR
Generate /retest {pipeline-name} commands for each failed run
Post the retest commands as MR comments (unless –dry-run)
Wait for the pipeline runs to start (unless –dry-run)

Examples

# Using --token-path option (highest priority)
ci/retrigger_failed_checks.py --token-path /path/to/token https://gitlab.com/group/project/-/merge_requests/1234

# Using GITLAB_TOKEN_PATH environment variable
GITLAB_TOKEN_PATH=/path/to/token ci/retrigger_failed_checks.py https://gitlab.com/group/project/-/merge_requests/1234

# Using GITLAB_TOKEN environment variable (legacy)
GITLAB_TOKEN=your-token-here ci/retrigger_failed_checks.py https://gitlab.com/group/project/-/merge_requests/1234

# Dry-run mode (show what would be done, don't post or wait)
ci/retrigger_failed_checks.py --token-path /path/to/token --dry-run https://gitlab.com/group/project/-/merge_requests/1234

5 - gitlab_sync.py

Sync a generated file to an external GitLab repository via merge request, with automatic merge wait, pipeline retry, and preemption support

Purpose

Sync a file from a public source URL to a target GitLab repository via merge request. The script always waits for the MR to be merged and for the post-merge pipeline to complete before returning. It is invoked by infrastructure CI jobs to keep external repos (pyxis-repo-configs, konflux-release-data) in sync with generated files from the source repos.

Usage

Run gitlab_sync.py --help for full usage information.

usage: gitlab_sync.py [-h] --source-url SOURCE_URL
                      --target-project TARGET_PROJECT
                      --target-file TARGET_FILE --sync-branch SYNC_BRANCH
                      --mr-title MR_TITLE --gitlab-url GITLAB_URL
                      [--merge-timeout MERGE_TIMEOUT]
                      [--post-merge-timeout POST_MERGE_TIMEOUT]
                      [--mr-description MR_DESCRIPTION] [--squash-on-merge]
                      [--dry-run]

Authentication:
  GITLAB_TOKEN    Environment variable with GitLab API token (required)

Monitoring:
  SENTRY_DSN      Sentry DSN for alerting on failures (optional)

How It Works

Fetch source from the public --source-url (with HTTP retries)
Determine comparison ref: use the sync branch if it exists, otherwise the project’s default branch
Compare source content against the comparison ref (stripped whitespace)
Early exit if content matches and the comparison ref is the default branch (nothing to deploy)
Commit the source to the sync branch using the GitLab Commits API with force: true (creates a single-commit branch on top of the default branch). Also auto-squashes if the branch has accumulated multiple commits.
Create or update MR targeting the default branch
Self-approve the MR (best-effort; continues if already approved)
Post takeover comment with CI_JOB_ID for preemption tracking
Wait for MR merge (polling every 30s, up to --merge-timeout)
Wait for post-merge pipeline to succeed (up to --post-merge-timeout)

Exit Codes

Code	Meaning
0	Success (MR merged) or no changes needed
1	Failure (timeout, pipeline failure, unrecoverable error)
42	Preempted by a newer CI job managing the same MR

Preemption

When multiple CI jobs target the same sync branch, the script uses MR comments to coordinate. Each job posts a comment containing its CI_JOB_ID (monotonically increasing within a GitLab instance). Before each merge attempt, the script checks for comments with a higher job ID. If found, it yields by exiting with code 42.

The caller should handle exit 42 to stop processing (a newer job will handle all remaining syncs). When running outside CI (CI_JOB_ID not set), preemption is disabled.

Error Recovery

Pipeline failure: Retries the MR pipeline with exponential backoff (60s, 300s, 900s). Sends a Sentry alert if all retries are exhausted.
Merge conflict: Rebases the MR and re-approves (rebase resets approvals in GitLab). Up to 3 attempts before failing with a Sentry alert.
Post-merge pipeline failure: Retries once, then returns exit code 1. This blocks downstream sync steps (e.g., Pyxis must succeed before RPA).
Draft MR: Keeps polling without attempting to merge (allows manual intervention).

Examples

# Dry run: show diff without modifying anything
GITLAB_TOKEN=$TOKEN gitlab_sync.py \
  --source-url "https://gitlab.com/redhat/hummingbird/containers/-/raw/main/releng/pyxis-hummingbird.yaml" \
  --target-project "releng/pyxis-repo-configs" \
  --target-file "products/hummingbird/hummingbird.yaml" \
  --sync-branch "hummingbird/sync-containers-pyxis" \
  --mr-title "chore: Update hummingbird Pyxis config" \
  --gitlab-url "https://gitlab.cee.redhat.com" \
  --dry-run

# Sync Pyxis config (requires squash merge)
gitlab_sync.py \
  --gitlab-url "https://gitlab.cee.redhat.com" \
  --source-url "https://gitlab.com/redhat/hummingbird/containers/-/raw/main/releng/pyxis-hummingbird.yaml" \
  --target-project "releng/pyxis-repo-configs" \
  --target-file "products/hummingbird/hummingbird.yaml" \
  --sync-branch "hummingbird/sync-containers-pyxis" \
  --mr-title "chore: Update hummingbird Pyxis config" \
  --squash-on-merge

# Sync RPM RPA to konflux-release-data
gitlab_sync.py \
  --gitlab-url "https://gitlab.cee.redhat.com" \
  --source-url "https://gitlab.com/redhat/hummingbird/rpms/-/raw/main/releng/hummingbird-rpms-staging.yaml" \
  --target-project "releng/konflux-release-data" \
  --target-file "config/kflux-prd-rh03.nnv1.p1/product/ReleasePlanAdmission/hummingbird/hummingbird-rpms-staging.yaml" \
  --sync-branch "hummingbird/sync-rpms-rpa" \
  --mr-title "Update hummingbird RPM ReleasePlanAdmission"

Preemption wrapper for sequential syncs in a CI job:

gitlab_sync.py --gitlab-url ... --source-url ... --squash-on-merge
rc=$?
if [ "$rc" -eq 42 ]; then
  echo "Preempted by newer job, stopping"
  exit 0
elif [ "$rc" -ne 0 ]; then
  exit "$rc"
fi
# Proceed to next sync only on success
gitlab_sync.py --gitlab-url ... --source-url ...

Environment Variables

Variable	Required	Description
`GITLAB_TOKEN`	Yes	API token with Developer+ access on the target project
`CI_JOB_ID`	No	Set automatically in CI; enables preemption detection
`SENTRY_DSN`	No	Sentry DSN for failure alerting

Development

The script and gitlab-ci tool image are maintained in the Hummingbird tools repository, not this repository. Make implementation and test changes there.