Testing Guide

How to run and write integration tests locally and work with CI

Container tests validate image functionality with Docker and Podman. K8s tests validate images in real Kubernetes environments.

Running Container Tests Locally

Prerequisites

Container Engine: Podman or Docker
Python: PyYAML package (pip install PyYAML or dnf install python3-pyyaml)

With Podman (Recommended)

Tests work directly with Podman:

ci/run_tests_container.sh <image-name>

# Test specific distro
ci/run_tests_container.sh <image-name>/rawhide

# Test specific distro/variant
ci/run_tests_container.sh <image-name>/rawhide/default

# Verbose output (shows passing test output and bash trace)
ci/run_tests_container.sh --verbose <image-name>

With Docker

Use --setup to automatically configure Docker-in-Docker:

ci/run_tests_container.sh --engine docker --setup <image-name>

The --setup flag:

Starts hummingbird-docker-dind container with mirror.gcr.io/docker:dind
Configures environment variables (DOCKER_HOST, DOCKER_CERT_PATH, etc.)
Waits for Docker daemon to be ready
Reuses existing container on subsequent runs

Prerequisites:

Docker CLI (dnf install docker-cli)
Podman (to run the dind container)

Building Images

Build images before testing:

# With Podman
ci/build_images.sh <image-name>

# With Docker
ci/build_images.sh --engine docker --setup <image-name>

Testing Base Images

When modifying base images, test dependent images:

ci/build_images.sh --include-reverse-deps core-runtime
ci/run_tests_container.sh --include-reverse-deps core-runtime

Reproducing CI Failures

Reproduce Testing Farm failures locally:

# Single distro/variant tests
ci/run_tests_container.sh --include-reverse-deps --engine podman <image-name>/rawhide/default
ci/run_tests_container.sh --include-reverse-deps --engine docker --setup <image-name>/rawhide/default

# Group tests (all distros/variants)
ci/run_tests_container.sh --engine podman <image-name>/group
ci/run_tests_container.sh --engine docker --setup <image-name>/group

Troubleshooting Container Tests

Inspecting Failed Tests

Use --pause to inspect containers before cleanup:

ci/run_tests_container.sh --pause <image-name>

Viewing Test Output

By default, only failing tests show output. Use --verbose to see passing test output:

ci/run_tests_container.sh --verbose <image-name>

This also enables bash trace mode (set -x), showing each command as it executes.

Running K8s Tests Locally

K8s tests require kubectl and access to a Kubernetes cluster (via kubeconfig or context).

Basic Usage

ci/run_tests_k8s.sh --context <context> <image-name>

# Test specific distro/variant
ci/run_tests_k8s.sh --context <context> <image-name>/rawhide/default

# Verbose output
ci/run_tests_k8s.sh --verbose --context <context> <image-name>

Safety: The script requires explicit --context or --kubeconfig to prevent accidental operations on production clusters.

Local Development Workflow

Build images locally and push to the internal registry:

# Build the image
podman build -t my-gitlab-ci:dev ci/images/gitlab-ci/

# Push to internal registry and test
ci/run_tests_k8s.sh --context mpp-preprod --push-image my-gitlab-ci:dev gitlab-ci/hummingbird/default

This requires oc login to the target cluster and oc project <namespace> to set the target namespace.

Testing Published Images

Test published images without building locally:

IMAGE_URL=quay.io/hummingbird-ci/gitlab-ci:latest \
IMAGE_NAME=gitlab-ci--hummingbird--default \
    ci/run_tests_k8s.sh --context mpp-preprod gitlab-ci/hummingbird/default

Troubleshooting K8s Tests

Use --pause to inspect resources before cleanup, or --verbose to see passing test output:

ci/run_tests_k8s.sh --pause --context <context> <image-name>
ci/run_tests_k8s.sh --verbose --context <context> <image-name>

Writing Tests

Test File Locations

Container tests: images/<name>/tests-container.yml
K8s tests: images/<name>/tests-k8s.yml

Both use the same YAML format with different available environment variables.

Basic Tests

Create a test file with named tests:

---
version-check:
  command: |
    test_engine_run --rm "${TEST_IMAGE}" your-service --version

basic-functionality:
  command: |
    test_engine_run --rm "${TEST_IMAGE}" your-service --help

For K8s tests:

---
cluster-access:
  command: |
    kubectl auth can-i get pods || test_fail "No pod access"

run-as-pod:
  command: |
    name="test-pod-${TEST_RUN_ID}"
    kubectl run "${name}" --image="${TEST_IMAGE}" --restart=Never --labels="${TEST_RUN_LABEL}"
    kubectl wait --for=jsonpath='{.status.phase}'=Succeeded "pod/${name}" --timeout=60s

External Test Scripts

For complex tests, use a separate shell script:

---
complex-test:
  command: ./test-complex-scenario.sh

Create images/<name>/test-complex-scenario.sh:

#!/bin/bash
set -euo pipefail

# Load TEST_IMAGES array for cross-variant testing
# shellcheck disable=SC1090
source "${TEST_IMAGES_PATH:?}"

# Run test
test_engine_run --rm "${TEST_IMAGE:?}" your-service test

Variant-Specific Tests

Limit tests to specific variants:

build-tools-test:
  variants: [builder]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" make --version

Cross-Variant Tests

Test interactions between variants:

cross-variant-test:
  variants: [group]
  command: |
    builder="${TEST_IMAGES[nginx/builder]:?}"
    default="${TEST_IMAGES[nginx/default]:?}"

    test_engine_run --rm "${builder}" nginx -version
    test_engine_run --rm "${default}" nginx -version

Known Issues (Container Tests Only)

Mark container tests with known failures:

test-with-known-issue:
  command: |
    result=$(test_engine_run --rm "${TEST_IMAGE}" some-command)
    [[ "$result" == "expected" ]] || test_fail "Custom error message"
  known_issues:
    - description: "Known configuration issue"
      issue: "https://issues.redhat.com/browse/PROJ-1234"
      pattern: "Custom error message"
      fails: "sometimes"

See the Test Configuration Reference for complete configuration options.

Working with CI Tests

Container Tests (Testing Farm)

Container tests run on Testing Farm with RHEL-9-Nightly systems for both x86_64 and aarch64 architectures.

Viewing CI Test Results

Open the merge request in GitLab
Navigate to the Jobs page for the pipeline
Identify the Testing Farm job (e.g., containers-rawhide-testing-farm-x86-64)
Click on the job name to go to the Konflux PipelineRun page
If wait-for-results is already finished, select it in the pipeline details, and then switch to the Testing Farm job via the ARTIFACTS_URL link in the right side pane; otherwise, select the scheduler job, switch to the Testing Farm API details via the tf-request link in the right side pane, and then follow the run.artifacts link in the JSON data

K8s Tests (Konflux Ephemeral Namespace)

K8s tests run in Konflux ephemeral namespaces provisioned via EaaS (Environment as a Service).

Viewing K8s Test Results

Open the merge request in GitLab
Navigate to the Jobs page for the pipeline
Identify the K8s test job (e.g., containers-rawhide-k8s-test)
Click on the job name to go to the Konflux PipelineRun page
View the run-tests task logs for test output

Rerunning CI Tests

Retrigger a specific pipeline using slash commands in merge request comments:

/retest gitlab-ci--default--main-on-pull-request

For automated retriggers of only failed checks, see Retrying Konflux Checks.

Next Steps

Test Configuration Reference - Complete test configuration reference
Adding Images - How to add new images with tests