run_tests_k8s.sh

Run K8s tests for image groups using kubectl with support for local development and CI environments

Purpose

Run K8s tests for image groups using kubectl. Tests execute in a real Kubernetes environment, validating that images work correctly in Kubernetes.

Usage

Run ci/run_tests_k8s.sh --help for full usage information.

Usage: ci/run_tests_k8s.sh [OPTIONS] [GROUP_NAMES...]

OPTIONS:
    --context CONTEXT    K8s context to use (REQUIRED unless --kubeconfig set)
    --kubeconfig FILE    Path to kubeconfig file (REQUIRED unless --context set)
    --push-image IMAGE   Push local image to OpenShift internal registry before testing
                         Sets TEST_IMAGE to the internal registry reference
    --verbose, -v        Show full output for passing tests
    --pause, -p          Pause on test failure before cleanup to allow debugging
    --component-name NAME
                         Parse component name (format: group--distro--variant)
                         Example: curl--rawhide--default → curl/rawhide/default
    --group-component-name NAME
                         Parse component name and add /group variant
                         Example: curl--rawhide--default → curl/rawhide/group
    --include-reverse-deps
                         Also run tests that use the specified image variants
    --output FILE        Write JSON test results to FILE (for CI integration)
                         When set, exits 0 after writing results regardless of test outcome
    --help, -h           Show this help message

Safety: The script requires explicit --context or --kubeconfig to prevent accidental operations on production clusters.

Note: If no distro/variant is specified, all combinations will be tested. To test only a specific variant, use <group_name>/<distro>/<variant>. To test only a specific test, use <group_name>/<distro>/<variant>/<test>.

Local Development Workflow

For local development, build images and push them to the internal registry:

# Build the image locally
ci/build_images.sh nginx/hummingbird/default
podman tag registry.access.redhat.com/hi/nginx:latest my-nginx:dev

# Push to internal registry and test
ci/run_tests_k8s.sh --context mpp-preprod --push-image my-nginx:dev nginx/hummingbird/default

The --push-image flag:

  • Compares local image digest with remote
  • Skips push if image already exists with same digest
  • Uses port-forward to registry-proxy for pushing
  • Sets TEST_IMAGE to the internal registry reference

Prerequisites:

  • oc login to the target cluster
  • oc project <namespace> to set the target namespace
  • Port-forward access to registry-proxy in hummingbird--internal

Testing with Published Images

Test published images without building locally:

# Test using published images (resolved via IMAGE_URL/IMAGE_NAME)
IMAGE_URL=registry.access.redhat.com/hi/nginx:latest \
IMAGE_NAME=nginx--hummingbird--default \
    ci/run_tests_k8s.sh --context mpp-preprod nginx/hummingbird/default

Reverse dependency selection

--include-reverse-deps uses ci/internal/image_test_plan.py, shared with the container runner, to select dependent tests individually. It reads Kubernetes commands from tests-k8s.yml and follows shell scripts and sourced fixtures for TEST_IMAGES[group/variant] references. Container-only references do not select Kubernetes tests.

ci/run_tests_k8s.sh --context kind-kind --include-reverse-deps nginx/hummingbird/default

This selects all tests for the requested nginx variant and only dependent tests that use nginx/default in the same distro. For example, Python custom CA tests run, but standalone Python version checks do not. Explicit test selectors remain restricted to the requested test. reverse_dependency_tests: false disables selection of consumers of that image.

--include-reverse-deps runs the planner in single-component mode: reverse dependency tests that need a same-group sibling variant are excluded. Such a test references TEST_IMAGES[<same-group>/variant] for a variant other than the one under test (or is itself defined for a same-group variant), but a per-component run provides only the image under test. The sibling would fall back to a released image — silently testing the changed component against an older sibling — or fail to pull when no release exists. Those combinations run in the group pipelines (pr group), which provide every variant of the group at the exact commit. Cross-group reverse dependencies are still selected, and tests of the requested variant itself are unaffected.

Kubernetes variants lists match exact variant names. Tests without a list apply to all image variants except group. Tests with variants: [group] run as group tests. The planner does not apply container engine or host FIPS filters to Kubernetes definitions, or infer the cluster architecture from the local host.

Environment Variables

Tests have access to these environment variables:

Variable Description
TEST_IMAGE Container image under test (with digest)
TEST_IMAGES Associative array with group/variant image URLs (uses TEST_DISTRO context)
TEST_IMAGES_PATH Path to file containing serialized TEST_IMAGES array
TEST_GROUP The image group being tested
TEST_DISTRO The distro being tested (e.g., rawhide)
TEST_VARIANT The variant being tested (e.g., default)
TEST_VERBOSE Show test command output (true or false)
TEST_RUN_ID Unique ID for this test run (for resource naming)
TEST_RUN_LABEL Label selector for cleanup (hum-k8s-test=<id>)

Cluster Access

The kubectl command is pre-configured with the context/kubeconfig from CLI args, so tests can use it directly without additional configuration.

Helper Function

Function Description
test_fail Fail the test with a custom error message

Examples

# Test single group with explicit context (all distro/variants)
ci/run_tests_k8s.sh --context mpp-preprod nginx

# Test specific distro/variant
ci/run_tests_k8s.sh --context mpp-preprod nginx/hummingbird/default

# Test specific test
ci/run_tests_k8s.sh --context mpp-preprod nginx/hummingbird/default/readiness-probe

# Test with verbose output for debugging
ci/run_tests_k8s.sh --verbose --context mpp-preprod nginx

# Test with kubeconfig file (CI environments)
ci/run_tests_k8s.sh --kubeconfig /workspace/kubeconfig nginx/hummingbird/default

# Build locally and push to internal registry
ci/build_images.sh nginx/hummingbird/default
podman tag registry.access.redhat.com/hi/nginx:latest my-nginx:dev
ci/run_tests_k8s.sh --context mpp-preprod --push-image my-nginx:dev nginx/hummingbird/default

# Test from CI component name format
ci/run_tests_k8s.sh --kubeconfig /workspace/kubeconfig --component-name nginx--hummingbird--default

# Write JSON results for CI integration
ci/run_tests_k8s.sh --output /tmp/results.json --kubeconfig /workspace/kubeconfig nginx/hummingbird/default

Resource Cleanup

Tests should label resources with TEST_RUN_LABEL for automatic cleanup:

kubectl create configmap my-config --from-literal=key=value
kubectl label configmap my-config "${TEST_RUN_LABEL}"

The test runner automatically cleans up all resources with the test run label after each test and at script exit.

CI Integration

In CI environments (Konflux), the script receives:

  • --kubeconfig pointing to the ephemeral namespace kubeconfig
  • --component-name or --group-component-name for component identification
  • --output for structured JSON results

The --output flag writes results in Konflux-compatible format and exits 0, allowing the pipeline to read results without relying on exit codes.