run_tests_k8s.sh

Run K8s tests for image groups using kubectl with support for local development and CI environments

Purpose

Run K8s tests for image groups using kubectl. Tests execute in a real Kubernetes environment, validating that images work correctly in Kubernetes.

Usage

Run ci/run_tests_k8s.sh --help for full usage information.

Usage: ci/run_tests_k8s.sh [OPTIONS] [GROUP_NAMES...]

OPTIONS:
    --context CONTEXT    K8s context to use (REQUIRED unless --kubeconfig set)
    --kubeconfig FILE    Path to kubeconfig file (REQUIRED unless --context set)
    --push-image IMAGE   Push local image to OpenShift internal registry before testing
                         Sets TEST_IMAGE to the internal registry reference
    --verbose, -v        Show full output for passing tests
    --pause, -p          Pause on test failure before cleanup to allow debugging
    --component-name NAME
                         Parse component name (format: group--distro--variant)
                         Example: curl--rawhide--default → curl/rawhide/default
    --group-component-name NAME
                         Parse component name and add /group variant
                         Example: curl--rawhide--default → curl/group
    --include-reverse-deps
                         Also test groups that depend on specified groups
    --output FILE        Write JSON test results to FILE (for CI integration)
                         When set, exits 0 after writing results regardless of test outcome
    --help, -h           Show this help message

Safety: The script requires explicit --context or --kubeconfig to prevent accidental operations on production clusters.

Note: If no distro/variant is specified, all combinations will be tested. To test only a specific variant, use <group_name>/<distro>/<variant>. To test only a specific test, use <group_name>/<distro>/<variant>/<test>.

Local Development Workflow

For local development, build images and push them to the internal registry:

# Build the image locally
podman build -t my-gitlab-ci:dev ci/images/gitlab-ci/

# Push to internal registry and test
ci/run_tests_k8s.sh --context mpp-preprod --push-image my-gitlab-ci:dev gitlab-ci/default

The --push-image flag:

Compares local image digest with remote
Skips push if image already exists with same digest
Uses port-forward to registry-proxy for pushing
Sets TEST_IMAGE to the internal registry reference

Prerequisites:

oc login to the target cluster
oc project <namespace> to set the target namespace
Port-forward access to registry-proxy in hummingbird--internal

Testing with Published Images

Test published images without building locally:

# Test using published images (resolved via IMAGE_URL/IMAGE_NAME)
IMAGE_URL=quay.io/hummingbird-ci/gitlab-ci:latest \
IMAGE_NAME=gitlab-ci--default--main \
    ci/run_tests_k8s.sh --context mpp-preprod gitlab-ci/default

Environment Variables

Tests have access to these environment variables:

Variable	Description
`TEST_IMAGE`	Container image under test (with digest)
`TEST_IMAGES`	Associative array with `group/variant` image URLs (uses TEST_DISTRO context)
`TEST_IMAGES_PATH`	Path to file containing serialized `TEST_IMAGES` array
`TEST_GROUP`	The image group being tested
`TEST_DISTRO`	The distro being tested (e.g., `rawhide`)
`TEST_VARIANT`	The variant being tested (e.g., `default`)
`TEST_VERBOSE`	Show test command output (`true` or `false`)
`TEST_RUN_ID`	Unique ID for this test run (for resource naming)
`TEST_RUN_LABEL`	Label selector for cleanup (`hum-k8s-test=<id>`)

Cluster Access

The kubectl command is pre-configured with the context/kubeconfig from CLI args, so tests can use it directly without additional configuration.

Helper Function

Function	Description
`test_fail`	Fail the test with a custom error message

Examples

# Test single group with explicit context (all distro/variants)
ci/run_tests_k8s.sh --context mpp-preprod gitlab-ci

# Test specific distro/variant
ci/run_tests_k8s.sh --context mpp-preprod gitlab-ci/hummingbird/default

# Test specific test
ci/run_tests_k8s.sh --context mpp-preprod gitlab-ci/hummingbird/default/kubectl-cluster-access

# Test with verbose output for debugging
ci/run_tests_k8s.sh --verbose --context mpp-preprod gitlab-ci

# Test with kubeconfig file (CI environments)
ci/run_tests_k8s.sh --kubeconfig /workspace/kubeconfig gitlab-ci/hummingbird/default

# Build locally and push to internal registry
podman build -t my-gitlab-ci:dev ci/images/gitlab-ci/hummingbird/default
ci/run_tests_k8s.sh --context mpp-preprod --push-image my-gitlab-ci:dev gitlab-ci/hummingbird/default

# Test from CI component name format
ci/run_tests_k8s.sh --kubeconfig /workspace/kubeconfig --component-name gitlab-ci--hummingbird--default

# Write JSON results for CI integration
ci/run_tests_k8s.sh --output /tmp/results.json --kubeconfig /workspace/kubeconfig gitlab-ci/hummingbird/default

Resource Cleanup

Tests should label resources with TEST_RUN_LABEL for automatic cleanup:

kubectl create configmap my-config --from-literal=key=value
kubectl label configmap my-config "${TEST_RUN_LABEL}"

The test runner automatically cleans up all resources with the test run label after each test and at script exit.

CI Integration

In CI environments (Konflux), the script receives:

--kubeconfig pointing to the ephemeral namespace kubeconfig
--component-name or --group-component-name for component identification
--output for structured JSON results

The --output flag writes results in Konflux-compatible format and exits 0, allowing the pipeline to read results without relying on exit codes.