# Testing Guide

> How to run and write integration tests locally and work with CI

---

LLMS index: [llms.txt](/llms.txt) | Full content: [llms-full.txt](/llms-full.txt)

---

Container tests validate image functionality with Docker and Podman. K8s tests validate
images in real Kubernetes environments.

## Running Container Tests Locally

### Prerequisites

- **Container Engine**: [Podman](https://podman.io/) or [Docker](https://www.docker.com/)
- **Python**: `PyYAML` package (`pip install PyYAML` or `dnf install python3-pyyaml`)

### With Podman (Recommended)

Tests work directly with Podman:

```bash
ci/run_tests_container.sh <image-name>

# Test specific distro
ci/run_tests_container.sh <image-name>/rawhide

# Test specific distro/variant
ci/run_tests_container.sh <image-name>/rawhide/default

# Verbose output (shows passing test output and bash trace)
ci/run_tests_container.sh --verbose <image-name>
```

### With Docker

Use `--setup` to automatically configure Docker-in-Docker:

```bash
ci/run_tests_container.sh --engine docker --setup <image-name>
```

The `--setup` flag:

- Starts `hummingbird-docker-dind` container with `mirror.gcr.io/docker:dind`
- Configures environment variables (`DOCKER_HOST`, `DOCKER_CERT_PATH`, etc.)
- Waits for Docker daemon to be ready
- Reuses existing container on subsequent runs

**Prerequisites:**

- Docker CLI (`dnf install docker-cli`)
- Podman (to run the dind container)

### Building Images

Build images before testing:

```bash
# With Podman
ci/build_images.sh <image-name>

# With Docker
ci/build_images.sh --engine docker --setup <image-name>
```

### Testing Base Images

When modifying base images, test dependent images:

```bash
ci/build_images.sh core-runtime
ci/build_images.sh --build-deps core-runtime
ci/run_tests_container.sh --include-reverse-deps core-runtime
```

### Reproducing CI Failures

Reproduce Testing Farm failures locally:

```bash
# Single distro/variant tests
ci/run_tests_container.sh --include-reverse-deps --engine podman <image-name>/rawhide/default
ci/run_tests_container.sh --include-reverse-deps --engine docker --setup <image-name>/rawhide/default

# Group tests (all distros/variants)
ci/run_tests_container.sh --engine podman <image-name>/rawhide/group
ci/run_tests_container.sh --engine docker --setup <image-name>/rawhide/group
```

### Troubleshooting Container Tests

#### Inspecting Failed Tests

Use `--pause` to inspect containers before cleanup:

```bash
ci/run_tests_container.sh --pause <image-name>
```

#### Viewing Test Output

By default, only failing tests show output. Use `--verbose` to see passing test output:

```bash
ci/run_tests_container.sh --verbose <image-name>
```

This also enables bash trace mode (`set -x`), showing each command as it executes.

## Running K8s Tests Locally

K8s tests require kubectl and access to a Kubernetes cluster (via kubeconfig or context).

### Basic Usage

```bash
ci/run_tests_k8s.sh --context <context> <image-name>

# Test specific distro/variant
ci/run_tests_k8s.sh --context <context> <image-name>/rawhide/default

# Verbose output
ci/run_tests_k8s.sh --verbose --context <context> <image-name>
```

**Safety**: The script requires explicit `--context` or `--kubeconfig` to prevent
accidental operations on production clusters.

### Local Development Workflow

Build images locally and push to the internal registry:

```bash
# Build the image
podman build -t my-nginx:dev images/nginx/hummingbird/default/

# Push to internal registry and test
ci/run_tests_k8s.sh --context mpp-preprod --push-image my-nginx:dev nginx/hummingbird/default
```

This requires `oc login` to the target cluster and `oc project <namespace>` to set
the target namespace.

### Testing Published Images

Test published images without building locally:

```bash
IMAGE_URL=quay.io/hummingbird/nginx:latest \
IMAGE_NAME=nginx--hummingbird--default \
    ci/run_tests_k8s.sh --context mpp-preprod nginx/hummingbird/default
```

### Troubleshooting K8s Tests

Use `--pause` to inspect resources before cleanup, or `--verbose` to see passing
test output:

```bash
ci/run_tests_k8s.sh --pause --context <context> <image-name>
ci/run_tests_k8s.sh --verbose --context <context> <image-name>
```

Tests that encounter recognized transient infrastructure failures run up to three times. The
runner cleans up test resources and waits 60 seconds before each retry. Application assertion
failures are reported immediately. Recognized failures include transient registry and container
daemon errors, package-download timeouts from the Hummingbird repository, and transient Go
`crypto/rand` entropy-read failures on FIPS-enabled hosts.

## Writing Tests

### Test File Locations

- **Container tests**: `images/<name>/tests-container.yml`
- **K8s tests**: `images/<name>/tests-k8s.yml`

Both use the same YAML format with different available environment variables.

### Basic Tests

Create a test file with named tests:

```yaml
---
version-check:
  command: |
    test_engine_run --rm "${TEST_IMAGE}" your-service --version

basic-functionality:
  command: |
    test_engine_run --rm "${TEST_IMAGE}" your-service --help
```

Container tests must use `test_engine_run` to start containers. When a detached container has both
a name and a network, the helper waits for Podman network DNS registration automatically before
returning. Fresh lookup processes avoid negative DNS caching in a single client process:

```bash
test_engine_run -d --network "${NETWORK_NAME}" --name "${CONTAINER_NAME}" "${TEST_IMAGE}"
```

For K8s tests:

```yaml
---
cluster-access:
  command: |
    kubectl auth can-i get pods || test_fail "No pod access"

run-as-pod:
  command: |
    name="test-pod-${TEST_RUN_ID}"
    kubectl run "${name}" --image="${TEST_IMAGE}" --restart=Never --labels="${TEST_RUN_LABEL}"
    kubectl wait --for=jsonpath='{.status.phase}'=Succeeded "pod/${name}" --timeout=60s
```

### External Test Scripts

For complex tests, use a separate shell script:

```yaml
---
complex-test:
  command: ./test-complex-scenario.sh
```

Create `images/<name>/test-complex-scenario.sh`:

```bash
#!/bin/bash
set -euo pipefail

# Load TEST_IMAGES array for cross-variant testing
# shellcheck disable=SC1090
source "${TEST_IMAGES_PATH:?}"

# Run test
test_engine_run --rm "${TEST_IMAGE:?}" your-service test
```

### Variant-Specific Tests

Limit tests to specific variants:

```yaml
build-tools-test:
  variants: [builder]
  command: |
    test_engine_run --rm "${TEST_IMAGE}" make --version
```

### Cross-Variant Tests

Test interactions between variants:

```yaml
cross-variant-test:
  variants: [group]
  command: |
    builder="${TEST_IMAGES[nginx/builder]:?}"
    default="${TEST_IMAGES[nginx/default]:?}"

    test_engine_run --rm "${builder}" nginx -version
    test_engine_run --rm "${default}" nginx -version
```

### Known Issues (Container Tests Only)

Mark container tests with known failures:

```yaml
test-with-known-issue:
  command: |
    result=$(test_engine_run --rm "${TEST_IMAGE}" some-command)
    [[ "$result" == "expected" ]] || test_fail "Custom error message"
  known_issues:
    - description: "Known configuration issue"
      issue: "https://issues.redhat.com/browse/PROJ-1234"
      pattern: "Custom error message"
      fails: "sometimes"
```

See the [Test Configuration Reference](test-configuration-reference.md) for complete
configuration options.

## Working with CI Tests

### Container Tests (Testing Farm)

Container tests run on Testing Farm with RHEL-9-Nightly systems for both `x86_64`
and `aarch64` architectures.

#### Viewing CI Test Results

1. Open the merge request in GitLab
2. Navigate to the Jobs page for the pipeline
3. Identify the Testing Farm job (e.g., `containers-rawhide-testing-farm-x86-64`)
4. Click on the job name to go to the Konflux PipelineRun page
5. If `wait-for-results` is already finished, select it in the pipeline details, and then switch to
   the Testing Farm job via the `ARTIFACTS_URL` link in the right side pane; otherwise, select the
   `scheduler` job, switch to the Testing Farm API details via the `tf-request` link in the right
   side pane, and then follow the `run.artifacts` link in the JSON data

### K8s Tests (Konflux Ephemeral Namespace)

K8s tests run in Konflux ephemeral namespaces provisioned via EaaS (Environment as a Service).

#### Viewing K8s Test Results

1. Open the merge request in GitLab
2. Navigate to the Jobs page for the pipeline
3. Identify the K8s test job (e.g., `containers-rawhide-k8s-test`)
4. Click on the job name to go to the Konflux PipelineRun page
5. View the `run-tests` task logs for test output

### Rerunning CI Tests

Retrigger a specific pipeline using slash commands in merge request comments:

```text
/retest nginx--hummingbird--default-on-pull-request
```

For automated retriggers of only failed checks, see [Retrying Konflux
Checks](https://hummingbird-project.io/l/retrying-konflux-checks).

## Next Steps

- [Test Configuration Reference](test-configuration-reference.md) - Complete test configuration
  reference
- [Adding Images](adding-images.md) - How to add new images with tests