# run_tests_container.sh

> Run tests for image groups using containerized test environment with support for both Docker and Podman engines

---

LLMS index: [llms.txt](https://hummingbird-project.io/llms.txt) | Full content: [llms-full.txt](https://hummingbird-project.io/llms-full.txt)

---

## Purpose

Run tests for image groups using containerized test environment with support
for both Docker and Podman engines.

## Usage

Run `ci/run_tests_container.sh --help` for full usage information.

```text
Usage: ci/run_tests_container.sh [OPTIONS] [GROUP_NAMES...]

OPTIONS:
    --verbose, -v        Enable verbose output during testing
    --engine ENGINE      Specify container engine (podman or docker)
    --setup              Set up Docker-in-Docker environment before running tests
    --pause, -p          Pause failed tests before cleanup to allow debugging
                         Prints a message and waits for Enter before cleaning up containers
    --hermetic           Use hermetic builds (--pull=never, only use prefetched/built images)
                         Default for CI. Without this, missing images are pulled on demand.
    --component-name NAME
                         Parse component name (format: group--distro--variant)
                         Example: curl--rawhide--default → curl/rawhide/default
    --group-component-name NAME
                         Parse component name and add /group variant
                         Example: curl--rawhide--default → curl/rawhide/group
    --affected-tests     Run only affected dependent image checks, retaining globals
    --dryrun             Print selected tests without invoking container engines
    --include-reverse-deps
                         Also test groups that depend on specified groups
    --help, -h           Show this help message
```

**Note**: If no distro/variant is specified, all combinations will be tested. To test
only a specific variant, use `<group_name>/<distro>/<variant>`. To test only a
specific test, use `<group_name>/<distro>/<variant>/<test>`.

**Engine Selection**: Use `--engine` to specify the container engine (`podman`
or `docker`). When using Docker, add `--setup` for automatic Docker-in-Docker
environment setup.

**Hermetic vs Non-Hermetic Testing**:

- **Local development** (default): Without `--hermetic`, the test runner allows
  pulling missing images from the registry. This is convenient for testing
  individual images without building all dependencies first.
- **CI environment**: Use `--hermetic` to enforce that tests only use
  prefetched or locally-built images (via `--pull=never`). This ensures
  reproducible builds and prevents accidentally using images from the registry
  that differ from what Konflux built.

## Building and Testing

For local development, you'll typically want to build the images first, then
test them:

```bash
# Build the images
ci/build_images.sh <group_name>[/distro/variant]
ci/build_images.sh <group_name1> <group_name2>  # Build multiple image groups

# Test the images
ci/run_tests_container.sh <group_name>[/distro/variant]
ci/run_tests_container.sh <group_name1> <group_name2>  # Test multiple image groups
```

The build script uses the same syntax as the test script, making it easy to
build and test the same image/variant combination.

When working on base images (like `core-runtime`) that other images depend on,
build the base image and its dependencies before running reverse-dependency tests:

```bash
# Build core-runtime, then its forward and reverse dependencies
ci/build_images.sh core-runtime
ci/build_images.sh --build-deps core-runtime

# Then test them all
ci/run_tests_container.sh --include-reverse-deps core-runtime
```

The `-p`/`--pause` option pauses the script on failed tests before cleaning up
to allow interactive and efficient debugging.

## Testing with Specific Image Builds

To reproduce CI failures, you can run the test with mapping the image under test to the Konflux build. Set `IMAGE_URL_<GROUP>__<DISTRO>__<VARIANT>` environment variables (uppercase, hyphens/dots → underscores, `__` separates parts):

```bash
# Test with a specific CI build
IMAGE_URL_TOMCAT_10__HUMMINGBIRD__BUILDER='quay.io/redhat-user-workloads/.../tomcat-10--hummingbird--builder@sha256:...' \
  ci/run_tests_container.sh tomcat-10/hummingbird/builder
```

## Automatic Retries for Transient Infrastructure Failures

The test runner automatically retries tests that fail with certain transient
infrastructure errors. This helps avoid false test failures caused by temporary
issues with external services like container registries or network problems.

**Retry Behavior**:

- Failed tests are checked against a list of retriable error patterns
- If a match is found, the test is automatically retried
- If the test still fails after all attempts, it's reported as a normal failure

## Examples

```bash
# Test single group (all distro/variants)
ci/run_tests_container.sh curl
ci/run_tests_container.sh nginx

# Test single group (specific distro/variant)
ci/run_tests_container.sh curl/rawhide/default
ci/run_tests_container.sh nodejs-20/hummingbird/builder

# Test multiple groups (all distro/variants)
ci/run_tests_container.sh curl nginx mariadb

# Test multiple groups (mixed variants)
ci/run_tests_container.sh curl/rawhide/default nginx mariadb/hummingbird/builder

# Test with different engines
ci/run_tests_container.sh --engine docker --setup git
ci/run_tests_container.sh --engine podman --verbose nginx

# Test with verbose output for debugging
ci/run_tests_container.sh --verbose nginx mariadb dotnet-runtime-8-0

# Test specific tests
ci/run_tests_container.sh curl/rawhide/default/version
ci/run_tests_container.sh nginx/rawhide/default/port
ci/run_tests_container.sh curl/rawhide/default/version nginx/hummingbird/default/port

# Also test groups with reverse dependencies
ci/run_tests_container.sh --include-reverse-deps core-runtime

# Test from CI component name format (used in CI environments)
ci/run_tests_container.sh --component-name curl--rawhide--default

# Test across variants from CI component name format
ci/run_tests_container.sh --group-component-name curl--rawhide--default
```

Reverse dependency testing is enabled by default for all images. When enabled,
changes to an image will automatically:

- Find images that depend on it by scanning for `TEST_IMAGES[group/distro/variant]`
  references
- Run tests for both the changed image and all its dependents

This catches breaking changes in base images early. Images can opt-out by setting
`reverse_dependency_tests: false` in `properties.yml` (e.g., `curl`, which is
widely used but typically doesn't need reverse dependency testing).

## Affected dependency tests

Testing Farm uses `--affected-tests` for single-component jobs. Dependency pulls
use `ci/build_images.sh --build-deps --affected-tests --pull` with the same selector.
The requested component runs its complete suite. Dependent variants run only
image checks that reference the changed component variant, plus all applicable
global checks.

Use `--dryrun` to list runnable checks without invoking container engines.
Use `--include-reverse-deps` without `--affected-tests` to run full dependent suites.

Run each suite with an isolated container engine. Cleanup preserves resources
present at startup and removes resources created during the suite, including
resources created concurrently by other processes.