run_tests_container.sh

Run tests for image groups using containerized test environment with support for both Docker and Podman engines

Purpose

Run tests for image groups using containerized test environment with support for both Docker and Podman engines.

Usage

Run ci/run_tests_container.sh --help for full usage information.

Usage: ci/run_tests_container.sh [OPTIONS] [GROUP_NAMES...]

OPTIONS:
    --verbose, -v        Enable verbose output during testing
    --engine ENGINE      Specify container engine (podman or docker)
    --setup              Set up Docker-in-Docker environment before running tests
    --pause, -p          Pause failed tests before cleanup to allow debugging
                         Prints a message and waits for Enter before cleaning up containers
    --component-name NAME
                         Parse component name (format: group--variant--branch)
                         Example: curl--default--main → curl/default
    --group-component-name NAME
                         Parse component name and add /group variant
                         Example: curl--default--main → curl/group
    --include-reverse-deps
                         Also test groups that depend on specified groups
    --help, -h           Show this help message

Note: If no variant is specified, all variants will be tested. To test only a specific variant, use <group_name>/<variant>. To test only a specific test, use <group_name>/<variant>/<test>.

Engine Selection: Use --engine to specify the container engine (podman or docker). When using Docker, add --setup for automatic Docker-in-Docker environment setup.

Building and Testing

For local development, you’ll typically want to build the images first, then test them:

# Build the images
ci/build_images.sh <group_name>[/variant]
ci/build_images.sh <group_name1> <group_name2>  # Build multiple image groups

# Test the images
ci/run_tests_container.sh <group_name>[/variant]
ci/run_tests_container.sh <group_name1> <group_name2>  # Test multiple image groups

The build script uses the same syntax as the test script, making it easy to build and test the same image/variant combination.

When working on base images (like core-runtime) that other images depend on, use --include-reverse-deps to build both the base image and all images that depend on it in one command:

# Build core-runtime and all images that depend on it
ci/build_images.sh --include-reverse-deps core-runtime

# Then test them all
ci/run_tests_container.sh --include-reverse-deps core-runtime

The -p/--pause option pauses the script on failed tests before cleaning up to allow interactive and efficient debugging.

Automatic Retries for Transient Infrastructure Failures

The test runner automatically retries tests that fail with certain transient infrastructure errors. This helps avoid false test failures caused by temporary issues with external services like container registries or network problems.

Retry Behavior:

Failed tests are checked against a list of retriable error patterns
If a match is found, the test is automatically retried
If the test still fails after all attempts, it’s reported as a normal failure

Examples

# Test single group (all variants)
ci/run_tests_container.sh curl
ci/run_tests_container.sh nginx

# Test single group (specific variant)
ci/run_tests_container.sh curl/default
ci/run_tests_container.sh nodejs-20/builder

# Test multiple groups (all variants)
ci/run_tests_container.sh curl nginx mariadb

# Test multiple groups (mixed variants)
ci/run_tests_container.sh curl/default nginx mariadb/builder

# Test with different engines
ci/run_tests_container.sh --engine docker --setup git
ci/run_tests_container.sh --engine podman --verbose nginx

# Test with verbose output for debugging
ci/run_tests_container.sh --verbose nginx mariadb dotnet-runtime-8-0

# Test specific tests
ci/run_tests_container.sh curl/default/version
ci/run_tests_container.sh nginx/default/port
ci/run_tests_container.sh curl/default/version nginx/default/port

# Also test groups with reverse dependencies
ci/run_tests_container.sh --include-reverse-deps core-runtime

# Test from CI component name format (used in CI environments)
ci/run_tests_container.sh --component-name curl--default--main

# Test across variants from CI component name format
ci/run_tests_container.sh --group-component-name curl--default--main

Reverse dependency testing is enabled by default for all images. When enabled, changes to an image will automatically:

Find images that depend on it by scanning for TEST_IMAGES[group/variant] references
Run tests for both the changed image and all its dependents

This catches breaking changes in base images early. Images can opt-out by setting reverse_dependency_tests: false in properties.yml (e.g., curl, which is widely used but typically doesn’t need reverse dependency testing).