Adding Kubernetes Tests

This guide describes how to add Kubernetes tests to container images.

Overview

Kubernetes tests validate that container images work correctly in Kubernetes and OpenShift environments, particularly:

  • Directory permissions (important for OpenShift arbitrary UIDs)
  • Version checks
  • Application functionality (web servers, databases, etc.)
  • Multi-container scenarios (sidecar patterns)

Confirm Test Design

Before implementing:

  1. Read images/{image_name}/tests-container.yml to understand existing tests
  2. Identify which tests need K8s validation:
    • Directory permissions (important for OpenShift arbitrary UIDs)
    • Version check
    • Application functionality (web server, database, etc.)
    • Multi-container scenarios (sidecar patterns)
  3. Confirm test approach before creating files

Rules

  • Script naming: test-k8s-*.sh prefix (e.g., test-k8s-version.sh)
  • Script location: images/{image_name}/test-scripts/
  • YAML location: images/{image_name}/tests-k8s.yml
  • Keep YAML lean: complex logic goes in scripts, not inline YAML
  • No external downloads: avoid network flakes (exception: container images)
  • OpenShift compatibility: writable dirs must use gid=0 + g+w permissions

Validation

  • If images/{image_name}/ doesn’t exist → error “Image not found”
  • If no tests-container.yml → error “Add container tests first”

Create Test Scripts

If directory doesn’t exist → create images/{image_name}/test-scripts/

Test Script Template

#!/bin/bash
set -euo pipefail

# Test logic
test_result=$(command)

if [[ ! "${test_result}" =~ "expected" ]]; then
    echo "FAIL: Description"
    echo "Got: ${test_result}"
    exit 1
fi

echo "SUCCESS: Test passed"
exit 0

Directory Permissions Test

If testing OpenShift compatibility → create test-k8s-directory-permissions.sh:

  • Check writable directories exist and are writable
  • Verify running as non-root (UID != 0)
  • Test actual write with touch/rm

Version Check Test

If testing version → create test-k8s-version.sh:

  • Run version command
  • Check major version only (not major.minor.patch)
  • Avoids breaking on patch updates

Application-Specific Tests

  • If web server → deployment test with sidecar (curl container)
  • If database → connection/persistence test
  • If tool → primary functionality test

Create tests-k8s.yml

Pattern for each test:

test-name:
  variants: [default]
  command: |
    # Create ConfigMap with script
    kubectl create configmap "${TEST_GROUP}"-scripts-"${TEST_RUN_ID}" \
      --from-file=test-k8s-name.sh=test-scripts/test-k8s-name.sh \
      --dry-run=client -o yaml | kubectl label -f - --local -o yaml hum-k8s-test="${TEST_RUN_ID}" | kubectl apply -f -

    # Create Pod
    kubectl apply -f - <<EOF
    apiVersion: v1
    kind: Pod
    metadata:
      name: ${TEST_GROUP}-name-${TEST_RUN_ID}
      labels:
        hum-k8s-test: "${TEST_RUN_ID}"
    spec:
      restartPolicy: Never
      containers:
      - name: test
        image: ${TEST_IMAGE:?}
        command: ["sh", "/scripts/test-k8s-name.sh"]
        volumeMounts:
        - name: test-scripts
          mountPath: /scripts
      volumes:
      - name: test-scripts
        configMap:
          name: ${TEST_GROUP}-scripts-${TEST_RUN_ID}
          defaultMode: 0755
    EOF

    # Wait for completion
    kubectl wait --for=jsonpath='{.status.containerStatuses[?(@.name=="test")].state.terminated.reason}'=Completed \
      pod/"${TEST_GROUP}"-name-"${TEST_RUN_ID}" --timeout=120s || TEST_FAIL "Test did not complete"

    # Check logs
    out=$(kubectl logs "${TEST_GROUP}"-name-"${TEST_RUN_ID}" -c test)
    [[ "${out}" =~ "SUCCESS" ]] || TEST_FAIL "Test failed: ${out}"

Multi-Container Tests

If test needs multiple containers (app + client):

  • Use emptyDir volume mounted in both containers
  • emptyDir has 777 permissions, works with arbitrary UIDs

Readiness Probes

If testing app functionality:

  • Add readinessProbe to app container
  • Set timeoutSeconds: 5 for OpenShift network latency

OpenShift Compatibility

Read images/{image_name}/Containerfile.j2:

  • If writable dirs use user:user ownership → change to user:0
  • If writable dirs lack g+w → add chmod -R g+w
  • Why gid=0? OpenShift runs containers with arbitrary UIDs (for security) but always assigns gid=0 (root group). Directories must be group-owned by 0 with group-write permissions so the arbitrary UID can write to them (since it’s in group 0)
  • IMPORTANT: Use {{ default_user }} template variable, never hardcode UIDs:
    • ✅ Correct: chown -R {{ default_user }}:0 ${NEWROOT}/dir
    • ❌ Wrong: chown -R 65532:0 ${NEWROOT}/dir
    • Template variables follow project conventions and allow centralized config changes

If Containerfile changed:

touch images/{image_name}/Containerfile.j2
make images/{image_name}/{distro}/default/Containerfile
ci/build_images.sh {image_name}/{distro}/default

Lint and Test

  1. Run make -j$(nproc) check → if fails, fix errors before proceeding

  2. Test in minikube:

    minikube start
ci/run_tests_k8s.sh --context minikube {image_name}

  3. Monitor pods immediately: kubectl get pods -w

  4. If Containerfile changed → test in OpenShift:

    ci/build_images.sh {image_name}/{distro}/default
# Tag with simple localhost name for --push-image
podman tag quay.io/hummingbird/{image_name}:latest localhost/{image_name}:test
ci/run_tests_k8s.sh --context <ocp> --push-image localhost/{image_name}:test {image_name}/{distro}/default

    Note: Push images to localhost/{image_name}:test, not to quay.io. Tests should never push to the official registry.

  5. Monitor pods: oc get pods -w

Common Issues

  • If tests fail OpenShift with “Permission denied” → check Containerfile uses gid=0 and g+w
  • If wrong image used in OpenShift → verify ci/run_tests_k8s.sh doesn’t override TEST_IMAGE when --push-image set
  • If multi-container test can’t see files → add emptyDir volume
  • If readiness probe times out in OpenShift → add timeoutSeconds: 5

Automated Skill

For AI-assisted implementation, use the /add-k8s-tests <image-name> command which follows these patterns automatically.