Adding Kubernetes Tests

This guide describes how to add Kubernetes tests to container images.

Overview

Kubernetes tests validate that container images work correctly in Kubernetes and OpenShift environments, particularly:

• Directory permissions (important for OpenShift arbitrary UIDs)
• Version checks
• Application functionality (web servers, databases, etc.)
• Multi-container scenarios (sidecar patterns)

Confirm Test Design

Before implementing:

1. Read images/{image_name}/tests-container.yml to understand existing tests
2. Identify which tests need K8s validation:
   • Directory permissions (important for OpenShift arbitrary UIDs)
   • Version check
   • Application functionality (web server, database, etc.)
   • Multi-container scenarios (sidecar patterns)
3. Confirm test approach before creating files

Rules

• Script naming: test-k8s-*.sh prefix (e.g., test-k8s-version.sh)
• Script location: images/{image_name}/test-scripts/
• YAML location: images/{image_name}/tests-k8s.yml
• Keep YAML lean: complex logic goes in scripts, not inline YAML
• No external downloads: avoid network flakes (exception: container images)
• OpenShift compatibility: writable dirs must use gid=0 + g+w permissions

Validation

• If images/{image_name}/ doesn’t exist → error “Image not found”
• If no tests-container.yml → error “Add container tests first”

Create Test Scripts

If directory doesn’t exist → create images/{image_name}/test-scripts/

Test Script Template

#!/bin/bash
set -euo pipefail

# Test logic
test_result=$(command)

if [[ ! "${test_result}" =~ "expected" ]]; then
    echo "FAIL: Description"
    echo "Got: ${test_result}"
    exit 1
fi

echo "SUCCESS: Test passed"
exit 0

Directory Permissions Test

If testing OpenShift compatibility → create test-k8s-directory-permissions.sh:

• Check writable directories exist and are writable
• Verify running as non-root (UID != 0)
• Test actual write with touch/rm

Version Check Test

If testing version → create test-k8s-version.sh:

• Run version command
• Check major version only (not major.minor.patch)
• Avoids breaking on patch updates

Application-Specific Tests

• If web server → deployment test with sidecar (curl container)
• If database → connection/persistence test
• If tool → primary functionality test

Create tests-k8s.yml

Pattern for each test:

test-name:
  variants: [default]
  command: |
    # Create ConfigMap with script
    kubectl create configmap "${TEST_GROUP}"-scripts-"${TEST_RUN_ID}" \
      --from-file=test-k8s-name.sh=test-scripts/test-k8s-name.sh \
      --dry-run=client -o yaml | kubectl label -f - --local -o yaml hum-k8s-test="${TEST_RUN_ID}" | kubectl apply -f -

    # Create Pod
    kubectl apply -f - <<EOF
    apiVersion: v1
    kind: Pod
    metadata:
      name: ${TEST_GROUP}-name-${TEST_RUN_ID}
      labels:
        hum-k8s-test: "${TEST_RUN_ID}"
    spec:
      restartPolicy: Never
      containers:
      - name: test
        image: ${TEST_IMAGE:?}
        command: ["sh", "/scripts/test-k8s-name.sh"]
        volumeMounts:
        - name: test-scripts
          mountPath: /scripts
      volumes:
      - name: test-scripts
        configMap:
          name: ${TEST_GROUP}-scripts-${TEST_RUN_ID}
          defaultMode: 0755
    EOF

    # Wait for completion
    kubectl wait --for=jsonpath='{.status.containerStatuses[?(@.name=="test")].state.terminated.reason}'=Completed \
      pod/"${TEST_GROUP}"-name-"${TEST_RUN_ID}" --timeout=120s || TEST_FAIL "Test did not complete"

    # Check logs
    out=$(kubectl logs "${TEST_GROUP}"-name-"${TEST_RUN_ID}" -c test)
    [[ "${out}" =~ "SUCCESS" ]] || TEST_FAIL "Test failed: ${out}"

Multi-Container Tests

If test needs multiple containers (app + client):

• Use emptyDir volume mounted in both containers
• emptyDir has 777 permissions, works with arbitrary UIDs

Readiness Probes

If testing app functionality:

• Add readinessProbe to app container
• Set timeoutSeconds: 5 for OpenShift network latency

OpenShift Compatibility

Read images/{image_name}/Containerfile.j2:

• If writable dirs use user:user ownership → change to user:0
• If writable dirs lack g+w → add chmod -R g+w
• Why gid=0? OpenShift runs containers with arbitrary UIDs (for security) but always assigns gid=0 (root group). Directories must be group-owned by 0 with group-write permissions so the arbitrary UID can write to them (since it’s in group 0)
• IMPORTANT: Use {{ default_user }} template variable, never hardcode UIDs:
  • ✅ Correct: chown -R {{ default_user }}:0 ${NEWROOT}/dir
  • ❌ Wrong: chown -R 65532:0 ${NEWROOT}/dir
  • Template variables follow project conventions and allow centralized config changes

If Containerfile changed:

touch images/{image_name}/Containerfile.j2
make images/{image_name}/{distro}/default/Containerfile
ci/build_images.sh {image_name}/{distro}/default

Lint and Test

1. Run make -j$(nproc) check → if fails, fix errors before proceeding

2. Test in minikube:

   minikube start
   ci/run_tests_k8s.sh --context minikube {image_name}
   

3. Monitor pods immediately: kubectl get pods -w

4. If Containerfile changed → test in OpenShift:

   ci/build_images.sh {image_name}/{distro}/default
   # Tag with simple localhost name for --push-image
   podman tag quay.io/hummingbird/{image_name}:latest localhost/{image_name}:test
   ci/run_tests_k8s.sh --context <ocp> --push-image localhost/{image_name}:test {image_name}/{distro}/default
   

   Note: Push images to localhost/{image_name}:test, not to quay.io. Tests should never push to the official registry.

5. Monitor pods: oc get pods -w

Common Issues

• If tests fail OpenShift with “Permission denied” → check Containerfile uses gid=0 and g+w
• If wrong image used in OpenShift → verify ci/run_tests_k8s.sh doesn’t override TEST_IMAGE when --push-image set
• If multi-container test can’t see files → add emptyDir volume
• If readiness probe times out in OpenShift → add timeoutSeconds: 5

Automated Skill

For AI-assisted implementation, use the /add-k8s-tests <image-name> command which follows these patterns automatically.