# Adding Kubernetes Tests

> How to add Kubernetes and OpenShift integration tests to container images

---

LLMS index: [llms.txt](/llms.txt) | Full content: [llms-full.txt](/llms-full.txt)

---

This guide describes how to add Kubernetes tests to container images.

## Overview

Kubernetes tests validate that container images work correctly in Kubernetes and OpenShift environments, particularly:

- Directory permissions (important for OpenShift arbitrary UIDs)
- Version checks
- Application functionality (web servers, databases, etc.)
- Multi-container scenarios (sidecar patterns)

## Confirm Test Design

Before implementing:

1. Read `images/{image_name}/tests-container.yml` to understand existing tests
2. Identify which tests need K8s validation:
   - Directory permissions (important for OpenShift arbitrary UIDs)
   - Version check
   - Application functionality (web server, database, etc.)
   - Multi-container scenarios (sidecar patterns)
3. Confirm test approach before creating files

## Rules

- Script naming: `test-k8s-*.sh` prefix (e.g., `test-k8s-version.sh`)
- Script location: `images/{image_name}/test-scripts/`
- YAML location: `images/{image_name}/tests-k8s.yml`
- Keep YAML lean: complex logic goes in scripts, not inline YAML
- No external downloads: avoid network flakes (exception: container images)
- OpenShift compatibility: writable dirs must use gid=0 + g+w permissions

## Validation

- If `images/{image_name}/` doesn't exist → error "Image not found"
- If no `tests-container.yml` → error "Add container tests first"

## Create Test Scripts

If directory doesn't exist → create `images/{image_name}/test-scripts/`

### Test Script Template

```bash
#!/bin/bash
set -euo pipefail

# Test logic
test_result=$(command)

if [[ ! "${test_result}" =~ "expected" ]]; then
    echo "FAIL: Description"
    echo "Got: ${test_result}"
    exit 1
fi

echo "SUCCESS: Test passed"
exit 0
```

### Directory Permissions Test

If testing OpenShift compatibility → create `test-k8s-directory-permissions.sh`:

- Check writable directories exist and are writable
- Verify running as non-root (UID != 0)
- Test actual write with touch/rm

### Version Check Test

If testing version → create `test-k8s-version.sh`:

- Run version command
- Check major version only (not major.minor.patch)
- Avoids breaking on patch updates

### Application-Specific Tests

- If web server → deployment test with sidecar (curl container)
- If database → connection/persistence test
- If tool → primary functionality test

## Create tests-k8s.yml

Pattern for each test:

```yaml
test-name:
  variants: [default]
  command: |
    # Create ConfigMap with script
    kubectl create configmap "${TEST_GROUP}"-scripts-"${TEST_RUN_ID}" \
      --from-file=test-k8s-name.sh=test-scripts/test-k8s-name.sh \
      --dry-run=client -o yaml | kubectl label -f - --local -o yaml hum-k8s-test="${TEST_RUN_ID}" | kubectl apply -f -

    # Create Pod
    kubectl apply -f - <<EOF
    apiVersion: v1
    kind: Pod
    metadata:
      name: ${TEST_GROUP}-name-${TEST_RUN_ID}
      labels:
        hum-k8s-test: "${TEST_RUN_ID}"
    spec:
      restartPolicy: Never
      containers:
      - name: test
        image: ${TEST_IMAGE:?}
        command: ["sh", "/scripts/test-k8s-name.sh"]
        volumeMounts:
        - name: test-scripts
          mountPath: /scripts
      volumes:
      - name: test-scripts
        configMap:
          name: ${TEST_GROUP}-scripts-${TEST_RUN_ID}
          defaultMode: 0755
    EOF

    # Wait for completion
    kubectl wait --for=jsonpath='{.status.containerStatuses[?(@.name=="test")].state.terminated.reason}'=Completed \
      pod/"${TEST_GROUP}"-name-"${TEST_RUN_ID}" --timeout=120s || TEST_FAIL "Test did not complete"

    # Check logs
    out=$(kubectl logs "${TEST_GROUP}"-name-"${TEST_RUN_ID}" -c test)
    [[ "${out}" =~ "SUCCESS" ]] || TEST_FAIL "Test failed: ${out}"
```

### Multi-Container Tests

If test needs multiple containers (app + client):

- Use `emptyDir` volume mounted in both containers
- emptyDir has 777 permissions, works with arbitrary UIDs

### Readiness Probes

If testing app functionality:

- Add `readinessProbe` to app container
- Set `timeoutSeconds: 5` for OpenShift network latency

## OpenShift Compatibility

Read `images/{image_name}/Containerfile.j2`:

- If writable dirs use `user:user` ownership → change to `user:0`
- If writable dirs lack `g+w` → add `chmod -R g+w`
- **Why gid=0?** OpenShift runs containers with arbitrary UIDs (for security) but always
  assigns gid=0 (root group). Directories must be group-owned by 0 with group-write
  permissions so the arbitrary UID can write to them (since it's in group 0)
- **IMPORTANT**: Use `{{ default_user }}` template variable, never hardcode UIDs:
  - ✅ Correct: `chown -R {{ default_user }}:0 ${NEWROOT}/dir`
  - ❌ Wrong: `chown -R 65532:0 ${NEWROOT}/dir`
  - Template variables follow project conventions and allow centralized config changes

If Containerfile changed:

```bash
touch images/{image_name}/Containerfile.j2
make images/{image_name}/{distro}/default/Containerfile
ci/build_images.sh {image_name}/{distro}/default
```

## Lint and Test

1. Run `make -j$(nproc) check` → if fails, fix errors before proceeding
2. Test in minikube:

   ```bash
   minikube start
   ci/run_tests_k8s.sh --context minikube {image_name}
   ```

3. Monitor pods immediately: `kubectl get pods -w`
4. If Containerfile changed → test in OpenShift:

   ```bash
   ci/build_images.sh {image_name}/{distro}/default
   # Tag with simple localhost name for --push-image
   podman tag quay.io/hummingbird/{image_name}:latest localhost/{image_name}:test
   ci/run_tests_k8s.sh --context <ocp> --push-image localhost/{image_name}:test {image_name}/{distro}/default
   ```

   **Note**: Push images to `localhost/{image_name}:test`, not to `quay.io`. Tests should
   never push to the official registry.

5. Monitor pods: `oc get pods -w`

## Common Issues

- If tests fail OpenShift with "Permission denied" → check Containerfile uses gid=0 and g+w
- If wrong image used in OpenShift → verify `ci/run_tests_k8s.sh` doesn't override TEST_IMAGE when `--push-image` set
- If multi-container test can't see files → add emptyDir volume
- If readiness probe times out in OpenShift → add `timeoutSeconds: 5`

## Automated Skill

For AI-assisted implementation, use the `/add-k8s-tests <image-name>` command which follows these patterns automatically.