K8s Test Pipeline

• 1: Pipeline Design
• 2: Test Format
• 3: EaaS and Debugging

1 - Pipeline Design

Task/Step Overview

Both check-for-tests and run-tests independently fetch source — each task gets its own emptyDir volume, so no state carries across tasks.

Design Guidelines

1. Filesystem over results

Pass data between steps via files on a shared emptyDir, not Tekton results. Tekton results have a 4KB termination message limit (tektoncd/pipeline#4060) and large $(params.*) substitutions hit ARG_MAX.

Each task mounts an emptyDir at /workdir. Steps communicate via files: snapshot.json, component, image_names, src/, kubeconfig, result, error. Zero $(params.*) references remain inside script: blocks — all param values are passed via env vars or written to /workdir.

2. Distinguish retryable from non-retryable errors

Use exit 1 for transient failures (network, registry) that benefit from retry. Use exit 0 + a sentinel file for permanent failures (bad data) to prevent wasted retries.

Non-retryable errors write Konflux-format JSON to /workdir/error and exit 0. Subsequent steps check for this sentinel and skip. The write-results step copies it to TEST_OUTPUT. Tasks default to retries: 2, but fail-pipeline-on-test-failure has retries: 0 — test failures should not be retried.

3. Fail fast

Every shell script enables set -euo pipefail so unexpected command failures surface immediately rather than cascading silently.

4. Separate data from status

Report test results as structured data for the CI system to consume, and separately translate results into pipeline pass/fail for source control integration.

Konflux reads TEST_OUTPUT results as data but does not fail the pipeline based on them. The fail-pipeline-on-test-failure finally-task reads both TEST_OUTPUT values and exit 1 on non-SUCCESS. This is needed because GitLab PAC integration requires the pipeline itself to fail for MR status reporting.

5. Gate expensive work behind cheap checks

Run a fast, cheap check before provisioning resources. Skip the expensive path when there is nothing to do.

provision-namespace and run-tests are gated on check-for-tests.results.HAS_TESTS == "true" via Tekton when expressions. If no tests-k8s.yml exists, the pipeline completes with just check-for-tests, skipping EaaS provisioning. This is the common case for components without K8s tests.

6. Make failures reproducible

Preserve all diagnostic output (do not swallow stderr). Log exact commands so developers can copy-paste to reproduce locally.

Stderr from cosign/jq/oras is not swallowed. The full run_tests_k8s.sh command line is logged.

7. Set task timeouts deliberately

Tekton applies a global default TaskRun timeout (typically 1h from default-timeout-minutes in config-defaults) to every task that has no explicit timeout, regardless of pipeline-level timeouts. This means omitting a task timeout does NOT let the pipeline timeout control the bound — the task gets killed at 1h.

Every task in this pipeline has an explicit timeout. Bounded operations (check-for-tests, provision-namespace, fail-pipeline-on-test-failure) use 10m. The variable-duration run-tests task uses 24h so the pipeline-level timeout (controlled per-application via ITS annotations like test.appstudio.openshift.io/pipeline_timeout) is the effective bound, following the Testing Farm pipeline pattern.

8. Define shared logic once

When multiple tasks need identical steps, use YAML anchors to define them once and reuse via aliases. Tekton pipelines have no function/import mechanism, so anchors are the only DRY tool available within a single pipeline file. Without this, step logic drifts between tasks when one copy is updated but not the other.

write-snapshot, resolve-component, and fetch-source are defined as anchors (&write-snapshot-step, etc.) in check-for-tests and reused via *write-snapshot-step in run-tests. Both tasks independently execute the same steps to fetch and process source, staying in sync.

Trusted Artifacts Source Fetch

Source code is fetched via the Konflux Trusted Artifacts chain:

1. cosign download attestation — retrieves build attestations (handles multiple JSONL attestations)
2. Extract SOURCE_ARTIFACT from .predicate.buildConfig.tasks[].results[]
3. oras blob fetch — downloads the source tarball
4. tar -xzf — extracts to /workdir/src

The oci: prefix is stripped from the artifact URI. --no-same-owner avoids permission issues during extraction.

2 - Test Format

Test Discovery

The pipeline locates tests-k8s.yml using source.git.context from the Konflux Snapshot. For example, if source.git.context is images/curl, the pipeline looks for images/curl/tests-k8s.yml in the source tree.

When source.git.context is not populated (currently the case for the triggering component in PR-triggered pipelines, tracked as KONFLUX-12674), a heuristic fallback derives the context from the component name: it tries images/<base> and then <base>.

The actual tests-k8s.yml format is opaque to the pipeline — it is passed to the repository’s test runner script, which defines the structure and semantics. See the respective repository documentation for the YAML format.

Environment Variables

The test runner receives these variables from the pipeline:

Pipeline Parameters

Group Snapshot Handling

For group snapshots (post-merge), the pipeline exports:

• IMAGE_NAMES — space-separated list of all component names
• IMAGE_URL_<COMPONENT> — image URL for each component (name uppercased, dashes replaced with underscores)

The test runner script is called once with multiple --group-component-name arguments. Test discovery checks all components for tests-k8s.yml. Source is fetched from the first component’s build attestation (shared source tree).

3 - EaaS and Debugging

Environment as a Service (EaaS)

EaaS is a Konflux-managed service that provisions ephemeral OpenShift namespaces on member clusters for integration testing.

Provisioning Flow

1. The provision-namespace task creates an eaas.konflux-ci.dev/v1alpha1 Namespace claim, owned by the PipelineRun.
2. The EaaS controller allocates a namespace on a member cluster (currently kflux-prd-es01) and writes a scoped kubeconfig into a Kubernetes Secret.
3. The run-tests step reads the kubeconfig from the Secret to get cluster access.
4. When the PipelineRun is deleted, the owner reference triggers garbage collection of the claim, namespace, and Secret.

The member cluster may change — it is dynamically assigned by the EaaS controller.

Debugging Test Failures

Using Kubearchive for Historical PLRs

Kubearchive stores historical PipelineRun, TaskRun, and Pod data. The containers repo includes a helper script at ci/internal/k8s_helper.py.

Example: find PLRs for a specific PR:

cd /path/to/rprm-containers
python3 -c "
from ci.internal import k8s_helper
h = k8s_helper.K8sHelper('https://konflux-ui.apps.kflux-prd-rh03.nnv1.p1.openshiftapps.com/ns/hummingbird-tenant/')
endpoint = h._tekton_endpoint('pipelineruns')
items = h._fetch_from_both_sources(endpoint, 'pac.test.appstudio.openshift.io/pull-request=421,test.appstudio.openshift.io/scenario=tools-k8s-test')
for plr in items:
    print(plr['metadata']['name'], plr['status']['conditions'][0]['reason'])
"

Useful label selectors:

• test.appstudio.openshift.io/scenario=<scenario-name> — filter by ITS name
• pac.test.appstudio.openshift.io/pull-request=<number> — filter by MR
• appstudio.openshift.io/component=<component> — filter by component

Accessing the EaaS Namespace During a Live PLR

While a PipelineRun is running, you can access the ephemeral namespace:

1. Find the provision secret name from the provision-namespace task results (visible in the PLR status or Konflux UI).

2. Extract the kubeconfig:

   kubectl get secret <secret-name> -n hummingbird-tenant \
     -o jsonpath='{.data.kubeconfig}' | base64 -d > /tmp/eaas-kubeconfig
   

3. Use it to inspect the namespace:

   export KUBECONFIG=/tmp/eaas-kubeconfig
   kubectl get pods
   kubectl get events --sort-by='.lastTimestamp'
   kubectl logs <pod-name>
   

Finding Which Cluster EaaS Uses

To determine the current EaaS member cluster, extract the server URL from a kubeconfig provisioned by EaaS:

kubectl get secret <secret-name> -n hummingbird-tenant \
  -o jsonpath='{.data.kubeconfig}' | base64 -d | grep server

As of this writing, the EaaS member cluster is kflux-prd-es01.1ion.p1.openshiftapps.com. This may change as the EaaS controller dynamically assigns clusters.