Pipeline Timing

Independent SQS worker for RPM and container execution timing and cross-system lineage. It shares the Status events database but owns only the pipeline_timing schema.

Features

  • Current state, timestamps, and separate attempts for GitLab jobs and Konflux PipelineRuns
  • Commit, merge request, PipelineRun, TaskRun, Snapshot, and Release lineage
  • Source-derived execution creation, start, and completion times
  • Alembic-managed schema, dedicated queue, and worker metrics

Prerequisites

  • Python 3.11+ and PostgreSQL 16
  • A dedicated timing SQS queue subscribed to the environment’s SNS topic with RawMessageDelivery: false

The existing Status worker and event forwarders keep their own images and queues. Infrastructure provisions the timeline queues, IAM, and alerts separately from the Status worker. Each environment has a separate queue named arr-hummingbird-<env>-pipeline-timing-sqs and a matching -sqs-dlq queue. The source queue retains messages for 4 days; its DLQ retains them for 14 days from the original enqueue time when SQS automatically moves a message to the DLQ. Operator redrive from the DLQ resets the enqueue timestamp. The source queue moves a message to its DLQ after 3 receives and uses a 300-second visibility timeout, matching Status. The SNS subscription preserves the SNS envelope (RawMessageDelivery: false) and does not filter events. See Hummingbird Events Topic for queue policy and redrive details. Provision both environments’ queues and SNS subscriptions before deploying the workers. Messages collected before a worker starts may expire after four days; this pre-launch loss is accepted. A queue without a consumer does not move messages to its DLQ because no receives occur. Validate the staging worker before deploying the production worker or enabling new production event sources.

Installation

cd pipeline-timing
pip install -e ".[dev]"

The pipeline-timing Containerfile builds the separate quay.io/hummingbird-ci/pipeline-timing image. Both the initializer and worker run from this image.

Usage

Schema initialization

Run the initializer as the init container of the single-replica timing worker Deployment. Use a Recreate strategy so Kubernetes stops the old worker before starting a Pod whose init container may upgrade the schema:

DATABASE_URL="$DATABASE_URL" python3 -m pipeline_timing.worker.init_db

The initializer creates the pipeline_timing schema and runs alembic upgrade head each time a Pod starts. This is a no-op at the expected revision; a new image can apply a forward migration during rollout. If migration fails, the initializer exits nonzero and Kubernetes does not start the worker. The worker checks the revision again before polling. Fix migration errors with a forward revision; do not downgrade or drop the schema during rollback.

Staging deployment from a merge request

Wait for the tools MR’s tests and Konflux image build to pass, then run the manual deploy_pipeline_timing_staging job. The job does not wait for those checks automatically. It triggers the infrastructure pipeline with ONLY_JOB_NAME=pipeline-timing and ONLY_DEPLOYMENT_ENVIRONMENT=staging, using the MR commit’s image for both the migration init container and worker. Production is not deployed by this job.

Verify the init container completed, the worker accepts the new Alembic revision, and its queue is catching up before promoting the published image to production. Normal infrastructure pipelines use a shared image pin for both environments. Coordinate those deployments during MR validation so an older pinned image does not replace the MR image. The worker requires an exact Alembic revision match; an older image will refuse a newer revision even when the table layout is unchanged.

Execution-only schema and catalogue removal

Under HUM-10942, revision 003_execution_records added nullable created_at to TaskRuns and PipelineRuns, stopped catalogue writes, and preserved old recovery pointers temporarily. Historical creation-timestamp recovery completed in staging and production using a one-time local tool. The tool was intentionally not retained.

Revision 004_drop_event_catalogue removes the legacy latest_event_id foreign keys and columns from the reporting tables, then drops pipeline_timing.events and its remaining indexes. It does not change reporting rows, their lifecycle fields, parentage, or ordering metadata. The database named events and legacy Status tables in public remain unchanged. Original migrations remain intact so fresh initialization follows the same revision chain.

Before deployment, confirm recovery is complete or its gaps accepted in both environments, no external consumer needs the catalogue, and the normal backup is available. Record the catalogue allocation and PVC free space, then verify the space recovery and continued ingestion in staging before production promotion. The table/index allocation is released when the migration commits without a bulk delete, table rewrite, or VACUUM FULL. Dropped pointer columns may retain small amounts of physical row space; that does not justify a rewrite.

The migration limits lock waits to five seconds. A conflicting reader or unexpected dependency makes initialization fail and the transactional DDL roll back. Resolve contention or the dependency and retry with the compatible image. Downgrade is explicitly unsupported: recreating empty history would not restore the deleted catalogue or pointers. Use a compatible forward fix or the backup recovery procedure.

Kubernetes creation time comes from metadata.creationTimestamp; start and completion come from status.startTime and status.completionTime. Missing source timestamps remain null, not inferred from SNS publication time. Pending conditions are distinct from running, and failure reasons retain cancellation details. GitLab jobs and pipelines retain their source lifecycle fields and status.

Only execution state and lineage are maintained. There is no event-history array, per-step timing, new archive-location table, or permanent S3 pointer on each row. Resource deletion preserves the execution’s result, timings, and parentage.

Timing worker

Deploy one worker replica with the environment’s dedicated queue. Kubernetes starts it only after the init container succeeds:

SQS_QUEUE_URL="$TIMING_QUEUE_URL" python3 -m pipeline_timing.worker

The worker polls when it starts. Accepted notifications update reporting rows in one committed transaction before SQS acknowledgement, without event-table writes. Malformed events remain for retry and eventual dead-letter-queue redrive; database or revision failures stop polling without affecting legacy Status. Scale the timing worker to zero replicas to pause ingestion. After launch, monitor queue age against the four-day source retention limit during a pause to avoid losing events. SQS preserves the original enqueue time for automatic DLQ transfer, and an operator redrive resets it. A new rollout uses Recreate: the old worker stops, Alembic upgrades, and the new worker starts. The S3 archiver receives its own SNS copy and remains independent of timing ingestion.

The worker follows Status’s polling pattern: a 20-second long poll receives up to 10 messages, processed sequentially with boto3’s default request retries. Each accepted event commits in its own transaction. At the end of each receive, one SQS batch-delete request acknowledges committed events and ignored messages. Short receives flush immediately; failed processing never adds a receipt to the delete request. Receive errors are logged and polling continues. Partial delete failures and uncertain request outcomes leave committed events available for idempotent redelivery, and polling continues. If a database failure interrupts a receive, the worker first attempts to acknowledge earlier confirmed commits. Database connection or schema failures stop the worker for Kubernetes to restart it. SQS visibility controls when an unacknowledged message becomes available again.

The worker caches up to 128 UPSERT statement shapes by table and ordered columns. Each execution binds fresh event values; the cache does not retain payloads, connections, or cursors. Source update timestamps order messages when both the incoming and existing row have them; SNS publication timestamps break ties or provide the fallback. Kubernetes condition transition times supply source ordering. Late pending/running snapshots cannot regress a completed Kubernetes execution. Sparse payloads preserve existing values, and creation time stays immutable once known.

Queue age and DLQ depth are monitored independently from the Status queue. Fix malformed messages before using the documented redrive procedure. If the worker is paused or not yet deployed, remove only a known test message after confirming that redrive returned it to the timing queue.

Configuration

Variable Default Description
DATABASE_URL - Existing Status events database URL
SQS_QUEUE_URL - Environment-specific timing queue URL
SQS_WAIT_TIME 20 SQS long-poll interval in seconds
METRICS_PORT 9090 Prometheus port when the worker is enabled

Infrastructure supplies queue values for each environment:

Environment Queue URL variable
Staging ARR_HUMMINGBIRD_STAGING_PIPELINE_TIMING_SQS_QUEUE_URL
Production ARR_HUMMINGBIRD_PROD_PIPELINE_TIMING_SQS_QUEUE_URL

S3_BUCKET is no longer required or used by the worker. Remove the old deployment variable only after the compatible worker is deployed in both environments. The independent S3 archiver and its configuration remain unchanged.

POSTGRES_URL is also accepted when DATABASE_URL is not set. For local development only, the database URL defaults to postgresql://postgres:dev@localhost:5432/events.

Schema and lineage

Table Identity Purpose
merge_requests GitLab project ID and MR IID Current state and lifecycle times
commits GitLab project ID and SHA Commit metadata
merge_request_commits Project ID, IID, SHA, relation Head and merge-commit links
gitlab_pipelines Project ID and pipeline ID Pipeline status and timing
gitlab_jobs Project ID and job ID Individual job attempts
pipeline_runs Cluster and Kubernetes UID Build, test, and Release attempts
task_runs Cluster and Kubernetes UID Task timing and parent UID
snapshots Cluster and Kubernetes UID Build-to-test bridge
releases Cluster and Kubernetes UID Snapshot-to-Release bridge

Source timestamps accept ISO 8601 and GitLab’s YYYY-MM-DD HH:MM:SS UTC format and are normalized to UTC. Timestamps without a timezone are interpreted as UTC.

latest_observed_at and source_updated_at order updates without a notification catalogue or event foreign keys. The schema stores selected reporting fields, not raw event JSON. Join MR head and merge commits to GitLab pipelines by canonical HTTPS repository URL and SHA; join jobs by project and pipeline ID. Konflux PipelineRuns use the same repository URL and SHA, then fan out by component and execution UID. TaskRuns link through PipelineRun owner UIDs; Snapshot source-run names, test-run Snapshot labels, and Release references connect the remaining phases. The Release completion time closes the current timeline. Distinct job IDs and PipelineRun UIDs retain each retry’s status and timing.

PipelineRun SHAs come from supported labels and annotations, including pac.test.appstudio.openshift.io/sha, before falling back to the revision parameter.

Promotion Snapshots populate source_pipeline_run_name from hummingbird-project.io/promotion-pipelinerun. This annotation takes precedence over the usual pac.test.appstudio.openshift.io/log-url, so the parent is the promotion run rather than the original MR build. Other Snapshots keep the log-URL behavior. This connects the promotion run to its Snapshot and Releases without changing the schema. References to the original MR build and source Snapshot in promotion annotations are not stored separately.

Existing promotion Snapshots need a separate backfill if no newer notification arrives after this extraction change. Deploying the worker does not rescan stored Snapshots or automatically repair historical links.

Finding archived notifications

The independent S3 archiver still receives its own SNS copy. The timing database does not store a direct archive pointer or a complete notification history.

For Kubernetes executions, use the environment’s archive bucket and latest_observed_at as a UTC SNS timestamp with milliseconds to narrow the object prefix, for example sns/2026/01/02/03/04/2026-01-02T03:04:05.678Z#. Match source, kind, cluster, namespace, and resource name in candidate filenames, then verify the UID in the downloaded payload. For GitLab, verify the project and execution/commit identity in candidate payloads.

This is candidate discovery, not a guaranteed unique or per-field provenance link. Several notifications may share a timestamp, and sparse updates can retain facts from earlier payloads. Lookup depends on archive delivery, naming, and retention; it does not recreate the removed database catalogue.

Metrics include hummingbird_pipeline_timing_records_total{kind,result}, hummingbird_pipeline_timing_failures_total{reason}, hummingbird_pipeline_timing_missing_lineage_total{kind}, and hummingbird_pipeline_timing_worker_stopped. success and ignored record results count only confirmed SQS acknowledgements, including duplicates. Each failed or uncertain batch entry increments the sqs failure counter. These counters do not measure unique event rows. Monitor queue age and dead-letter-queue depth separately from the legacy Status queue. Fix malformed events before redriving them. Identity-based upserts and ordering guards keep redelivery idempotent; there is no database row for every SNS message.

Development

Run tests only against the disposable PostgreSQL service provided in CI or a disposable local database. Tests truncate timing data and exercise schema upgrades and historical downgrades; never use a staging or production database URL. Migration tests create uniquely named disposable databases and need CREATEDB on that test server. Older-revision tests stop at revision 003; they do not downgrade through irreversible catalogue removal:

DATABASE_URL=postgresql://postgres:dev@localhost:5432/events \
  python3 -m unittest discover -s tests -t .

Tests also load shared publisher payload fixtures from sns-s3-archiver/tests/fixtures/archive-key-contract.json; timing ingestion no longer calculates or persists archive keys. CI provides DATABASE_URL for the PostgreSQL integration tests. The timing image has a Kubernetes import smoke test in tests-k8s.yml.

License

This project is licensed under the GNU General Public License v3.0 or later. See the LICENSE.