Pipeline Timing
Independent SQS worker for RPM and container execution timing and cross-system lineage.
It shares the Status events database but owns only the pipeline_timing schema.
Features
- Current state, timestamps, and separate attempts for GitLab jobs and Konflux PipelineRuns
- Commit, merge request, PipelineRun, TaskRun, Snapshot, and Release lineage
- Source-derived execution creation, start, and completion times
- Alembic-managed schema, dedicated queue, and worker metrics
Prerequisites
- Python 3.11+ and PostgreSQL 16
- A dedicated timing SQS queue subscribed to the environment’s SNS topic with
RawMessageDelivery: false
The existing Status worker and event forwarders keep their own images and queues.
Infrastructure provisions the timeline queues, IAM, and alerts separately from
the Status worker.
Each environment has a separate queue named
arr-hummingbird-<env>-pipeline-timing-sqs and a matching -sqs-dlq queue.
The source queue retains messages for 4 days; its DLQ retains them for 14 days
from the original enqueue time when SQS automatically moves a message to the
DLQ. Operator redrive from the DLQ resets the enqueue timestamp. The source queue
moves a message to its DLQ after 3 receives and uses a 300-second visibility
timeout, matching Status.
The SNS subscription preserves the SNS envelope (RawMessageDelivery: false)
and does not filter events. See Hummingbird Events Topic
for queue policy and redrive details.
Provision both environments’ queues and SNS subscriptions before deploying the
workers. Messages collected before a worker starts may expire after four days;
this pre-launch loss is accepted. A queue without a consumer does not move
messages to its DLQ because no receives occur. Validate the staging worker
before deploying the production worker or enabling new production event sources.
Installation
cd pipeline-timing
pip install -e ".[dev]"
The pipeline-timing Containerfile builds the separate
quay.io/hummingbird-ci/pipeline-timing image. Both the initializer and worker
run from this image.
Usage
Schema initialization
Run the initializer as the init container of the single-replica timing worker
Deployment. Use a Recreate strategy so Kubernetes stops the old worker before
starting a Pod whose init container may upgrade the schema:
DATABASE_URL="$DATABASE_URL" python3 -m pipeline_timing.worker.init_db
The initializer creates the pipeline_timing schema and runs
alembic upgrade head each time a Pod starts. This is a no-op at the expected
revision; a new image can apply a forward migration during rollout. If migration
fails, the initializer exits nonzero and Kubernetes does not start the worker.
The worker checks the revision again before polling. Fix migration errors with
a forward revision; do not downgrade or drop the schema during rollback.
Staging deployment from a merge request
Wait for the tools MR’s tests and Konflux image build to pass, then run the
manual deploy_pipeline_timing_staging job. The job does not wait for those
checks automatically. It triggers the infrastructure pipeline with
ONLY_JOB_NAME=pipeline-timing and ONLY_DEPLOYMENT_ENVIRONMENT=staging, using
the MR commit’s image for both the migration init container and worker.
Production is not deployed by this job.
Verify the init container completed, the worker accepts the new Alembic revision, and its queue is catching up before promoting the published image to production. Normal infrastructure pipelines use a shared image pin for both environments. Coordinate those deployments during MR validation so an older pinned image does not replace the MR image. The worker requires an exact Alembic revision match; an older image will refuse a newer revision even when the table layout is unchanged.
Execution-only schema and catalogue removal
Under HUM-10942, revision 003_execution_records added nullable created_at to
TaskRuns and PipelineRuns, stopped catalogue writes, and preserved old recovery
pointers temporarily. Historical creation-timestamp recovery completed in staging
and production using a one-time local tool. The tool was intentionally not retained.
Revision 004_drop_event_catalogue removes the legacy latest_event_id foreign
keys and columns from the reporting tables, then drops pipeline_timing.events
and its remaining indexes. It does not change reporting rows, their lifecycle
fields, parentage, or ordering metadata. The database named events and legacy
Status tables in public remain unchanged. Original migrations remain intact so
fresh initialization follows the same revision chain.
Before deployment, confirm recovery is complete or its gaps accepted in both
environments, no external consumer needs the catalogue, and the normal backup is
available. Record the catalogue allocation and PVC free space, then verify the
space recovery and continued ingestion in staging before production promotion.
The table/index allocation is released when the migration commits without a bulk
delete, table rewrite, or VACUUM FULL. Dropped pointer columns may retain small
amounts of physical row space; that does not justify a rewrite.
The migration limits lock waits to five seconds. A conflicting reader or unexpected dependency makes initialization fail and the transactional DDL roll back. Resolve contention or the dependency and retry with the compatible image. Downgrade is explicitly unsupported: recreating empty history would not restore the deleted catalogue or pointers. Use a compatible forward fix or the backup recovery procedure.
Kubernetes creation time comes from metadata.creationTimestamp; start and
completion come from status.startTime and status.completionTime. Missing
source timestamps remain null, not inferred from SNS publication time. Pending
conditions are distinct from running, and failure reasons retain cancellation
details. GitLab jobs and pipelines retain their source lifecycle fields and status.
Only execution state and lineage are maintained. There is no event-history array, per-step timing, new archive-location table, or permanent S3 pointer on each row. Resource deletion preserves the execution’s result, timings, and parentage.
Timing worker
Deploy one worker replica with the environment’s dedicated queue. Kubernetes starts it only after the init container succeeds:
SQS_QUEUE_URL="$TIMING_QUEUE_URL" python3 -m pipeline_timing.worker
The worker polls when it starts. Accepted notifications update reporting rows in
one committed transaction before SQS acknowledgement, without event-table writes. Malformed
events remain for retry and eventual dead-letter-queue redrive; database or
revision failures stop polling without affecting legacy Status. Scale the timing
worker to zero replicas to pause ingestion. After launch, monitor queue age
against the four-day source retention limit during a pause to avoid losing
events. SQS preserves the original enqueue time for automatic DLQ transfer, and
an operator redrive resets it. A new rollout uses Recreate: the old worker
stops, Alembic upgrades, and the new worker starts. The S3 archiver receives its
own SNS copy and remains independent of timing ingestion.
The worker follows Status’s polling pattern: a 20-second long poll receives up to 10 messages, processed sequentially with boto3’s default request retries. Each accepted event commits in its own transaction. At the end of each receive, one SQS batch-delete request acknowledges committed events and ignored messages. Short receives flush immediately; failed processing never adds a receipt to the delete request. Receive errors are logged and polling continues. Partial delete failures and uncertain request outcomes leave committed events available for idempotent redelivery, and polling continues. If a database failure interrupts a receive, the worker first attempts to acknowledge earlier confirmed commits. Database connection or schema failures stop the worker for Kubernetes to restart it. SQS visibility controls when an unacknowledged message becomes available again.
The worker caches up to 128 UPSERT statement shapes by table and ordered columns. Each execution binds fresh event values; the cache does not retain payloads, connections, or cursors. Source update timestamps order messages when both the incoming and existing row have them; SNS publication timestamps break ties or provide the fallback. Kubernetes condition transition times supply source ordering. Late pending/running snapshots cannot regress a completed Kubernetes execution. Sparse payloads preserve existing values, and creation time stays immutable once known.
Queue age and DLQ depth are monitored independently from the Status queue. Fix malformed messages before using the documented redrive procedure. If the worker is paused or not yet deployed, remove only a known test message after confirming that redrive returned it to the timing queue.
Configuration
| Variable | Default | Description |
|---|---|---|
DATABASE_URL |
- | Existing Status events database URL |
SQS_QUEUE_URL |
- | Environment-specific timing queue URL |
SQS_WAIT_TIME |
20 |
SQS long-poll interval in seconds |
METRICS_PORT |
9090 |
Prometheus port when the worker is enabled |
Infrastructure supplies queue values for each environment:
| Environment | Queue URL variable |
|---|---|
| Staging | ARR_HUMMINGBIRD_STAGING_PIPELINE_TIMING_SQS_QUEUE_URL |
| Production | ARR_HUMMINGBIRD_PROD_PIPELINE_TIMING_SQS_QUEUE_URL |
S3_BUCKET is no longer required or used by the worker. Remove the old deployment
variable only after the compatible worker is deployed in both environments.
The independent S3 archiver and its configuration remain unchanged.
POSTGRES_URL is also accepted when DATABASE_URL is not set. For local
development only, the database URL defaults to
postgresql://postgres:dev@localhost:5432/events.
Schema and lineage
| Table | Identity | Purpose |
|---|---|---|
merge_requests |
GitLab project ID and MR IID | Current state and lifecycle times |
commits |
GitLab project ID and SHA | Commit metadata |
merge_request_commits |
Project ID, IID, SHA, relation | Head and merge-commit links |
gitlab_pipelines |
Project ID and pipeline ID | Pipeline status and timing |
gitlab_jobs |
Project ID and job ID | Individual job attempts |
pipeline_runs |
Cluster and Kubernetes UID | Build, test, and Release attempts |
task_runs |
Cluster and Kubernetes UID | Task timing and parent UID |
snapshots |
Cluster and Kubernetes UID | Build-to-test bridge |
releases |
Cluster and Kubernetes UID | Snapshot-to-Release bridge |
Source timestamps accept ISO 8601 and GitLab’s YYYY-MM-DD HH:MM:SS UTC format
and are normalized to UTC. Timestamps without a timezone are interpreted as UTC.
latest_observed_at and source_updated_at order updates without a notification
catalogue or event foreign keys. The schema stores selected reporting fields,
not raw event JSON. Join MR head and merge
commits to GitLab pipelines by canonical HTTPS repository URL and SHA; join
jobs by project and pipeline ID. Konflux PipelineRuns use the same repository
URL and SHA, then fan out by component and execution UID. TaskRuns link through
PipelineRun owner UIDs; Snapshot source-run names, test-run Snapshot labels,
and Release references connect the remaining phases. The Release completion
time closes the current timeline. Distinct job IDs and PipelineRun UIDs retain
each retry’s status and timing.
PipelineRun SHAs come from supported labels and annotations, including
pac.test.appstudio.openshift.io/sha, before falling back to the revision
parameter.
Promotion Snapshots populate source_pipeline_run_name from
hummingbird-project.io/promotion-pipelinerun. This annotation takes precedence
over the usual pac.test.appstudio.openshift.io/log-url, so the parent is the
promotion run rather than the original MR build. Other Snapshots keep the
log-URL behavior. This connects the promotion run to its Snapshot and Releases
without changing the schema. References to the original MR build and source
Snapshot in promotion annotations are not stored separately.
Existing promotion Snapshots need a separate backfill if no newer notification arrives after this extraction change. Deploying the worker does not rescan stored Snapshots or automatically repair historical links.
Finding archived notifications
The independent S3 archiver still receives its own SNS copy. The timing database does not store a direct archive pointer or a complete notification history.
For Kubernetes executions, use the environment’s archive bucket and
latest_observed_at as a UTC SNS timestamp with milliseconds to narrow the object
prefix, for example sns/2026/01/02/03/04/2026-01-02T03:04:05.678Z#. Match source,
kind, cluster, namespace, and resource name in candidate filenames, then verify the
UID in the downloaded payload. For GitLab, verify the project and execution/commit
identity in candidate payloads.
This is candidate discovery, not a guaranteed unique or per-field provenance link. Several notifications may share a timestamp, and sparse updates can retain facts from earlier payloads. Lookup depends on archive delivery, naming, and retention; it does not recreate the removed database catalogue.
Metrics include hummingbird_pipeline_timing_records_total{kind,result},
hummingbird_pipeline_timing_failures_total{reason},
hummingbird_pipeline_timing_missing_lineage_total{kind}, and
hummingbird_pipeline_timing_worker_stopped.
success and ignored record results count only confirmed SQS acknowledgements,
including duplicates. Each failed or uncertain batch entry increments the sqs
failure counter. These counters do not measure unique event rows. Monitor queue age and
dead-letter-queue depth separately from the legacy Status queue. Fix malformed
events before redriving them. Identity-based upserts and ordering guards keep
redelivery idempotent; there is no database row for every SNS message.
Development
Run tests only against the disposable PostgreSQL service provided in CI or a
disposable local database. Tests truncate timing data and exercise schema upgrades
and historical downgrades; never use a staging or production database URL.
Migration tests create uniquely named disposable databases and need CREATEDB on
that test server. Older-revision tests stop at revision 003; they do not downgrade
through irreversible catalogue removal:
DATABASE_URL=postgresql://postgres:dev@localhost:5432/events \
python3 -m unittest discover -s tests -t .
Tests also load shared publisher payload fixtures from
sns-s3-archiver/tests/fixtures/archive-key-contract.json; timing ingestion no
longer calculates or persists archive keys. CI provides
DATABASE_URL for the PostgreSQL integration tests. The timing image has a
Kubernetes import smoke test in tests-k8s.yml.
License
This project is licensed under the GNU General Public License v3.0 or later. See the LICENSE.