This is the multi-page printable view of this section.
Click here to print.
Return to the regular view of this page.
Pipeline Timing
Independent SQS worker for RPM and container execution timing and cross-system lineage.
It shares the Status events database but owns only the pipeline_timing schema.
Features
- Current state, timestamps, and separate attempts for GitLab jobs and Konflux PipelineRuns
- Commit, merge request, PipelineRun, TaskRun, Snapshot, and Release lineage
- Source-derived execution creation, start, and completion times
- Alembic-managed schema, dedicated queue, and worker metrics
Prerequisites
- Python 3.11+ and PostgreSQL 16
- A dedicated timing SQS queue subscribed to the environment’s SNS topic with
RawMessageDelivery: false
The existing Status worker and event forwarders keep their own images and queues.
Infrastructure provisions the timeline queues, IAM, and alerts separately from
the Status worker.
Each environment has a separate queue named
arr-hummingbird-<env>-pipeline-timing-sqs and a matching -sqs-dlq queue.
The source queue retains messages for 4 days; its DLQ retains them for 14 days
from the original enqueue time when SQS automatically moves a message to the
DLQ. Operator redrive from the DLQ resets the enqueue timestamp. The source queue
moves a message to its DLQ after 3 receives and uses a 300-second visibility
timeout, matching Status.
The SNS subscription preserves the SNS envelope (RawMessageDelivery: false)
and does not filter events. See Hummingbird Events Topic
for queue policy and redrive details.
Provision both environments’ queues and SNS subscriptions before deploying the
workers. Messages collected before a worker starts may expire after four days;
this pre-launch loss is accepted. A queue without a consumer does not move
messages to its DLQ because no receives occur. Validate the staging worker
before deploying the production worker or enabling new production event sources.
Installation
cd pipeline-timing
pip install -e ".[dev]"
The pipeline-timing Containerfile builds the separate
quay.io/hummingbird-ci/pipeline-timing image. Both the initializer and worker
run from this image.
Usage
Schema initialization
Run the initializer as the init container of the single-replica timing worker
Deployment. Use a Recreate strategy so Kubernetes stops the old worker before
starting a Pod whose init container may upgrade the schema:
DATABASE_URL="$DATABASE_URL" python3 -m pipeline_timing.worker.init_db
The initializer creates the pipeline_timing schema and runs
alembic upgrade head each time a Pod starts. This is a no-op at the expected
revision; a new image can apply a forward migration during rollout. If migration
fails, the initializer exits nonzero and Kubernetes does not start the worker.
The worker checks the revision again before polling. Fix migration errors with
a forward revision; do not downgrade or drop the schema during rollback.
Staging deployment from a merge request
Wait for the tools MR’s tests and Konflux image build to pass, then run the
manual deploy_pipeline_timing_staging job. The job does not wait for those
checks automatically. It triggers the infrastructure pipeline with
ONLY_JOB_NAME=pipeline-timing and ONLY_DEPLOYMENT_ENVIRONMENT=staging, using
the MR commit’s image for both the migration init container and worker.
Production is not deployed by this job.
Verify the init container completed, the worker accepts the new Alembic revision,
and its queue is catching up before promoting the published image to production.
Normal infrastructure pipelines use a shared image pin for both environments.
Coordinate those deployments during MR validation so an older pinned image does
not replace the MR image. The worker requires an exact Alembic revision match;
an older image will refuse a newer revision even when the table layout is unchanged.
Execution-only schema and catalogue removal
Under HUM-10942, revision 003_execution_records added nullable created_at to
TaskRuns and PipelineRuns, stopped catalogue writes, and preserved old recovery
pointers temporarily. Historical creation-timestamp recovery completed in staging
and production using a one-time local tool. The tool was intentionally not retained.
Revision 004_drop_event_catalogue removes the legacy latest_event_id foreign
keys and columns from the reporting tables, then drops pipeline_timing.events
and its remaining indexes. It does not change reporting rows, their lifecycle
fields, parentage, or ordering metadata. The database named events and legacy
Status tables in public remain unchanged. Original migrations remain intact so
fresh initialization follows the same revision chain.
Before deployment, confirm recovery is complete or its gaps accepted in both
environments, no external consumer needs the catalogue, and the normal backup is
available. Record the catalogue allocation and PVC free space, then verify the
space recovery and continued ingestion in staging before production promotion.
The table/index allocation is released when the migration commits without a bulk
delete, table rewrite, or VACUUM FULL. Dropped pointer columns may retain small
amounts of physical row space; that does not justify a rewrite.
The migration limits lock waits to five seconds. A conflicting reader or unexpected
dependency makes initialization fail and the transactional DDL roll back. Resolve
contention or the dependency and retry with the compatible image. Downgrade is
explicitly unsupported: recreating empty history would not restore the deleted
catalogue or pointers. Use a compatible forward fix or the backup recovery procedure.
Kubernetes creation time comes from metadata.creationTimestamp; start and
completion come from status.startTime and status.completionTime. Missing
source timestamps remain null, not inferred from SNS publication time. Pending
conditions are distinct from running, and failure reasons retain cancellation
details. GitLab jobs and pipelines retain their source lifecycle fields and status.
Only execution state and lineage are maintained. There is no event-history array,
per-step timing, new archive-location table, or permanent S3 pointer on each row.
Resource deletion preserves the execution’s result, timings, and parentage.
Timing worker
Deploy one worker replica with the environment’s dedicated queue. Kubernetes
starts it only after the init container succeeds:
SQS_QUEUE_URL="$TIMING_QUEUE_URL" python3 -m pipeline_timing.worker
The worker polls when it starts. Accepted notifications update reporting rows in
one committed transaction before SQS acknowledgement, without event-table writes. Malformed
events remain for retry and eventual dead-letter-queue redrive; database or
revision failures stop polling without affecting legacy Status. Scale the timing
worker to zero replicas to pause ingestion. After launch, monitor queue age
against the four-day source retention limit during a pause to avoid losing
events. SQS preserves the original enqueue time for automatic DLQ transfer, and
an operator redrive resets it. A new rollout uses Recreate: the old worker
stops, Alembic upgrades, and the new worker starts. The S3 archiver receives its
own SNS copy and remains independent of timing ingestion.
The worker follows Status’s polling pattern: a 20-second long poll receives up
to 10 messages, processed sequentially with boto3’s default request retries.
Each accepted event commits in its own transaction. At the end of each receive,
one SQS batch-delete request acknowledges committed events and ignored messages.
Short receives flush immediately; failed processing never adds a receipt to the
delete request. Receive errors are logged and polling continues. Partial delete
failures and uncertain request outcomes leave committed events available for
idempotent redelivery, and polling continues. If a database failure interrupts a
receive, the worker first attempts to acknowledge earlier confirmed commits.
Database connection or schema failures stop the worker
for Kubernetes to restart it. SQS visibility controls when an unacknowledged
message becomes available again.
The worker caches up to 128 UPSERT statement shapes by table and ordered columns.
Each execution binds fresh event values; the cache does not retain payloads,
connections, or cursors. Source update timestamps order messages when both the
incoming and existing row have them; SNS publication timestamps break ties or
provide the fallback. Kubernetes condition transition times supply source ordering.
Late pending/running snapshots cannot regress a completed Kubernetes execution.
Sparse payloads preserve existing values, and creation time stays immutable once known.
Queue age and DLQ depth are monitored independently from the Status queue. Fix
malformed messages before using the documented redrive procedure. If the worker
is paused or not yet deployed, remove only a known test message after confirming
that redrive returned it to the timing queue.
Configuration
| Variable |
Default |
Description |
DATABASE_URL |
- |
Existing Status events database URL |
SQS_QUEUE_URL |
- |
Environment-specific timing queue URL |
SQS_WAIT_TIME |
20 |
SQS long-poll interval in seconds |
METRICS_PORT |
9090 |
Prometheus port when the worker is enabled |
Infrastructure supplies queue values for each environment:
| Environment |
Queue URL variable |
| Staging |
ARR_HUMMINGBIRD_STAGING_PIPELINE_TIMING_SQS_QUEUE_URL |
| Production |
ARR_HUMMINGBIRD_PROD_PIPELINE_TIMING_SQS_QUEUE_URL |
S3_BUCKET is no longer required or used by the worker. Remove the old deployment
variable only after the compatible worker is deployed in both environments.
The independent S3 archiver and its configuration remain unchanged.
POSTGRES_URL is also accepted when DATABASE_URL is not set. For local
development only, the database URL defaults to
postgresql://postgres:dev@localhost:5432/events.
Schema and lineage
| Table |
Identity |
Purpose |
merge_requests |
GitLab project ID and MR IID |
Current state and lifecycle times |
commits |
GitLab project ID and SHA |
Commit metadata |
merge_request_commits |
Project ID, IID, SHA, relation |
Head and merge-commit links |
gitlab_pipelines |
Project ID and pipeline ID |
Pipeline status and timing |
gitlab_jobs |
Project ID and job ID |
Individual job attempts |
pipeline_runs |
Cluster and Kubernetes UID |
Build, test, and Release attempts |
task_runs |
Cluster and Kubernetes UID |
Task timing and parent UID |
snapshots |
Cluster and Kubernetes UID |
Build-to-test bridge |
releases |
Cluster and Kubernetes UID |
Snapshot-to-Release bridge |
Source timestamps accept ISO 8601 and GitLab’s YYYY-MM-DD HH:MM:SS UTC format
and are normalized to UTC. Timestamps without a timezone are interpreted as UTC.
latest_observed_at and source_updated_at order updates without a notification
catalogue or event foreign keys. The schema stores selected reporting fields,
not raw event JSON. Join MR head and merge
commits to GitLab pipelines by canonical HTTPS repository URL and SHA; join
jobs by project and pipeline ID. Konflux PipelineRuns use the same repository
URL and SHA, then fan out by component and execution UID. TaskRuns link through
PipelineRun owner UIDs; Snapshot source-run names, test-run Snapshot labels,
and Release references connect the remaining phases. The Release completion
time closes the current timeline. Distinct job IDs and PipelineRun UIDs retain
each retry’s status and timing.
PipelineRun SHAs come from supported labels and annotations, including
pac.test.appstudio.openshift.io/sha, before falling back to the revision
parameter.
Promotion Snapshots populate source_pipeline_run_name from
hummingbird-project.io/promotion-pipelinerun. This annotation takes precedence
over the usual pac.test.appstudio.openshift.io/log-url, so the parent is the
promotion run rather than the original MR build. Other Snapshots keep the
log-URL behavior. This connects the promotion run to its Snapshot and Releases
without changing the schema. References to the original MR build and source
Snapshot in promotion annotations are not stored separately.
Existing promotion Snapshots need a separate backfill if no newer notification
arrives after this extraction change. Deploying the worker does not rescan stored
Snapshots or automatically repair historical links.
Finding archived notifications
The independent S3 archiver still receives its own SNS copy. The timing database
does not store a direct archive pointer or a complete notification history.
For Kubernetes executions, use the environment’s archive bucket and
latest_observed_at as a UTC SNS timestamp with milliseconds to narrow the object
prefix, for example sns/2026/01/02/03/04/2026-01-02T03:04:05.678Z#. Match source,
kind, cluster, namespace, and resource name in candidate filenames, then verify the
UID in the downloaded payload. For GitLab, verify the project and execution/commit
identity in candidate payloads.
This is candidate discovery, not a guaranteed unique or per-field provenance link.
Several notifications may share a timestamp, and sparse updates can retain facts
from earlier payloads. Lookup depends on archive delivery, naming, and retention;
it does not recreate the removed database catalogue.
Metrics include hummingbird_pipeline_timing_records_total{kind,result},
hummingbird_pipeline_timing_failures_total{reason},
hummingbird_pipeline_timing_missing_lineage_total{kind}, and
hummingbird_pipeline_timing_worker_stopped.
success and ignored record results count only confirmed SQS acknowledgements,
including duplicates. Each failed or uncertain batch entry increments the sqs
failure counter. These counters do not measure unique event rows. Monitor queue age and
dead-letter-queue depth separately from the legacy Status queue. Fix malformed
events before redriving them. Identity-based upserts and ordering guards keep
redelivery idempotent; there is no database row for every SNS message.
Development
Run tests only against the disposable PostgreSQL service provided in CI or a
disposable local database. Tests truncate timing data and exercise schema upgrades
and historical downgrades; never use a staging or production database URL.
Migration tests create uniquely named disposable databases and need CREATEDB on
that test server. Older-revision tests stop at revision 003; they do not downgrade
through irreversible catalogue removal:
DATABASE_URL=postgresql://postgres:dev@localhost:5432/events \
python3 -m unittest discover -s tests -t .
Tests also load shared publisher payload fixtures from
sns-s3-archiver/tests/fixtures/archive-key-contract.json; timing ingestion no
longer calculates or persists archive keys. CI provides
DATABASE_URL for the PostgreSQL integration tests. The timing image has a
Kubernetes import smoke test in tests-k8s.yml.
License
This project is licensed under the GNU General Public License v3.0 or later.
See the LICENSE.
2 - Pipeline timing implementation plan
Purpose
The original RPM pipeline timing design records
the HUM-5820 design and remains unchanged. This implementation plan records the
HUM-10942 revisions to retain execution timing and lineage for both RPM and
container pipelines, not a permanent notification catalogue or permanent S3
references on reporting rows. Its execution-only requirements supersede the
original design’s archive-reference and RPM-only scope decisions.
The Status events database remains the storage location, but pipeline-timing is a separate
project, image, and worker with its own deployment lifecycle and resource quotas. Hummingbird Status
continues legacy ingestion; the GitLab and Kubernetes event forwarders remain independent
publishers. This service split follows HUM-7529 MR
1261 review without changing the
event contracts or lineage model.
Prior Exploration
Closed tools MR 1165 and closed
infrastructure MR 1381
informed this plan. They are not implementation vehicles for the Stories above. New focused merge
requests should reference the Story that owns their work.
Current execution-only transition
The original HUM-5820 delivery stories are complete. HUM-10942 owns the revised
execution-only requirements and supersedes the old archive-reference and RPM-only
scope decisions. Existing Status tables, publishers, the archiver, and queue
contracts remain unchanged.
- Release one removes the unused composite S3 lookup index.
- Release two, revision
003_execution_records, adds nullable creation times to
TaskRuns and PipelineRuns and makes the reporting tables’ latest_event_id
nullable. The worker stops inserting events and never changes old pointers.
New rows use null pointers; the old foreign keys remain valid temporarily.
- Operator-run local recovery filled historical creation timestamps through those
frozen pointers without replaying state. Both environments were verified with
zero missing TaskRun/PipelineRun creation times or creation-time ordering anomalies.
The one-time tooling was intentionally removed rather than retained in the repo.
- Release three, revision
004_drop_event_catalogue, explicitly removes the
legacy foreign keys and latest_event_id columns, then drops events and its
remaining indexes. All reporting values and ordering metadata are preserved.
Catalogue removal is transactional, uses five-second lock waits, and has no
supported downgrade. Confirm recovery/backup readiness and no remaining external
consumer, validate staging ingestion and physical space recovery, then use the
existing image promotion path for production. Do not recreate empty history as a
rollback. The events database and legacy public schema are not removed.
The worker does not require S3_BUCKET or calculate archive keys. Remove unused
infrastructure bucket configuration only after the compatible app is deployed in
both environments. Do not change the worker’s IAM permissions for local recovery.
Before each promotion, verify migration revision, execution timing/lineage,
commit-before-acknowledgement, queue catch-up, and legacy isolation. During MR
validation, prevent ordinary infrastructure pipelines from restoring the older
shared image pin. Merge can lead to automatic Konflux publication, Renovate image
pin updates, and deployment to both environments; it is not a recovery hold.
Test fresh initialization and populated-schema upgrades, preserving all reporting
values and relation files while removing only the catalogue and legacy references.
Keep revision-001/002/003 regression tests in isolated disposable databases that
stop before irreversible removal. Verify a forced cleanup failure rolls back all
constraint/column drops, and an attempted downgrade cannot fabricate history.
Continue testing lifecycle times, states, lineage, redelivery, and ingestion with
the catalogue absent. No change to the worker’s ingestion logic is required.
Implementation Plan
Scope Decisions
- The timeline tracks through completion of the Konflux
Release resource.
- The Release completion timestamp is the final tracked milestone.
- PipelineRun and TaskRun detail in
rhtap-releng-tenant requires separate provisioning and is
required before HUM-5820 is complete.
- PipelineRuns and TaskRuns in
hummingbird-tenant remain in scope.
Snapshot and Release events are required as lineage bridges even though they were omitted
from the initial agreed-event list.
- A Snapshot links a build PipelineRun to test PipelineRuns.
- A Release links a Snapshot to Release pipeline completion and, where visible, its PipelineRun.
- The new schema stores only their reporting and lineage fields, not raw JSON.
- The new model stores the latest known state of each execution and its relevant timestamps. It
does not duplicate every status transition as a domain row.
- A retry is a new PipelineRun and has its own UID, row, status, start time, and completion time.
It does not replace the failed PipelineRun.
- For example, a PipelineRun that fails after four hours and a retry that succeeds in ten minutes
remain two separate records. A report must not show only the ten-minute retry as the time for
both attempts.
- Both RPM and container pipeline executions are in scope. Retain the existing
source mix rather than introducing repository filters during this transition.
- Full event replay and per-step timing are out of scope. Historical recovery is
limited to best-effort creation timestamps and runs outside startup migrations.
- Each environment needs a dedicated timing SQS queue, dead-letter queue, and
pipeline-timing
worker deployment using its own image. The existing Status queue, image, and worker continue
legacy ingestion unchanged.
PostgreSQL Schema
Create a pipeline_timing schema in the existing Status events database. Do not add columns,
indexes, constraints, or triggers to existing tables in the public schema. The Dashboard continues
to query only the existing tables.
Domain tables contain selected scalar fields, source timestamps, lineage, and
ordering metadata, not a raw payload, history array, or catalogue foreign key.
| Table |
Identity columns |
Purpose |
pipeline_timing.merge_requests |
gitlab_project_id, iid |
Current MR state and lifecycle timestamps |
pipeline_timing.commits |
gitlab_project_id, sha |
Commit identity and available commit metadata |
pipeline_timing.merge_request_commits |
gitlab_project_id, iid, sha, relationship |
MR head and merge-commit lineage |
pipeline_timing.gitlab_pipelines |
gitlab_project_id, pipeline_id |
Pipeline status, source, ref, SHA, and execution timestamps |
pipeline_timing.gitlab_jobs |
gitlab_project_id, job_id |
Job attempt status, stage, timing, failure reason, and retry information |
pipeline_timing.pipeline_runs |
cluster, uid |
Konflux PipelineRun state, source identity, timing, and Snapshot lineage |
pipeline_timing.task_runs |
cluster, uid |
Tekton task state, timing, and parent PipelineRun UID |
pipeline_timing.snapshots |
cluster, uid |
Build PipelineRun to Snapshot lineage |
pipeline_timing.releases |
cluster, uid |
Snapshot to Release completion and PipelineRun lineage |
Store both GitLab project ID and a canonical HTTPS repository URL where available. GitLab project ID
is the stable GitLab identity. The repository URL is needed because Konflux resources contain source
URLs rather than GitLab project IDs.
004_drop_event_catalogue removes the temporary recovery catalogue and pointers
after recovery completion. latest_observed_at and source_updated_at remain on
all reporting tables for ordering, independently of notification-history storage.
The S3 archiver remains an independent subscriber; execution fields require no
archive lookup during normal ingestion or reporting.
Schema Management
Use Alembic (a database schema migration tool already used by
Hummingbird’s workqueue service) to manage only the pipeline_timing schema. Existing tables in the
public schema remain under the Hummingbird Status db.create_schema() path; adopting Alembic for
those tables is separate work.
- Add Alembic configuration and revisions to the
pipeline-timing project and image.
- Add
pipeline_timing.worker.init_db for the worker Deployment’s init container. It executes
CREATE SCHEMA IF NOT EXISTS pipeline_timing directly and commits before invoking Alembic. Do
not rely on Alembic to create the schema. Alembic manages every object inside the schema after
that bootstrap.
- Set Alembic’s
version_table_schema="pipeline_timing" and qualify every migration object with
the schema. The version table and all managed objects must remain outside public.
- Create the tables, constraints, and required indexes in the initial revision.
- Run
alembic upgrade head unconditionally from pipeline_timing.worker.init_db in the
single-replica Deployment’s init container. Use Recreate so the old worker stops before a new
image migrates the schema. Alembic is a no-op when the schema is already at head.
- On migration failure, log the exception and expected revision at ERROR level and exit nonzero.
Kubernetes must not start the worker container after a failed init container; the separate
Hummingbird Status Deployment continues unchanged.
- Have the timing worker independently verify that the database is at the expected Alembic revision
before polling its queue. If the revision check fails, log the expected and observed revision at
ERROR level and stop the timing worker. The legacy worker continues unchanged.
- Deploy one replica with the timing image and init container once the queue and archiver are
ready. To pause polling, scale the Deployment to zero and monitor queue age against the source
queue’s 4-day retention. SQS preserves a standard message’s enqueue timestamp when automatically
moving it to the DLQ, whose 14-day retention is a maximum total age. Operator redrive from the
DLQ resets the enqueue timestamp.
- Do not automatically downgrade or drop the schema during application rollback. Stop the timeline
worker and use a forward migration to correct schema problems.
Selected Fields
The detailed schema should include at least the following reporting fields. These are proposed
database column names, not source-payload field names.
| Record type |
Required columns |
| Merge request |
gitlab_project_id, repository_url, iid, state, source_branch, target_branch, head_sha, merge_commit_sha, created_at, updated_at, merged_at, closed_at, web_url |
| Commit |
gitlab_project_id, repository_url, sha, title, message, committed_at, web_url |
| MR commit link |
gitlab_project_id, repository_url, iid, sha, relationship |
| GitLab pipeline |
gitlab_project_id, repository_url, pipeline_id, mr_iid, sha, ref, source, status, created_at, updated_at, started_at, completed_at, web_url |
| GitLab job |
gitlab_project_id, repository_url, job_id, pipeline_id, sha, name, stage, status, failure_reason, allow_failure, created_at, started_at, completed_at, duration_seconds, queued_duration_seconds, retry_count, web_url |
| PipelineRun |
cluster, namespace, uid, name, application, component, pipeline_type, pipeline_name, trigger_type, repository_url, sha, mr_iid, snapshot, status, reason, created_at, started_at, completed_at |
| TaskRun |
cluster, namespace, uid, name, pipeline_run_uid, pipeline_run_name, pipeline_task, task_reference, status, reason, pod_name, created_at, started_at, completed_at |
| Snapshot |
cluster, namespace, uid, name, application, component, sha, source_pipeline_run_name, created_at |
| Release |
cluster, namespace, uid, name, snapshot, status, pipeline_run_namespace, pipeline_run_name, completed_at |
TaskRun and PipelineRun creation time is metadata.creationTimestamp; start and
completion are the resource’s own status timestamps. Missing values remain null.
Unknown conditions with a pending reason report Pending, not Running.
Cancellation remains distinguishable through the failure reason. GitLab execution
timestamps and status remain source-derived. Non-execution entities retain their
meaningful lifecycle milestones rather than fabricated start/completion fields.
GitLab job retries have distinct job IDs and therefore distinct rows. Store retry metadata when
GitLab supplies it, but do not invent a parent job identity. Consumers can group attempts by
project, pipeline, stage, and job name.
PipelineRun retries also have distinct rows because each PipelineRun has a distinct Kubernetes UID.
Keep the timing and result for every attempt. This plan does not define a retry-group column or a
single combined retry duration. Consumers that need an end-to-end duration across retries must keep
the duration of each attempt visible and calculate the full elapsed time from the first attempt’s
start to the final attempt’s completion.
GitLab Job Hook payloads do not provide a dedicated job URL. Derive web_url without an API request
by appending /-/jobs/{build_id} to project.web_url.
merged_at and closed_at are new nullable TIMESTAMPTZ columns on
pipeline_timing.merge_requests; they are not additions to the existing
public.gitlab_merge_requests table. Use explicit GitLab merge and close timestamps when the
webhook supplies them. Otherwise, derive them from the first event that reports the corresponding
action or state, preferring the source updated_at timestamp and falling back to observed_at.
Preserve the first derived terminal timestamp across later sparse events. This makes MR
creation-to-merge timing available without requiring a GitLab API lookup.
Repository URL Normalization
Normalize repository identity to an HTTPS URL without a trailing slash, .git suffix, or GitLab
resource path. Prefer GitLab’s project.web_url when it is available. Accept HTTPS, SSH, SCP-style
Git, and GitLab tree URLs from Konflux metadata.
These source values:
https://gitlab.com/redhat/hummingbird/rpms.git
git@gitlab.com:redhat/hummingbird/rpms.git
https://gitlab.com/redhat/hummingbird/rpms/-/tree/main
all normalize to:
https://gitlab.com/redhat/hummingbird/rpms
Use repository_url consistently in the new schema. Do not use a bare project path or
source-specific Git URL as the cross-system identity.
Identity and Update Ordering
- Use
(cluster, uid) as the primary execution identity for Kubernetes resources.
- Names are not sufficient because Kubernetes can recreate a resource with the same name.
- Store namespace and name for display and fallback joins.
- Use source IDs for GitLab records and retain
repository_url for cross-system joins.
- Store the latest accepted SNS timestamp as
latest_observed_at on reporting rows.
- Use source update timestamps from GitLab and Kubernetes condition transition times.
- Prevent an older event delivered later by SQS from replacing newer state.
- Compare source timestamps only when both records have them; use SNS publication
timestamps as the fallback and tie-breaker, including migration of legacy rows.
- Late non-terminal snapshots must not regress completed Kubernetes executions.
- Preserve source fields absent from sparse updates and keep known creation time
immutable. Ordering metadata advances independently of notification history.
- Do not require foreign keys for cross-source lineage that may arrive out of order.
Use indexed identity columns and late joins. There are no legacy event foreign
keys or pointer columns after catalogue removal.
Required Indexes
The schema must provide indexes for the known lineage and timeline queries.
Exact index names and partial predicates are implementation details, but the
following column order is required as a baseline:
| Table |
Columns |
Supports |
merge_request_commits |
gitlab_project_id, iid |
commits for one MR |
merge_request_commits |
repository_url, sha |
MR lookup from a commit |
gitlab_pipelines |
repository_url, sha |
pipelines for a commit |
gitlab_jobs |
gitlab_project_id, pipeline_id |
jobs for a pipeline |
pipeline_runs |
repository_url, sha |
Konflux runs for a commit |
pipeline_runs |
cluster, namespace, name |
name-based fallback lineage |
pipeline_runs |
cluster, namespace, snapshot |
test PipelineRuns for a Snapshot |
task_runs |
cluster, pipeline_run_uid |
tasks for a PipelineRun |
snapshots |
cluster, namespace, source_pipeline_run_name |
Snapshot produced by a build PipelineRun |
releases |
cluster, namespace, snapshot |
Releases for a Snapshot |
Primary keys provide the identity indexes listed in the schema table and do not need duplicate
secondary indexes. Validate the final indexes with the representative acceptance query and
PostgreSQL EXPLAIN before source enablement.
The initial design required a reverse archive lookup index on s3_bucket, s3_key.
The staging storage investigation on October 22, 2026 found that this index occupied
about 8.1 GiB, roughly 40% of the event table’s total allocation including indexes,
with zero recorded scans. No application query using the reverse lookup was found.
The temporary legacy recovery used frozen latest_event_id values to read the
archived creation metadata. Runtime ingestion never needed that reverse index.
Revision 002_drop_events_s3_index removes this unused application index to reclaim
disk space and avoid maintaining it for every new event. It preserves event rows,
columns, and archive references. Scan statistics do not prove that no external
consumer needs the index; confirm that before deployment. Deploy and verify index
space recovery in staging and production before the execution-only transition.
Revision 004_drop_event_catalogue drops the remaining catalogue indexes with the
table. No execution-reporting index is removed or rebuilt.
Lineage Model
Consumers reconstruct a timeline with the following relationships:
- A merge request links to its head commits and merge commit through
merge_request_commits.
- A commit links to GitLab pipelines by
repository_url and SHA. A direct pipeline MR IID is used
when the webhook supplies it.
- GitLab jobs link to their pipeline by project and pipeline ID.
- Konflux PipelineRuns link to a commit by
repository_url and SHA.
- Extract the repository from Pipelines as Code annotations first, then the
git-url parameter.
- Extract SHA and MR IID from the current Pipelines as Code and build-service labels and annotations.
- TaskRuns link to PipelineRuns by Kubernetes owner UID. Parent name is a fallback for malformed or
older resources without an owner UID.
- A Snapshot links to its source build PipelineRun by name and cluster/namespace context.
- Test PipelineRuns link to a Snapshot through the Snapshot label.
- A Release links to its Snapshot and to any referenced Release PipelineRun.
- In the current scope, the Release completion timestamp closes the timeline even when its managed
PipelineRun is in an inaccessible namespace.
The primary cross-system correlation key within one source pipeline is canonical repository_url
plus commit SHA, not SHA alone. One commit can produce several component PipelineRuns, so the
resulting timeline fans out by component and execution rather than assuming one build per commit.
Event Processing and Failure Isolation
Subscribe a dedicated timing queue to the SNS topic already used by the existing Status queue. The
timing queue receives its own SQS copy of each SNS envelope. The existing Status queue and worker
remain unchanged. The pipeline-timing worker parses and upserts only pipeline_timing records.
For each timeline message:
- Start polling only after the Pod’s migration init container succeeds and the worker verifies the
expected revision. Scaling the single-replica Deployment to zero leaves the queue untouched.
- Decode the SNS envelope. A valid event outside the accepted timeline event types is ignored and acknowledged.
- For an accepted notification, parse and upsert the reporting records in one
transaction. Do not append a notification-history row or require a catalogue link.
- Commit the transaction, then delete the SQS message.
- On malformed data for an accepted event, roll back and leave the message on the queue. SQS
retries it and moves it to the timeline dead-letter queue after the configured receive limit.
- On a database connection, schema, or migration failure, roll back, stop polling, and leave the
queue intact for recovery. Do not send valid events to the dead-letter queue during an
infrastructure outage.
A timeline failure must not delay, retry, or change legacy ingestion. Operators fix a malformed
event or parser error, then redrive messages from the timeline dead-letter queue. They fix an
infrastructure failure, then restart the worker to resume polling. Configure the source queue with
4-day retention and its DLQ with 14-day retention, a finite receive limit, and a visibility timeout
at least twice the worker’s per-message processing budget. Automatic source-to-DLQ transfer
preserves a standard message’s original enqueue timestamp; operator redrive from the DLQ resets it.
Add queue-age and dead-letter-queue alerts, and document and test the redrive procedure.
Completed recovery and archive inspection
S3 archival remains independent and unchanged. The timing worker neither computes
archive keys nor needs bucket configuration or S3 read permissions. The operator-run
recovery read immutable creation timestamps from archived snapshots, updated only
null creation fields, and finished before catalogue removal. It was not part of a
migration, and its tooling is not retained.
latest_observed_at and resource identity can narrow an archive search while
objects are retained. Validate candidate payload identities; this is not guaranteed
unique lookup or per-field provenance because sparse updates can merge older facts.
See finding archived notifications.
No permanent per-execution S3 pointer or replacement history table is added.
Original HUM-5820 delivery and source rollout
The remaining delivery-story sections record the original rollout. Their permanent
archive-reference, bucket-configuration, and RPM-only assumptions are historical
and superseded by HUM-10942 and the execution-only transition above. The publishers
and independent archiver retain their existing event contracts.
This section describes code, tests, and documentation in the tools repository. Environment-specific
source infrastructure is owned by HUM-7535 and activated under HUM-7533.
GitLab Event Forwarder
The forwarder already publishes arbitrary GitLab webhook bodies and derives event_type from
object_kind. No event-specific forwarding implementation is required.
Add fixtures and tests for:
- Pipeline Hooks with
object_kind: pipeline
- Job Hooks with
object_kind: build
project_path and event_type SNS message attributes
- compressed payload round trips
Kubernetes Event Forwarder
- Add
TaskRun to the configured resources only after the deployed image understands its payload requirements.
- Extend the resource configuration schema with optional
labelSelector and fieldSelector keys.
- Pass configured selectors to the existing
ResourceWatcher label_selector and field_selector
parameters.
- Remove
metadata.managedFields from TaskRun messages before SNS publication because it is
transport metadata and can dominate payload size.
- Keep status, results, owner references, task identity, and timing data needed for extraction and
future diagnosis.
- Measure compressed and base64-encoded TaskRun payloads from representative RPM build, test, and
tenant Release PipelineRuns.
- Do not enable the source if representative payloads approach the SNS 256 KiB limit. Define and
review any additional projection before rollout rather than silently dropping fields.
- Preserve spooled events on publish failure so a transient SNS error does not discard them.
Use a TaskRun label selector for the RPM application if production samples confirm that Tekton
propagates a stable application label. If no reliable selector exists, estimate and approve the
namespace-wide event rate before enabling the watch because every matching TaskRun will also be
archived to S3.
S3 Archiver
The archiver is already generic, but its resource identity logic must change for the new event
types. Extend _extract_resource_name() and _build_s3_key() so GitLab pipeline and job keys
include their numeric pipeline or job ID and Kubernetes keys include logical cluster name,
namespace, and name. URL-valued cluster attributes from older forwarders retain name-only
identities. Cover those identities with the canonical contract fixture.
Continue archiving the complete event as received from SNS. The Kubernetes forwarder’s documented
removal of managedFields occurs before that boundary. Keep explicit test coverage for PipelineRun,
TaskRun, Snapshot, Release, merge request, commit/push, pipeline, and job events.
Pipeline Timing and Hummingbird Status Changes
- Add a
pipeline-timing/ project, package, image, and Alembic configuration with an initial
pipeline_timing revision. Generate its independent Konflux build resources.
- Add
pipeline_timing/timeline.py for extraction, ordering rules, and upserts.
- Keep Hummingbird Status extraction, schema creation, upsert functions, dependencies, and image
responsible only for the
public schema.
- Add timing dispatch for GitLab
push, merge_request, pipeline, and build events and
Kubernetes PipelineRun, TaskRun, Snapshot, and Release events.
- Mark the newly handled event types as known to the legacy Status parser so expected timing events
do not generate unknown-kind warnings; do not route them to legacy tables.
- Add
pipeline_timing.worker.init_db to the timing Deployment as an init container. Leave
hummingbird_status.worker.init_db and legacy worker/sqs.py responsible only for public.
- Deploy the independent timing image to poll only the timing queue and write only the
pipeline_timing schema in the shared events database.
- Add archive bucket configuration to the timing worker deployment.
- Start worker polling only after its init container succeeds; retain the worker’s own startup
revision check before queue access.
- Keep Alembic migration and revision-check failures isolated to the timing worker so they cannot
prevent legacy worker startup.
- Provide separate
pipeline_timing database counts without changing existing Status statistics queries.
Observability
Add metrics and structured logs for:
- timeline records processed by kind and result
- timeline extraction and database failures
- events missing expected lineage fields
- oversized Kubernetes payloads and SNS publish failures
- timeline queue age and dead-letter-queue depth
- timeline worker stopped because of a migration or revision-check failure
Failure logs must include the SNS message ID, source, kind, resource identity, and S3 location
without logging the full raw event. Continue monitoring legacy queue age and DLQ metrics, archiver
failures, PostgreSQL availability, and PVC usage.
Documentation Updates
Update the existing documentation for every component changed by implementation. Do not create a
separate documentation Story. Each delivery Story owns the pages for the behavior it changes:
| Story |
Required documentation updates |
| HUM-7529 |
New pipeline-timing.md, legacy isolation in hummingbird-status.md, and the timing schema and lineage sections of data-flow.md |
| HUM-7530 |
gitlab-event-forwarder.md, kubernetes-event-forwarder.md, sns-s3-archiver.md, and event-flow portions of data-flow.md |
| HUM-7531 |
staging worker deployment, monitoring, and recovery details in pipeline-timing.md |
| HUM-7532 |
managed Release namespace access in kubernetes-event-forwarder.md and data-flow.md |
| HUM-7533 |
final documentation audit across all affected pages before HUM-5820 closes |
| HUM-7534 |
production worker deployment, monitoring, and recovery details in pipeline-timing.md |
| HUM-7535 |
both environments’ timeline queues, dead-letter queues, subscriptions, IAM, and alerts in hummingbird-events-topic.md and infrastructure monitoring documentation |
HUM-8015 already updated message-bus.md for the independent staging message bus. HUM-7535
clarifies that timing queue subscriptions can precede worker deployment.
Testing
Unit Tests
Cover:
- every accepted GitLab and Kubernetes event type
- status and timestamp extraction
- explicit and derived MR merge and close timestamps
- repository URL normalization across HTTPS, SSH, SCP-style Git, and tree URLs
- GitLab job URL derivation from
project.web_url and build_id
- MR head and merge-commit lineage
- GitLab job retries as distinct job IDs
- a failed PipelineRun followed by a successful retry, with separate statuses and timings for both attempts
- PipelineRun-to-TaskRun owner UID lineage
- Snapshot and Release bridge extraction
- sparse and out-of-order updates
- deterministic archive keys
- missing optional lineage without rejecting otherwise valid records
- TaskRun serialization without
managedFields
- init-container migration gating and zero-replica pause behavior
- Alembic upgrade failure that blocks worker startup, and startup revision failure that stops only
the timing worker
- timeline acknowledgement only after a committed timeline transaction
- timeline retry and dead-letter-queue handling for parsing, data, and database failures
- an archive key from a timeline queue copy that matches the archiver’s key for the same SNS event
- ignored acknowledgement of valid events outside the accepted timeline event types
- stopped polling on database, schema, or migration failure
PostgreSQL Integration Tests
Use the PostgreSQL service already available in repository CI to:
- Apply the initial Alembic revision to an empty database and to a database with populated legacy tables.
- Verify Alembic reports the expected head revision and that its version table exists in
pipeline_timing, not public.
- Confirm the migration does not alter existing
public table definitions or representative legacy
queries.
- Ingest representative events in and out of order.
- Verify idempotent current-state upserts.
- Verify that a timeline failure leaves its message available for retry without affecting legacy ingestion.
- Query one RPM timeline from MR creation through Release completion and calculate each available
interval without raw JSON.
- Verify all domain rows resolve to event records with the expected S3 references.
- Inspect the new schema to confirm it has no raw JSON columns.
- Verify that timeline redrive produces idempotent rows and retains the same S3 reference.
End-to-End Acceptance Test
After source enablement, use a controlled RPM merge request that runs GitLab CI, Konflux build and
test PipelineRuns, and a Release. Record the MR and component selected for the test.
The acceptance query must demonstrate:
- MR creation and merge timestamps
- GitLab pipeline and job timing
- derived GitLab job URLs that match
project.web_url and build_id
- pre-merge and post-merge commit lineage
- component build PipelineRun and TaskRun timing
- a failed PipelineRun and successful retry shown as separate attempts, without losing the failed
attempt’s timing
- Snapshot and test PipelineRun lineage
- Release completion
- calculable durations between phases
- retrievable S3 objects for every row used by the timeline
- paused timeline processing, recovery, and dead-letter-queue redrive without affecting legacy Status
Run make test, make coverage, make check, and SAM template validation for the affected projects.
Infrastructure Implementation
HUM-8015 delivered the independent staging message bus: its own SNS topic, GitLab and Kubernetes
publishers, S3 archiver and bucket, and consumer cutover. HUM-7535 adds only the timeline queue,
dead-letter queue, subscription, IAM, and alerts in staging and production.
Staging Event Path
Use the staging message bus delivered by HUM-8015 before enabling the new sources in production:
- Verify that the HUM-8015 staging topic, publishers, archiver, bucket, and existing Status
consumer are deployed and that
message-bus.md describes them.
- Under HUM-7535, subscribe dedicated staging and production timeline queues to their respective
topics with
RawMessageDelivery: false. Configure each source queue with a finite receive limit,
a visibility timeout at least twice the worker’s per-message processing budget, its own
dead-letter queue, and least-privilege worker credentials. Production queue provisioning does not
depend on staging worker acceptance.
- Configure 4-day retention for both source queues and 14-day retention for their dead-letter
queues, then add queue-age and dead-letter-queue alerts. Messages collected before worker
deployment may expire after four days; this pre-launch loss is accepted. Without receives,
messages do not automatically move to the DLQ. Verify redrive before source enablement.
- Under HUM-7531, deploy one staging
pipeline-timing replica with Recreate strategy, its
migration init container, timing queue, resource limits, and explicit staging S3_BUCKET; do not
use the production bucket. Verify the init container reached the expected revision and the worker
is polling before enabling new event sources. Once the worker runs, monitor queue age to prevent
loss during pauses.
- Keep new production event sources disabled until staging acceptance succeeds.
The staging Kubernetes event forwarder may observe production Konflux resources, but it publishes
only to the staging topic. Staging and production forwarders must not publish the same watch stream
to the same topic.
Production Event Path
- Keep Pipeline Hooks already enabled by the shared Hummingbird webhook configuration.
- Add RPM Job Hooks only after the production Status and archiver images understand Job Hook events.
- Add PipelineRun and TaskRun watches only after compatible forwarder and Status images are deployed.
- Use the production queue, subscription, dead-letter queue, IAM, retention, and alerts provisioned
under HUM-7535. After staging worker acceptance, deploy one
pipeline-timing replica with
Recreate strategy, a migration init container, resource limits, and an explicit production
S3_BUCKET. Pre-launch queue messages may have expired; no historical backfill is required.
Verify the revision and worker before activating new event sources, and monitor queue age after
launch.
- Verify the production redrive procedure before enabling new sources.
- Enable new production event sources only through HUM-7533 after staging acceptance; the timing
worker starts under HUM-7534.
Managed Release Pipeline Access
HUM-7532 owns the access needed for rhtap-releng-tenant and is required for Epic completion.
- Obtain read-only
get, list, and watch permissions for PipelineRuns and TaskRuns in the
managed namespace.
- Add managed-namespace contexts to the staging and production Kubernetes event forwarders.
- Verify each event-forwarder service account with
auth can-i before enabling its watches.
- Link the managed PipelineRun to the Release record through its namespaced reference.
- Include signing and Pulp publication TaskRuns in the final acceptance timeline.
Infrastructure Verification Gates
- Render and resolve each final RPM webhook configuration before deployment.
- The production endpoint must retain its token and existing push, merge request, note,
confidential note, and pipeline events while adding job events.
- If same-URL inheritance does not deep-merge those settings, use an explicit webhook block
containing the complete configuration.
- Deploy compatible Hummingbird Status, Pipeline Timing, event-forwarder, and S3 archiver images
before enabling Job Hook or TaskRun events in that environment.
- Pass the canonical archive-key contract tests in both the archiver and Pipeline Timing projects.
- Verify representative TaskRun payload size after removing
managedFields.
- Use an RPM application label selector if production samples confirm Tekton propagates a stable
label. Otherwise, document approval of the namespace-wide event rate before enabling the watch.
- Use a single replica and
Recreate strategy; verify the migration init container succeeded and
the worker checked the expected Alembic revision before it polls.
- Verify the source queue has 4-day retention and the dead-letter queue has 14-day retention, with
alerts and a tested redrive procedure.
Rollout
- Complete the independent Pipeline Timing image and the legacy Status parser compatibility change
in HUM-7529, and event-bus support in HUM-7530.
- Verify the HUM-8015 staging message bus and provision both environments’ timing queues,
subscriptions, IAM, and alerts in HUM-7535, independent of staging worker acceptance. Pre-launch
messages may expire without reaching the DLQ; this loss is accepted.
- Under HUM-7531, deploy the single-replica staging timing worker with
Recreate strategy and its
migration init container. Verify the new schema and legacy isolation before enabling new sources.
- Complete
rhtap-releng-tenant provisioning in HUM-7532 and add it to the staging event forwarder.
- Under HUM-7533, activate staging Job Hook, PipelineRun, and TaskRun sources after validating the
running timing worker.
- Run a controlled RPM merge request and verify staging lineage, duration calculations, managed
Release tasks, and S3 references.
- Under HUM-7534, deploy the single-replica production timing worker with
Recreate strategy and
its migration init container after staging worker validation.
- Verify the production Alembic revision, legacy ingestion, timeline queue alerts, and metrics.
- Add the managed namespace to the production event forwarder, then activate new production sources
under HUM-7533 after verifying the timing worker is running.
- Expect each Kubernetes watcher to emit an initial
ADDED event for every existing matching
resource. Schedule source activation during a monitored window.
- Monitor legacy and timeline queue age, dead-letter-queue depth, worker and archiver errors,
forwarder spool growth, database size, and PVC free space.
- Run the final controlled RPM acceptance merge request in production before closing HUM-5820.
Rollback does not require dropping or downgrading the schema. Scale the timing worker to zero and
disable the new TaskRun and Job Hook sources in the affected environment. Legacy ingestion
continues. After launch, monitor queue age while paused: messages expire after four days unless the
worker resumes in time. Malformed messages move to the DLQ after failed receives and can be redriven
after repair.
Epic Completion Criteria
The approved design’s acceptance criteria remain authoritative. HUM-5820 is complete when:
- HUM-7529 through HUM-7535 are complete.
- The design’s acceptance criteria pass in both staging and production.
- Representative timelines include managed PipelineRun and TaskRun timing from
rhtap-releng-tenant.
- The staging-first source rollout and final production acceptance run are documented under HUM-7533.
- HUM-7533 has verified that each Story’s required documentation update is complete.
Out of Scope
- Creating a UI for visualization is out of scope.
- The data may later support a new application, Dashboard changes, or Grafana. UX should be
involved in that decision.
- Tracking the containers repository pipeline is out of scope.
- The pipeline-neutral schema, event contracts, and execution identities are intended to support
its GitLab and Konflux events in a future follow-up without creating a parallel storage model.
- The RPM and containers pipelines do not connect directly. A separate scheduled process
discovers updated RPMs and commits changes to
rpms.lock.yaml files in the containers
repository.
- Timeline replay and historical backfill are out of scope.
- Existing Hummingbird Status S3 ingestion bootstraps the
public schema and collapses
Kubernetes history, so it is not suitable for timeline recovery.
- Future work must define scoped S3 read permissions and chronological ingestion into the
Alembic-managed
pipeline_timing schema. Events that were never collected cannot be
reconstructed.
Delivery Stories
| Story |
Responsibility |
| HUM-7529 Make RPM pipeline timing data queryable |
Independent Pipeline Timing image and schema, Alembic, worker, ingestion, lineage, tests, and legacy Status compatibility |
| HUM-7530 Carry RPM pipeline timing events through the event bus |
GitLab and Kubernetes forwarders, S3 archiver event identities, payload handling, and tests |
| HUM-7531 Validate RPM pipeline timeline storage in staging |
Staging Pipeline Timing deployment, Alembic verification, legacy isolation, and representative queries |
| HUM-7534 Make RPM pipeline timeline storage available in production |
Production Pipeline Timing deployment with event sources still disabled |
| HUM-7535 Support environment-specific RPM pipeline event collection |
Both environments’ timeline queues, dead-letter queues, subscriptions, IAM, alerts, and configuration before worker rollout |
| HUM-7532 Expose managed Release pipeline timing |
Read-only rhtap-releng-tenant access and managed PipelineRun and TaskRun event collection |
| HUM-7533 Roll out RPM pipeline event collection |
Staging source activation, production source activation, monitoring, recovery, redrive, and end-to-end validation |
Dependency Order
- HUM-7529 and HUM-7530 can proceed in parallel.
- HUM-8015 is the completed staging message-bus prerequisite. HUM-7535 depends on HUM-7530’s event
contracts and provisions timeline-specific infrastructure in both environments without waiting
for staging worker acceptance.
- HUM-7531 depends on HUM-7529, HUM-7530, and the staging path from HUM-7535.
- HUM-7534 depends on successful staging validation in HUM-7531.
- HUM-7532 depends on the event support from HUM-7530 and the environment-specific infrastructure
from HUM-7535. Managed namespace access can proceed alongside the Pipeline Timing deployments,
but completion gates final acceptance under HUM-7533.
- HUM-7533 depends on HUM-7531, HUM-7534, HUM-7535, and HUM-7532. HUM-5820 remains open until the
final rollout is verified.