Pipeline timing implementation plan
Purpose
The original RPM pipeline timing design records the HUM-5820 design and remains unchanged. This implementation plan records the HUM-10942 revisions to retain execution timing and lineage for both RPM and container pipelines, not a permanent notification catalogue or permanent S3 references on reporting rows. Its execution-only requirements supersede the original design’s archive-reference and RPM-only scope decisions.
The Status events database remains the storage location, but pipeline-timing is a separate
project, image, and worker with its own deployment lifecycle and resource quotas. Hummingbird Status
continues legacy ingestion; the GitLab and Kubernetes event forwarders remain independent
publishers. This service split follows HUM-7529 MR
1261 review without changing the
event contracts or lineage model.
Prior Exploration
Closed tools MR 1165 and closed infrastructure MR 1381 informed this plan. They are not implementation vehicles for the Stories above. New focused merge requests should reference the Story that owns their work.
Current execution-only transition
The original HUM-5820 delivery stories are complete. HUM-10942 owns the revised execution-only requirements and supersedes the old archive-reference and RPM-only scope decisions. Existing Status tables, publishers, the archiver, and queue contracts remain unchanged.
- Release one removes the unused composite S3 lookup index.
- Release two, revision
003_execution_records, adds nullable creation times to TaskRuns and PipelineRuns and makes the reporting tables’latest_event_idnullable. The worker stops inserting events and never changes old pointers. New rows use null pointers; the old foreign keys remain valid temporarily. - Operator-run local recovery filled historical creation timestamps through those frozen pointers without replaying state. Both environments were verified with zero missing TaskRun/PipelineRun creation times or creation-time ordering anomalies. The one-time tooling was intentionally removed rather than retained in the repo.
- Release three, revision
004_drop_event_catalogue, explicitly removes the legacy foreign keys andlatest_event_idcolumns, then dropseventsand its remaining indexes. All reporting values and ordering metadata are preserved.
Catalogue removal is transactional, uses five-second lock waits, and has no
supported downgrade. Confirm recovery/backup readiness and no remaining external
consumer, validate staging ingestion and physical space recovery, then use the
existing image promotion path for production. Do not recreate empty history as a
rollback. The events database and legacy public schema are not removed.
The worker does not require S3_BUCKET or calculate archive keys. Remove unused
infrastructure bucket configuration only after the compatible app is deployed in
both environments. Do not change the worker’s IAM permissions for local recovery.
Before each promotion, verify migration revision, execution timing/lineage, commit-before-acknowledgement, queue catch-up, and legacy isolation. During MR validation, prevent ordinary infrastructure pipelines from restoring the older shared image pin. Merge can lead to automatic Konflux publication, Renovate image pin updates, and deployment to both environments; it is not a recovery hold.
Test fresh initialization and populated-schema upgrades, preserving all reporting values and relation files while removing only the catalogue and legacy references. Keep revision-001/002/003 regression tests in isolated disposable databases that stop before irreversible removal. Verify a forced cleanup failure rolls back all constraint/column drops, and an attempted downgrade cannot fabricate history. Continue testing lifecycle times, states, lineage, redelivery, and ingestion with the catalogue absent. No change to the worker’s ingestion logic is required.
Implementation Plan
Scope Decisions
- The timeline tracks through completion of the Konflux
Releaseresource.- The Release completion timestamp is the final tracked milestone.
- PipelineRun and TaskRun detail in
rhtap-releng-tenantrequires separate provisioning and is required before HUM-5820 is complete. - PipelineRuns and TaskRuns in
hummingbird-tenantremain in scope.
SnapshotandReleaseevents are required as lineage bridges even though they were omitted from the initial agreed-event list.- A Snapshot links a build PipelineRun to test PipelineRuns.
- A Release links a Snapshot to Release pipeline completion and, where visible, its PipelineRun.
- The new schema stores only their reporting and lineage fields, not raw JSON.
- The new model stores the latest known state of each execution and its relevant timestamps. It
does not duplicate every status transition as a domain row.
- A retry is a new PipelineRun and has its own UID, row, status, start time, and completion time. It does not replace the failed PipelineRun.
- For example, a PipelineRun that fails after four hours and a retry that succeeds in ten minutes remain two separate records. A report must not show only the ten-minute retry as the time for both attempts.
- Both RPM and container pipeline executions are in scope. Retain the existing source mix rather than introducing repository filters during this transition.
- Full event replay and per-step timing are out of scope. Historical recovery is limited to best-effort creation timestamps and runs outside startup migrations.
- Each environment needs a dedicated timing SQS queue, dead-letter queue, and
pipeline-timingworker deployment using its own image. The existing Status queue, image, and worker continue legacy ingestion unchanged.
PostgreSQL Schema
Create a pipeline_timing schema in the existing Status events database. Do not add columns,
indexes, constraints, or triggers to existing tables in the public schema. The Dashboard continues
to query only the existing tables.
Domain tables contain selected scalar fields, source timestamps, lineage, and
ordering metadata, not a raw payload, history array, or catalogue foreign key.
| Table | Identity columns | Purpose |
|---|---|---|
pipeline_timing.merge_requests |
gitlab_project_id, iid |
Current MR state and lifecycle timestamps |
pipeline_timing.commits |
gitlab_project_id, sha |
Commit identity and available commit metadata |
pipeline_timing.merge_request_commits |
gitlab_project_id, iid, sha, relationship |
MR head and merge-commit lineage |
pipeline_timing.gitlab_pipelines |
gitlab_project_id, pipeline_id |
Pipeline status, source, ref, SHA, and execution timestamps |
pipeline_timing.gitlab_jobs |
gitlab_project_id, job_id |
Job attempt status, stage, timing, failure reason, and retry information |
pipeline_timing.pipeline_runs |
cluster, uid |
Konflux PipelineRun state, source identity, timing, and Snapshot lineage |
pipeline_timing.task_runs |
cluster, uid |
Tekton task state, timing, and parent PipelineRun UID |
pipeline_timing.snapshots |
cluster, uid |
Build PipelineRun to Snapshot lineage |
pipeline_timing.releases |
cluster, uid |
Snapshot to Release completion and PipelineRun lineage |
Store both GitLab project ID and a canonical HTTPS repository URL where available. GitLab project ID is the stable GitLab identity. The repository URL is needed because Konflux resources contain source URLs rather than GitLab project IDs.
004_drop_event_catalogue removes the temporary recovery catalogue and pointers
after recovery completion. latest_observed_at and source_updated_at remain on
all reporting tables for ordering, independently of notification-history storage.
The S3 archiver remains an independent subscriber; execution fields require no
archive lookup during normal ingestion or reporting.
Schema Management
Use Alembic (a database schema migration tool already used by
Hummingbird’s workqueue service) to manage only the pipeline_timing schema. Existing tables in the
public schema remain under the Hummingbird Status db.create_schema() path; adopting Alembic for
those tables is separate work.
- Add Alembic configuration and revisions to the
pipeline-timingproject and image. - Add
pipeline_timing.worker.init_dbfor the worker Deployment’s init container. It executesCREATE SCHEMA IF NOT EXISTS pipeline_timingdirectly and commits before invoking Alembic. Do not rely on Alembic to create the schema. Alembic manages every object inside the schema after that bootstrap. - Set Alembic’s
version_table_schema="pipeline_timing"and qualify every migration object with the schema. The version table and all managed objects must remain outsidepublic. - Create the tables, constraints, and required indexes in the initial revision.
- Run
alembic upgrade headunconditionally frompipeline_timing.worker.init_dbin the single-replica Deployment’s init container. UseRecreateso the old worker stops before a new image migrates the schema. Alembic is a no-op when the schema is already at head. - On migration failure, log the exception and expected revision at ERROR level and exit nonzero. Kubernetes must not start the worker container after a failed init container; the separate Hummingbird Status Deployment continues unchanged.
- Have the timing worker independently verify that the database is at the expected Alembic revision before polling its queue. If the revision check fails, log the expected and observed revision at ERROR level and stop the timing worker. The legacy worker continues unchanged.
- Deploy one replica with the timing image and init container once the queue and archiver are ready. To pause polling, scale the Deployment to zero and monitor queue age against the source queue’s 4-day retention. SQS preserves a standard message’s enqueue timestamp when automatically moving it to the DLQ, whose 14-day retention is a maximum total age. Operator redrive from the DLQ resets the enqueue timestamp.
- Do not automatically downgrade or drop the schema during application rollback. Stop the timeline worker and use a forward migration to correct schema problems.
Selected Fields
The detailed schema should include at least the following reporting fields. These are proposed database column names, not source-payload field names.
| Record type | Required columns |
|---|---|
| Merge request | gitlab_project_id, repository_url, iid, state, source_branch, target_branch, head_sha, merge_commit_sha, created_at, updated_at, merged_at, closed_at, web_url |
| Commit | gitlab_project_id, repository_url, sha, title, message, committed_at, web_url |
| MR commit link | gitlab_project_id, repository_url, iid, sha, relationship |
| GitLab pipeline | gitlab_project_id, repository_url, pipeline_id, mr_iid, sha, ref, source, status, created_at, updated_at, started_at, completed_at, web_url |
| GitLab job | gitlab_project_id, repository_url, job_id, pipeline_id, sha, name, stage, status, failure_reason, allow_failure, created_at, started_at, completed_at, duration_seconds, queued_duration_seconds, retry_count, web_url |
| PipelineRun | cluster, namespace, uid, name, application, component, pipeline_type, pipeline_name, trigger_type, repository_url, sha, mr_iid, snapshot, status, reason, created_at, started_at, completed_at |
| TaskRun | cluster, namespace, uid, name, pipeline_run_uid, pipeline_run_name, pipeline_task, task_reference, status, reason, pod_name, created_at, started_at, completed_at |
| Snapshot | cluster, namespace, uid, name, application, component, sha, source_pipeline_run_name, created_at |
| Release | cluster, namespace, uid, name, snapshot, status, pipeline_run_namespace, pipeline_run_name, completed_at |
TaskRun and PipelineRun creation time is metadata.creationTimestamp; start and
completion are the resource’s own status timestamps. Missing values remain null.
Unknown conditions with a pending reason report Pending, not Running.
Cancellation remains distinguishable through the failure reason. GitLab execution
timestamps and status remain source-derived. Non-execution entities retain their
meaningful lifecycle milestones rather than fabricated start/completion fields.
GitLab job retries have distinct job IDs and therefore distinct rows. Store retry metadata when GitLab supplies it, but do not invent a parent job identity. Consumers can group attempts by project, pipeline, stage, and job name.
PipelineRun retries also have distinct rows because each PipelineRun has a distinct Kubernetes UID. Keep the timing and result for every attempt. This plan does not define a retry-group column or a single combined retry duration. Consumers that need an end-to-end duration across retries must keep the duration of each attempt visible and calculate the full elapsed time from the first attempt’s start to the final attempt’s completion.
GitLab Job Hook payloads do not provide a dedicated job URL. Derive web_url without an API request
by appending /-/jobs/{build_id} to project.web_url.
merged_at and closed_at are new nullable TIMESTAMPTZ columns on
pipeline_timing.merge_requests; they are not additions to the existing
public.gitlab_merge_requests table. Use explicit GitLab merge and close timestamps when the
webhook supplies them. Otherwise, derive them from the first event that reports the corresponding
action or state, preferring the source updated_at timestamp and falling back to observed_at.
Preserve the first derived terminal timestamp across later sparse events. This makes MR
creation-to-merge timing available without requiring a GitLab API lookup.
Repository URL Normalization
Normalize repository identity to an HTTPS URL without a trailing slash, .git suffix, or GitLab
resource path. Prefer GitLab’s project.web_url when it is available. Accept HTTPS, SSH, SCP-style
Git, and GitLab tree URLs from Konflux metadata.
These source values:
https://gitlab.com/redhat/hummingbird/rpms.gitgit@gitlab.com:redhat/hummingbird/rpms.githttps://gitlab.com/redhat/hummingbird/rpms/-/tree/main
all normalize to:
https://gitlab.com/redhat/hummingbird/rpms
Use repository_url consistently in the new schema. Do not use a bare project path or
source-specific Git URL as the cross-system identity.
Identity and Update Ordering
- Use
(cluster, uid)as the primary execution identity for Kubernetes resources.- Names are not sufficient because Kubernetes can recreate a resource with the same name.
- Store namespace and name for display and fallback joins.
- Use source IDs for GitLab records and retain
repository_urlfor cross-system joins. - Store the latest accepted SNS timestamp as
latest_observed_aton reporting rows. - Use source update timestamps from GitLab and Kubernetes condition transition times.
- Prevent an older event delivered later by SQS from replacing newer state.
- Compare source timestamps only when both records have them; use SNS publication timestamps as the fallback and tie-breaker, including migration of legacy rows.
- Late non-terminal snapshots must not regress completed Kubernetes executions.
- Preserve source fields absent from sparse updates and keep known creation time immutable. Ordering metadata advances independently of notification history.
- Do not require foreign keys for cross-source lineage that may arrive out of order. Use indexed identity columns and late joins. There are no legacy event foreign keys or pointer columns after catalogue removal.
Required Indexes
The schema must provide indexes for the known lineage and timeline queries. Exact index names and partial predicates are implementation details, but the following column order is required as a baseline:
| Table | Columns | Supports |
|---|---|---|
merge_request_commits |
gitlab_project_id, iid |
commits for one MR |
merge_request_commits |
repository_url, sha |
MR lookup from a commit |
gitlab_pipelines |
repository_url, sha |
pipelines for a commit |
gitlab_jobs |
gitlab_project_id, pipeline_id |
jobs for a pipeline |
pipeline_runs |
repository_url, sha |
Konflux runs for a commit |
pipeline_runs |
cluster, namespace, name |
name-based fallback lineage |
pipeline_runs |
cluster, namespace, snapshot |
test PipelineRuns for a Snapshot |
task_runs |
cluster, pipeline_run_uid |
tasks for a PipelineRun |
snapshots |
cluster, namespace, source_pipeline_run_name |
Snapshot produced by a build PipelineRun |
releases |
cluster, namespace, snapshot |
Releases for a Snapshot |
Primary keys provide the identity indexes listed in the schema table and do not need duplicate
secondary indexes. Validate the final indexes with the representative acceptance query and
PostgreSQL EXPLAIN before source enablement.
The initial design required a reverse archive lookup index on s3_bucket, s3_key.
The staging storage investigation on October 22, 2026 found that this index occupied
about 8.1 GiB, roughly 40% of the event table’s total allocation including indexes,
with zero recorded scans. No application query using the reverse lookup was found.
The temporary legacy recovery used frozen latest_event_id values to read the
archived creation metadata. Runtime ingestion never needed that reverse index.
Revision 002_drop_events_s3_index removes this unused application index to reclaim
disk space and avoid maintaining it for every new event. It preserves event rows,
columns, and archive references. Scan statistics do not prove that no external
consumer needs the index; confirm that before deployment. Deploy and verify index
space recovery in staging and production before the execution-only transition.
Revision 004_drop_event_catalogue drops the remaining catalogue indexes with the
table. No execution-reporting index is removed or rebuilt.
Lineage Model
Consumers reconstruct a timeline with the following relationships:
- A merge request links to its head commits and merge commit through
merge_request_commits. - A commit links to GitLab pipelines by
repository_urland SHA. A direct pipeline MR IID is used when the webhook supplies it. - GitLab jobs link to their pipeline by project and pipeline ID.
- Konflux PipelineRuns link to a commit by
repository_urland SHA.- Extract the repository from Pipelines as Code annotations first, then the
git-urlparameter. - Extract SHA and MR IID from the current Pipelines as Code and build-service labels and annotations.
- Extract the repository from Pipelines as Code annotations first, then the
- TaskRuns link to PipelineRuns by Kubernetes owner UID. Parent name is a fallback for malformed or older resources without an owner UID.
- A Snapshot links to its source build PipelineRun by name and cluster/namespace context.
- Test PipelineRuns link to a Snapshot through the Snapshot label.
- A Release links to its Snapshot and to any referenced Release PipelineRun.
- In the current scope, the Release completion timestamp closes the timeline even when its managed PipelineRun is in an inaccessible namespace.
The primary cross-system correlation key within one source pipeline is canonical repository_url
plus commit SHA, not SHA alone. One commit can produce several component PipelineRuns, so the
resulting timeline fans out by component and execution rather than assuming one build per commit.
Event Processing and Failure Isolation
Subscribe a dedicated timing queue to the SNS topic already used by the existing Status queue. The
timing queue receives its own SQS copy of each SNS envelope. The existing Status queue and worker
remain unchanged. The pipeline-timing worker parses and upserts only pipeline_timing records.
For each timeline message:
- Start polling only after the Pod’s migration init container succeeds and the worker verifies the expected revision. Scaling the single-replica Deployment to zero leaves the queue untouched.
- Decode the SNS envelope. A valid event outside the accepted timeline event types is ignored and acknowledged.
- For an accepted notification, parse and upsert the reporting records in one transaction. Do not append a notification-history row or require a catalogue link.
- Commit the transaction, then delete the SQS message.
- On malformed data for an accepted event, roll back and leave the message on the queue. SQS retries it and moves it to the timeline dead-letter queue after the configured receive limit.
- On a database connection, schema, or migration failure, roll back, stop polling, and leave the queue intact for recovery. Do not send valid events to the dead-letter queue during an infrastructure outage.
A timeline failure must not delay, retry, or change legacy ingestion. Operators fix a malformed event or parser error, then redrive messages from the timeline dead-letter queue. They fix an infrastructure failure, then restart the worker to resume polling. Configure the source queue with 4-day retention and its DLQ with 14-day retention, a finite receive limit, and a visibility timeout at least twice the worker’s per-message processing budget. Automatic source-to-DLQ transfer preserves a standard message’s original enqueue timestamp; operator redrive from the DLQ resets it. Add queue-age and dead-letter-queue alerts, and document and test the redrive procedure.
Completed recovery and archive inspection
S3 archival remains independent and unchanged. The timing worker neither computes archive keys nor needs bucket configuration or S3 read permissions. The operator-run recovery read immutable creation timestamps from archived snapshots, updated only null creation fields, and finished before catalogue removal. It was not part of a migration, and its tooling is not retained.
latest_observed_at and resource identity can narrow an archive search while
objects are retained. Validate candidate payload identities; this is not guaranteed
unique lookup or per-field provenance because sparse updates can merge older facts.
See finding archived notifications.
No permanent per-execution S3 pointer or replacement history table is added.
Original HUM-5820 delivery and source rollout
The remaining delivery-story sections record the original rollout. Their permanent archive-reference, bucket-configuration, and RPM-only assumptions are historical and superseded by HUM-10942 and the execution-only transition above. The publishers and independent archiver retain their existing event contracts.
Tools Repository: Publisher and Archiver Changes
This section describes code, tests, and documentation in the tools repository. Environment-specific source infrastructure is owned by HUM-7535 and activated under HUM-7533.
GitLab Event Forwarder
The forwarder already publishes arbitrary GitLab webhook bodies and derives event_type from
object_kind. No event-specific forwarding implementation is required.
Add fixtures and tests for:
- Pipeline Hooks with
object_kind: pipeline - Job Hooks with
object_kind: build project_pathandevent_typeSNS message attributes- compressed payload round trips
Kubernetes Event Forwarder
- Add
TaskRunto the configured resources only after the deployed image understands its payload requirements. - Extend the resource configuration schema with optional
labelSelectorandfieldSelectorkeys. - Pass configured selectors to the existing
ResourceWatcherlabel_selectorandfield_selectorparameters. - Remove
metadata.managedFieldsfrom TaskRun messages before SNS publication because it is transport metadata and can dominate payload size. - Keep status, results, owner references, task identity, and timing data needed for extraction and future diagnosis.
- Measure compressed and base64-encoded TaskRun payloads from representative RPM build, test, and tenant Release PipelineRuns.
- Do not enable the source if representative payloads approach the SNS 256 KiB limit. Define and review any additional projection before rollout rather than silently dropping fields.
- Preserve spooled events on publish failure so a transient SNS error does not discard them.
Use a TaskRun label selector for the RPM application if production samples confirm that Tekton propagates a stable application label. If no reliable selector exists, estimate and approve the namespace-wide event rate before enabling the watch because every matching TaskRun will also be archived to S3.
S3 Archiver
The archiver is already generic, but its resource identity logic must change for the new event
types. Extend _extract_resource_name() and _build_s3_key() so GitLab pipeline and job keys
include their numeric pipeline or job ID and Kubernetes keys include logical cluster name,
namespace, and name. URL-valued cluster attributes from older forwarders retain name-only
identities. Cover those identities with the canonical contract fixture.
Continue archiving the complete event as received from SNS. The Kubernetes forwarder’s documented
removal of managedFields occurs before that boundary. Keep explicit test coverage for PipelineRun,
TaskRun, Snapshot, Release, merge request, commit/push, pipeline, and job events.
Pipeline Timing and Hummingbird Status Changes
- Add a
pipeline-timing/project, package, image, and Alembic configuration with an initialpipeline_timingrevision. Generate its independent Konflux build resources. - Add
pipeline_timing/timeline.pyfor extraction, ordering rules, and upserts. - Keep Hummingbird Status extraction, schema creation, upsert functions, dependencies, and image
responsible only for the
publicschema. - Add timing dispatch for GitLab
push,merge_request,pipeline, andbuildevents and KubernetesPipelineRun,TaskRun,Snapshot, andReleaseevents. - Mark the newly handled event types as known to the legacy Status parser so expected timing events do not generate unknown-kind warnings; do not route them to legacy tables.
- Add
pipeline_timing.worker.init_dbto the timing Deployment as an init container. Leavehummingbird_status.worker.init_dband legacyworker/sqs.pyresponsible only forpublic. - Deploy the independent timing image to poll only the timing queue and write only the
pipeline_timingschema in the shared events database. - Add archive bucket configuration to the timing worker deployment.
- Start worker polling only after its init container succeeds; retain the worker’s own startup revision check before queue access.
- Keep Alembic migration and revision-check failures isolated to the timing worker so they cannot prevent legacy worker startup.
- Provide separate
pipeline_timingdatabase counts without changing existing Status statistics queries.
Observability
Add metrics and structured logs for:
- timeline records processed by kind and result
- timeline extraction and database failures
- events missing expected lineage fields
- oversized Kubernetes payloads and SNS publish failures
- timeline queue age and dead-letter-queue depth
- timeline worker stopped because of a migration or revision-check failure
Failure logs must include the SNS message ID, source, kind, resource identity, and S3 location without logging the full raw event. Continue monitoring legacy queue age and DLQ metrics, archiver failures, PostgreSQL availability, and PVC usage.
Documentation Updates
Update the existing documentation for every component changed by implementation. Do not create a separate documentation Story. Each delivery Story owns the pages for the behavior it changes:
| Story | Required documentation updates |
|---|---|
| HUM-7529 | New pipeline-timing.md, legacy isolation in hummingbird-status.md, and the timing schema and lineage sections of data-flow.md |
| HUM-7530 | gitlab-event-forwarder.md, kubernetes-event-forwarder.md, sns-s3-archiver.md, and event-flow portions of data-flow.md |
| HUM-7531 | staging worker deployment, monitoring, and recovery details in pipeline-timing.md |
| HUM-7532 | managed Release namespace access in kubernetes-event-forwarder.md and data-flow.md |
| HUM-7533 | final documentation audit across all affected pages before HUM-5820 closes |
| HUM-7534 | production worker deployment, monitoring, and recovery details in pipeline-timing.md |
| HUM-7535 | both environments’ timeline queues, dead-letter queues, subscriptions, IAM, and alerts in hummingbird-events-topic.md and infrastructure monitoring documentation |
HUM-8015 already updated message-bus.md for the independent staging message bus. HUM-7535
clarifies that timing queue subscriptions can precede worker deployment.
Testing
Unit Tests
Cover:
- every accepted GitLab and Kubernetes event type
- status and timestamp extraction
- explicit and derived MR merge and close timestamps
- repository URL normalization across HTTPS, SSH, SCP-style Git, and tree URLs
- GitLab job URL derivation from
project.web_urlandbuild_id - MR head and merge-commit lineage
- GitLab job retries as distinct job IDs
- a failed PipelineRun followed by a successful retry, with separate statuses and timings for both attempts
- PipelineRun-to-TaskRun owner UID lineage
- Snapshot and Release bridge extraction
- sparse and out-of-order updates
- deterministic archive keys
- missing optional lineage without rejecting otherwise valid records
- TaskRun serialization without
managedFields - init-container migration gating and zero-replica pause behavior
- Alembic upgrade failure that blocks worker startup, and startup revision failure that stops only the timing worker
- timeline acknowledgement only after a committed timeline transaction
- timeline retry and dead-letter-queue handling for parsing, data, and database failures
- an archive key from a timeline queue copy that matches the archiver’s key for the same SNS event
- ignored acknowledgement of valid events outside the accepted timeline event types
- stopped polling on database, schema, or migration failure
PostgreSQL Integration Tests
Use the PostgreSQL service already available in repository CI to:
- Apply the initial Alembic revision to an empty database and to a database with populated legacy tables.
- Verify Alembic reports the expected head revision and that its version table exists in
pipeline_timing, notpublic. - Confirm the migration does not alter existing
publictable definitions or representative legacy queries. - Ingest representative events in and out of order.
- Verify idempotent current-state upserts.
- Verify that a timeline failure leaves its message available for retry without affecting legacy ingestion.
- Query one RPM timeline from MR creation through Release completion and calculate each available interval without raw JSON.
- Verify all domain rows resolve to event records with the expected S3 references.
- Inspect the new schema to confirm it has no raw JSON columns.
- Verify that timeline redrive produces idempotent rows and retains the same S3 reference.
End-to-End Acceptance Test
After source enablement, use a controlled RPM merge request that runs GitLab CI, Konflux build and test PipelineRuns, and a Release. Record the MR and component selected for the test.
The acceptance query must demonstrate:
- MR creation and merge timestamps
- GitLab pipeline and job timing
- derived GitLab job URLs that match
project.web_urlandbuild_id - pre-merge and post-merge commit lineage
- component build PipelineRun and TaskRun timing
- a failed PipelineRun and successful retry shown as separate attempts, without losing the failed attempt’s timing
- Snapshot and test PipelineRun lineage
- Release completion
- calculable durations between phases
- retrievable S3 objects for every row used by the timeline
- paused timeline processing, recovery, and dead-letter-queue redrive without affecting legacy Status
Run make test, make coverage, make check, and SAM template validation for the affected projects.
Infrastructure Implementation
HUM-8015 delivered the independent staging message bus: its own SNS topic, GitLab and Kubernetes publishers, S3 archiver and bucket, and consumer cutover. HUM-7535 adds only the timeline queue, dead-letter queue, subscription, IAM, and alerts in staging and production.
Staging Event Path
Use the staging message bus delivered by HUM-8015 before enabling the new sources in production:
- Verify that the HUM-8015 staging topic, publishers, archiver, bucket, and existing Status
consumer are deployed and that
message-bus.mddescribes them. - Under HUM-7535, subscribe dedicated staging and production timeline queues to their respective
topics with
RawMessageDelivery: false. Configure each source queue with a finite receive limit, a visibility timeout at least twice the worker’s per-message processing budget, its own dead-letter queue, and least-privilege worker credentials. Production queue provisioning does not depend on staging worker acceptance. - Configure 4-day retention for both source queues and 14-day retention for their dead-letter queues, then add queue-age and dead-letter-queue alerts. Messages collected before worker deployment may expire after four days; this pre-launch loss is accepted. Without receives, messages do not automatically move to the DLQ. Verify redrive before source enablement.
- Under HUM-7531, deploy one staging
pipeline-timingreplica withRecreatestrategy, its migration init container, timing queue, resource limits, and explicit stagingS3_BUCKET; do not use the production bucket. Verify the init container reached the expected revision and the worker is polling before enabling new event sources. Once the worker runs, monitor queue age to prevent loss during pauses. - Keep new production event sources disabled until staging acceptance succeeds.
The staging Kubernetes event forwarder may observe production Konflux resources, but it publishes only to the staging topic. Staging and production forwarders must not publish the same watch stream to the same topic.
Production Event Path
- Keep Pipeline Hooks already enabled by the shared Hummingbird webhook configuration.
- Add RPM Job Hooks only after the production Status and archiver images understand Job Hook events.
- Add PipelineRun and TaskRun watches only after compatible forwarder and Status images are deployed.
- Use the production queue, subscription, dead-letter queue, IAM, retention, and alerts provisioned
under HUM-7535. After staging worker acceptance, deploy one
pipeline-timingreplica withRecreatestrategy, a migration init container, resource limits, and an explicit productionS3_BUCKET. Pre-launch queue messages may have expired; no historical backfill is required. Verify the revision and worker before activating new event sources, and monitor queue age after launch. - Verify the production redrive procedure before enabling new sources.
- Enable new production event sources only through HUM-7533 after staging acceptance; the timing worker starts under HUM-7534.
Managed Release Pipeline Access
HUM-7532 owns the access needed for rhtap-releng-tenant and is required for Epic completion.
- Obtain read-only
get,list, andwatchpermissions for PipelineRuns and TaskRuns in the managed namespace. - Add managed-namespace contexts to the staging and production Kubernetes event forwarders.
- Verify each event-forwarder service account with
auth can-ibefore enabling its watches. - Link the managed PipelineRun to the Release record through its namespaced reference.
- Include signing and Pulp publication TaskRuns in the final acceptance timeline.
Infrastructure Verification Gates
- Render and resolve each final RPM webhook configuration before deployment.
- The production endpoint must retain its token and existing push, merge request, note, confidential note, and pipeline events while adding job events.
- If same-URL inheritance does not deep-merge those settings, use an explicit webhook block containing the complete configuration.
- Deploy compatible Hummingbird Status, Pipeline Timing, event-forwarder, and S3 archiver images before enabling Job Hook or TaskRun events in that environment.
- Pass the canonical archive-key contract tests in both the archiver and Pipeline Timing projects.
- Verify representative TaskRun payload size after removing
managedFields. - Use an RPM application label selector if production samples confirm Tekton propagates a stable label. Otherwise, document approval of the namespace-wide event rate before enabling the watch.
- Use a single replica and
Recreatestrategy; verify the migration init container succeeded and the worker checked the expected Alembic revision before it polls. - Verify the source queue has 4-day retention and the dead-letter queue has 14-day retention, with alerts and a tested redrive procedure.
Rollout
- Complete the independent Pipeline Timing image and the legacy Status parser compatibility change in HUM-7529, and event-bus support in HUM-7530.
- Verify the HUM-8015 staging message bus and provision both environments’ timing queues, subscriptions, IAM, and alerts in HUM-7535, independent of staging worker acceptance. Pre-launch messages may expire without reaching the DLQ; this loss is accepted.
- Under HUM-7531, deploy the single-replica staging timing worker with
Recreatestrategy and its migration init container. Verify the new schema and legacy isolation before enabling new sources. - Complete
rhtap-releng-tenantprovisioning in HUM-7532 and add it to the staging event forwarder. - Under HUM-7533, activate staging Job Hook, PipelineRun, and TaskRun sources after validating the running timing worker.
- Run a controlled RPM merge request and verify staging lineage, duration calculations, managed Release tasks, and S3 references.
- Under HUM-7534, deploy the single-replica production timing worker with
Recreatestrategy and its migration init container after staging worker validation. - Verify the production Alembic revision, legacy ingestion, timeline queue alerts, and metrics.
- Add the managed namespace to the production event forwarder, then activate new production sources under HUM-7533 after verifying the timing worker is running.
- Expect each Kubernetes watcher to emit an initial
ADDEDevent for every existing matching resource. Schedule source activation during a monitored window. - Monitor legacy and timeline queue age, dead-letter-queue depth, worker and archiver errors, forwarder spool growth, database size, and PVC free space.
- Run the final controlled RPM acceptance merge request in production before closing HUM-5820.
Rollback does not require dropping or downgrading the schema. Scale the timing worker to zero and disable the new TaskRun and Job Hook sources in the affected environment. Legacy ingestion continues. After launch, monitor queue age while paused: messages expire after four days unless the worker resumes in time. Malformed messages move to the DLQ after failed receives and can be redriven after repair.
Epic Completion Criteria
The approved design’s acceptance criteria remain authoritative. HUM-5820 is complete when:
- HUM-7529 through HUM-7535 are complete.
- The design’s acceptance criteria pass in both staging and production.
- Representative timelines include managed PipelineRun and TaskRun timing from
rhtap-releng-tenant. - The staging-first source rollout and final production acceptance run are documented under HUM-7533.
- HUM-7533 has verified that each Story’s required documentation update is complete.
Out of Scope
- Creating a UI for visualization is out of scope.
- The data may later support a new application, Dashboard changes, or Grafana. UX should be involved in that decision.
- Tracking the containers repository pipeline is out of scope.
- The pipeline-neutral schema, event contracts, and execution identities are intended to support its GitLab and Konflux events in a future follow-up without creating a parallel storage model.
- The RPM and containers pipelines do not connect directly. A separate scheduled process
discovers updated RPMs and commits changes to
rpms.lock.yamlfiles in the containers repository.
- Timeline replay and historical backfill are out of scope.
- Existing Hummingbird Status S3 ingestion bootstraps the
publicschema and collapses Kubernetes history, so it is not suitable for timeline recovery. - Future work must define scoped S3 read permissions and chronological ingestion into the
Alembic-managed
pipeline_timingschema. Events that were never collected cannot be reconstructed.
- Existing Hummingbird Status S3 ingestion bootstraps the
Delivery Stories
| Story | Responsibility |
|---|---|
| HUM-7529 Make RPM pipeline timing data queryable | Independent Pipeline Timing image and schema, Alembic, worker, ingestion, lineage, tests, and legacy Status compatibility |
| HUM-7530 Carry RPM pipeline timing events through the event bus | GitLab and Kubernetes forwarders, S3 archiver event identities, payload handling, and tests |
| HUM-7531 Validate RPM pipeline timeline storage in staging | Staging Pipeline Timing deployment, Alembic verification, legacy isolation, and representative queries |
| HUM-7534 Make RPM pipeline timeline storage available in production | Production Pipeline Timing deployment with event sources still disabled |
| HUM-7535 Support environment-specific RPM pipeline event collection | Both environments’ timeline queues, dead-letter queues, subscriptions, IAM, alerts, and configuration before worker rollout |
| HUM-7532 Expose managed Release pipeline timing | Read-only rhtap-releng-tenant access and managed PipelineRun and TaskRun event collection |
| HUM-7533 Roll out RPM pipeline event collection | Staging source activation, production source activation, monitoring, recovery, redrive, and end-to-end validation |
Dependency Order
- HUM-7529 and HUM-7530 can proceed in parallel.
- HUM-8015 is the completed staging message-bus prerequisite. HUM-7535 depends on HUM-7530’s event contracts and provisions timeline-specific infrastructure in both environments without waiting for staging worker acceptance.
- HUM-7531 depends on HUM-7529, HUM-7530, and the staging path from HUM-7535.
- HUM-7534 depends on successful staging validation in HUM-7531.
- HUM-7532 depends on the event support from HUM-7530 and the environment-specific infrastructure from HUM-7535. Managed namespace access can proceed alongside the Pipeline Timing deployments, but completion gates final acceptance under HUM-7533.
- HUM-7533 depends on HUM-7531, HUM-7534, HUM-7535, and HUM-7532. HUM-5820 remains open until the final rollout is verified.