Pipeline timing implementation plan

Purpose

The original RPM pipeline timing design records the HUM-5820 design and remains unchanged. This implementation plan records the HUM-10942 revisions to retain execution timing and lineage for both RPM and container pipelines, not a permanent notification catalogue or permanent S3 references on reporting rows. Its execution-only requirements supersede the original design’s archive-reference and RPM-only scope decisions.

The Status events database remains the storage location, but pipeline-timing is a separate project, image, and worker with its own deployment lifecycle and resource quotas. Hummingbird Status continues legacy ingestion; the GitLab and Kubernetes event forwarders remain independent publishers. This service split follows HUM-7529 MR 1261 review without changing the event contracts or lineage model.

Prior Exploration

Closed tools MR 1165 and closed infrastructure MR 1381 informed this plan. They are not implementation vehicles for the Stories above. New focused merge requests should reference the Story that owns their work.

Current execution-only transition

The original HUM-5820 delivery stories are complete. HUM-10942 owns the revised execution-only requirements and supersedes the old archive-reference and RPM-only scope decisions. Existing Status tables, publishers, the archiver, and queue contracts remain unchanged.

  1. Release one removes the unused composite S3 lookup index.
  2. Release two, revision 003_execution_records, adds nullable creation times to TaskRuns and PipelineRuns and makes the reporting tables’ latest_event_id nullable. The worker stops inserting events and never changes old pointers. New rows use null pointers; the old foreign keys remain valid temporarily.
  3. Operator-run local recovery filled historical creation timestamps through those frozen pointers without replaying state. Both environments were verified with zero missing TaskRun/PipelineRun creation times or creation-time ordering anomalies. The one-time tooling was intentionally removed rather than retained in the repo.
  4. Release three, revision 004_drop_event_catalogue, explicitly removes the legacy foreign keys and latest_event_id columns, then drops events and its remaining indexes. All reporting values and ordering metadata are preserved.

Catalogue removal is transactional, uses five-second lock waits, and has no supported downgrade. Confirm recovery/backup readiness and no remaining external consumer, validate staging ingestion and physical space recovery, then use the existing image promotion path for production. Do not recreate empty history as a rollback. The events database and legacy public schema are not removed.

The worker does not require S3_BUCKET or calculate archive keys. Remove unused infrastructure bucket configuration only after the compatible app is deployed in both environments. Do not change the worker’s IAM permissions for local recovery.

Before each promotion, verify migration revision, execution timing/lineage, commit-before-acknowledgement, queue catch-up, and legacy isolation. During MR validation, prevent ordinary infrastructure pipelines from restoring the older shared image pin. Merge can lead to automatic Konflux publication, Renovate image pin updates, and deployment to both environments; it is not a recovery hold.

Test fresh initialization and populated-schema upgrades, preserving all reporting values and relation files while removing only the catalogue and legacy references. Keep revision-001/002/003 regression tests in isolated disposable databases that stop before irreversible removal. Verify a forced cleanup failure rolls back all constraint/column drops, and an attempted downgrade cannot fabricate history. Continue testing lifecycle times, states, lineage, redelivery, and ingestion with the catalogue absent. No change to the worker’s ingestion logic is required.

Implementation Plan

Scope Decisions

  1. The timeline tracks through completion of the Konflux Release resource.
    • The Release completion timestamp is the final tracked milestone.
    • PipelineRun and TaskRun detail in rhtap-releng-tenant requires separate provisioning and is required before HUM-5820 is complete.
    • PipelineRuns and TaskRuns in hummingbird-tenant remain in scope.
  2. Snapshot and Release events are required as lineage bridges even though they were omitted from the initial agreed-event list.
    • A Snapshot links a build PipelineRun to test PipelineRuns.
    • A Release links a Snapshot to Release pipeline completion and, where visible, its PipelineRun.
    • The new schema stores only their reporting and lineage fields, not raw JSON.
  3. The new model stores the latest known state of each execution and its relevant timestamps. It does not duplicate every status transition as a domain row.
    • A retry is a new PipelineRun and has its own UID, row, status, start time, and completion time. It does not replace the failed PipelineRun.
    • For example, a PipelineRun that fails after four hours and a retry that succeeds in ten minutes remain two separate records. A report must not show only the ten-minute retry as the time for both attempts.
  4. Both RPM and container pipeline executions are in scope. Retain the existing source mix rather than introducing repository filters during this transition.
  5. Full event replay and per-step timing are out of scope. Historical recovery is limited to best-effort creation timestamps and runs outside startup migrations.
  6. Each environment needs a dedicated timing SQS queue, dead-letter queue, and pipeline-timing worker deployment using its own image. The existing Status queue, image, and worker continue legacy ingestion unchanged.

PostgreSQL Schema

Create a pipeline_timing schema in the existing Status events database. Do not add columns, indexes, constraints, or triggers to existing tables in the public schema. The Dashboard continues to query only the existing tables.

Domain tables contain selected scalar fields, source timestamps, lineage, and ordering metadata, not a raw payload, history array, or catalogue foreign key.

Table Identity columns Purpose
pipeline_timing.merge_requests gitlab_project_id, iid Current MR state and lifecycle timestamps
pipeline_timing.commits gitlab_project_id, sha Commit identity and available commit metadata
pipeline_timing.merge_request_commits gitlab_project_id, iid, sha, relationship MR head and merge-commit lineage
pipeline_timing.gitlab_pipelines gitlab_project_id, pipeline_id Pipeline status, source, ref, SHA, and execution timestamps
pipeline_timing.gitlab_jobs gitlab_project_id, job_id Job attempt status, stage, timing, failure reason, and retry information
pipeline_timing.pipeline_runs cluster, uid Konflux PipelineRun state, source identity, timing, and Snapshot lineage
pipeline_timing.task_runs cluster, uid Tekton task state, timing, and parent PipelineRun UID
pipeline_timing.snapshots cluster, uid Build PipelineRun to Snapshot lineage
pipeline_timing.releases cluster, uid Snapshot to Release completion and PipelineRun lineage

Store both GitLab project ID and a canonical HTTPS repository URL where available. GitLab project ID is the stable GitLab identity. The repository URL is needed because Konflux resources contain source URLs rather than GitLab project IDs.

004_drop_event_catalogue removes the temporary recovery catalogue and pointers after recovery completion. latest_observed_at and source_updated_at remain on all reporting tables for ordering, independently of notification-history storage. The S3 archiver remains an independent subscriber; execution fields require no archive lookup during normal ingestion or reporting.

Schema Management

Use Alembic (a database schema migration tool already used by Hummingbird’s workqueue service) to manage only the pipeline_timing schema. Existing tables in the public schema remain under the Hummingbird Status db.create_schema() path; adopting Alembic for those tables is separate work.

  1. Add Alembic configuration and revisions to the pipeline-timing project and image.
  2. Add pipeline_timing.worker.init_db for the worker Deployment’s init container. It executes CREATE SCHEMA IF NOT EXISTS pipeline_timing directly and commits before invoking Alembic. Do not rely on Alembic to create the schema. Alembic manages every object inside the schema after that bootstrap.
  3. Set Alembic’s version_table_schema="pipeline_timing" and qualify every migration object with the schema. The version table and all managed objects must remain outside public.
  4. Create the tables, constraints, and required indexes in the initial revision.
  5. Run alembic upgrade head unconditionally from pipeline_timing.worker.init_db in the single-replica Deployment’s init container. Use Recreate so the old worker stops before a new image migrates the schema. Alembic is a no-op when the schema is already at head.
  6. On migration failure, log the exception and expected revision at ERROR level and exit nonzero. Kubernetes must not start the worker container after a failed init container; the separate Hummingbird Status Deployment continues unchanged.
  7. Have the timing worker independently verify that the database is at the expected Alembic revision before polling its queue. If the revision check fails, log the expected and observed revision at ERROR level and stop the timing worker. The legacy worker continues unchanged.
  8. Deploy one replica with the timing image and init container once the queue and archiver are ready. To pause polling, scale the Deployment to zero and monitor queue age against the source queue’s 4-day retention. SQS preserves a standard message’s enqueue timestamp when automatically moving it to the DLQ, whose 14-day retention is a maximum total age. Operator redrive from the DLQ resets the enqueue timestamp.
  9. Do not automatically downgrade or drop the schema during application rollback. Stop the timeline worker and use a forward migration to correct schema problems.

Selected Fields

The detailed schema should include at least the following reporting fields. These are proposed database column names, not source-payload field names.

Record type Required columns
Merge request gitlab_project_id, repository_url, iid, state, source_branch, target_branch, head_sha, merge_commit_sha, created_at, updated_at, merged_at, closed_at, web_url
Commit gitlab_project_id, repository_url, sha, title, message, committed_at, web_url
MR commit link gitlab_project_id, repository_url, iid, sha, relationship
GitLab pipeline gitlab_project_id, repository_url, pipeline_id, mr_iid, sha, ref, source, status, created_at, updated_at, started_at, completed_at, web_url
GitLab job gitlab_project_id, repository_url, job_id, pipeline_id, sha, name, stage, status, failure_reason, allow_failure, created_at, started_at, completed_at, duration_seconds, queued_duration_seconds, retry_count, web_url
PipelineRun cluster, namespace, uid, name, application, component, pipeline_type, pipeline_name, trigger_type, repository_url, sha, mr_iid, snapshot, status, reason, created_at, started_at, completed_at
TaskRun cluster, namespace, uid, name, pipeline_run_uid, pipeline_run_name, pipeline_task, task_reference, status, reason, pod_name, created_at, started_at, completed_at
Snapshot cluster, namespace, uid, name, application, component, sha, source_pipeline_run_name, created_at
Release cluster, namespace, uid, name, snapshot, status, pipeline_run_namespace, pipeline_run_name, completed_at

TaskRun and PipelineRun creation time is metadata.creationTimestamp; start and completion are the resource’s own status timestamps. Missing values remain null. Unknown conditions with a pending reason report Pending, not Running. Cancellation remains distinguishable through the failure reason. GitLab execution timestamps and status remain source-derived. Non-execution entities retain their meaningful lifecycle milestones rather than fabricated start/completion fields.

GitLab job retries have distinct job IDs and therefore distinct rows. Store retry metadata when GitLab supplies it, but do not invent a parent job identity. Consumers can group attempts by project, pipeline, stage, and job name.

PipelineRun retries also have distinct rows because each PipelineRun has a distinct Kubernetes UID. Keep the timing and result for every attempt. This plan does not define a retry-group column or a single combined retry duration. Consumers that need an end-to-end duration across retries must keep the duration of each attempt visible and calculate the full elapsed time from the first attempt’s start to the final attempt’s completion.

GitLab Job Hook payloads do not provide a dedicated job URL. Derive web_url without an API request by appending /-/jobs/{build_id} to project.web_url.

merged_at and closed_at are new nullable TIMESTAMPTZ columns on pipeline_timing.merge_requests; they are not additions to the existing public.gitlab_merge_requests table. Use explicit GitLab merge and close timestamps when the webhook supplies them. Otherwise, derive them from the first event that reports the corresponding action or state, preferring the source updated_at timestamp and falling back to observed_at. Preserve the first derived terminal timestamp across later sparse events. This makes MR creation-to-merge timing available without requiring a GitLab API lookup.

Repository URL Normalization

Normalize repository identity to an HTTPS URL without a trailing slash, .git suffix, or GitLab resource path. Prefer GitLab’s project.web_url when it is available. Accept HTTPS, SSH, SCP-style Git, and GitLab tree URLs from Konflux metadata.

These source values:

  • https://gitlab.com/redhat/hummingbird/rpms.git
  • git@gitlab.com:redhat/hummingbird/rpms.git
  • https://gitlab.com/redhat/hummingbird/rpms/-/tree/main

all normalize to:

https://gitlab.com/redhat/hummingbird/rpms

Use repository_url consistently in the new schema. Do not use a bare project path or source-specific Git URL as the cross-system identity.

Identity and Update Ordering

  1. Use (cluster, uid) as the primary execution identity for Kubernetes resources.
    • Names are not sufficient because Kubernetes can recreate a resource with the same name.
    • Store namespace and name for display and fallback joins.
  2. Use source IDs for GitLab records and retain repository_url for cross-system joins.
  3. Store the latest accepted SNS timestamp as latest_observed_at on reporting rows.
  4. Use source update timestamps from GitLab and Kubernetes condition transition times.
  5. Prevent an older event delivered later by SQS from replacing newer state.
    • Compare source timestamps only when both records have them; use SNS publication timestamps as the fallback and tie-breaker, including migration of legacy rows.
    • Late non-terminal snapshots must not regress completed Kubernetes executions.
  6. Preserve source fields absent from sparse updates and keep known creation time immutable. Ordering metadata advances independently of notification history.
  7. Do not require foreign keys for cross-source lineage that may arrive out of order. Use indexed identity columns and late joins. There are no legacy event foreign keys or pointer columns after catalogue removal.

Required Indexes

The schema must provide indexes for the known lineage and timeline queries. Exact index names and partial predicates are implementation details, but the following column order is required as a baseline:

Table Columns Supports
merge_request_commits gitlab_project_id, iid commits for one MR
merge_request_commits repository_url, sha MR lookup from a commit
gitlab_pipelines repository_url, sha pipelines for a commit
gitlab_jobs gitlab_project_id, pipeline_id jobs for a pipeline
pipeline_runs repository_url, sha Konflux runs for a commit
pipeline_runs cluster, namespace, name name-based fallback lineage
pipeline_runs cluster, namespace, snapshot test PipelineRuns for a Snapshot
task_runs cluster, pipeline_run_uid tasks for a PipelineRun
snapshots cluster, namespace, source_pipeline_run_name Snapshot produced by a build PipelineRun
releases cluster, namespace, snapshot Releases for a Snapshot

Primary keys provide the identity indexes listed in the schema table and do not need duplicate secondary indexes. Validate the final indexes with the representative acceptance query and PostgreSQL EXPLAIN before source enablement.

The initial design required a reverse archive lookup index on s3_bucket, s3_key. The staging storage investigation on October 22, 2026 found that this index occupied about 8.1 GiB, roughly 40% of the event table’s total allocation including indexes, with zero recorded scans. No application query using the reverse lookup was found. The temporary legacy recovery used frozen latest_event_id values to read the archived creation metadata. Runtime ingestion never needed that reverse index.

Revision 002_drop_events_s3_index removes this unused application index to reclaim disk space and avoid maintaining it for every new event. It preserves event rows, columns, and archive references. Scan statistics do not prove that no external consumer needs the index; confirm that before deployment. Deploy and verify index space recovery in staging and production before the execution-only transition. Revision 004_drop_event_catalogue drops the remaining catalogue indexes with the table. No execution-reporting index is removed or rebuilt.

Lineage Model

Consumers reconstruct a timeline with the following relationships:

  1. A merge request links to its head commits and merge commit through merge_request_commits.
  2. A commit links to GitLab pipelines by repository_url and SHA. A direct pipeline MR IID is used when the webhook supplies it.
  3. GitLab jobs link to their pipeline by project and pipeline ID.
  4. Konflux PipelineRuns link to a commit by repository_url and SHA.
    • Extract the repository from Pipelines as Code annotations first, then the git-url parameter.
    • Extract SHA and MR IID from the current Pipelines as Code and build-service labels and annotations.
  5. TaskRuns link to PipelineRuns by Kubernetes owner UID. Parent name is a fallback for malformed or older resources without an owner UID.
  6. A Snapshot links to its source build PipelineRun by name and cluster/namespace context.
  7. Test PipelineRuns link to a Snapshot through the Snapshot label.
  8. A Release links to its Snapshot and to any referenced Release PipelineRun.
  9. In the current scope, the Release completion timestamp closes the timeline even when its managed PipelineRun is in an inaccessible namespace.

The primary cross-system correlation key within one source pipeline is canonical repository_url plus commit SHA, not SHA alone. One commit can produce several component PipelineRuns, so the resulting timeline fans out by component and execution rather than assuming one build per commit.

Event Processing and Failure Isolation

Subscribe a dedicated timing queue to the SNS topic already used by the existing Status queue. The timing queue receives its own SQS copy of each SNS envelope. The existing Status queue and worker remain unchanged. The pipeline-timing worker parses and upserts only pipeline_timing records.

For each timeline message:

  1. Start polling only after the Pod’s migration init container succeeds and the worker verifies the expected revision. Scaling the single-replica Deployment to zero leaves the queue untouched.
  2. Decode the SNS envelope. A valid event outside the accepted timeline event types is ignored and acknowledged.
  3. For an accepted notification, parse and upsert the reporting records in one transaction. Do not append a notification-history row or require a catalogue link.
  4. Commit the transaction, then delete the SQS message.
  5. On malformed data for an accepted event, roll back and leave the message on the queue. SQS retries it and moves it to the timeline dead-letter queue after the configured receive limit.
  6. On a database connection, schema, or migration failure, roll back, stop polling, and leave the queue intact for recovery. Do not send valid events to the dead-letter queue during an infrastructure outage.

A timeline failure must not delay, retry, or change legacy ingestion. Operators fix a malformed event or parser error, then redrive messages from the timeline dead-letter queue. They fix an infrastructure failure, then restart the worker to resume polling. Configure the source queue with 4-day retention and its DLQ with 14-day retention, a finite receive limit, and a visibility timeout at least twice the worker’s per-message processing budget. Automatic source-to-DLQ transfer preserves a standard message’s original enqueue timestamp; operator redrive from the DLQ resets it. Add queue-age and dead-letter-queue alerts, and document and test the redrive procedure.

Completed recovery and archive inspection

S3 archival remains independent and unchanged. The timing worker neither computes archive keys nor needs bucket configuration or S3 read permissions. The operator-run recovery read immutable creation timestamps from archived snapshots, updated only null creation fields, and finished before catalogue removal. It was not part of a migration, and its tooling is not retained.

latest_observed_at and resource identity can narrow an archive search while objects are retained. Validate candidate payload identities; this is not guaranteed unique lookup or per-field provenance because sparse updates can merge older facts. See finding archived notifications. No permanent per-execution S3 pointer or replacement history table is added.

Original HUM-5820 delivery and source rollout

The remaining delivery-story sections record the original rollout. Their permanent archive-reference, bucket-configuration, and RPM-only assumptions are historical and superseded by HUM-10942 and the execution-only transition above. The publishers and independent archiver retain their existing event contracts.

Tools Repository: Publisher and Archiver Changes

This section describes code, tests, and documentation in the tools repository. Environment-specific source infrastructure is owned by HUM-7535 and activated under HUM-7533.

GitLab Event Forwarder

The forwarder already publishes arbitrary GitLab webhook bodies and derives event_type from object_kind. No event-specific forwarding implementation is required.

Add fixtures and tests for:

  • Pipeline Hooks with object_kind: pipeline
  • Job Hooks with object_kind: build
  • project_path and event_type SNS message attributes
  • compressed payload round trips

Kubernetes Event Forwarder

  1. Add TaskRun to the configured resources only after the deployed image understands its payload requirements.
  2. Extend the resource configuration schema with optional labelSelector and fieldSelector keys.
  3. Pass configured selectors to the existing ResourceWatcher label_selector and field_selector parameters.
  4. Remove metadata.managedFields from TaskRun messages before SNS publication because it is transport metadata and can dominate payload size.
  5. Keep status, results, owner references, task identity, and timing data needed for extraction and future diagnosis.
  6. Measure compressed and base64-encoded TaskRun payloads from representative RPM build, test, and tenant Release PipelineRuns.
  7. Do not enable the source if representative payloads approach the SNS 256 KiB limit. Define and review any additional projection before rollout rather than silently dropping fields.
  8. Preserve spooled events on publish failure so a transient SNS error does not discard them.

Use a TaskRun label selector for the RPM application if production samples confirm that Tekton propagates a stable application label. If no reliable selector exists, estimate and approve the namespace-wide event rate before enabling the watch because every matching TaskRun will also be archived to S3.

S3 Archiver

The archiver is already generic, but its resource identity logic must change for the new event types. Extend _extract_resource_name() and _build_s3_key() so GitLab pipeline and job keys include their numeric pipeline or job ID and Kubernetes keys include logical cluster name, namespace, and name. URL-valued cluster attributes from older forwarders retain name-only identities. Cover those identities with the canonical contract fixture.

Continue archiving the complete event as received from SNS. The Kubernetes forwarder’s documented removal of managedFields occurs before that boundary. Keep explicit test coverage for PipelineRun, TaskRun, Snapshot, Release, merge request, commit/push, pipeline, and job events.

Pipeline Timing and Hummingbird Status Changes

  1. Add a pipeline-timing/ project, package, image, and Alembic configuration with an initial pipeline_timing revision. Generate its independent Konflux build resources.
  2. Add pipeline_timing/timeline.py for extraction, ordering rules, and upserts.
  3. Keep Hummingbird Status extraction, schema creation, upsert functions, dependencies, and image responsible only for the public schema.
  4. Add timing dispatch for GitLab push, merge_request, pipeline, and build events and Kubernetes PipelineRun, TaskRun, Snapshot, and Release events.
  5. Mark the newly handled event types as known to the legacy Status parser so expected timing events do not generate unknown-kind warnings; do not route them to legacy tables.
  6. Add pipeline_timing.worker.init_db to the timing Deployment as an init container. Leave hummingbird_status.worker.init_db and legacy worker/sqs.py responsible only for public.
  7. Deploy the independent timing image to poll only the timing queue and write only the pipeline_timing schema in the shared events database.
  8. Add archive bucket configuration to the timing worker deployment.
  9. Start worker polling only after its init container succeeds; retain the worker’s own startup revision check before queue access.
  10. Keep Alembic migration and revision-check failures isolated to the timing worker so they cannot prevent legacy worker startup.
  11. Provide separate pipeline_timing database counts without changing existing Status statistics queries.

Observability

Add metrics and structured logs for:

  • timeline records processed by kind and result
  • timeline extraction and database failures
  • events missing expected lineage fields
  • oversized Kubernetes payloads and SNS publish failures
  • timeline queue age and dead-letter-queue depth
  • timeline worker stopped because of a migration or revision-check failure

Failure logs must include the SNS message ID, source, kind, resource identity, and S3 location without logging the full raw event. Continue monitoring legacy queue age and DLQ metrics, archiver failures, PostgreSQL availability, and PVC usage.

Documentation Updates

Update the existing documentation for every component changed by implementation. Do not create a separate documentation Story. Each delivery Story owns the pages for the behavior it changes:

Story Required documentation updates
HUM-7529 New pipeline-timing.md, legacy isolation in hummingbird-status.md, and the timing schema and lineage sections of data-flow.md
HUM-7530 gitlab-event-forwarder.md, kubernetes-event-forwarder.md, sns-s3-archiver.md, and event-flow portions of data-flow.md
HUM-7531 staging worker deployment, monitoring, and recovery details in pipeline-timing.md
HUM-7532 managed Release namespace access in kubernetes-event-forwarder.md and data-flow.md
HUM-7533 final documentation audit across all affected pages before HUM-5820 closes
HUM-7534 production worker deployment, monitoring, and recovery details in pipeline-timing.md
HUM-7535 both environments’ timeline queues, dead-letter queues, subscriptions, IAM, and alerts in hummingbird-events-topic.md and infrastructure monitoring documentation

HUM-8015 already updated message-bus.md for the independent staging message bus. HUM-7535 clarifies that timing queue subscriptions can precede worker deployment.

Testing

Unit Tests

Cover:

  • every accepted GitLab and Kubernetes event type
  • status and timestamp extraction
  • explicit and derived MR merge and close timestamps
  • repository URL normalization across HTTPS, SSH, SCP-style Git, and tree URLs
  • GitLab job URL derivation from project.web_url and build_id
  • MR head and merge-commit lineage
  • GitLab job retries as distinct job IDs
  • a failed PipelineRun followed by a successful retry, with separate statuses and timings for both attempts
  • PipelineRun-to-TaskRun owner UID lineage
  • Snapshot and Release bridge extraction
  • sparse and out-of-order updates
  • deterministic archive keys
  • missing optional lineage without rejecting otherwise valid records
  • TaskRun serialization without managedFields
  • init-container migration gating and zero-replica pause behavior
  • Alembic upgrade failure that blocks worker startup, and startup revision failure that stops only the timing worker
  • timeline acknowledgement only after a committed timeline transaction
  • timeline retry and dead-letter-queue handling for parsing, data, and database failures
  • an archive key from a timeline queue copy that matches the archiver’s key for the same SNS event
  • ignored acknowledgement of valid events outside the accepted timeline event types
  • stopped polling on database, schema, or migration failure

PostgreSQL Integration Tests

Use the PostgreSQL service already available in repository CI to:

  1. Apply the initial Alembic revision to an empty database and to a database with populated legacy tables.
  2. Verify Alembic reports the expected head revision and that its version table exists in pipeline_timing, not public.
  3. Confirm the migration does not alter existing public table definitions or representative legacy queries.
  4. Ingest representative events in and out of order.
  5. Verify idempotent current-state upserts.
  6. Verify that a timeline failure leaves its message available for retry without affecting legacy ingestion.
  7. Query one RPM timeline from MR creation through Release completion and calculate each available interval without raw JSON.
  8. Verify all domain rows resolve to event records with the expected S3 references.
  9. Inspect the new schema to confirm it has no raw JSON columns.
  10. Verify that timeline redrive produces idempotent rows and retains the same S3 reference.

End-to-End Acceptance Test

After source enablement, use a controlled RPM merge request that runs GitLab CI, Konflux build and test PipelineRuns, and a Release. Record the MR and component selected for the test.

The acceptance query must demonstrate:

  • MR creation and merge timestamps
  • GitLab pipeline and job timing
  • derived GitLab job URLs that match project.web_url and build_id
  • pre-merge and post-merge commit lineage
  • component build PipelineRun and TaskRun timing
  • a failed PipelineRun and successful retry shown as separate attempts, without losing the failed attempt’s timing
  • Snapshot and test PipelineRun lineage
  • Release completion
  • calculable durations between phases
  • retrievable S3 objects for every row used by the timeline
  • paused timeline processing, recovery, and dead-letter-queue redrive without affecting legacy Status

Run make test, make coverage, make check, and SAM template validation for the affected projects.

Infrastructure Implementation

HUM-8015 delivered the independent staging message bus: its own SNS topic, GitLab and Kubernetes publishers, S3 archiver and bucket, and consumer cutover. HUM-7535 adds only the timeline queue, dead-letter queue, subscription, IAM, and alerts in staging and production.

Staging Event Path

Use the staging message bus delivered by HUM-8015 before enabling the new sources in production:

  1. Verify that the HUM-8015 staging topic, publishers, archiver, bucket, and existing Status consumer are deployed and that message-bus.md describes them.
  2. Under HUM-7535, subscribe dedicated staging and production timeline queues to their respective topics with RawMessageDelivery: false. Configure each source queue with a finite receive limit, a visibility timeout at least twice the worker’s per-message processing budget, its own dead-letter queue, and least-privilege worker credentials. Production queue provisioning does not depend on staging worker acceptance.
  3. Configure 4-day retention for both source queues and 14-day retention for their dead-letter queues, then add queue-age and dead-letter-queue alerts. Messages collected before worker deployment may expire after four days; this pre-launch loss is accepted. Without receives, messages do not automatically move to the DLQ. Verify redrive before source enablement.
  4. Under HUM-7531, deploy one staging pipeline-timing replica with Recreate strategy, its migration init container, timing queue, resource limits, and explicit staging S3_BUCKET; do not use the production bucket. Verify the init container reached the expected revision and the worker is polling before enabling new event sources. Once the worker runs, monitor queue age to prevent loss during pauses.
  5. Keep new production event sources disabled until staging acceptance succeeds.

The staging Kubernetes event forwarder may observe production Konflux resources, but it publishes only to the staging topic. Staging and production forwarders must not publish the same watch stream to the same topic.

Production Event Path

  1. Keep Pipeline Hooks already enabled by the shared Hummingbird webhook configuration.
  2. Add RPM Job Hooks only after the production Status and archiver images understand Job Hook events.
  3. Add PipelineRun and TaskRun watches only after compatible forwarder and Status images are deployed.
  4. Use the production queue, subscription, dead-letter queue, IAM, retention, and alerts provisioned under HUM-7535. After staging worker acceptance, deploy one pipeline-timing replica with Recreate strategy, a migration init container, resource limits, and an explicit production S3_BUCKET. Pre-launch queue messages may have expired; no historical backfill is required. Verify the revision and worker before activating new event sources, and monitor queue age after launch.
  5. Verify the production redrive procedure before enabling new sources.
  6. Enable new production event sources only through HUM-7533 after staging acceptance; the timing worker starts under HUM-7534.

Managed Release Pipeline Access

HUM-7532 owns the access needed for rhtap-releng-tenant and is required for Epic completion.

  1. Obtain read-only get, list, and watch permissions for PipelineRuns and TaskRuns in the managed namespace.
  2. Add managed-namespace contexts to the staging and production Kubernetes event forwarders.
  3. Verify each event-forwarder service account with auth can-i before enabling its watches.
  4. Link the managed PipelineRun to the Release record through its namespaced reference.
  5. Include signing and Pulp publication TaskRuns in the final acceptance timeline.

Infrastructure Verification Gates

  1. Render and resolve each final RPM webhook configuration before deployment.
    • The production endpoint must retain its token and existing push, merge request, note, confidential note, and pipeline events while adding job events.
    • If same-URL inheritance does not deep-merge those settings, use an explicit webhook block containing the complete configuration.
  2. Deploy compatible Hummingbird Status, Pipeline Timing, event-forwarder, and S3 archiver images before enabling Job Hook or TaskRun events in that environment.
  3. Pass the canonical archive-key contract tests in both the archiver and Pipeline Timing projects.
  4. Verify representative TaskRun payload size after removing managedFields.
  5. Use an RPM application label selector if production samples confirm Tekton propagates a stable label. Otherwise, document approval of the namespace-wide event rate before enabling the watch.
  6. Use a single replica and Recreate strategy; verify the migration init container succeeded and the worker checked the expected Alembic revision before it polls.
  7. Verify the source queue has 4-day retention and the dead-letter queue has 14-day retention, with alerts and a tested redrive procedure.

Rollout

  1. Complete the independent Pipeline Timing image and the legacy Status parser compatibility change in HUM-7529, and event-bus support in HUM-7530.
  2. Verify the HUM-8015 staging message bus and provision both environments’ timing queues, subscriptions, IAM, and alerts in HUM-7535, independent of staging worker acceptance. Pre-launch messages may expire without reaching the DLQ; this loss is accepted.
  3. Under HUM-7531, deploy the single-replica staging timing worker with Recreate strategy and its migration init container. Verify the new schema and legacy isolation before enabling new sources.
  4. Complete rhtap-releng-tenant provisioning in HUM-7532 and add it to the staging event forwarder.
  5. Under HUM-7533, activate staging Job Hook, PipelineRun, and TaskRun sources after validating the running timing worker.
  6. Run a controlled RPM merge request and verify staging lineage, duration calculations, managed Release tasks, and S3 references.
  7. Under HUM-7534, deploy the single-replica production timing worker with Recreate strategy and its migration init container after staging worker validation.
  8. Verify the production Alembic revision, legacy ingestion, timeline queue alerts, and metrics.
  9. Add the managed namespace to the production event forwarder, then activate new production sources under HUM-7533 after verifying the timing worker is running.
  10. Expect each Kubernetes watcher to emit an initial ADDED event for every existing matching resource. Schedule source activation during a monitored window.
  11. Monitor legacy and timeline queue age, dead-letter-queue depth, worker and archiver errors, forwarder spool growth, database size, and PVC free space.
  12. Run the final controlled RPM acceptance merge request in production before closing HUM-5820.

Rollback does not require dropping or downgrading the schema. Scale the timing worker to zero and disable the new TaskRun and Job Hook sources in the affected environment. Legacy ingestion continues. After launch, monitor queue age while paused: messages expire after four days unless the worker resumes in time. Malformed messages move to the DLQ after failed receives and can be redriven after repair.

Epic Completion Criteria

The approved design’s acceptance criteria remain authoritative. HUM-5820 is complete when:

  1. HUM-7529 through HUM-7535 are complete.
  2. The design’s acceptance criteria pass in both staging and production.
  3. Representative timelines include managed PipelineRun and TaskRun timing from rhtap-releng-tenant.
  4. The staging-first source rollout and final production acceptance run are documented under HUM-7533.
  5. HUM-7533 has verified that each Story’s required documentation update is complete.

Out of Scope

  1. Creating a UI for visualization is out of scope.
    • The data may later support a new application, Dashboard changes, or Grafana. UX should be involved in that decision.
  2. Tracking the containers repository pipeline is out of scope.
    • The pipeline-neutral schema, event contracts, and execution identities are intended to support its GitLab and Konflux events in a future follow-up without creating a parallel storage model.
    • The RPM and containers pipelines do not connect directly. A separate scheduled process discovers updated RPMs and commits changes to rpms.lock.yaml files in the containers repository.
  3. Timeline replay and historical backfill are out of scope.
    • Existing Hummingbird Status S3 ingestion bootstraps the public schema and collapses Kubernetes history, so it is not suitable for timeline recovery.
    • Future work must define scoped S3 read permissions and chronological ingestion into the Alembic-managed pipeline_timing schema. Events that were never collected cannot be reconstructed.

Delivery Stories

Story Responsibility
HUM-7529 Make RPM pipeline timing data queryable Independent Pipeline Timing image and schema, Alembic, worker, ingestion, lineage, tests, and legacy Status compatibility
HUM-7530 Carry RPM pipeline timing events through the event bus GitLab and Kubernetes forwarders, S3 archiver event identities, payload handling, and tests
HUM-7531 Validate RPM pipeline timeline storage in staging Staging Pipeline Timing deployment, Alembic verification, legacy isolation, and representative queries
HUM-7534 Make RPM pipeline timeline storage available in production Production Pipeline Timing deployment with event sources still disabled
HUM-7535 Support environment-specific RPM pipeline event collection Both environments’ timeline queues, dead-letter queues, subscriptions, IAM, alerts, and configuration before worker rollout
HUM-7532 Expose managed Release pipeline timing Read-only rhtap-releng-tenant access and managed PipelineRun and TaskRun event collection
HUM-7533 Roll out RPM pipeline event collection Staging source activation, production source activation, monitoring, recovery, redrive, and end-to-end validation

Dependency Order

  1. HUM-7529 and HUM-7530 can proceed in parallel.
  2. HUM-8015 is the completed staging message-bus prerequisite. HUM-7535 depends on HUM-7530’s event contracts and provisions timeline-specific infrastructure in both environments without waiting for staging worker acceptance.
  3. HUM-7531 depends on HUM-7529, HUM-7530, and the staging path from HUM-7535.
  4. HUM-7534 depends on successful staging validation in HUM-7531.
  5. HUM-7532 depends on the event support from HUM-7530 and the environment-specific infrastructure from HUM-7535. Managed namespace access can proceed alongside the Pipeline Timing deployments, but completion gates final acceptance under HUM-7533.
  6. HUM-7533 depends on HUM-7531, HUM-7534, HUM-7535, and HUM-7532. HUM-5820 remains open until the final rollout is verified.