This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Pipeline Timing

Independent SQS worker for RPM and container execution timing and cross-system lineage. It shares the Status events database but owns only the pipeline_timing schema.

Features

  • Current state, timestamps, and separate attempts for GitLab jobs and Konflux PipelineRuns
  • Commit, merge request, PipelineRun, TaskRun, Snapshot, and Release lineage
  • Source-derived execution creation, start, and completion times
  • Alembic-managed schema, dedicated queue, and worker metrics

Prerequisites

  • Python 3.11+ and PostgreSQL 16
  • A dedicated timing SQS queue subscribed to the environment’s SNS topic with RawMessageDelivery: false

The existing Status worker and event forwarders keep their own images and queues. Infrastructure provisions the timeline queues, IAM, and alerts separately from the Status worker. Each environment has a separate queue named arr-hummingbird-<env>-pipeline-timing-sqs and a matching -sqs-dlq queue. The source queue retains messages for 4 days; its DLQ retains them for 14 days from the original enqueue time when SQS automatically moves a message to the DLQ. Operator redrive from the DLQ resets the enqueue timestamp. The source queue moves a message to its DLQ after 3 receives and uses a 300-second visibility timeout, matching Status. The SNS subscription preserves the SNS envelope (RawMessageDelivery: false) and does not filter events. See Hummingbird Events Topic for queue policy and redrive details. Provision both environments’ queues and SNS subscriptions before deploying the workers. Messages collected before a worker starts may expire after four days; this pre-launch loss is accepted. A queue without a consumer does not move messages to its DLQ because no receives occur. Validate the staging worker before deploying the production worker or enabling new production event sources.

Installation

cd pipeline-timing
pip install -e ".[dev]"

The pipeline-timing Containerfile builds the separate quay.io/hummingbird-ci/pipeline-timing image. Both the initializer and worker run from this image.

Usage

Schema initialization

Run the initializer as the init container of the single-replica timing worker Deployment. Use a Recreate strategy so Kubernetes stops the old worker before starting a Pod whose init container may upgrade the schema:

DATABASE_URL="$DATABASE_URL" python3 -m pipeline_timing.worker.init_db

The initializer creates the pipeline_timing schema and runs alembic upgrade head each time a Pod starts. This is a no-op at the expected revision; a new image can apply a forward migration during rollout. If migration fails, the initializer exits nonzero and Kubernetes does not start the worker. The worker checks the revision again before polling. Fix migration errors with a forward revision; do not downgrade or drop the schema during rollback.

Staging deployment from a merge request

Wait for the tools MR’s tests and Konflux image build to pass, then run the manual deploy_pipeline_timing_staging job. The job does not wait for those checks automatically. It triggers the infrastructure pipeline with ONLY_JOB_NAME=pipeline-timing and ONLY_DEPLOYMENT_ENVIRONMENT=staging, using the MR commit’s image for both the migration init container and worker. Production is not deployed by this job.

Verify the init container completed, the worker accepts the new Alembic revision, and its queue is catching up before promoting the published image to production. Normal infrastructure pipelines use a shared image pin for both environments. Coordinate those deployments during MR validation so an older pinned image does not replace the MR image. The worker requires an exact Alembic revision match; an older image will refuse a newer revision even when the table layout is unchanged.

Execution-only schema and catalogue removal

Under HUM-10942, revision 003_execution_records added nullable created_at to TaskRuns and PipelineRuns, stopped catalogue writes, and preserved old recovery pointers temporarily. Historical creation-timestamp recovery completed in staging and production using a one-time local tool. The tool was intentionally not retained.

Revision 004_drop_event_catalogue removes the legacy latest_event_id foreign keys and columns from the reporting tables, then drops pipeline_timing.events and its remaining indexes. It does not change reporting rows, their lifecycle fields, parentage, or ordering metadata. The database named events and legacy Status tables in public remain unchanged. Original migrations remain intact so fresh initialization follows the same revision chain.

Before deployment, confirm recovery is complete or its gaps accepted in both environments, no external consumer needs the catalogue, and the normal backup is available. Record the catalogue allocation and PVC free space, then verify the space recovery and continued ingestion in staging before production promotion. The table/index allocation is released when the migration commits without a bulk delete, table rewrite, or VACUUM FULL. Dropped pointer columns may retain small amounts of physical row space; that does not justify a rewrite.

The migration limits lock waits to five seconds. A conflicting reader or unexpected dependency makes initialization fail and the transactional DDL roll back. Resolve contention or the dependency and retry with the compatible image. Downgrade is explicitly unsupported: recreating empty history would not restore the deleted catalogue or pointers. Use a compatible forward fix or the backup recovery procedure.

Kubernetes creation time comes from metadata.creationTimestamp; start and completion come from status.startTime and status.completionTime. Missing source timestamps remain null, not inferred from SNS publication time. Pending conditions are distinct from running, and failure reasons retain cancellation details. GitLab jobs and pipelines retain their source lifecycle fields and status.

Only execution state and lineage are maintained. There is no event-history array, per-step timing, new archive-location table, or permanent S3 pointer on each row. Resource deletion preserves the execution’s result, timings, and parentage.

Timing worker

Deploy one worker replica with the environment’s dedicated queue. Kubernetes starts it only after the init container succeeds:

SQS_QUEUE_URL="$TIMING_QUEUE_URL" python3 -m pipeline_timing.worker

The worker polls when it starts. Accepted notifications update reporting rows in one committed transaction before SQS acknowledgement, without event-table writes. Malformed events remain for retry and eventual dead-letter-queue redrive; database or revision failures stop polling without affecting legacy Status. Scale the timing worker to zero replicas to pause ingestion. After launch, monitor queue age against the four-day source retention limit during a pause to avoid losing events. SQS preserves the original enqueue time for automatic DLQ transfer, and an operator redrive resets it. A new rollout uses Recreate: the old worker stops, Alembic upgrades, and the new worker starts. The S3 archiver receives its own SNS copy and remains independent of timing ingestion.

The worker follows Status’s polling pattern: a 20-second long poll receives up to 10 messages, processed sequentially with boto3’s default request retries. Each accepted event commits in its own transaction. At the end of each receive, one SQS batch-delete request acknowledges committed events and ignored messages. Short receives flush immediately; failed processing never adds a receipt to the delete request. Receive errors are logged and polling continues. Partial delete failures and uncertain request outcomes leave committed events available for idempotent redelivery, and polling continues. If a database failure interrupts a receive, the worker first attempts to acknowledge earlier confirmed commits. Database connection or schema failures stop the worker for Kubernetes to restart it. SQS visibility controls when an unacknowledged message becomes available again.

The worker caches up to 128 UPSERT statement shapes by table and ordered columns. Each execution binds fresh event values; the cache does not retain payloads, connections, or cursors. Source update timestamps order messages when both the incoming and existing row have them; SNS publication timestamps break ties or provide the fallback. Kubernetes condition transition times supply source ordering. Late pending/running snapshots cannot regress a completed Kubernetes execution. Sparse payloads preserve existing values, and creation time stays immutable once known.

Queue age and DLQ depth are monitored independently from the Status queue. Fix malformed messages before using the documented redrive procedure. If the worker is paused or not yet deployed, remove only a known test message after confirming that redrive returned it to the timing queue.

Configuration

Variable Default Description
DATABASE_URL - Existing Status events database URL
SQS_QUEUE_URL - Environment-specific timing queue URL
SQS_WAIT_TIME 20 SQS long-poll interval in seconds
METRICS_PORT 9090 Prometheus port when the worker is enabled

Infrastructure supplies queue values for each environment:

Environment Queue URL variable
Staging ARR_HUMMINGBIRD_STAGING_PIPELINE_TIMING_SQS_QUEUE_URL
Production ARR_HUMMINGBIRD_PROD_PIPELINE_TIMING_SQS_QUEUE_URL

S3_BUCKET is no longer required or used by the worker. Remove the old deployment variable only after the compatible worker is deployed in both environments. The independent S3 archiver and its configuration remain unchanged.

POSTGRES_URL is also accepted when DATABASE_URL is not set. For local development only, the database URL defaults to postgresql://postgres:dev@localhost:5432/events.

Schema and lineage

Table Identity Purpose
merge_requests GitLab project ID and MR IID Current state and lifecycle times
commits GitLab project ID and SHA Commit metadata
merge_request_commits Project ID, IID, SHA, relation Head and merge-commit links
gitlab_pipelines Project ID and pipeline ID Pipeline status and timing
gitlab_jobs Project ID and job ID Individual job attempts
pipeline_runs Cluster and Kubernetes UID Build, test, and Release attempts
task_runs Cluster and Kubernetes UID Task timing and parent UID
snapshots Cluster and Kubernetes UID Build-to-test bridge
releases Cluster and Kubernetes UID Snapshot-to-Release bridge

Source timestamps accept ISO 8601 and GitLab’s YYYY-MM-DD HH:MM:SS UTC format and are normalized to UTC. Timestamps without a timezone are interpreted as UTC.

latest_observed_at and source_updated_at order updates without a notification catalogue or event foreign keys. The schema stores selected reporting fields, not raw event JSON. Join MR head and merge commits to GitLab pipelines by canonical HTTPS repository URL and SHA; join jobs by project and pipeline ID. Konflux PipelineRuns use the same repository URL and SHA, then fan out by component and execution UID. TaskRuns link through PipelineRun owner UIDs; Snapshot source-run names, test-run Snapshot labels, and Release references connect the remaining phases. The Release completion time closes the current timeline. Distinct job IDs and PipelineRun UIDs retain each retry’s status and timing.

PipelineRun SHAs come from supported labels and annotations, including pac.test.appstudio.openshift.io/sha, before falling back to the revision parameter.

Promotion Snapshots populate source_pipeline_run_name from hummingbird-project.io/promotion-pipelinerun. This annotation takes precedence over the usual pac.test.appstudio.openshift.io/log-url, so the parent is the promotion run rather than the original MR build. Other Snapshots keep the log-URL behavior. This connects the promotion run to its Snapshot and Releases without changing the schema. References to the original MR build and source Snapshot in promotion annotations are not stored separately.

Existing promotion Snapshots need a separate backfill if no newer notification arrives after this extraction change. Deploying the worker does not rescan stored Snapshots or automatically repair historical links.

Finding archived notifications

The independent S3 archiver still receives its own SNS copy. The timing database does not store a direct archive pointer or a complete notification history.

For Kubernetes executions, use the environment’s archive bucket and latest_observed_at as a UTC SNS timestamp with milliseconds to narrow the object prefix, for example sns/2026/01/02/03/04/2026-01-02T03:04:05.678Z#. Match source, kind, cluster, namespace, and resource name in candidate filenames, then verify the UID in the downloaded payload. For GitLab, verify the project and execution/commit identity in candidate payloads.

This is candidate discovery, not a guaranteed unique or per-field provenance link. Several notifications may share a timestamp, and sparse updates can retain facts from earlier payloads. Lookup depends on archive delivery, naming, and retention; it does not recreate the removed database catalogue.

Metrics include hummingbird_pipeline_timing_records_total{kind,result}, hummingbird_pipeline_timing_failures_total{reason}, hummingbird_pipeline_timing_missing_lineage_total{kind}, and hummingbird_pipeline_timing_worker_stopped. success and ignored record results count only confirmed SQS acknowledgements, including duplicates. Each failed or uncertain batch entry increments the sqs failure counter. These counters do not measure unique event rows. Monitor queue age and dead-letter-queue depth separately from the legacy Status queue. Fix malformed events before redriving them. Identity-based upserts and ordering guards keep redelivery idempotent; there is no database row for every SNS message.

Development

Run tests only against the disposable PostgreSQL service provided in CI or a disposable local database. Tests truncate timing data and exercise schema upgrades and historical downgrades; never use a staging or production database URL. Migration tests create uniquely named disposable databases and need CREATEDB on that test server. Older-revision tests stop at revision 003; they do not downgrade through irreversible catalogue removal:

DATABASE_URL=postgresql://postgres:dev@localhost:5432/events \
  python3 -m unittest discover -s tests -t .

Tests also load shared publisher payload fixtures from sns-s3-archiver/tests/fixtures/archive-key-contract.json; timing ingestion no longer calculates or persists archive keys. CI provides DATABASE_URL for the PostgreSQL integration tests. The timing image has a Kubernetes import smoke test in tests-k8s.yml.

License

This project is licensed under the GNU General Public License v3.0 or later. See the LICENSE.

1 - RPM pipeline timing design

Problem

We lack a clear way to see where time is spent in the RPM build pipeline.

Background

References: HUM-5819 - Create a diagram that describes all phases and steps of various pipelines from start to finish. Then review with team. HUM-5820 - Time breakdown from MR creation to completion of Release pipeline.

During the F44 rebuild, RPM builds had problems whose causes were not immediately clear. The sprint retrospective and a later F44-specific retrospective created HUM-5819 and HUM-5820 to improve visibility into pipeline behavior and time spent in each phase.

An initial pass through HUM-5819 identified key areas for time tracking. HUM-5820 records those areas.

Findings

The team initially considered a small addition to the tools/hummingbird-status database. A draft MR explored that approach.

Findings from the draft MR:

  1. The existing Status events database is a suitable home because the system forwards GitLab and Konflux events through SNS, archives them in S3, and updates PostgreSQL. The timing worker runs as an independent service with its own image and deployment lifecycle.
  2. Some hummingbird-dashboard functions do directly query the raw JSON.
  3. The original purpose of hummingbird-status was to diagnose Konflux event issues, not long-term system diagnosis.
  4. The raw event JSON for Konflux PipelineRuns consumes significant space, about 97% of the 47 GB database, despite redundant event storage in S3.
  5. hummingbird-status PVC monitoring is now in place; the draft MR highlighted its absence.
  6. The solution to our problem statement is not to mimic the existing approach.

Proposal

Track required event data in new tables in the existing database and update event processing to populate them without changing existing tables or Dashboard behavior.

Events Already Agreed Upon for Tracking

HUM-5820 outlines the events already agreed upon for tracking.

  1. GitLab MRs and commits
  2. GitLab pipelines
  3. GitLab pipeline jobs
  4. Konflux PipelineRuns
  5. Konflux Tekton TaskRuns

The model must support reporting of:

  1. Status
  2. Relevant start and end times, including MR creation and merge times
  3. Event timestamps
  4. Event lineage across GitLab and Konflux, sufficient to reconstruct a single pipeline timeline

This is not a database design. It identifies the reporting requirements that detailed design must meet.

Approach

  1. Preserve existing event ingestion, database tables, and Dashboard behavior. Run timing ingestion in a separate pipeline-timing service and image while retaining the existing publishers, archiver, and events database.
  2. Create a new schema in the existing PostgreSQL database.
    • The schema provides logical isolation so existing consumers do not accidentally use the new tables.
  3. Create tables that hold the data needed to meet the agreed reporting requirements without duplicating raw event JSON.
  4. Treat the S3 archive as the authoritative raw-event record for the new schema.
    • Store a reference that lets consumers retrieve the archived event for detail, display, or later data-model improvements.
  5. Configure GitLab webhooks and the Kubernetes event forwarder to include the agreed event types, then route them through the existing SNS event pipeline.
  6. Ensure the S3 archiver can archive and identify every agreed event type.

Acceptance Criteria

  1. Existing Status database tables and Dashboard behavior remain unchanged.
  2. The new PostgreSQL schema provides the agreed event coverage.
  3. Representative data demonstrates an RPM pipeline timeline from MR creation through completion of the Release pipeline.
  4. Consumers can calculate time spent between tracked pipeline phases without querying raw event JSON.
  5. Consumers can trace lineage across all agreed event types and retrieve the corresponding authoritative S3 event when needed.
  6. Tests verify the new data model, lineage, and S3 references.

Out of Scope

  1. Creating a UI for visualization is out of scope.
    • The data may later support a new application, Dashboard changes, or Grafana. UX should be involved in that decision.
  2. Tracking the containers repository pipeline is out of scope.
    • That pipeline builds OCI images from RPMs produced by the RPM pipeline and is a potential follow-up effort.

2 - Pipeline timing implementation plan

Purpose

The original RPM pipeline timing design records the HUM-5820 design and remains unchanged. This implementation plan records the HUM-10942 revisions to retain execution timing and lineage for both RPM and container pipelines, not a permanent notification catalogue or permanent S3 references on reporting rows. Its execution-only requirements supersede the original design’s archive-reference and RPM-only scope decisions.

The Status events database remains the storage location, but pipeline-timing is a separate project, image, and worker with its own deployment lifecycle and resource quotas. Hummingbird Status continues legacy ingestion; the GitLab and Kubernetes event forwarders remain independent publishers. This service split follows HUM-7529 MR 1261 review without changing the event contracts or lineage model.

Prior Exploration

Closed tools MR 1165 and closed infrastructure MR 1381 informed this plan. They are not implementation vehicles for the Stories above. New focused merge requests should reference the Story that owns their work.

Current execution-only transition

The original HUM-5820 delivery stories are complete. HUM-10942 owns the revised execution-only requirements and supersedes the old archive-reference and RPM-only scope decisions. Existing Status tables, publishers, the archiver, and queue contracts remain unchanged.

  1. Release one removes the unused composite S3 lookup index.
  2. Release two, revision 003_execution_records, adds nullable creation times to TaskRuns and PipelineRuns and makes the reporting tables’ latest_event_id nullable. The worker stops inserting events and never changes old pointers. New rows use null pointers; the old foreign keys remain valid temporarily.
  3. Operator-run local recovery filled historical creation timestamps through those frozen pointers without replaying state. Both environments were verified with zero missing TaskRun/PipelineRun creation times or creation-time ordering anomalies. The one-time tooling was intentionally removed rather than retained in the repo.
  4. Release three, revision 004_drop_event_catalogue, explicitly removes the legacy foreign keys and latest_event_id columns, then drops events and its remaining indexes. All reporting values and ordering metadata are preserved.

Catalogue removal is transactional, uses five-second lock waits, and has no supported downgrade. Confirm recovery/backup readiness and no remaining external consumer, validate staging ingestion and physical space recovery, then use the existing image promotion path for production. Do not recreate empty history as a rollback. The events database and legacy public schema are not removed.

The worker does not require S3_BUCKET or calculate archive keys. Remove unused infrastructure bucket configuration only after the compatible app is deployed in both environments. Do not change the worker’s IAM permissions for local recovery.

Before each promotion, verify migration revision, execution timing/lineage, commit-before-acknowledgement, queue catch-up, and legacy isolation. During MR validation, prevent ordinary infrastructure pipelines from restoring the older shared image pin. Merge can lead to automatic Konflux publication, Renovate image pin updates, and deployment to both environments; it is not a recovery hold.

Test fresh initialization and populated-schema upgrades, preserving all reporting values and relation files while removing only the catalogue and legacy references. Keep revision-001/002/003 regression tests in isolated disposable databases that stop before irreversible removal. Verify a forced cleanup failure rolls back all constraint/column drops, and an attempted downgrade cannot fabricate history. Continue testing lifecycle times, states, lineage, redelivery, and ingestion with the catalogue absent. No change to the worker’s ingestion logic is required.

Implementation Plan

Scope Decisions

  1. The timeline tracks through completion of the Konflux Release resource.
    • The Release completion timestamp is the final tracked milestone.
    • PipelineRun and TaskRun detail in rhtap-releng-tenant requires separate provisioning and is required before HUM-5820 is complete.
    • PipelineRuns and TaskRuns in hummingbird-tenant remain in scope.
  2. Snapshot and Release events are required as lineage bridges even though they were omitted from the initial agreed-event list.
    • A Snapshot links a build PipelineRun to test PipelineRuns.
    • A Release links a Snapshot to Release pipeline completion and, where visible, its PipelineRun.
    • The new schema stores only their reporting and lineage fields, not raw JSON.
  3. The new model stores the latest known state of each execution and its relevant timestamps. It does not duplicate every status transition as a domain row.
    • A retry is a new PipelineRun and has its own UID, row, status, start time, and completion time. It does not replace the failed PipelineRun.
    • For example, a PipelineRun that fails after four hours and a retry that succeeds in ten minutes remain two separate records. A report must not show only the ten-minute retry as the time for both attempts.
  4. Both RPM and container pipeline executions are in scope. Retain the existing source mix rather than introducing repository filters during this transition.
  5. Full event replay and per-step timing are out of scope. Historical recovery is limited to best-effort creation timestamps and runs outside startup migrations.
  6. Each environment needs a dedicated timing SQS queue, dead-letter queue, and pipeline-timing worker deployment using its own image. The existing Status queue, image, and worker continue legacy ingestion unchanged.

PostgreSQL Schema

Create a pipeline_timing schema in the existing Status events database. Do not add columns, indexes, constraints, or triggers to existing tables in the public schema. The Dashboard continues to query only the existing tables.

Domain tables contain selected scalar fields, source timestamps, lineage, and ordering metadata, not a raw payload, history array, or catalogue foreign key.

Table Identity columns Purpose
pipeline_timing.merge_requests gitlab_project_id, iid Current MR state and lifecycle timestamps
pipeline_timing.commits gitlab_project_id, sha Commit identity and available commit metadata
pipeline_timing.merge_request_commits gitlab_project_id, iid, sha, relationship MR head and merge-commit lineage
pipeline_timing.gitlab_pipelines gitlab_project_id, pipeline_id Pipeline status, source, ref, SHA, and execution timestamps
pipeline_timing.gitlab_jobs gitlab_project_id, job_id Job attempt status, stage, timing, failure reason, and retry information
pipeline_timing.pipeline_runs cluster, uid Konflux PipelineRun state, source identity, timing, and Snapshot lineage
pipeline_timing.task_runs cluster, uid Tekton task state, timing, and parent PipelineRun UID
pipeline_timing.snapshots cluster, uid Build PipelineRun to Snapshot lineage
pipeline_timing.releases cluster, uid Snapshot to Release completion and PipelineRun lineage

Store both GitLab project ID and a canonical HTTPS repository URL where available. GitLab project ID is the stable GitLab identity. The repository URL is needed because Konflux resources contain source URLs rather than GitLab project IDs.

004_drop_event_catalogue removes the temporary recovery catalogue and pointers after recovery completion. latest_observed_at and source_updated_at remain on all reporting tables for ordering, independently of notification-history storage. The S3 archiver remains an independent subscriber; execution fields require no archive lookup during normal ingestion or reporting.

Schema Management

Use Alembic (a database schema migration tool already used by Hummingbird’s workqueue service) to manage only the pipeline_timing schema. Existing tables in the public schema remain under the Hummingbird Status db.create_schema() path; adopting Alembic for those tables is separate work.

  1. Add Alembic configuration and revisions to the pipeline-timing project and image.
  2. Add pipeline_timing.worker.init_db for the worker Deployment’s init container. It executes CREATE SCHEMA IF NOT EXISTS pipeline_timing directly and commits before invoking Alembic. Do not rely on Alembic to create the schema. Alembic manages every object inside the schema after that bootstrap.
  3. Set Alembic’s version_table_schema="pipeline_timing" and qualify every migration object with the schema. The version table and all managed objects must remain outside public.
  4. Create the tables, constraints, and required indexes in the initial revision.
  5. Run alembic upgrade head unconditionally from pipeline_timing.worker.init_db in the single-replica Deployment’s init container. Use Recreate so the old worker stops before a new image migrates the schema. Alembic is a no-op when the schema is already at head.
  6. On migration failure, log the exception and expected revision at ERROR level and exit nonzero. Kubernetes must not start the worker container after a failed init container; the separate Hummingbird Status Deployment continues unchanged.
  7. Have the timing worker independently verify that the database is at the expected Alembic revision before polling its queue. If the revision check fails, log the expected and observed revision at ERROR level and stop the timing worker. The legacy worker continues unchanged.
  8. Deploy one replica with the timing image and init container once the queue and archiver are ready. To pause polling, scale the Deployment to zero and monitor queue age against the source queue’s 4-day retention. SQS preserves a standard message’s enqueue timestamp when automatically moving it to the DLQ, whose 14-day retention is a maximum total age. Operator redrive from the DLQ resets the enqueue timestamp.
  9. Do not automatically downgrade or drop the schema during application rollback. Stop the timeline worker and use a forward migration to correct schema problems.

Selected Fields

The detailed schema should include at least the following reporting fields. These are proposed database column names, not source-payload field names.

Record type Required columns
Merge request gitlab_project_id, repository_url, iid, state, source_branch, target_branch, head_sha, merge_commit_sha, created_at, updated_at, merged_at, closed_at, web_url
Commit gitlab_project_id, repository_url, sha, title, message, committed_at, web_url
MR commit link gitlab_project_id, repository_url, iid, sha, relationship
GitLab pipeline gitlab_project_id, repository_url, pipeline_id, mr_iid, sha, ref, source, status, created_at, updated_at, started_at, completed_at, web_url
GitLab job gitlab_project_id, repository_url, job_id, pipeline_id, sha, name, stage, status, failure_reason, allow_failure, created_at, started_at, completed_at, duration_seconds, queued_duration_seconds, retry_count, web_url
PipelineRun cluster, namespace, uid, name, application, component, pipeline_type, pipeline_name, trigger_type, repository_url, sha, mr_iid, snapshot, status, reason, created_at, started_at, completed_at
TaskRun cluster, namespace, uid, name, pipeline_run_uid, pipeline_run_name, pipeline_task, task_reference, status, reason, pod_name, created_at, started_at, completed_at
Snapshot cluster, namespace, uid, name, application, component, sha, source_pipeline_run_name, created_at
Release cluster, namespace, uid, name, snapshot, status, pipeline_run_namespace, pipeline_run_name, completed_at

TaskRun and PipelineRun creation time is metadata.creationTimestamp; start and completion are the resource’s own status timestamps. Missing values remain null. Unknown conditions with a pending reason report Pending, not Running. Cancellation remains distinguishable through the failure reason. GitLab execution timestamps and status remain source-derived. Non-execution entities retain their meaningful lifecycle milestones rather than fabricated start/completion fields.

GitLab job retries have distinct job IDs and therefore distinct rows. Store retry metadata when GitLab supplies it, but do not invent a parent job identity. Consumers can group attempts by project, pipeline, stage, and job name.

PipelineRun retries also have distinct rows because each PipelineRun has a distinct Kubernetes UID. Keep the timing and result for every attempt. This plan does not define a retry-group column or a single combined retry duration. Consumers that need an end-to-end duration across retries must keep the duration of each attempt visible and calculate the full elapsed time from the first attempt’s start to the final attempt’s completion.

GitLab Job Hook payloads do not provide a dedicated job URL. Derive web_url without an API request by appending /-/jobs/{build_id} to project.web_url.

merged_at and closed_at are new nullable TIMESTAMPTZ columns on pipeline_timing.merge_requests; they are not additions to the existing public.gitlab_merge_requests table. Use explicit GitLab merge and close timestamps when the webhook supplies them. Otherwise, derive them from the first event that reports the corresponding action or state, preferring the source updated_at timestamp and falling back to observed_at. Preserve the first derived terminal timestamp across later sparse events. This makes MR creation-to-merge timing available without requiring a GitLab API lookup.

Repository URL Normalization

Normalize repository identity to an HTTPS URL without a trailing slash, .git suffix, or GitLab resource path. Prefer GitLab’s project.web_url when it is available. Accept HTTPS, SSH, SCP-style Git, and GitLab tree URLs from Konflux metadata.

These source values:

  • https://gitlab.com/redhat/hummingbird/rpms.git
  • git@gitlab.com:redhat/hummingbird/rpms.git
  • https://gitlab.com/redhat/hummingbird/rpms/-/tree/main

all normalize to:

https://gitlab.com/redhat/hummingbird/rpms

Use repository_url consistently in the new schema. Do not use a bare project path or source-specific Git URL as the cross-system identity.

Identity and Update Ordering

  1. Use (cluster, uid) as the primary execution identity for Kubernetes resources.
    • Names are not sufficient because Kubernetes can recreate a resource with the same name.
    • Store namespace and name for display and fallback joins.
  2. Use source IDs for GitLab records and retain repository_url for cross-system joins.
  3. Store the latest accepted SNS timestamp as latest_observed_at on reporting rows.
  4. Use source update timestamps from GitLab and Kubernetes condition transition times.
  5. Prevent an older event delivered later by SQS from replacing newer state.
    • Compare source timestamps only when both records have them; use SNS publication timestamps as the fallback and tie-breaker, including migration of legacy rows.
    • Late non-terminal snapshots must not regress completed Kubernetes executions.
  6. Preserve source fields absent from sparse updates and keep known creation time immutable. Ordering metadata advances independently of notification history.
  7. Do not require foreign keys for cross-source lineage that may arrive out of order. Use indexed identity columns and late joins. There are no legacy event foreign keys or pointer columns after catalogue removal.

Required Indexes

The schema must provide indexes for the known lineage and timeline queries. Exact index names and partial predicates are implementation details, but the following column order is required as a baseline:

Table Columns Supports
merge_request_commits gitlab_project_id, iid commits for one MR
merge_request_commits repository_url, sha MR lookup from a commit
gitlab_pipelines repository_url, sha pipelines for a commit
gitlab_jobs gitlab_project_id, pipeline_id jobs for a pipeline
pipeline_runs repository_url, sha Konflux runs for a commit
pipeline_runs cluster, namespace, name name-based fallback lineage
pipeline_runs cluster, namespace, snapshot test PipelineRuns for a Snapshot
task_runs cluster, pipeline_run_uid tasks for a PipelineRun
snapshots cluster, namespace, source_pipeline_run_name Snapshot produced by a build PipelineRun
releases cluster, namespace, snapshot Releases for a Snapshot

Primary keys provide the identity indexes listed in the schema table and do not need duplicate secondary indexes. Validate the final indexes with the representative acceptance query and PostgreSQL EXPLAIN before source enablement.

The initial design required a reverse archive lookup index on s3_bucket, s3_key. The staging storage investigation on October 22, 2026 found that this index occupied about 8.1 GiB, roughly 40% of the event table’s total allocation including indexes, with zero recorded scans. No application query using the reverse lookup was found. The temporary legacy recovery used frozen latest_event_id values to read the archived creation metadata. Runtime ingestion never needed that reverse index.

Revision 002_drop_events_s3_index removes this unused application index to reclaim disk space and avoid maintaining it for every new event. It preserves event rows, columns, and archive references. Scan statistics do not prove that no external consumer needs the index; confirm that before deployment. Deploy and verify index space recovery in staging and production before the execution-only transition. Revision 004_drop_event_catalogue drops the remaining catalogue indexes with the table. No execution-reporting index is removed or rebuilt.

Lineage Model

Consumers reconstruct a timeline with the following relationships:

  1. A merge request links to its head commits and merge commit through merge_request_commits.
  2. A commit links to GitLab pipelines by repository_url and SHA. A direct pipeline MR IID is used when the webhook supplies it.
  3. GitLab jobs link to their pipeline by project and pipeline ID.
  4. Konflux PipelineRuns link to a commit by repository_url and SHA.
    • Extract the repository from Pipelines as Code annotations first, then the git-url parameter.
    • Extract SHA and MR IID from the current Pipelines as Code and build-service labels and annotations.
  5. TaskRuns link to PipelineRuns by Kubernetes owner UID. Parent name is a fallback for malformed or older resources without an owner UID.
  6. A Snapshot links to its source build PipelineRun by name and cluster/namespace context.
  7. Test PipelineRuns link to a Snapshot through the Snapshot label.
  8. A Release links to its Snapshot and to any referenced Release PipelineRun.
  9. In the current scope, the Release completion timestamp closes the timeline even when its managed PipelineRun is in an inaccessible namespace.

The primary cross-system correlation key within one source pipeline is canonical repository_url plus commit SHA, not SHA alone. One commit can produce several component PipelineRuns, so the resulting timeline fans out by component and execution rather than assuming one build per commit.

Event Processing and Failure Isolation

Subscribe a dedicated timing queue to the SNS topic already used by the existing Status queue. The timing queue receives its own SQS copy of each SNS envelope. The existing Status queue and worker remain unchanged. The pipeline-timing worker parses and upserts only pipeline_timing records.

For each timeline message:

  1. Start polling only after the Pod’s migration init container succeeds and the worker verifies the expected revision. Scaling the single-replica Deployment to zero leaves the queue untouched.
  2. Decode the SNS envelope. A valid event outside the accepted timeline event types is ignored and acknowledged.
  3. For an accepted notification, parse and upsert the reporting records in one transaction. Do not append a notification-history row or require a catalogue link.
  4. Commit the transaction, then delete the SQS message.
  5. On malformed data for an accepted event, roll back and leave the message on the queue. SQS retries it and moves it to the timeline dead-letter queue after the configured receive limit.
  6. On a database connection, schema, or migration failure, roll back, stop polling, and leave the queue intact for recovery. Do not send valid events to the dead-letter queue during an infrastructure outage.

A timeline failure must not delay, retry, or change legacy ingestion. Operators fix a malformed event or parser error, then redrive messages from the timeline dead-letter queue. They fix an infrastructure failure, then restart the worker to resume polling. Configure the source queue with 4-day retention and its DLQ with 14-day retention, a finite receive limit, and a visibility timeout at least twice the worker’s per-message processing budget. Automatic source-to-DLQ transfer preserves a standard message’s original enqueue timestamp; operator redrive from the DLQ resets it. Add queue-age and dead-letter-queue alerts, and document and test the redrive procedure.

Completed recovery and archive inspection

S3 archival remains independent and unchanged. The timing worker neither computes archive keys nor needs bucket configuration or S3 read permissions. The operator-run recovery read immutable creation timestamps from archived snapshots, updated only null creation fields, and finished before catalogue removal. It was not part of a migration, and its tooling is not retained.

latest_observed_at and resource identity can narrow an archive search while objects are retained. Validate candidate payload identities; this is not guaranteed unique lookup or per-field provenance because sparse updates can merge older facts. See finding archived notifications. No permanent per-execution S3 pointer or replacement history table is added.

Original HUM-5820 delivery and source rollout

The remaining delivery-story sections record the original rollout. Their permanent archive-reference, bucket-configuration, and RPM-only assumptions are historical and superseded by HUM-10942 and the execution-only transition above. The publishers and independent archiver retain their existing event contracts.

Tools Repository: Publisher and Archiver Changes

This section describes code, tests, and documentation in the tools repository. Environment-specific source infrastructure is owned by HUM-7535 and activated under HUM-7533.

GitLab Event Forwarder

The forwarder already publishes arbitrary GitLab webhook bodies and derives event_type from object_kind. No event-specific forwarding implementation is required.

Add fixtures and tests for:

  • Pipeline Hooks with object_kind: pipeline
  • Job Hooks with object_kind: build
  • project_path and event_type SNS message attributes
  • compressed payload round trips

Kubernetes Event Forwarder

  1. Add TaskRun to the configured resources only after the deployed image understands its payload requirements.
  2. Extend the resource configuration schema with optional labelSelector and fieldSelector keys.
  3. Pass configured selectors to the existing ResourceWatcher label_selector and field_selector parameters.
  4. Remove metadata.managedFields from TaskRun messages before SNS publication because it is transport metadata and can dominate payload size.
  5. Keep status, results, owner references, task identity, and timing data needed for extraction and future diagnosis.
  6. Measure compressed and base64-encoded TaskRun payloads from representative RPM build, test, and tenant Release PipelineRuns.
  7. Do not enable the source if representative payloads approach the SNS 256 KiB limit. Define and review any additional projection before rollout rather than silently dropping fields.
  8. Preserve spooled events on publish failure so a transient SNS error does not discard them.

Use a TaskRun label selector for the RPM application if production samples confirm that Tekton propagates a stable application label. If no reliable selector exists, estimate and approve the namespace-wide event rate before enabling the watch because every matching TaskRun will also be archived to S3.

S3 Archiver

The archiver is already generic, but its resource identity logic must change for the new event types. Extend _extract_resource_name() and _build_s3_key() so GitLab pipeline and job keys include their numeric pipeline or job ID and Kubernetes keys include logical cluster name, namespace, and name. URL-valued cluster attributes from older forwarders retain name-only identities. Cover those identities with the canonical contract fixture.

Continue archiving the complete event as received from SNS. The Kubernetes forwarder’s documented removal of managedFields occurs before that boundary. Keep explicit test coverage for PipelineRun, TaskRun, Snapshot, Release, merge request, commit/push, pipeline, and job events.

Pipeline Timing and Hummingbird Status Changes

  1. Add a pipeline-timing/ project, package, image, and Alembic configuration with an initial pipeline_timing revision. Generate its independent Konflux build resources.
  2. Add pipeline_timing/timeline.py for extraction, ordering rules, and upserts.
  3. Keep Hummingbird Status extraction, schema creation, upsert functions, dependencies, and image responsible only for the public schema.
  4. Add timing dispatch for GitLab push, merge_request, pipeline, and build events and Kubernetes PipelineRun, TaskRun, Snapshot, and Release events.
  5. Mark the newly handled event types as known to the legacy Status parser so expected timing events do not generate unknown-kind warnings; do not route them to legacy tables.
  6. Add pipeline_timing.worker.init_db to the timing Deployment as an init container. Leave hummingbird_status.worker.init_db and legacy worker/sqs.py responsible only for public.
  7. Deploy the independent timing image to poll only the timing queue and write only the pipeline_timing schema in the shared events database.
  8. Add archive bucket configuration to the timing worker deployment.
  9. Start worker polling only after its init container succeeds; retain the worker’s own startup revision check before queue access.
  10. Keep Alembic migration and revision-check failures isolated to the timing worker so they cannot prevent legacy worker startup.
  11. Provide separate pipeline_timing database counts without changing existing Status statistics queries.

Observability

Add metrics and structured logs for:

  • timeline records processed by kind and result
  • timeline extraction and database failures
  • events missing expected lineage fields
  • oversized Kubernetes payloads and SNS publish failures
  • timeline queue age and dead-letter-queue depth
  • timeline worker stopped because of a migration or revision-check failure

Failure logs must include the SNS message ID, source, kind, resource identity, and S3 location without logging the full raw event. Continue monitoring legacy queue age and DLQ metrics, archiver failures, PostgreSQL availability, and PVC usage.

Documentation Updates

Update the existing documentation for every component changed by implementation. Do not create a separate documentation Story. Each delivery Story owns the pages for the behavior it changes:

Story Required documentation updates
HUM-7529 New pipeline-timing.md, legacy isolation in hummingbird-status.md, and the timing schema and lineage sections of data-flow.md
HUM-7530 gitlab-event-forwarder.md, kubernetes-event-forwarder.md, sns-s3-archiver.md, and event-flow portions of data-flow.md
HUM-7531 staging worker deployment, monitoring, and recovery details in pipeline-timing.md
HUM-7532 managed Release namespace access in kubernetes-event-forwarder.md and data-flow.md
HUM-7533 final documentation audit across all affected pages before HUM-5820 closes
HUM-7534 production worker deployment, monitoring, and recovery details in pipeline-timing.md
HUM-7535 both environments’ timeline queues, dead-letter queues, subscriptions, IAM, and alerts in hummingbird-events-topic.md and infrastructure monitoring documentation

HUM-8015 already updated message-bus.md for the independent staging message bus. HUM-7535 clarifies that timing queue subscriptions can precede worker deployment.

Testing

Unit Tests

Cover:

  • every accepted GitLab and Kubernetes event type
  • status and timestamp extraction
  • explicit and derived MR merge and close timestamps
  • repository URL normalization across HTTPS, SSH, SCP-style Git, and tree URLs
  • GitLab job URL derivation from project.web_url and build_id
  • MR head and merge-commit lineage
  • GitLab job retries as distinct job IDs
  • a failed PipelineRun followed by a successful retry, with separate statuses and timings for both attempts
  • PipelineRun-to-TaskRun owner UID lineage
  • Snapshot and Release bridge extraction
  • sparse and out-of-order updates
  • deterministic archive keys
  • missing optional lineage without rejecting otherwise valid records
  • TaskRun serialization without managedFields
  • init-container migration gating and zero-replica pause behavior
  • Alembic upgrade failure that blocks worker startup, and startup revision failure that stops only the timing worker
  • timeline acknowledgement only after a committed timeline transaction
  • timeline retry and dead-letter-queue handling for parsing, data, and database failures
  • an archive key from a timeline queue copy that matches the archiver’s key for the same SNS event
  • ignored acknowledgement of valid events outside the accepted timeline event types
  • stopped polling on database, schema, or migration failure

PostgreSQL Integration Tests

Use the PostgreSQL service already available in repository CI to:

  1. Apply the initial Alembic revision to an empty database and to a database with populated legacy tables.
  2. Verify Alembic reports the expected head revision and that its version table exists in pipeline_timing, not public.
  3. Confirm the migration does not alter existing public table definitions or representative legacy queries.
  4. Ingest representative events in and out of order.
  5. Verify idempotent current-state upserts.
  6. Verify that a timeline failure leaves its message available for retry without affecting legacy ingestion.
  7. Query one RPM timeline from MR creation through Release completion and calculate each available interval without raw JSON.
  8. Verify all domain rows resolve to event records with the expected S3 references.
  9. Inspect the new schema to confirm it has no raw JSON columns.
  10. Verify that timeline redrive produces idempotent rows and retains the same S3 reference.

End-to-End Acceptance Test

After source enablement, use a controlled RPM merge request that runs GitLab CI, Konflux build and test PipelineRuns, and a Release. Record the MR and component selected for the test.

The acceptance query must demonstrate:

  • MR creation and merge timestamps
  • GitLab pipeline and job timing
  • derived GitLab job URLs that match project.web_url and build_id
  • pre-merge and post-merge commit lineage
  • component build PipelineRun and TaskRun timing
  • a failed PipelineRun and successful retry shown as separate attempts, without losing the failed attempt’s timing
  • Snapshot and test PipelineRun lineage
  • Release completion
  • calculable durations between phases
  • retrievable S3 objects for every row used by the timeline
  • paused timeline processing, recovery, and dead-letter-queue redrive without affecting legacy Status

Run make test, make coverage, make check, and SAM template validation for the affected projects.

Infrastructure Implementation

HUM-8015 delivered the independent staging message bus: its own SNS topic, GitLab and Kubernetes publishers, S3 archiver and bucket, and consumer cutover. HUM-7535 adds only the timeline queue, dead-letter queue, subscription, IAM, and alerts in staging and production.

Staging Event Path

Use the staging message bus delivered by HUM-8015 before enabling the new sources in production:

  1. Verify that the HUM-8015 staging topic, publishers, archiver, bucket, and existing Status consumer are deployed and that message-bus.md describes them.
  2. Under HUM-7535, subscribe dedicated staging and production timeline queues to their respective topics with RawMessageDelivery: false. Configure each source queue with a finite receive limit, a visibility timeout at least twice the worker’s per-message processing budget, its own dead-letter queue, and least-privilege worker credentials. Production queue provisioning does not depend on staging worker acceptance.
  3. Configure 4-day retention for both source queues and 14-day retention for their dead-letter queues, then add queue-age and dead-letter-queue alerts. Messages collected before worker deployment may expire after four days; this pre-launch loss is accepted. Without receives, messages do not automatically move to the DLQ. Verify redrive before source enablement.
  4. Under HUM-7531, deploy one staging pipeline-timing replica with Recreate strategy, its migration init container, timing queue, resource limits, and explicit staging S3_BUCKET; do not use the production bucket. Verify the init container reached the expected revision and the worker is polling before enabling new event sources. Once the worker runs, monitor queue age to prevent loss during pauses.
  5. Keep new production event sources disabled until staging acceptance succeeds.

The staging Kubernetes event forwarder may observe production Konflux resources, but it publishes only to the staging topic. Staging and production forwarders must not publish the same watch stream to the same topic.

Production Event Path

  1. Keep Pipeline Hooks already enabled by the shared Hummingbird webhook configuration.
  2. Add RPM Job Hooks only after the production Status and archiver images understand Job Hook events.
  3. Add PipelineRun and TaskRun watches only after compatible forwarder and Status images are deployed.
  4. Use the production queue, subscription, dead-letter queue, IAM, retention, and alerts provisioned under HUM-7535. After staging worker acceptance, deploy one pipeline-timing replica with Recreate strategy, a migration init container, resource limits, and an explicit production S3_BUCKET. Pre-launch queue messages may have expired; no historical backfill is required. Verify the revision and worker before activating new event sources, and monitor queue age after launch.
  5. Verify the production redrive procedure before enabling new sources.
  6. Enable new production event sources only through HUM-7533 after staging acceptance; the timing worker starts under HUM-7534.

Managed Release Pipeline Access

HUM-7532 owns the access needed for rhtap-releng-tenant and is required for Epic completion.

  1. Obtain read-only get, list, and watch permissions for PipelineRuns and TaskRuns in the managed namespace.
  2. Add managed-namespace contexts to the staging and production Kubernetes event forwarders.
  3. Verify each event-forwarder service account with auth can-i before enabling its watches.
  4. Link the managed PipelineRun to the Release record through its namespaced reference.
  5. Include signing and Pulp publication TaskRuns in the final acceptance timeline.

Infrastructure Verification Gates

  1. Render and resolve each final RPM webhook configuration before deployment.
    • The production endpoint must retain its token and existing push, merge request, note, confidential note, and pipeline events while adding job events.
    • If same-URL inheritance does not deep-merge those settings, use an explicit webhook block containing the complete configuration.
  2. Deploy compatible Hummingbird Status, Pipeline Timing, event-forwarder, and S3 archiver images before enabling Job Hook or TaskRun events in that environment.
  3. Pass the canonical archive-key contract tests in both the archiver and Pipeline Timing projects.
  4. Verify representative TaskRun payload size after removing managedFields.
  5. Use an RPM application label selector if production samples confirm Tekton propagates a stable label. Otherwise, document approval of the namespace-wide event rate before enabling the watch.
  6. Use a single replica and Recreate strategy; verify the migration init container succeeded and the worker checked the expected Alembic revision before it polls.
  7. Verify the source queue has 4-day retention and the dead-letter queue has 14-day retention, with alerts and a tested redrive procedure.

Rollout

  1. Complete the independent Pipeline Timing image and the legacy Status parser compatibility change in HUM-7529, and event-bus support in HUM-7530.
  2. Verify the HUM-8015 staging message bus and provision both environments’ timing queues, subscriptions, IAM, and alerts in HUM-7535, independent of staging worker acceptance. Pre-launch messages may expire without reaching the DLQ; this loss is accepted.
  3. Under HUM-7531, deploy the single-replica staging timing worker with Recreate strategy and its migration init container. Verify the new schema and legacy isolation before enabling new sources.
  4. Complete rhtap-releng-tenant provisioning in HUM-7532 and add it to the staging event forwarder.
  5. Under HUM-7533, activate staging Job Hook, PipelineRun, and TaskRun sources after validating the running timing worker.
  6. Run a controlled RPM merge request and verify staging lineage, duration calculations, managed Release tasks, and S3 references.
  7. Under HUM-7534, deploy the single-replica production timing worker with Recreate strategy and its migration init container after staging worker validation.
  8. Verify the production Alembic revision, legacy ingestion, timeline queue alerts, and metrics.
  9. Add the managed namespace to the production event forwarder, then activate new production sources under HUM-7533 after verifying the timing worker is running.
  10. Expect each Kubernetes watcher to emit an initial ADDED event for every existing matching resource. Schedule source activation during a monitored window.
  11. Monitor legacy and timeline queue age, dead-letter-queue depth, worker and archiver errors, forwarder spool growth, database size, and PVC free space.
  12. Run the final controlled RPM acceptance merge request in production before closing HUM-5820.

Rollback does not require dropping or downgrading the schema. Scale the timing worker to zero and disable the new TaskRun and Job Hook sources in the affected environment. Legacy ingestion continues. After launch, monitor queue age while paused: messages expire after four days unless the worker resumes in time. Malformed messages move to the DLQ after failed receives and can be redriven after repair.

Epic Completion Criteria

The approved design’s acceptance criteria remain authoritative. HUM-5820 is complete when:

  1. HUM-7529 through HUM-7535 are complete.
  2. The design’s acceptance criteria pass in both staging and production.
  3. Representative timelines include managed PipelineRun and TaskRun timing from rhtap-releng-tenant.
  4. The staging-first source rollout and final production acceptance run are documented under HUM-7533.
  5. HUM-7533 has verified that each Story’s required documentation update is complete.

Out of Scope

  1. Creating a UI for visualization is out of scope.
    • The data may later support a new application, Dashboard changes, or Grafana. UX should be involved in that decision.
  2. Tracking the containers repository pipeline is out of scope.
    • The pipeline-neutral schema, event contracts, and execution identities are intended to support its GitLab and Konflux events in a future follow-up without creating a parallel storage model.
    • The RPM and containers pipelines do not connect directly. A separate scheduled process discovers updated RPMs and commits changes to rpms.lock.yaml files in the containers repository.
  3. Timeline replay and historical backfill are out of scope.
    • Existing Hummingbird Status S3 ingestion bootstraps the public schema and collapses Kubernetes history, so it is not suitable for timeline recovery.
    • Future work must define scoped S3 read permissions and chronological ingestion into the Alembic-managed pipeline_timing schema. Events that were never collected cannot be reconstructed.

Delivery Stories

Story Responsibility
HUM-7529 Make RPM pipeline timing data queryable Independent Pipeline Timing image and schema, Alembic, worker, ingestion, lineage, tests, and legacy Status compatibility
HUM-7530 Carry RPM pipeline timing events through the event bus GitLab and Kubernetes forwarders, S3 archiver event identities, payload handling, and tests
HUM-7531 Validate RPM pipeline timeline storage in staging Staging Pipeline Timing deployment, Alembic verification, legacy isolation, and representative queries
HUM-7534 Make RPM pipeline timeline storage available in production Production Pipeline Timing deployment with event sources still disabled
HUM-7535 Support environment-specific RPM pipeline event collection Both environments’ timeline queues, dead-letter queues, subscriptions, IAM, alerts, and configuration before worker rollout
HUM-7532 Expose managed Release pipeline timing Read-only rhtap-releng-tenant access and managed PipelineRun and TaskRun event collection
HUM-7533 Roll out RPM pipeline event collection Staging source activation, production source activation, monitoring, recovery, redrive, and end-to-end validation

Dependency Order

  1. HUM-7529 and HUM-7530 can proceed in parallel.
  2. HUM-8015 is the completed staging message-bus prerequisite. HUM-7535 depends on HUM-7530’s event contracts and provisions timeline-specific infrastructure in both environments without waiting for staging worker acceptance.
  3. HUM-7531 depends on HUM-7529, HUM-7530, and the staging path from HUM-7535.
  4. HUM-7534 depends on successful staging validation in HUM-7531.
  5. HUM-7532 depends on the event support from HUM-7530 and the environment-specific infrastructure from HUM-7535. Managed namespace access can proceed alongside the Pipeline Timing deployments, but completion gates final acceptance under HUM-7533.
  6. HUM-7533 depends on HUM-7531, HUM-7534, HUM-7535, and HUM-7532. HUM-5820 remains open until the final rollout is verified.