# Pipeline timing implementation plan

LLMS index: [llms.txt](https://hummingbird-project.io/llms.txt) | Full content: [llms-full.txt](https://hummingbird-project.io/llms-full.txt)

---

## Purpose

The [original RPM pipeline timing design](https://hummingbird-project.io/docs/background/tools/pipeline-timing/design/) records
the HUM-5820 design and remains unchanged. This implementation plan records the
HUM-10942 revisions to retain execution timing and lineage for both RPM and
container pipelines, not a permanent notification catalogue or permanent S3
references on reporting rows. Its execution-only requirements supersede the
original design's archive-reference and RPM-only scope decisions.

The Status events database remains the storage location, but `pipeline-timing` is a separate
project, image, and worker with its own deployment lifecycle and resource quotas. Hummingbird Status
continues legacy ingestion; the GitLab and Kubernetes event forwarders remain independent
publishers. This service split follows [HUM-7529 MR
1261](https://gitlab.com/redhat/hummingbird/tools/-/merge_requests/1261) review without changing the
event contracts or lineage model.

## Prior Exploration

Closed [tools MR 1165](https://gitlab.com/redhat/hummingbird/tools/-/merge_requests/1165) and closed
[infrastructure MR 1381](https://gitlab.com/redhat/hummingbird/infrastructure/-/merge_requests/1381)
informed this plan. They are not implementation vehicles for the Stories above. New focused merge
requests should reference the Story that owns their work.

## Current execution-only transition

The original HUM-5820 delivery stories are complete. HUM-10942 owns the revised
execution-only requirements and supersedes the old archive-reference and RPM-only
scope decisions. Existing Status tables, publishers, the archiver, and queue
contracts remain unchanged.

1. Release one removes the unused composite S3 lookup index.
2. Release two, revision `003_execution_records`, adds nullable creation times to
   TaskRuns and PipelineRuns and makes the reporting tables' `latest_event_id`
   nullable. The worker stops inserting events and never changes old pointers.
   New rows use null pointers; the old foreign keys remain valid temporarily.
3. Operator-run local recovery filled historical creation timestamps through those
   frozen pointers without replaying state. Both environments were verified with
   zero missing TaskRun/PipelineRun creation times or creation-time ordering anomalies.
   The one-time tooling was intentionally removed rather than retained in the repo.
4. Release three, revision `004_drop_event_catalogue`, explicitly removes the
   legacy foreign keys and `latest_event_id` columns, then drops `events` and its
   remaining indexes. All reporting values and ordering metadata are preserved.

Catalogue removal is transactional, uses five-second lock waits, and has no
supported downgrade. Confirm recovery/backup readiness and no remaining external
consumer, validate staging ingestion and physical space recovery, then use the
existing image promotion path for production. Do not recreate empty history as a
rollback. The `events` database and legacy `public` schema are not removed.

The worker does not require `S3_BUCKET` or calculate archive keys. Remove unused
infrastructure bucket configuration only after the compatible app is deployed in
both environments. Do not change the worker's IAM permissions for local recovery.

Before each promotion, verify migration revision, execution timing/lineage,
commit-before-acknowledgement, queue catch-up, and legacy isolation. During MR
validation, prevent ordinary infrastructure pipelines from restoring the older
shared image pin. Merge can lead to automatic Konflux publication, Renovate image
pin updates, and deployment to both environments; it is not a recovery hold.

Test fresh initialization and populated-schema upgrades, preserving all reporting
values and relation files while removing only the catalogue and legacy references.
Keep revision-001/002/003 regression tests in isolated disposable databases that
stop before irreversible removal. Verify a forced cleanup failure rolls back all
constraint/column drops, and an attempted downgrade cannot fabricate history.
Continue testing lifecycle times, states, lineage, redelivery, and ingestion with
the catalogue absent. No change to the worker's ingestion logic is required.

## Implementation Plan

### Scope Decisions

1. The timeline tracks through completion of the Konflux `Release` resource.
   - The Release completion timestamp is the final tracked milestone.
   - PipelineRun and TaskRun detail in `rhtap-releng-tenant` requires separate provisioning and is
     required before HUM-5820 is complete.
   - PipelineRuns and TaskRuns in `hummingbird-tenant` remain in scope.
2. `Snapshot` and `Release` events are required as lineage bridges even though they were omitted
   from the initial agreed-event list.
   - A Snapshot links a build PipelineRun to test PipelineRuns.
   - A Release links a Snapshot to Release pipeline completion and, where visible, its PipelineRun.
   - The new schema stores only their reporting and lineage fields, not raw JSON.
3. The new model stores the latest known state of each execution and its relevant timestamps. It
   does not duplicate every status transition as a domain row.
   - A retry is a new PipelineRun and has its own UID, row, status, start time, and completion time.
     It does not replace the failed PipelineRun.
   - For example, a PipelineRun that fails after four hours and a retry that succeeds in ten minutes
     remain two separate records. A report must not show only the ten-minute retry as the time for
     both attempts.
4. Both RPM and container pipeline executions are in scope. Retain the existing
   source mix rather than introducing repository filters during this transition.
5. Full event replay and per-step timing are out of scope. Historical recovery is
   limited to best-effort creation timestamps and runs outside startup migrations.
6. Each environment needs a dedicated timing SQS queue, dead-letter queue, and `pipeline-timing`
   worker deployment using its own image. The existing Status queue, image, and worker continue
   legacy ingestion unchanged.

### PostgreSQL Schema

Create a `pipeline_timing` schema in the existing Status events database. Do not add columns,
indexes, constraints, or triggers to existing tables in the `public` schema. The Dashboard continues
to query only the existing tables.

Domain tables contain selected scalar fields, source timestamps, lineage, and
ordering metadata, not a `raw` payload, history array, or catalogue foreign key.

| Table                                   | Identity columns                                  | Purpose                                                                  |
| --------------------------------------- | ------------------------------------------------- | ------------------------------------------------------------------------ |
| `pipeline_timing.merge_requests`        | `gitlab_project_id`, `iid`                        | Current MR state and lifecycle timestamps                                |
| `pipeline_timing.commits`               | `gitlab_project_id`, `sha`                        | Commit identity and available commit metadata                            |
| `pipeline_timing.merge_request_commits` | `gitlab_project_id`, `iid`, `sha`, `relationship` | MR head and merge-commit lineage                                         |
| `pipeline_timing.gitlab_pipelines`      | `gitlab_project_id`, `pipeline_id`                | Pipeline status, source, ref, SHA, and execution timestamps              |
| `pipeline_timing.gitlab_jobs`           | `gitlab_project_id`, `job_id`                     | Job attempt status, stage, timing, failure reason, and retry information |
| `pipeline_timing.pipeline_runs`         | `cluster`, `uid`                                  | Konflux PipelineRun state, source identity, timing, and Snapshot lineage |
| `pipeline_timing.task_runs`             | `cluster`, `uid`                                  | Tekton task state, timing, and parent PipelineRun UID                    |
| `pipeline_timing.snapshots`             | `cluster`, `uid`                                  | Build PipelineRun to Snapshot lineage                                    |
| `pipeline_timing.releases`              | `cluster`, `uid`                                  | Snapshot to Release completion and PipelineRun lineage                   |

Store both GitLab project ID and a canonical HTTPS repository URL where available. GitLab project ID
is the stable GitLab identity. The repository URL is needed because Konflux resources contain source
URLs rather than GitLab project IDs.

`004_drop_event_catalogue` removes the temporary recovery catalogue and pointers
after recovery completion. `latest_observed_at` and `source_updated_at` remain on
all reporting tables for ordering, independently of notification-history storage.
The S3 archiver remains an independent subscriber; execution fields require no
archive lookup during normal ingestion or reporting.

### Schema Management

Use [Alembic](https://alembic.sqlalchemy.org/) (a database schema migration tool already used by
Hummingbird's workqueue service) to manage only the `pipeline_timing` schema. Existing tables in the
`public` schema remain under the Hummingbird Status `db.create_schema()` path; adopting Alembic for
those tables is separate work.

1. Add Alembic configuration and revisions to the `pipeline-timing` project and image.
2. Add `pipeline_timing.worker.init_db` for the worker Deployment's init container. It executes
   `CREATE SCHEMA IF NOT EXISTS pipeline_timing` directly and commits before invoking Alembic. Do
   not rely on Alembic to create the schema. Alembic manages every object inside the schema after
   that bootstrap.
3. Set Alembic's `version_table_schema="pipeline_timing"` and qualify every migration object with
   the schema. The version table and all managed objects must remain outside `public`.
4. Create the tables, constraints, and required indexes in the initial revision.
5. Run `alembic upgrade head` unconditionally from `pipeline_timing.worker.init_db` in the
   single-replica Deployment's init container. Use `Recreate` so the old worker stops before a new
   image migrates the schema. Alembic is a no-op when the schema is already at head.
6. On migration failure, log the exception and expected revision at ERROR level and exit nonzero.
   Kubernetes must not start the worker container after a failed init container; the separate
   Hummingbird Status Deployment continues unchanged.
7. Have the timing worker independently verify that the database is at the expected Alembic revision
   before polling its queue. If the revision check fails, log the expected and observed revision at
   ERROR level and stop the timing worker. The legacy worker continues unchanged.
8. Deploy one replica with the timing image and init container once the queue and archiver are
   ready. To pause polling, scale the Deployment to zero and monitor queue age against the source
   queue's 4-day retention. SQS preserves a standard message's enqueue timestamp when automatically
   moving it to the DLQ, whose 14-day retention is a maximum total age. Operator redrive from the
   DLQ resets the enqueue timestamp.
9. Do not automatically downgrade or drop the schema during application rollback. Stop the timeline
   worker and use a forward migration to correct schema problems.

### Selected Fields

The detailed schema should include at least the following reporting fields. These are proposed
database column names, not source-payload field names.

| Record type     | Required columns                                                                                                                                                                                                                                         |
| --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Merge request   | `gitlab_project_id`, `repository_url`, `iid`, `state`, `source_branch`, `target_branch`, `head_sha`, `merge_commit_sha`, `created_at`, `updated_at`, `merged_at`, `closed_at`, `web_url`                                                                 |
| Commit          | `gitlab_project_id`, `repository_url`, `sha`, `title`, `message`, `committed_at`, `web_url`                                                                                                                                                              |
| MR commit link  | `gitlab_project_id`, `repository_url`, `iid`, `sha`, `relationship`                                                                                                                                                                                      |
| GitLab pipeline | `gitlab_project_id`, `repository_url`, `pipeline_id`, `mr_iid`, `sha`, `ref`, `source`, `status`, `created_at`, `updated_at`, `started_at`, `completed_at`, `web_url`                                                                                    |
| GitLab job      | `gitlab_project_id`, `repository_url`, `job_id`, `pipeline_id`, `sha`, `name`, `stage`, `status`, `failure_reason`, `allow_failure`, `created_at`, `started_at`, `completed_at`, `duration_seconds`, `queued_duration_seconds`, `retry_count`, `web_url` |
| PipelineRun     | `cluster`, `namespace`, `uid`, `name`, `application`, `component`, `pipeline_type`, `pipeline_name`, `trigger_type`, `repository_url`, `sha`, `mr_iid`, `snapshot`, `status`, `reason`, `created_at`, `started_at`, `completed_at`                       |
| TaskRun         | `cluster`, `namespace`, `uid`, `name`, `pipeline_run_uid`, `pipeline_run_name`, `pipeline_task`, `task_reference`, `status`, `reason`, `pod_name`, `created_at`, `started_at`, `completed_at`                                                            |
| Snapshot        | `cluster`, `namespace`, `uid`, `name`, `application`, `component`, `sha`, `source_pipeline_run_name`, `created_at`                                                                                                                                       |
| Release         | `cluster`, `namespace`, `uid`, `name`, `snapshot`, `status`, `pipeline_run_namespace`, `pipeline_run_name`, `completed_at`                                                                                                                               |

TaskRun and PipelineRun creation time is `metadata.creationTimestamp`; start and
completion are the resource's own status timestamps. Missing values remain null.
`Unknown` conditions with a pending reason report `Pending`, not `Running`.
Cancellation remains distinguishable through the failure reason. GitLab execution
timestamps and status remain source-derived. Non-execution entities retain their
meaningful lifecycle milestones rather than fabricated start/completion fields.

GitLab job retries have distinct job IDs and therefore distinct rows. Store retry metadata when
GitLab supplies it, but do not invent a parent job identity. Consumers can group attempts by
project, pipeline, stage, and job name.

PipelineRun retries also have distinct rows because each PipelineRun has a distinct Kubernetes UID.
Keep the timing and result for every attempt. This plan does not define a retry-group column or a
single combined retry duration. Consumers that need an end-to-end duration across retries must keep
the duration of each attempt visible and calculate the full elapsed time from the first attempt's
start to the final attempt's completion.

GitLab Job Hook payloads do not provide a dedicated job URL. Derive `web_url` without an API request
by appending `/-/jobs/{build_id}` to `project.web_url`.

`merged_at` and `closed_at` are new nullable `TIMESTAMPTZ` columns on
`pipeline_timing.merge_requests`; they are not additions to the existing
`public.gitlab_merge_requests` table. Use explicit GitLab merge and close timestamps when the
webhook supplies them. Otherwise, derive them from the first event that reports the corresponding
action or state, preferring the source `updated_at` timestamp and falling back to `observed_at`.
Preserve the first derived terminal timestamp across later sparse events. This makes MR
creation-to-merge timing available without requiring a GitLab API lookup.

### Repository URL Normalization

Normalize repository identity to an HTTPS URL without a trailing slash, `.git` suffix, or GitLab
resource path. Prefer GitLab's `project.web_url` when it is available. Accept HTTPS, SSH, SCP-style
Git, and GitLab tree URLs from Konflux metadata.

These source values:

- `https://gitlab.com/redhat/hummingbird/rpms.git`
- `git@gitlab.com:redhat/hummingbird/rpms.git`
- `https://gitlab.com/redhat/hummingbird/rpms/-/tree/main`

all normalize to:

```text
https://gitlab.com/redhat/hummingbird/rpms
```

Use `repository_url` consistently in the new schema. Do not use a bare project path or
source-specific Git URL as the cross-system identity.

### Identity and Update Ordering

1. Use `(cluster, uid)` as the primary execution identity for Kubernetes resources.
   - Names are not sufficient because Kubernetes can recreate a resource with the same name.
   - Store namespace and name for display and fallback joins.
2. Use source IDs for GitLab records and retain `repository_url` for cross-system joins.
3. Store the latest accepted SNS timestamp as `latest_observed_at` on reporting rows.
4. Use source update timestamps from GitLab and Kubernetes condition transition times.
5. Prevent an older event delivered later by SQS from replacing newer state.
   - Compare source timestamps only when both records have them; use SNS publication
     timestamps as the fallback and tie-breaker, including migration of legacy rows.
   - Late non-terminal snapshots must not regress completed Kubernetes executions.
6. Preserve source fields absent from sparse updates and keep known creation time
   immutable. Ordering metadata advances independently of notification history.
7. Do not require foreign keys for cross-source lineage that may arrive out of order.
   Use indexed identity columns and late joins. There are no legacy event foreign
   keys or pointer columns after catalogue removal.

### Required Indexes

The schema must provide indexes for the known lineage and timeline queries.
Exact index names and partial predicates are implementation details, but the
following column order is required as a baseline:

| Table                   | Columns                                            | Supports                                 |
| ----------------------- | -------------------------------------------------- | ---------------------------------------- |
| `merge_request_commits` | `gitlab_project_id`, `iid`                         | commits for one MR                       |
| `merge_request_commits` | `repository_url`, `sha`                            | MR lookup from a commit                  |
| `gitlab_pipelines`      | `repository_url`, `sha`                            | pipelines for a commit                   |
| `gitlab_jobs`           | `gitlab_project_id`, `pipeline_id`                 | jobs for a pipeline                      |
| `pipeline_runs`         | `repository_url`, `sha`                            | Konflux runs for a commit                |
| `pipeline_runs`         | `cluster`, `namespace`, `name`                     | name-based fallback lineage              |
| `pipeline_runs`         | `cluster`, `namespace`, `snapshot`                 | test PipelineRuns for a Snapshot         |
| `task_runs`             | `cluster`, `pipeline_run_uid`                      | tasks for a PipelineRun                  |
| `snapshots`             | `cluster`, `namespace`, `source_pipeline_run_name` | Snapshot produced by a build PipelineRun |
| `releases`              | `cluster`, `namespace`, `snapshot`                 | Releases for a Snapshot                  |

Primary keys provide the identity indexes listed in the schema table and do not need duplicate
secondary indexes. Validate the final indexes with the representative acceptance query and
PostgreSQL `EXPLAIN` before source enablement.

The initial design required a reverse archive lookup index on `s3_bucket`, `s3_key`.
The staging storage investigation on October 22, 2026 found that this index occupied
about 8.1 GiB, roughly 40% of the event table's total allocation including indexes,
with zero recorded scans. No application query using the reverse lookup was found.
The temporary legacy recovery used frozen `latest_event_id` values to read the
archived creation metadata. Runtime ingestion never needed that reverse index.

Revision `002_drop_events_s3_index` removes this unused application index to reclaim
disk space and avoid maintaining it for every new event. It preserves event rows,
columns, and archive references. Scan statistics do not prove that no external
consumer needs the index; confirm that before deployment. Deploy and verify index
space recovery in staging and production before the execution-only transition.
Revision `004_drop_event_catalogue` drops the remaining catalogue indexes with the
table. No execution-reporting index is removed or rebuilt.

### Lineage Model

Consumers reconstruct a timeline with the following relationships:

1. A merge request links to its head commits and merge commit through `merge_request_commits`.
2. A commit links to GitLab pipelines by `repository_url` and SHA. A direct pipeline MR IID is used
   when the webhook supplies it.
3. GitLab jobs link to their pipeline by project and pipeline ID.
4. Konflux PipelineRuns link to a commit by `repository_url` and SHA.
   - Extract the repository from Pipelines as Code annotations first, then the `git-url` parameter.
   - Extract SHA and MR IID from the current Pipelines as Code and build-service labels and annotations.
5. TaskRuns link to PipelineRuns by Kubernetes owner UID. Parent name is a fallback for malformed or
   older resources without an owner UID.
6. A Snapshot links to its source build PipelineRun by name and cluster/namespace context.
7. Test PipelineRuns link to a Snapshot through the Snapshot label.
8. A Release links to its Snapshot and to any referenced Release PipelineRun.
9. In the current scope, the Release completion timestamp closes the timeline even when its managed
   PipelineRun is in an inaccessible namespace.

The primary cross-system correlation key within one source pipeline is canonical `repository_url`
plus commit SHA, not SHA alone. One commit can produce several component PipelineRuns, so the
resulting timeline fans out by component and execution rather than assuming one build per commit.

### Event Processing and Failure Isolation

Subscribe a dedicated timing queue to the SNS topic already used by the existing Status queue. The
timing queue receives its own SQS copy of each SNS envelope. The existing Status queue and worker
remain unchanged. The `pipeline-timing` worker parses and upserts only `pipeline_timing` records.

For each timeline message:

1. Start polling only after the Pod's migration init container succeeds and the worker verifies the
   expected revision. Scaling the single-replica Deployment to zero leaves the queue untouched.
2. Decode the SNS envelope. A valid event outside the accepted timeline event types is ignored and acknowledged.
3. For an accepted notification, parse and upsert the reporting records in one
   transaction. Do not append a notification-history row or require a catalogue link.
4. Commit the transaction, then delete the SQS message.
5. On malformed data for an accepted event, roll back and leave the message on the queue. SQS
   retries it and moves it to the timeline dead-letter queue after the configured receive limit.
6. On a database connection, schema, or migration failure, roll back, stop polling, and leave the
   queue intact for recovery. Do not send valid events to the dead-letter queue during an
   infrastructure outage.

A timeline failure must not delay, retry, or change legacy ingestion. Operators fix a malformed
event or parser error, then redrive messages from the timeline dead-letter queue. They fix an
infrastructure failure, then restart the worker to resume polling. Configure the source queue with
4-day retention and its DLQ with 14-day retention, a finite receive limit, and a visibility timeout
at least twice the worker's per-message processing budget. Automatic source-to-DLQ transfer
preserves a standard message's original enqueue timestamp; operator redrive from the DLQ resets it.
Add queue-age and dead-letter-queue alerts, and document and test the redrive procedure.

### Completed recovery and archive inspection

S3 archival remains independent and unchanged. The timing worker neither computes
archive keys nor needs bucket configuration or S3 read permissions. The operator-run
recovery read immutable creation timestamps from archived snapshots, updated only
null creation fields, and finished before catalogue removal. It was not part of a
migration, and its tooling is not retained.

`latest_observed_at` and resource identity can narrow an archive search while
objects are retained. Validate candidate payload identities; this is not guaranteed
unique lookup or per-field provenance because sparse updates can merge older facts.
See [finding archived notifications](https://hummingbird-project.io/l/pipeline-timing#finding-archived-notifications).
No permanent per-execution S3 pointer or replacement history table is added.

## Original HUM-5820 delivery and source rollout

The remaining delivery-story sections record the original rollout. Their permanent
archive-reference, bucket-configuration, and RPM-only assumptions are historical
and superseded by HUM-10942 and the execution-only transition above. The publishers
and independent archiver retain their existing event contracts.

### Tools Repository: Publisher and Archiver Changes

This section describes code, tests, and documentation in the tools repository. Environment-specific
source infrastructure is owned by HUM-7535 and activated under HUM-7533.

#### GitLab Event Forwarder

The forwarder already publishes arbitrary GitLab webhook bodies and derives `event_type` from
`object_kind`. No event-specific forwarding implementation is required.

Add fixtures and tests for:

- Pipeline Hooks with `object_kind: pipeline`
- Job Hooks with `object_kind: build`
- `project_path` and `event_type` SNS message attributes
- compressed payload round trips

#### Kubernetes Event Forwarder

1. Add `TaskRun` to the configured resources only after the deployed image understands its payload requirements.
2. Extend the resource configuration schema with optional `labelSelector` and `fieldSelector` keys.
3. Pass configured selectors to the existing `ResourceWatcher` `label_selector` and `field_selector`
   parameters.
4. Remove `metadata.managedFields` from TaskRun messages before SNS publication because it is
   transport metadata and can dominate payload size.
5. Keep status, results, owner references, task identity, and timing data needed for extraction and
   future diagnosis.
6. Measure compressed and base64-encoded TaskRun payloads from representative RPM build, test, and
   tenant Release PipelineRuns.
7. Do not enable the source if representative payloads approach the SNS 256 KiB limit. Define and
   review any additional projection before rollout rather than silently dropping fields.
8. Preserve spooled events on publish failure so a transient SNS error does not discard them.

Use a TaskRun label selector for the RPM application if production samples confirm that Tekton
propagates a stable application label. If no reliable selector exists, estimate and approve the
namespace-wide event rate before enabling the watch because every matching TaskRun will also be
archived to S3.

#### S3 Archiver

The archiver is already generic, but its resource identity logic must change for the new event
types. Extend `_extract_resource_name()` and `_build_s3_key()` so GitLab pipeline and job keys
include their numeric pipeline or job ID and Kubernetes keys include logical cluster name,
namespace, and name. URL-valued `cluster` attributes from older forwarders retain name-only
identities. Cover those identities with the canonical contract fixture.

Continue archiving the complete event as received from SNS. The Kubernetes forwarder's documented
removal of `managedFields` occurs before that boundary. Keep explicit test coverage for PipelineRun,
TaskRun, Snapshot, Release, merge request, commit/push, pipeline, and job events.

### Pipeline Timing and Hummingbird Status Changes

1. Add a `pipeline-timing/` project, package, image, and Alembic configuration with an initial
   `pipeline_timing` revision. Generate its independent Konflux build resources.
2. Add `pipeline_timing/timeline.py` for extraction, ordering rules, and upserts.
3. Keep Hummingbird Status extraction, schema creation, upsert functions, dependencies, and image
   responsible only for the `public` schema.
4. Add timing dispatch for GitLab `push`, `merge_request`, `pipeline`, and `build` events and
   Kubernetes `PipelineRun`, `TaskRun`, `Snapshot`, and `Release` events.
5. Mark the newly handled event types as known to the legacy Status parser so expected timing events
   do not generate unknown-kind warnings; do not route them to legacy tables.
6. Add `pipeline_timing.worker.init_db` to the timing Deployment as an init container. Leave
   `hummingbird_status.worker.init_db` and legacy `worker/sqs.py` responsible only for `public`.
7. Deploy the independent timing image to poll only the timing queue and write only the
   `pipeline_timing` schema in the shared events database.
8. Add archive bucket configuration to the timing worker deployment.
9. Start worker polling only after its init container succeeds; retain the worker's own startup
   revision check before queue access.
10. Keep Alembic migration and revision-check failures isolated to the timing worker so they cannot
    prevent legacy worker startup.
11. Provide separate `pipeline_timing` database counts without changing existing Status statistics queries.

### Observability

Add metrics and structured logs for:

- timeline records processed by kind and result
- timeline extraction and database failures
- events missing expected lineage fields
- oversized Kubernetes payloads and SNS publish failures
- timeline queue age and dead-letter-queue depth
- timeline worker stopped because of a migration or revision-check failure

Failure logs must include the SNS message ID, source, kind, resource identity, and S3 location
without logging the full raw event. Continue monitoring legacy queue age and DLQ metrics, archiver
failures, PostgreSQL availability, and PVC usage.

### Documentation Updates

Update the existing documentation for every component changed by implementation. Do not create a
separate documentation Story. Each delivery Story owns the pages for the behavior it changes:

| Story    | Required documentation updates                                                                                                                                      |
| -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| HUM-7529 | New `pipeline-timing.md`, legacy isolation in `hummingbird-status.md`, and the timing schema and lineage sections of `data-flow.md`                                 |
| HUM-7530 | `gitlab-event-forwarder.md`, `kubernetes-event-forwarder.md`, `sns-s3-archiver.md`, and event-flow portions of `data-flow.md`                                       |
| HUM-7531 | staging worker deployment, monitoring, and recovery details in `pipeline-timing.md`                                                                                 |
| HUM-7532 | managed Release namespace access in `kubernetes-event-forwarder.md` and `data-flow.md`                                                                              |
| HUM-7533 | final documentation audit across all affected pages before HUM-5820 closes                                                                                          |
| HUM-7534 | production worker deployment, monitoring, and recovery details in `pipeline-timing.md`                                                                              |
| HUM-7535 | both environments' timeline queues, dead-letter queues, subscriptions, IAM, and alerts in `hummingbird-events-topic.md` and infrastructure monitoring documentation |

HUM-8015 already updated `message-bus.md` for the independent staging message bus. HUM-7535
clarifies that timing queue subscriptions can precede worker deployment.

### Testing

#### Unit Tests

Cover:

- every accepted GitLab and Kubernetes event type
- status and timestamp extraction
- explicit and derived MR merge and close timestamps
- repository URL normalization across HTTPS, SSH, SCP-style Git, and tree URLs
- GitLab job URL derivation from `project.web_url` and `build_id`
- MR head and merge-commit lineage
- GitLab job retries as distinct job IDs
- a failed PipelineRun followed by a successful retry, with separate statuses and timings for both attempts
- PipelineRun-to-TaskRun owner UID lineage
- Snapshot and Release bridge extraction
- sparse and out-of-order updates
- deterministic archive keys
- missing optional lineage without rejecting otherwise valid records
- TaskRun serialization without `managedFields`
- init-container migration gating and zero-replica pause behavior
- Alembic upgrade failure that blocks worker startup, and startup revision failure that stops only
  the timing worker
- timeline acknowledgement only after a committed timeline transaction
- timeline retry and dead-letter-queue handling for parsing, data, and database failures
- an archive key from a timeline queue copy that matches the archiver's key for the same SNS event
- ignored acknowledgement of valid events outside the accepted timeline event types
- stopped polling on database, schema, or migration failure

#### PostgreSQL Integration Tests

Use the PostgreSQL service already available in repository CI to:

1. Apply the initial Alembic revision to an empty database and to a database with populated legacy tables.
2. Verify Alembic reports the expected head revision and that its version table exists in
   `pipeline_timing`, not `public`.
3. Confirm the migration does not alter existing `public` table definitions or representative legacy
   queries.
4. Ingest representative events in and out of order.
5. Verify idempotent current-state upserts.
6. Verify that a timeline failure leaves its message available for retry without affecting legacy ingestion.
7. Query one RPM timeline from MR creation through Release completion and calculate each available
   interval without raw JSON.
8. Verify all domain rows resolve to event records with the expected S3 references.
9. Inspect the new schema to confirm it has no raw JSON columns.
10. Verify that timeline redrive produces idempotent rows and retains the same S3 reference.

#### End-to-End Acceptance Test

After source enablement, use a controlled RPM merge request that runs GitLab CI, Konflux build and
test PipelineRuns, and a Release. Record the MR and component selected for the test.

The acceptance query must demonstrate:

- MR creation and merge timestamps
- GitLab pipeline and job timing
- derived GitLab job URLs that match `project.web_url` and `build_id`
- pre-merge and post-merge commit lineage
- component build PipelineRun and TaskRun timing
- a failed PipelineRun and successful retry shown as separate attempts, without losing the failed
  attempt's timing
- Snapshot and test PipelineRun lineage
- Release completion
- calculable durations between phases
- retrievable S3 objects for every row used by the timeline
- paused timeline processing, recovery, and dead-letter-queue redrive without affecting legacy Status

Run `make test`, `make coverage`, `make check`, and SAM template validation for the affected projects.

### Infrastructure Implementation

HUM-8015 delivered the independent staging message bus: its own SNS topic, GitLab and Kubernetes
publishers, S3 archiver and bucket, and consumer cutover. HUM-7535 adds only the timeline queue,
dead-letter queue, subscription, IAM, and alerts in staging and production.

#### Staging Event Path

Use the staging message bus delivered by HUM-8015 before enabling the new sources in production:

1. Verify that the HUM-8015 staging topic, publishers, archiver, bucket, and existing Status
   consumer are deployed and that `message-bus.md` describes them.
2. Under HUM-7535, subscribe dedicated staging and production timeline queues to their respective
   topics with `RawMessageDelivery: false`. Configure each source queue with a finite receive limit,
   a visibility timeout at least twice the worker's per-message processing budget, its own
   dead-letter queue, and least-privilege worker credentials. Production queue provisioning does not
   depend on staging worker acceptance.
3. Configure 4-day retention for both source queues and 14-day retention for their dead-letter
   queues, then add queue-age and dead-letter-queue alerts. Messages collected before worker
   deployment may expire after four days; this pre-launch loss is accepted. Without receives,
   messages do not automatically move to the DLQ. Verify redrive before source enablement.
4. Under HUM-7531, deploy one staging `pipeline-timing` replica with `Recreate` strategy, its
   migration init container, timing queue, resource limits, and explicit staging `S3_BUCKET`; do not
   use the production bucket. Verify the init container reached the expected revision and the worker
   is polling before enabling new event sources. Once the worker runs, monitor queue age to prevent
   loss during pauses.
5. Keep new production event sources disabled until staging acceptance succeeds.

The staging Kubernetes event forwarder may observe production Konflux resources, but it publishes
only to the staging topic. Staging and production forwarders must not publish the same watch stream
to the same topic.

#### Production Event Path

1. Keep Pipeline Hooks already enabled by the shared Hummingbird webhook configuration.
2. Add RPM Job Hooks only after the production Status and archiver images understand Job Hook events.
3. Add PipelineRun and TaskRun watches only after compatible forwarder and Status images are deployed.
4. Use the production queue, subscription, dead-letter queue, IAM, retention, and alerts provisioned
   under HUM-7535. After staging worker acceptance, deploy one `pipeline-timing` replica with
   `Recreate` strategy, a migration init container, resource limits, and an explicit production
   `S3_BUCKET`. Pre-launch queue messages may have expired; no historical backfill is required.
   Verify the revision and worker before activating new event sources, and monitor queue age after
   launch.
5. Verify the production redrive procedure before enabling new sources.
6. Enable new production event sources only through HUM-7533 after staging acceptance; the timing
   worker starts under HUM-7534.

#### Managed Release Pipeline Access

HUM-7532 owns the access needed for `rhtap-releng-tenant` and is required for Epic completion.

1. Obtain read-only `get`, `list`, and `watch` permissions for PipelineRuns and TaskRuns in the
   managed namespace.
2. Add managed-namespace contexts to the staging and production Kubernetes event forwarders.
3. Verify each event-forwarder service account with `auth can-i` before enabling its watches.
4. Link the managed PipelineRun to the Release record through its namespaced reference.
5. Include signing and Pulp publication TaskRuns in the final acceptance timeline.

#### Infrastructure Verification Gates

1. Render and resolve each final RPM webhook configuration before deployment.
   - The production endpoint must retain its token and existing push, merge request, note,
     confidential note, and pipeline events while adding job events.
   - If same-URL inheritance does not deep-merge those settings, use an explicit webhook block
     containing the complete configuration.
2. Deploy compatible Hummingbird Status, Pipeline Timing, event-forwarder, and S3 archiver images
   before enabling Job Hook or TaskRun events in that environment.
3. Pass the canonical archive-key contract tests in both the archiver and Pipeline Timing projects.
4. Verify representative TaskRun payload size after removing `managedFields`.
5. Use an RPM application label selector if production samples confirm Tekton propagates a stable
   label. Otherwise, document approval of the namespace-wide event rate before enabling the watch.
6. Use a single replica and `Recreate` strategy; verify the migration init container succeeded and
   the worker checked the expected Alembic revision before it polls.
7. Verify the source queue has 4-day retention and the dead-letter queue has 14-day retention, with
   alerts and a tested redrive procedure.

### Rollout

1. Complete the independent Pipeline Timing image and the legacy Status parser compatibility change
   in HUM-7529, and event-bus support in HUM-7530.
2. Verify the HUM-8015 staging message bus and provision both environments' timing queues,
   subscriptions, IAM, and alerts in HUM-7535, independent of staging worker acceptance. Pre-launch
   messages may expire without reaching the DLQ; this loss is accepted.
3. Under HUM-7531, deploy the single-replica staging timing worker with `Recreate` strategy and its
   migration init container. Verify the new schema and legacy isolation before enabling new sources.
4. Complete `rhtap-releng-tenant` provisioning in HUM-7532 and add it to the staging event forwarder.
5. Under HUM-7533, activate staging Job Hook, PipelineRun, and TaskRun sources after validating the
   running timing worker.
6. Run a controlled RPM merge request and verify staging lineage, duration calculations, managed
   Release tasks, and S3 references.
7. Under HUM-7534, deploy the single-replica production timing worker with `Recreate` strategy and
   its migration init container after staging worker validation.
8. Verify the production Alembic revision, legacy ingestion, timeline queue alerts, and metrics.
9. Add the managed namespace to the production event forwarder, then activate new production sources
   under HUM-7533 after verifying the timing worker is running.
10. Expect each Kubernetes watcher to emit an initial `ADDED` event for every existing matching
    resource. Schedule source activation during a monitored window.
11. Monitor legacy and timeline queue age, dead-letter-queue depth, worker and archiver errors,
    forwarder spool growth, database size, and PVC free space.
12. Run the final controlled RPM acceptance merge request in production before closing HUM-5820.

Rollback does not require dropping or downgrading the schema. Scale the timing worker to zero and
disable the new TaskRun and Job Hook sources in the affected environment. Legacy ingestion
continues. After launch, monitor queue age while paused: messages expire after four days unless the
worker resumes in time. Malformed messages move to the DLQ after failed receives and can be redriven
after repair.

## Epic Completion Criteria

The approved design's acceptance criteria remain authoritative. HUM-5820 is complete when:

1. HUM-7529 through HUM-7535 are complete.
2. The design's acceptance criteria pass in both staging and production.
3. Representative timelines include managed PipelineRun and TaskRun timing from `rhtap-releng-tenant`.
4. The staging-first source rollout and final production acceptance run are documented under HUM-7533.
5. HUM-7533 has verified that each Story's required documentation update is complete.

## Out of Scope

1. Creating a UI for visualization is out of scope.
   - The data may later support a new application, Dashboard changes, or Grafana. UX should be
     involved in that decision.
2. Tracking the containers repository pipeline is out of scope.
   - The pipeline-neutral schema, event contracts, and execution identities are intended to support
     its GitLab and Konflux events in a future follow-up without creating a parallel storage model.
   - The RPM and containers pipelines do not connect directly. A separate scheduled process
     discovers updated RPMs and commits changes to `rpms.lock.yaml` files in the containers
     repository.
3. Timeline replay and historical backfill are out of scope.
   - Existing Hummingbird Status S3 ingestion bootstraps the `public` schema and collapses
     Kubernetes history, so it is not suitable for timeline recovery.
   - Future work must define scoped S3 read permissions and chronological ingestion into the
     Alembic-managed `pipeline_timing` schema. Events that were never collected cannot be
     reconstructed.

## Delivery Stories

| Story                                                                                                               | Responsibility                                                                                                              |
| ------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| [HUM-7529](https://redhat.atlassian.net/browse/HUM-7529) Make RPM pipeline timing data queryable                    | Independent Pipeline Timing image and schema, Alembic, worker, ingestion, lineage, tests, and legacy Status compatibility   |
| [HUM-7530](https://redhat.atlassian.net/browse/HUM-7530) Carry RPM pipeline timing events through the event bus     | GitLab and Kubernetes forwarders, S3 archiver event identities, payload handling, and tests                                 |
| [HUM-7531](https://redhat.atlassian.net/browse/HUM-7531) Validate RPM pipeline timeline storage in staging          | Staging Pipeline Timing deployment, Alembic verification, legacy isolation, and representative queries                      |
| [HUM-7534](https://redhat.atlassian.net/browse/HUM-7534) Make RPM pipeline timeline storage available in production | Production Pipeline Timing deployment with event sources still disabled                                                     |
| [HUM-7535](https://redhat.atlassian.net/browse/HUM-7535) Support environment-specific RPM pipeline event collection | Both environments' timeline queues, dead-letter queues, subscriptions, IAM, alerts, and configuration before worker rollout |
| [HUM-7532](https://redhat.atlassian.net/browse/HUM-7532) Expose managed Release pipeline timing                     | Read-only `rhtap-releng-tenant` access and managed PipelineRun and TaskRun event collection                                 |
| [HUM-7533](https://redhat.atlassian.net/browse/HUM-7533) Roll out RPM pipeline event collection                     | Staging source activation, production source activation, monitoring, recovery, redrive, and end-to-end validation           |

## Dependency Order

1. HUM-7529 and HUM-7530 can proceed in parallel.
2. HUM-8015 is the completed staging message-bus prerequisite. HUM-7535 depends on HUM-7530's event
   contracts and provisions timeline-specific infrastructure in both environments without waiting
   for staging worker acceptance.
3. HUM-7531 depends on HUM-7529, HUM-7530, and the staging path from HUM-7535.
4. HUM-7534 depends on successful staging validation in HUM-7531.
5. HUM-7532 depends on the event support from HUM-7530 and the environment-specific infrastructure
   from HUM-7535. Managed namespace access can proceed alongside the Pipeline Timing deployments,
   but completion gates final acceptance under HUM-7533.
6. HUM-7533 depends on HUM-7531, HUM-7534, HUM-7535, and HUM-7532. HUM-5820 remains open until the
   final rollout is verified.
