RPM pipeline timing design
Problem
We lack a clear way to see where time is spent in the RPM build pipeline.
Background
References: HUM-5819 - Create a diagram that describes all phases and steps of various pipelines from start to finish. Then review with team. HUM-5820 - Time breakdown from MR creation to completion of Release pipeline.
During the F44 rebuild, RPM builds had problems whose causes were not immediately clear. The sprint retrospective and a later F44-specific retrospective created HUM-5819 and HUM-5820 to improve visibility into pipeline behavior and time spent in each phase.
An initial pass through HUM-5819 identified key areas for time tracking. HUM-5820 records those areas.
Findings
The team initially considered a small addition to the tools/hummingbird-status database. A draft
MR explored that approach.
Findings from the draft MR:
- The existing Status events database is a suitable home because the system forwards GitLab and Konflux events through SNS, archives them in S3, and updates PostgreSQL. The timing worker runs as an independent service with its own image and deployment lifecycle.
- Some
hummingbird-dashboardfunctions do directly query therawJSON. - The original purpose of
hummingbird-statuswas to diagnose Konflux event issues, not long-term system diagnosis. - The
rawevent JSON for Konflux PipelineRuns consumes significant space, about 97% of the 47 GB database, despite redundant event storage in S3. hummingbird-statusPVC monitoring is now in place; the draft MR highlighted its absence.- The solution to our problem statement is not to mimic the existing approach.
Proposal
Track required event data in new tables in the existing database and update event processing to populate them without changing existing tables or Dashboard behavior.
Events Already Agreed Upon for Tracking
HUM-5820 outlines the events already agreed upon for tracking.
- GitLab MRs and commits
- GitLab pipelines
- GitLab pipeline jobs
- Konflux PipelineRuns
- Konflux Tekton TaskRuns
The model must support reporting of:
- Status
- Relevant start and end times, including MR creation and merge times
- Event timestamps
- Event lineage across GitLab and Konflux, sufficient to reconstruct a single pipeline timeline
This is not a database design. It identifies the reporting requirements that detailed design must meet.
Approach
- Preserve existing event ingestion, database tables, and Dashboard behavior. Run timing ingestion
in a separate
pipeline-timingservice and image while retaining the existing publishers, archiver, and events database. - Create a new schema in the existing PostgreSQL database.
- The schema provides logical isolation so existing consumers do not accidentally use the new tables.
- Create tables that hold the data needed to meet the agreed reporting requirements without duplicating raw event JSON.
- Treat the S3 archive as the authoritative raw-event record for the new schema.
- Store a reference that lets consumers retrieve the archived event for detail, display, or later data-model improvements.
- Configure GitLab webhooks and the Kubernetes event forwarder to include the agreed event types, then route them through the existing SNS event pipeline.
- Ensure the S3 archiver can archive and identify every agreed event type.
Acceptance Criteria
- Existing Status database tables and Dashboard behavior remain unchanged.
- The new PostgreSQL schema provides the agreed event coverage.
- Representative data demonstrates an RPM pipeline timeline from MR creation through completion of the Release pipeline.
- Consumers can calculate time spent between tracked pipeline phases without querying raw event JSON.
- Consumers can trace lineage across all agreed event types and retrieve the corresponding authoritative S3 event when needed.
- Tests verify the new data model, lineage, and S3 references.
Out of Scope
- Creating a UI for visualization is out of scope.
- The data may later support a new application, Dashboard changes, or Grafana. UX should be involved in that decision.
- Tracking the containers repository pipeline is out of scope.
- That pipeline builds OCI images from RPMs produced by the RPM pipeline and is a potential follow-up effort.