Hummingbird Dashboard: RPM upstream rollout tracker
Problem
We lack a clear view of which RPMs are behind upstream, how long they have been behind, and which update MRs are stuck. Staleness is found reactively.
Background
The upstream monitor service discovers, normalizes and
classifies upstream releases. It exposes a released_at timestamp per release,
groups versions into streams, and carries support lifecycle state sourced from
endoflife.date. The timestamp’s meaning depends on the upstream source. It
currently monitors about 200 projects.
The rpms repo package metadata carries upstream_repo and
sometimes track_upstream, upstream_branch, upstream_version_transform, or
version_suffix_strip. The packaged version comes from the spec file. The
existing ci/check_upstream_versions.py reads it with rpmspec, which resolves
macros. A scheduled upstream_version_update CI job bumps specs, fetches
sources, and opens update MRs.
hummingbird-status initializes the shared events PostgreSQL database and
ingests GitLab MRs into its public.gitlab_merge_requests table. The dashboard
reads that table and owns additional tables for its own features. The CVE
Tracker uses a collector POSTing to a token-authenticated endpoint. The RPM
tracker instead follows Pipeline Timing: an independent
schema owner writes directly to its schema in the shared database, and the
dashboard reads the resulting state.
Findings
- All 557 packages carry
upstream_repoin metadata. Read their packaged versions from the specs with the existingparse_spec_versionhelper rather than treating the metadataversionfield as authoritative. - The upstream monitor reports
released_atand the release source. The tracker does not need to fetch release dates from multiple upstream APIs, although a reported date is not always the instant a source became public. - The monitor has about 200 projects, not necessarily 200 of our packages. Package coverage must be counted after matching packages to projects. Its published API already exposes project definitions, paginated release history, stream lifecycle, and project scan status.
- Stream lifecycle (
active,security,extended,unsupported) is exposed per stream. An unsupported upstream stream is a support-risk signal. It does not change our downstream CVE policy on its own. - V1 can use an update MR’s actual merge time as a proxy for rollout when it is available. The planned Phase 2 definition is successful Release completion, confirmed with Benjamin Sherman (HUM-5820). The timeline data model and its resolution path are not yet implemented.
- A pipeline timeline alone cannot identify the target package and version of an update MR. That identity must be resolved from the rpms repo’s update MRs.
Proposal
Mirror upstream state from the monitor into the shared events database,
compare it against our packaged versions, and render an RPM Tracker tab. A
collector maintains the upstream and version rows through tracker-owned
transactional storage. The dashboard reads these rows and resolves open update
MRs from the existing Status data.
Data
RPM tracker owns and migrates its tables in a dedicated
rpm_tracker schema within the shared events database. Table names below
refer to this schema unless qualified otherwise. The original event tables in
public, including public.gitlab_merge_requests, are owned by
hummingbird-status. Keeping these use cases in distinct schemas preserves
table ownership while allowing the dashboard to join tracker and MR data
across schemas in one query. Run the tracker initializer before exposing the
dashboard read API and before each collector rollout; the collector checks the
schema revision before collecting or writing. A failed migration or revision
check stops collection without changing Status ingestion. Postgres can filter
and sort computed staleness alongside the existing MR data.
rpm_upstream_state is a tracker-owned read model. The monitor remains the
authority for upstream releases. Anything else that needs upstream state
consumes the monitor directly rather than reading our mirror.
rpm_upstream_state: Latest eligible upstream state and packaged stream
lifecycle per mapped package, computed from the monitor on every collector run.
Unmonitored packages have no upstream row and appear through a left join with
rpm_version_status.
| Column | Meaning |
|---|---|
package |
rpms repo package name (primary key) |
project_id |
Verified monitor project identifier |
upstream_version |
Latest eligible upstream version |
upstream_released_at |
Monitor timestamp for that release |
upstream_source |
Source reported for the upstream release |
packaged_stream |
Stream containing the packaged version, if known |
lifecycle_status |
Lifecycle of packaged_stream, or unknown |
monitor_last_success_at |
Last successful upstream scan for this project |
monitor_state |
Current upstream monitor scan state |
monitor_last_error |
Latest upstream monitor scan error, if any |
updated_at |
When we last wrote this row |
Map each package’s upstream_repo to a monitor project using its project
definition’s source and a verified repository mapping. If that mapping is not
unique, require an explicit upstream_monitor_project_id in package metadata.
A project ID is not derivable from upstream_repo alone.
Use track_upstream to select eligible releases as the existing update job
does. A version prefix restricts candidates to that line. latest considers
all streams. For a package without track_upstream, use an explicit
upstream_stream metadata override or the unique stream matching its packaged
version. An ambiguous choice yields unknown, not the project’s global latest.
When track_upstream is set, an upstream_stream override can identify the
packaged stream for lifecycle but cannot silently change the update target.
Do not treat upstream_branch as a stream name without a verified mapping.
Resolve packaged_stream independently of the update target so lifecycle
describes the version we package even when we track a newer stream. Respect
the monitor’s prerelease policy and the package’s version_suffix_strip and
upstream_version_transform when comparing with RPM versions. The unrelated
version_transform metadata field is for CVE version matching. If the monitor
already normalized a version in the same way, do not apply a transform twice.
Source artifact availability may delay update automation, but does not change
whether an upstream release is visible in this tracker.
rpm_version_status: One row per package, holding our side of the comparison
and the derived state. Upstream version and release date are read from
rpm_upstream_state via JOIN, and rollout timing from rpm_version_rollouts,
so neither is duplicated here.
| Column | Meaning |
|---|---|
package |
Package name (primary key) |
current_version |
Version currently packaged, from the spec file |
status |
current, behind, unmonitored, unknown |
first_behind_at |
Earliest newer upstream release date, if behind |
checked_at |
When the collector last successfully refreshed this row |
The dashboard resolves the blocking MR from public.gitlab_merge_requests for
display; MR identity and URL are not part of the replaceable inventory. MR
creation time is not stored in that table today, so the Stuck MRs view must
obtain it from GitLab or the MR-owning feed.
current means the packaged spec version is equal to or ahead of the latest
eligible upstream version. behind means an eligible upstream version is newer.
unmonitored means there is no verified monitor project mapping. unknown
means a mapped project exists, but its update target, release history, spec
version, or comparison is unusable.
rpm_sync_state: One row holding the last successful collection time, the rpms
checkout SHA, and a generation number. A successful full inventory replaces
current tracker rows and increments the generation in the same database
transaction. A failed run leaves the previous rows and generation intact.
| Column | Meaning |
|---|---|
id |
Primary key fixed to 1 for the singleton row |
generation |
Integer incremented after each successful ingest |
rpms_checkout_sha |
Default branch commit inspected by the collector |
last_successful_at |
Completion time of the last successful collection |
rpm_version_rollouts: Resolved rollout times, one row per
(package, version). A later tracker-owned reconciliation step can populate
them from the status-owned public.gitlab_merge_requests table. Keeping these
rows separate from the replaceable inventory preserves historical rollouts;
the dashboard reads them but does not write them.
| Column | Meaning |
|---|---|
package |
Package name (composite primary key) |
version |
The version that was rolled out (composite primary key) |
rolled_out_at |
Actual merge time of the matched MR (v1 proxy for publication) |
mr_iid |
IID of the matched MR |
mr_project |
Project of the matched MR |
resolved_at |
When this entry was resolved |
Match merged update MRs in public.gitlab_merge_requests by rpms project,
package, target version, and target branch. The update job uses a source
branch containing the package name and an MR title containing both package and
version. A branch match alone does not identify a version. Use GitLab’s actual
merged_at from the webhook payload when present. Obtain it from GitLab for
older MRs that lack it. Do not substitute the mutable MR updated_at. If the
version or merge time cannot be established, or multiple MRs still match, leave
rolled_out_at null and show rollout lag as unavailable. Manual bumps without
an identifiable package and version remain unresolved rather than silently
matching a different MR. Re-resolve unresolved rows as MR data arrives.
Collection and reconciliation
Every four hours, collect a fresh inventory from the rpms checkout and the
monitor’s existing REST API. GET /v1/projects lists projects.
GET /v1/projects/{projectID}/definition supplies the source configuration
used to verify repository mappings. For each distinct mapped project,
GET /v1/projects/{projectID}/status reports scan health,
GET /v1/projects/{projectID}/streams provides lifecycle, and
GET /v1/projects/{projectID}/releases provides dated release history with
X-Next-Cursor pagination. Fetch every release page and
GET /v1/projects/{projectID} for snapshot releases and observations, including
undated versions omitted from pagination. Merge by source and release ID, with
snapshot records taking precedence, including dates corrected to unknown. Treat
the monitor’s JSON null empty-stream representation as an empty list. Group
releases by project and reuse the results for packages sharing a project. Choose
the latest eligible version with RPM comparison rather than relying on API page
order. The combined releases give the first newer upstream release date; an
eligible newer release with no date makes staleness unavailable. This full read
also sees changed dates, stream lifecycle changes, and
new project mappings without an event cursor or a second database of historical
releases. The monitor’s historical catalog preserves observed releases after
they disappear upstream; a full read alone cannot identify withdrawals. Excluding
withdrawn releases requires an authoritative monitor eligibility signal.
The collector is implemented in rpm-tracker/rpm_tracker/collect.py, with monitor
reads in monitor.py and package selection in policy.py. It reads the expected
generation before cloning the default branch, reuses the checkout’s spec parser
and named transforms, and writes only through store.ingest. See
RPM Tracker for runtime
prerequisites and usage. Deployment
as an OpenShift CronJob remains a subsequent step.
The monitor’s per-project endpoints do not promise one snapshot across all
pages or projects. If any required request or page fails, do not replace the
last successful inventory. If releases change during collection,
the next full run reconciles them. Do not claim that each four-hour run is an
atomic view of all upstream projects. Expose the monitor’s project scan state
and last_success_at alongside the collector’s own freshness so a successful
API read does not conceal failed upstream polling. Prevent overlapping scheduled
runs and check the expected rpm_sync_state generation on ingest, including
runs with no release changes.
Metrics
- Currency (behind / current): compare the spec version against the latest
eligible upstream version with RPM semantics.
behindmeans upstream is newer. An equal or ahead packaged version iscurrent. If versions cannot be compared or the update target cannot be identified, useunknownrather than assumingcurrent. These are source repository statuses, not proof of publication. - Staleness (behind packages): Time since
first_behind_at, the earliest monitorreleased_atamong eligible releases newer than the spec version. A newer upstream release does not reset the clock. This is based on the monitor’s reported dates and does not measure security exposure. If history or its date is missing, show unavailable, not zero. - Rollout lag (current packages whose version equals
upstream_version): Time betweenupstream_released_atandrolled_out_at. An ahead packaged version is stillcurrent, but its lag is unavailable. In v1,rolled_out_atis the matched update MR’s merge time, not proof of successful publication. Historical rollout rows remain stored, but the v1 Rollout view stops showing their lag when a newer upstream release makes the packagebehindagain. - Lifecycle: flag an unsupported upstream stream as support risk. Whether
to fix or close CVEs follows the downstream
fix_statuspolicy in package metadata, not this upstream flag. - Stuck update MRs: all open update MRs older than 24 hours by GitLab MR creation time, independently of the single MR shown on a package row.
Approach
- Deploy the collector as a Kubernetes CronJob on OpenShift. Its
batch/v1schedule starts a Job every four hours, independently ofupstream_version_update. SetconcurrencyPolicy: Forbidto prevent overlapping scheduled runs. Run the tracker schema initializer before collection starts. Clone the rpms repo’s default branch read-only at the start of each run and record the checked commit SHA. - Read
metadata/*.jsonfrom that checkout and parse each package’s spec version with the existingparse_spec_versionhelper. If a spec cannot be parsed, mark that packageunknownrather than trusting a possibly stale metadata version. Map packages to projects using verified monitor definitions, fetch all needed project status, streams, and release history, and compute upstream state andfirst_behind_atwithrpm.labelCompare(provided by the collector image). Create a SQLAlchemy engine and read the generation withstore.get_generation(engine)before cloning or collecting external data. - Pass the complete package inventory, checked checkout SHA, and expected
generation to
store.ingest(engine, request). It rejects an outdated generation withGenerationConflictError, atomically replaces upstream and status rows, and increments the generation. Packages removed from a successfully read default branch checkout disappear. An incomplete fetch must not call ingest; a failed database write leaves the previous inventory unchanged. A conflict fails the Job visibly, and the next run collects a fresh inventory. Grant the CronJob database access limited to tracker-owned tables; use a separate migration role for schema changes. The dashboard remains a read-only consumer of tracker data. - A later tracker-owned reconciliation step resolves
rolled_out_atfor a package/version from matching merged update MRs. The dashboard may show one open update MR on a package row, but the Stuck MRs view queries all open rpms update MRs independently. Obtain MR creation time from the MR-owning feed or GitLab when missing; MRs without a known creation time appear asunknown age, not silently filtered out. - Render an RPM Tracker tab with a Rollout view (version, status, staleness
or rollout lag, lifecycle, blocking MR) and a Stuck MRs view, mirroring
the CVE Tracker, plus JSON API endpoints. Show
checked_at, the rpms checkout SHA, and the last successful collection time. Warn when collection exceeds eight hours without success. Show monitor scan failures and each project’s last successful upstream check separately.unmonitoredmeans no verified monitor project.unknownmeans a mapped project without a usable update target, release, or comparison. On collection failure, retain the last successful state and show its age. - Keep the existing Slack and failure-file output for update automation. Monitor failed or missed collector Jobs in OpenShift and show the last successful collection time in the dashboard.
MR reconciliation refinements
- Add
rpm_update_mrsin the tracker schema, keyed by(project, iid), as the dashboard’s MR read model. Reconciliation reads the Status table and stores package/version identity, MR state and links, explicit creation/merge times, and uncertainty reasons. Preserve merged identity evidence across later edits and retain known timestamps when later payloads omit them. Only currently qualifying MRs are rollout candidates. - Extend
rpm_sync_statewithmr_project,mr_target_branch, andlast_mr_reconciled_at. Fix project/branch scope on the first successful reconciliation and track MR freshness independently of inventory generation and collection freshness. Commit MR and rollout changes under the sync-row lock without changing the inventory generation. - Add
unavailable_reasontorpm_version_rolloutsand defineresolved_atas the time the resolution result last changed. Revisit historical keys after inventory advances and invalidate results when new ambiguity appears. - The initial reconciliation step uses only explicit timestamps in stored Status webhook payloads. The planned GitLab backfill remains follow-up work; missing timestamps stay unavailable until reliable evidence is supplied.
- Expose the independent open-MR listing through
GET /api/rpm-update-mrs, including itsstuck_onlyfilter, and include MR information inGET /api/rpm-upstream-state. Select the single package-row MR from the read model without restricting the independent listing.
Dependencies
- Package registration and mapping. Count actual package coverage by
verifying metadata against monitor definitions. Provision missing projects
separately.
POST /v1/projectsrequires a project ID, name, homepage, source provider configuration, version policy, and stream policy. Anupstream_repoURL alone is not sufficient to register a project. A package without a verified project mapping remainsunmonitored. - Monitor load. The existing API provides paginated release history, project streams, and project scan status. Confirm that a complete collection of mapped projects fits within the four-hour schedule at a bounded request rate. No new monitor endpoint or cursor protocol is required for v1.
- Collector runtime. The CronJob image needs Git,
rpmspec, the Python RPM bindings, and the dependencies needed to use the version parsing helper from the rpms repo. Install the tracker package with its bundled migrations, run the initializer before the collector, and give the CronJob schema-scoped database credentials. Provide a read-only GitLab credential if the repo requires one. - MR timestamps. MR reconciliation needs actual
merged_atand creation times from the MR-owning feed or GitLab for missing historical data. It cannot infer these times from mutableupdated_at.
Acceptance Criteria
- The Rollout view lists every package with its status. Behind packages show time since the first newer upstream release and the blocking update MR. Current packages show merge-time rollout lag only when their version equals the selected upstream version and has a matching MR. Unsupported upstream streams are flagged without changing CVE policy.
- Packages not registered with the monitor are shown as
unmonitored, not omitted or assumed current. Ambiguous stream mappings areunknown. A packaged version equal to or ahead of the latest eligible upstream version iscurrent. A package pinned withtrack_upstreamignores releases outside its pinned line. A package trackinglatestconsiders eligible releases in all streams while lifecycle reflects the packaged stream. - The Stuck MRs view lists every open update MR older than 24 hours with links and creation-based ages, including older MRs hidden by a newer MR on the same package. MRs with unknown creation time remain visible as such.
- The collector runs as a separate CronJob on OpenShift every four hours. It
recomputes every package from the current specs and complete upstream
history without slowing
upstream_version_update. Failed and missed runs are visible through OpenShift and dashboard freshness. - A release arriving during a run appears in that or the next complete run. Failed pagination or ingest retains the last successful inventory. Later runs pick up corrected release dates, lifecycle changes, and newly registered projects without replaying a release feed. Withdrawals require a monitor eligibility signal rather than disappearance from upstream. An outdated ingest generation is rejected even when upstream releases did not change.
- A newer upstream release does not reset staleness. An MR for a different version or with no reliable merge time does not acquire a rollout timestamp.
- Failed collection retains the last successful data with visible age. Monitor scan failures remain visible even after a successful collector run. Removed packages disappear only after a complete inventory run.
- Existing dashboard behavior, the update loop, and existing reporting are unchanged.
Phase 2
Future work, contingent on the independent pipeline-timing service from !1261
providing tested lineage and sufficient data in its pipeline_timing schema.
The joins needed by this tracker remain to be verified. !1261 preserves the
existing public tables and Dashboard reads. There is no planned removal of
public.gitlab_merge_requests that requires this phase.
What changes
Rollout timestamp: Replace the v1 merge-time proxy with completion time of
the first successful publication for the matched package and version. The
future model must support an exact component match, MR merge commit → build
PipelineRun → Snapshot → Release lineage, and identification of the intended
ReleasePlan/destination. A SHA alone or an arbitrary single run is insufficient.
If several builds or releases qualify, choose the earliest successful completion
for that destination. Failed releases and releases to other destinations must
not count. If the lineage or destination is ambiguous, leave rolled_out_at
null. Retry unresolved rows as new timeline events arrive. A version shipped
by a later repair commit needs a documented version-to-release association
rather than being attributed to the original MR by SHA alone. Use
pipeline_timing completion times only where the lineage is verified.
Backfill existing rollout rows and compare results with the v1 merge-time proxy before switching the timestamp. Retain the matched run and Release identifiers for traceability. Settle their column names against the implemented schema.
What stays the same
The collector CronJob, upstream mirror, and public.gitlab_merge_requests as
the source for versioned MR matching and the Stuck MRs view. The bot-convention
matching, rpm_version_status columns, and UI also stay the same. Phase 2 is a
rollout timestamp upgrade with no planned UI restructure.