Hummingbird Dashboard: RPM upstream rollout tracker

Problem

We lack a clear view of which RPMs are behind upstream, how long they have been behind, and which update MRs are stuck. Staleness is found reactively.

Background

The upstream monitor service discovers, normalizes and classifies upstream releases. It exposes a released_at timestamp per release, groups versions into streams, and carries support lifecycle state sourced from endoflife.date. The timestamp’s meaning depends on the upstream source. It currently monitors about 200 projects.

The rpms repo package metadata carries upstream_repo and sometimes track_upstream, upstream_branch, upstream_version_transform, or version_suffix_strip. The packaged version comes from the spec file. The existing ci/check_upstream_versions.py reads it with rpmspec, which resolves macros. A scheduled upstream_version_update CI job bumps specs, fetches sources, and opens update MRs.

hummingbird-status initializes the shared events PostgreSQL database and ingests GitLab MRs into its public.gitlab_merge_requests table. The dashboard reads that table and owns additional tables for its own features. The CVE Tracker uses a collector POSTing to a token-authenticated endpoint. The RPM tracker instead follows Pipeline Timing: an independent schema owner writes directly to its schema in the shared database, and the dashboard reads the resulting state.

Findings

  1. All 557 packages carry upstream_repo in metadata. Read their packaged versions from the specs with the existing parse_spec_version helper rather than treating the metadata version field as authoritative.
  2. The upstream monitor reports released_at and the release source. The tracker does not need to fetch release dates from multiple upstream APIs, although a reported date is not always the instant a source became public.
  3. The monitor has about 200 projects, not necessarily 200 of our packages. Package coverage must be counted after matching packages to projects. Its published API already exposes project definitions, paginated release history, stream lifecycle, and project scan status.
  4. Stream lifecycle (active, security, extended, unsupported) is exposed per stream. An unsupported upstream stream is a support-risk signal. It does not change our downstream CVE policy on its own.
  5. V1 can use an update MR’s actual merge time as a proxy for rollout when it is available. The planned Phase 2 definition is successful Release completion, confirmed with Benjamin Sherman (HUM-5820). The timeline data model and its resolution path are not yet implemented.
  6. A pipeline timeline alone cannot identify the target package and version of an update MR. That identity must be resolved from the rpms repo’s update MRs.

Proposal

Mirror upstream state from the monitor into the shared events database, compare it against our packaged versions, and render an RPM Tracker tab. A collector maintains the upstream and version rows through tracker-owned transactional storage. The dashboard reads these rows and resolves open update MRs from the existing Status data.

Data

RPM tracker owns and migrates its tables in a dedicated rpm_tracker schema within the shared events database. Table names below refer to this schema unless qualified otherwise. The original event tables in public, including public.gitlab_merge_requests, are owned by hummingbird-status. Keeping these use cases in distinct schemas preserves table ownership while allowing the dashboard to join tracker and MR data across schemas in one query. Run the tracker initializer before exposing the dashboard read API and before each collector rollout; the collector checks the schema revision before collecting or writing. A failed migration or revision check stops collection without changing Status ingestion. Postgres can filter and sort computed staleness alongside the existing MR data.

rpm_upstream_state is a tracker-owned read model. The monitor remains the authority for upstream releases. Anything else that needs upstream state consumes the monitor directly rather than reading our mirror.

rpm_upstream_state: Latest eligible upstream state and packaged stream lifecycle per mapped package, computed from the monitor on every collector run. Unmonitored packages have no upstream row and appear through a left join with rpm_version_status.

Column Meaning
package rpms repo package name (primary key)
project_id Verified monitor project identifier
upstream_version Latest eligible upstream version
upstream_released_at Monitor timestamp for that release
upstream_source Source reported for the upstream release
packaged_stream Stream containing the packaged version, if known
lifecycle_status Lifecycle of packaged_stream, or unknown
monitor_last_success_at Last successful upstream scan for this project
monitor_state Current upstream monitor scan state
monitor_last_error Latest upstream monitor scan error, if any
updated_at When we last wrote this row

Map each package’s upstream_repo to a monitor project using its project definition’s source and a verified repository mapping. If that mapping is not unique, require an explicit upstream_monitor_project_id in package metadata. A project ID is not derivable from upstream_repo alone.

Use track_upstream to select eligible releases as the existing update job does. A version prefix restricts candidates to that line. latest considers all streams. For a package without track_upstream, use an explicit upstream_stream metadata override or the unique stream matching its packaged version. An ambiguous choice yields unknown, not the project’s global latest. When track_upstream is set, an upstream_stream override can identify the packaged stream for lifecycle but cannot silently change the update target. Do not treat upstream_branch as a stream name without a verified mapping. Resolve packaged_stream independently of the update target so lifecycle describes the version we package even when we track a newer stream. Respect the monitor’s prerelease policy and the package’s version_suffix_strip and upstream_version_transform when comparing with RPM versions. The unrelated version_transform metadata field is for CVE version matching. If the monitor already normalized a version in the same way, do not apply a transform twice. Source artifact availability may delay update automation, but does not change whether an upstream release is visible in this tracker.

rpm_version_status: One row per package, holding our side of the comparison and the derived state. Upstream version and release date are read from rpm_upstream_state via JOIN, and rollout timing from rpm_version_rollouts, so neither is duplicated here.

Column Meaning
package Package name (primary key)
current_version Version currently packaged, from the spec file
status current, behind, unmonitored, unknown
first_behind_at Earliest newer upstream release date, if behind
checked_at When the collector last successfully refreshed this row

The dashboard resolves the blocking MR from public.gitlab_merge_requests for display; MR identity and URL are not part of the replaceable inventory. MR creation time is not stored in that table today, so the Stuck MRs view must obtain it from GitLab or the MR-owning feed.

current means the packaged spec version is equal to or ahead of the latest eligible upstream version. behind means an eligible upstream version is newer. unmonitored means there is no verified monitor project mapping. unknown means a mapped project exists, but its update target, release history, spec version, or comparison is unusable.

rpm_sync_state: One row holding the last successful collection time, the rpms checkout SHA, and a generation number. A successful full inventory replaces current tracker rows and increments the generation in the same database transaction. A failed run leaves the previous rows and generation intact.

Column Meaning
id Primary key fixed to 1 for the singleton row
generation Integer incremented after each successful ingest
rpms_checkout_sha Default branch commit inspected by the collector
last_successful_at Completion time of the last successful collection

rpm_version_rollouts: Resolved rollout times, one row per (package, version). A later tracker-owned reconciliation step can populate them from the status-owned public.gitlab_merge_requests table. Keeping these rows separate from the replaceable inventory preserves historical rollouts; the dashboard reads them but does not write them.

Column Meaning
package Package name (composite primary key)
version The version that was rolled out (composite primary key)
rolled_out_at Actual merge time of the matched MR (v1 proxy for publication)
mr_iid IID of the matched MR
mr_project Project of the matched MR
resolved_at When this entry was resolved

Match merged update MRs in public.gitlab_merge_requests by rpms project, package, target version, and target branch. The update job uses a source branch containing the package name and an MR title containing both package and version. A branch match alone does not identify a version. Use GitLab’s actual merged_at from the webhook payload when present. Obtain it from GitLab for older MRs that lack it. Do not substitute the mutable MR updated_at. If the version or merge time cannot be established, or multiple MRs still match, leave rolled_out_at null and show rollout lag as unavailable. Manual bumps without an identifiable package and version remain unresolved rather than silently matching a different MR. Re-resolve unresolved rows as MR data arrives.

Collection and reconciliation

Every four hours, collect a fresh inventory from the rpms checkout and the monitor’s existing REST API. GET /v1/projects lists projects. GET /v1/projects/{projectID}/definition supplies the source configuration used to verify repository mappings. For each distinct mapped project, GET /v1/projects/{projectID}/status reports scan health, GET /v1/projects/{projectID}/streams provides lifecycle, and GET /v1/projects/{projectID}/releases provides dated release history with X-Next-Cursor pagination. Fetch every release page and GET /v1/projects/{projectID} for snapshot releases and observations, including undated versions omitted from pagination. Merge by source and release ID, with snapshot records taking precedence, including dates corrected to unknown. Treat the monitor’s JSON null empty-stream representation as an empty list. Group releases by project and reuse the results for packages sharing a project. Choose the latest eligible version with RPM comparison rather than relying on API page order. The combined releases give the first newer upstream release date; an eligible newer release with no date makes staleness unavailable. This full read also sees changed dates, stream lifecycle changes, and new project mappings without an event cursor or a second database of historical releases. The monitor’s historical catalog preserves observed releases after they disappear upstream; a full read alone cannot identify withdrawals. Excluding withdrawn releases requires an authoritative monitor eligibility signal.

The collector is implemented in rpm-tracker/rpm_tracker/collect.py, with monitor reads in monitor.py and package selection in policy.py. It reads the expected generation before cloning the default branch, reuses the checkout’s spec parser and named transforms, and writes only through store.ingest. See RPM Tracker for runtime prerequisites and usage. Deployment as an OpenShift CronJob remains a subsequent step.

The monitor’s per-project endpoints do not promise one snapshot across all pages or projects. If any required request or page fails, do not replace the last successful inventory. If releases change during collection, the next full run reconciles them. Do not claim that each four-hour run is an atomic view of all upstream projects. Expose the monitor’s project scan state and last_success_at alongside the collector’s own freshness so a successful API read does not conceal failed upstream polling. Prevent overlapping scheduled runs and check the expected rpm_sync_state generation on ingest, including runs with no release changes.

Metrics

  1. Currency (behind / current): compare the spec version against the latest eligible upstream version with RPM semantics. behind means upstream is newer. An equal or ahead packaged version is current. If versions cannot be compared or the update target cannot be identified, use unknown rather than assuming current. These are source repository statuses, not proof of publication.
  2. Staleness (behind packages): Time since first_behind_at, the earliest monitor released_at among eligible releases newer than the spec version. A newer upstream release does not reset the clock. This is based on the monitor’s reported dates and does not measure security exposure. If history or its date is missing, show unavailable, not zero.
  3. Rollout lag (current packages whose version equals upstream_version): Time between upstream_released_at and rolled_out_at. An ahead packaged version is still current, but its lag is unavailable. In v1, rolled_out_at is the matched update MR’s merge time, not proof of successful publication. Historical rollout rows remain stored, but the v1 Rollout view stops showing their lag when a newer upstream release makes the package behind again.
  4. Lifecycle: flag an unsupported upstream stream as support risk. Whether to fix or close CVEs follows the downstream fix_status policy in package metadata, not this upstream flag.
  5. Stuck update MRs: all open update MRs older than 24 hours by GitLab MR creation time, independently of the single MR shown on a package row.

Approach

  1. Deploy the collector as a Kubernetes CronJob on OpenShift. Its batch/v1 schedule starts a Job every four hours, independently of upstream_version_update. Set concurrencyPolicy: Forbid to prevent overlapping scheduled runs. Run the tracker schema initializer before collection starts. Clone the rpms repo’s default branch read-only at the start of each run and record the checked commit SHA.
  2. Read metadata/*.json from that checkout and parse each package’s spec version with the existing parse_spec_version helper. If a spec cannot be parsed, mark that package unknown rather than trusting a possibly stale metadata version. Map packages to projects using verified monitor definitions, fetch all needed project status, streams, and release history, and compute upstream state and first_behind_at with rpm.labelCompare (provided by the collector image). Create a SQLAlchemy engine and read the generation with store.get_generation(engine) before cloning or collecting external data.
  3. Pass the complete package inventory, checked checkout SHA, and expected generation to store.ingest(engine, request). It rejects an outdated generation with GenerationConflictError, atomically replaces upstream and status rows, and increments the generation. Packages removed from a successfully read default branch checkout disappear. An incomplete fetch must not call ingest; a failed database write leaves the previous inventory unchanged. A conflict fails the Job visibly, and the next run collects a fresh inventory. Grant the CronJob database access limited to tracker-owned tables; use a separate migration role for schema changes. The dashboard remains a read-only consumer of tracker data.
  4. A later tracker-owned reconciliation step resolves rolled_out_at for a package/version from matching merged update MRs. The dashboard may show one open update MR on a package row, but the Stuck MRs view queries all open rpms update MRs independently. Obtain MR creation time from the MR-owning feed or GitLab when missing; MRs without a known creation time appear as unknown age, not silently filtered out.
  5. Render an RPM Tracker tab with a Rollout view (version, status, staleness or rollout lag, lifecycle, blocking MR) and a Stuck MRs view, mirroring the CVE Tracker, plus JSON API endpoints. Show checked_at, the rpms checkout SHA, and the last successful collection time. Warn when collection exceeds eight hours without success. Show monitor scan failures and each project’s last successful upstream check separately. unmonitored means no verified monitor project. unknown means a mapped project without a usable update target, release, or comparison. On collection failure, retain the last successful state and show its age.
  6. Keep the existing Slack and failure-file output for update automation. Monitor failed or missed collector Jobs in OpenShift and show the last successful collection time in the dashboard.

MR reconciliation refinements

  • Add rpm_update_mrs in the tracker schema, keyed by (project, iid), as the dashboard’s MR read model. Reconciliation reads the Status table and stores package/version identity, MR state and links, explicit creation/merge times, and uncertainty reasons. Preserve merged identity evidence across later edits and retain known timestamps when later payloads omit them. Only currently qualifying MRs are rollout candidates.
  • Extend rpm_sync_state with mr_project, mr_target_branch, and last_mr_reconciled_at. Fix project/branch scope on the first successful reconciliation and track MR freshness independently of inventory generation and collection freshness. Commit MR and rollout changes under the sync-row lock without changing the inventory generation.
  • Add unavailable_reason to rpm_version_rollouts and define resolved_at as the time the resolution result last changed. Revisit historical keys after inventory advances and invalidate results when new ambiguity appears.
  • The initial reconciliation step uses only explicit timestamps in stored Status webhook payloads. The planned GitLab backfill remains follow-up work; missing timestamps stay unavailable until reliable evidence is supplied.
  • Expose the independent open-MR listing through GET /api/rpm-update-mrs, including its stuck_only filter, and include MR information in GET /api/rpm-upstream-state. Select the single package-row MR from the read model without restricting the independent listing.

Dependencies

  1. Package registration and mapping. Count actual package coverage by verifying metadata against monitor definitions. Provision missing projects separately. POST /v1/projects requires a project ID, name, homepage, source provider configuration, version policy, and stream policy. An upstream_repo URL alone is not sufficient to register a project. A package without a verified project mapping remains unmonitored.
  2. Monitor load. The existing API provides paginated release history, project streams, and project scan status. Confirm that a complete collection of mapped projects fits within the four-hour schedule at a bounded request rate. No new monitor endpoint or cursor protocol is required for v1.
  3. Collector runtime. The CronJob image needs Git, rpmspec, the Python RPM bindings, and the dependencies needed to use the version parsing helper from the rpms repo. Install the tracker package with its bundled migrations, run the initializer before the collector, and give the CronJob schema-scoped database credentials. Provide a read-only GitLab credential if the repo requires one.
  4. MR timestamps. MR reconciliation needs actual merged_at and creation times from the MR-owning feed or GitLab for missing historical data. It cannot infer these times from mutable updated_at.

Acceptance Criteria

  1. The Rollout view lists every package with its status. Behind packages show time since the first newer upstream release and the blocking update MR. Current packages show merge-time rollout lag only when their version equals the selected upstream version and has a matching MR. Unsupported upstream streams are flagged without changing CVE policy.
  2. Packages not registered with the monitor are shown as unmonitored, not omitted or assumed current. Ambiguous stream mappings are unknown. A packaged version equal to or ahead of the latest eligible upstream version is current. A package pinned with track_upstream ignores releases outside its pinned line. A package tracking latest considers eligible releases in all streams while lifecycle reflects the packaged stream.
  3. The Stuck MRs view lists every open update MR older than 24 hours with links and creation-based ages, including older MRs hidden by a newer MR on the same package. MRs with unknown creation time remain visible as such.
  4. The collector runs as a separate CronJob on OpenShift every four hours. It recomputes every package from the current specs and complete upstream history without slowing upstream_version_update. Failed and missed runs are visible through OpenShift and dashboard freshness.
  5. A release arriving during a run appears in that or the next complete run. Failed pagination or ingest retains the last successful inventory. Later runs pick up corrected release dates, lifecycle changes, and newly registered projects without replaying a release feed. Withdrawals require a monitor eligibility signal rather than disappearance from upstream. An outdated ingest generation is rejected even when upstream releases did not change.
  6. A newer upstream release does not reset staleness. An MR for a different version or with no reliable merge time does not acquire a rollout timestamp.
  7. Failed collection retains the last successful data with visible age. Monitor scan failures remain visible even after a successful collector run. Removed packages disappear only after a complete inventory run.
  8. Existing dashboard behavior, the update loop, and existing reporting are unchanged.

Phase 2

Future work, contingent on the independent pipeline-timing service from !1261 providing tested lineage and sufficient data in its pipeline_timing schema. The joins needed by this tracker remain to be verified. !1261 preserves the existing public tables and Dashboard reads. There is no planned removal of public.gitlab_merge_requests that requires this phase.

What changes

Rollout timestamp: Replace the v1 merge-time proxy with completion time of the first successful publication for the matched package and version. The future model must support an exact component match, MR merge commit → build PipelineRun → Snapshot → Release lineage, and identification of the intended ReleasePlan/destination. A SHA alone or an arbitrary single run is insufficient. If several builds or releases qualify, choose the earliest successful completion for that destination. Failed releases and releases to other destinations must not count. If the lineage or destination is ambiguous, leave rolled_out_at null. Retry unresolved rows as new timeline events arrive. A version shipped by a later repair commit needs a documented version-to-release association rather than being attributed to the original MR by SHA alone. Use pipeline_timing completion times only where the lineage is verified.

Backfill existing rollout rows and compare results with the v1 merge-time proxy before switching the timestamp. Retain the matched run and Release identifiers for traceability. Settle their column names against the implemented schema.

What stays the same

The collector CronJob, upstream mirror, and public.gitlab_merge_requests as the source for versioned MR matching and the Stuck MRs view. The bot-convention matching, rpm_version_status columns, and UI also stay the same. Phase 2 is a rollout timestamp upgrade with no planned UI restructure.