# Hummingbird Dashboard: RPM upstream rollout tracker

LLMS index: [llms.txt](https://hummingbird-project.io/llms.txt) | Full content: [llms-full.txt](https://hummingbird-project.io/llms-full.txt)

---

## Problem

We lack a clear view of which RPMs are behind upstream, how long they have been
behind, and which update MRs are stuck. Staleness is found reactively.

## Background

The [upstream monitor][upstream-monitor] service discovers, normalizes and
classifies upstream releases. It exposes a `released_at` timestamp per release,
groups versions into streams, and carries support lifecycle state sourced from
endoflife.date. The timestamp's meaning depends on the upstream source. It
currently monitors about 200 projects.

The [rpms repo][rpms-repo] package metadata carries `upstream_repo` and
sometimes `track_upstream`, `upstream_branch`, `upstream_version_transform`, or
`version_suffix_strip`. The packaged version comes from the spec file. The
existing `ci/check_upstream_versions.py` reads it with `rpmspec`, which resolves
macros. A scheduled `upstream_version_update` CI job bumps specs, fetches
sources, and opens update MRs.

`hummingbird-status` initializes the shared `events` PostgreSQL database and
ingests GitLab MRs into its `public.gitlab_merge_requests` table. The dashboard
reads that table and owns additional tables for its own features. The CVE
Tracker uses a collector POSTing to a token-authenticated endpoint. The RPM
tracker instead follows [Pipeline Timing](https://hummingbird-project.io/l/pipeline-timing): an independent
schema owner writes directly to its schema in the shared database, and the
dashboard reads the resulting state.

## Findings

1. All 557 packages carry `upstream_repo` in metadata. Read their packaged
   versions from the specs with the existing `parse_spec_version` helper rather
   than treating the metadata `version` field as authoritative.
2. The upstream monitor reports `released_at` and the release source. The
   tracker does not need to fetch release dates from multiple upstream APIs,
   although a reported date is not always the instant a source became public.
3. The monitor has about 200 projects, not necessarily 200 of our packages.
   Package coverage must be counted after matching packages to projects. Its
   published API already exposes project definitions, paginated release
   history, stream lifecycle, and project scan status.
4. Stream lifecycle (`active`, `security`, `extended`, `unsupported`) is exposed
   per stream. An unsupported upstream stream is a support-risk signal. It does
   not change our downstream CVE policy on its own.
5. V1 can use an update MR's actual merge time as a proxy for rollout when it is
   available. The planned Phase 2 definition is successful Release completion,
   confirmed with Benjamin Sherman ([HUM-5820][hum-5820]). The timeline data
   model and its resolution path are not yet implemented.
6. A pipeline timeline alone cannot identify the target package and version of
   an update MR. That identity must be resolved from the rpms repo's update MRs.

## Proposal

Mirror upstream state from the monitor into the shared `events` database,
compare it against our packaged versions, and render an **RPM Tracker** tab. A
collector maintains the upstream and version rows through tracker-owned
transactional storage. The dashboard reads these rows and resolves open update
MRs from the existing Status data.

### Data

RPM tracker owns and migrates its tables in a dedicated
`rpm_tracker` schema within the shared `events` database. Table names below
refer to this schema unless qualified otherwise. The original event tables in
`public`, including `public.gitlab_merge_requests`, are owned by
`hummingbird-status`. Keeping these use cases in distinct schemas preserves
table ownership while allowing the dashboard to join tracker and MR data
across schemas in one query. Run the tracker initializer before exposing the
dashboard read API and before each collector rollout; the collector checks the
schema revision before collecting or writing. A failed migration or revision
check stops collection without changing Status ingestion. Postgres can filter
and sort computed staleness alongside the existing MR data.

`rpm_upstream_state` is a tracker-owned read model. The monitor remains the
authority for upstream releases. Anything else that needs upstream state
consumes the monitor directly rather than reading our mirror.

`rpm_upstream_state`: Latest eligible upstream state and packaged stream
lifecycle per mapped package, computed from the monitor on every collector run.
Unmonitored packages have no upstream row and appear through a left join with
`rpm_version_status`.

| Column                    | Meaning                                          |
| ------------------------- | ------------------------------------------------ |
| `package`                 | rpms repo package name (primary key)             |
| `project_id`              | Verified monitor project identifier              |
| `upstream_version`        | Latest eligible upstream version                 |
| `upstream_released_at`    | Monitor timestamp for that release               |
| `upstream_source`         | Source reported for the upstream release         |
| `packaged_stream`         | Stream containing the packaged version, if known |
| `lifecycle_status`        | Lifecycle of `packaged_stream`, or unknown       |
| `monitor_last_success_at` | Last successful upstream scan for this project   |
| `monitor_state`           | Current upstream monitor scan state              |
| `monitor_last_error`      | Latest upstream monitor scan error, if any       |
| `updated_at`              | When we last wrote this row                      |

Map each package's `upstream_repo` to a monitor project using its project
definition's source and a verified repository mapping. If that mapping is not
unique, require an explicit `upstream_monitor_project_id` in package metadata.
A project ID is not derivable from `upstream_repo` alone.

Use `track_upstream` to select eligible releases as the existing update job
does. A version prefix restricts candidates to that line. `latest` considers
all streams. For a package without `track_upstream`, use an explicit
`upstream_stream` metadata override or the unique stream matching its packaged
version. An ambiguous choice yields `unknown`, not the project's global latest.
When `track_upstream` is set, an `upstream_stream` override can identify the
packaged stream for lifecycle but cannot silently change the update target.
Do not treat `upstream_branch` as a stream name without a verified mapping.
Resolve `packaged_stream` independently of the update target so lifecycle
describes the version we package even when we track a newer stream. Respect
the monitor's prerelease policy and the package's `version_suffix_strip` and
`upstream_version_transform` when comparing with RPM versions. The unrelated
`version_transform` metadata field is for CVE version matching. If the monitor
already normalized a version in the same way, do not apply a transform twice.
Source artifact availability may delay update automation, but does not change
whether an upstream release is visible in this tracker.

`rpm_version_status`: One row per package, holding our side of the comparison
and the derived state. Upstream version and release date are read from
`rpm_upstream_state` via JOIN, and rollout timing from `rpm_version_rollouts`,
so neither is duplicated here.

| Column            | Meaning                                                 |
| ----------------- | ------------------------------------------------------- |
| `package`         | Package name (primary key)                              |
| `current_version` | Version currently packaged, from the spec file          |
| `status`          | current, behind, unmonitored, unknown                   |
| `first_behind_at` | Earliest newer upstream release date, if behind         |
| `checked_at`      | When the collector last successfully refreshed this row |

The dashboard resolves the blocking MR from `public.gitlab_merge_requests` for
display; MR identity and URL are not part of the replaceable inventory. MR
creation time is not stored in that table today, so the Stuck MRs view must
obtain it from GitLab or the MR-owning feed.

`current` means the packaged spec version is equal to or ahead of the latest
eligible upstream version. `behind` means an eligible upstream version is newer.
`unmonitored` means there is no verified monitor project mapping. `unknown`
means a mapped project exists, but its update target, release history, spec
version, or comparison is unusable.

`rpm_sync_state`: One row holding the last successful collection time, the rpms
checkout SHA, and a generation number. A successful full inventory replaces
current tracker rows and increments the generation in the same database
transaction. A failed run leaves the previous rows and generation intact.

| Column               | Meaning                                           |
| -------------------- | ------------------------------------------------- |
| `id`                 | Primary key fixed to 1 for the singleton row      |
| `generation`         | Integer incremented after each successful ingest  |
| `rpms_checkout_sha`  | Default branch commit inspected by the collector  |
| `last_successful_at` | Completion time of the last successful collection |

`rpm_version_rollouts`: Resolved rollout times, one row per
`(package, version)`. A later tracker-owned reconciliation step can populate
them from the status-owned `public.gitlab_merge_requests` table. Keeping these
rows separate from the replaceable inventory preserves historical rollouts;
the dashboard reads them but does not write them.

| Column          | Meaning                                                        |
| --------------- | -------------------------------------------------------------- |
| `package`       | Package name (composite primary key)                           |
| `version`       | The version that was rolled out (composite primary key)        |
| `rolled_out_at` | Actual merge time of the matched MR (v1 proxy for publication) |
| `mr_iid`        | IID of the matched MR                                          |
| `mr_project`    | Project of the matched MR                                      |
| `resolved_at`   | When this entry was resolved                                   |

Match merged update MRs in `public.gitlab_merge_requests` by rpms project,
package, **target version**, and target branch. The update job uses a source
branch containing the package name and an MR title containing both package and
version. A branch match alone does not identify a version. Use GitLab's actual
`merged_at` from the webhook payload when present. Obtain it from GitLab for
older MRs that lack it. Do not substitute the mutable MR `updated_at`. If the
version or merge time cannot be established, or multiple MRs still match, leave
`rolled_out_at` null and show rollout lag as unavailable. Manual bumps without
an identifiable package and version remain unresolved rather than silently
matching a different MR. Re-resolve unresolved rows as MR data arrives.

### Collection and reconciliation

Every four hours, collect a fresh inventory from the rpms checkout and the
monitor's existing REST API. `GET /v1/projects` lists projects.
`GET /v1/projects/{projectID}/definition` supplies the source configuration
used to verify repository mappings. For each distinct mapped project,
`GET /v1/projects/{projectID}/status` reports scan health,
`GET /v1/projects/{projectID}/streams` provides lifecycle, and
`GET /v1/projects/{projectID}/releases` provides dated release history with
`X-Next-Cursor` pagination. Fetch every release page and
`GET /v1/projects/{projectID}` for snapshot releases and observations, including
undated versions omitted from pagination. Merge by source and release ID, with
snapshot records taking precedence, including dates corrected to unknown. Treat
the monitor's JSON `null` empty-stream representation as an empty list. Group
releases by project and reuse the results for packages sharing a project. Choose
the latest eligible version with RPM comparison rather than relying on API page
order. The combined releases give the first newer upstream release date; an
eligible newer release with no date makes staleness unavailable. This full read
also sees changed dates, stream lifecycle changes, and
new project mappings without an event cursor or a second database of historical
releases. The monitor's historical catalog preserves observed releases after
they disappear upstream; a full read alone cannot identify withdrawals. Excluding
withdrawn releases requires an authoritative monitor eligibility signal.

The collector is implemented in `rpm-tracker/rpm_tracker/collect.py`, with monitor
reads in `monitor.py` and package selection in `policy.py`. It reads the expected
generation before cloning the default branch, reuses the checkout's spec parser
and named transforms, and writes only through `store.ingest`. See
[RPM Tracker](https://hummingbird-project.io/docs/background/tools/rpm-tracker/) for runtime
prerequisites and usage. Deployment
as an OpenShift CronJob remains a subsequent step.

The monitor's per-project endpoints do not promise one snapshot across all
pages or projects. If any required request or page fails, do not replace the
last successful inventory. If releases change during collection,
the next full run reconciles them. Do not claim that each four-hour run is an
atomic view of all upstream projects. Expose the monitor's project scan state
and `last_success_at` alongside the collector's own freshness so a successful
API read does not conceal failed upstream polling. Prevent overlapping scheduled
runs and check the expected `rpm_sync_state` generation on ingest, including
runs with no release changes.

### Metrics

1. **Currency** (behind / current): compare the spec version against the latest
   eligible upstream version with RPM semantics. `behind` means upstream is
   newer. An equal or ahead packaged version is `current`. If versions cannot
   be compared or the update target cannot be identified, use `unknown` rather
   than assuming `current`. These are source repository statuses, not proof of
   publication.
2. **Staleness** (behind packages): Time since `first_behind_at`, the earliest
   monitor `released_at` among eligible releases newer than the spec version.
   A newer upstream release does not reset the clock. This is based on the
   monitor's reported dates and does not measure security exposure. If history
   or its date is missing, show unavailable, not zero.
3. **Rollout lag** (current packages whose version equals `upstream_version`):
   Time between `upstream_released_at` and `rolled_out_at`. An ahead packaged
   version is still `current`, but its lag is unavailable. In v1,
   `rolled_out_at` is the matched update MR's merge time, not proof of
   successful publication. Historical rollout rows remain stored, but the v1
   Rollout view stops showing their lag when a newer upstream release makes
   the package `behind` again.
4. **Lifecycle**: flag an unsupported upstream stream as support risk. Whether
   to fix or close CVEs follows the downstream `fix_status` policy in package
   metadata, not this upstream flag.
5. **Stuck update MRs**: all open update MRs older than 24 hours by GitLab MR
   creation time, independently of the single MR shown on a package row.

### Approach

1. Deploy the collector as a **Kubernetes CronJob on OpenShift**. Its `batch/v1`
   schedule starts a Job every four hours, independently of
   `upstream_version_update`. Set `concurrencyPolicy: Forbid` to prevent
   overlapping scheduled runs. Run the tracker schema initializer before
   collection starts. Clone the rpms repo's default branch read-only at the
   start of each run and record the checked commit SHA.
2. Read `metadata/*.json` from that checkout and parse each
   package's spec version with the existing `parse_spec_version` helper. If a
   spec cannot be parsed, mark that package `unknown` rather than trusting a
   possibly stale metadata version. Map packages to projects using verified
   monitor definitions, fetch all needed project status, streams, and release
   history, and compute upstream state and `first_behind_at` with
   `rpm.labelCompare` (provided by the collector image). Create a SQLAlchemy
   engine and read the generation with `store.get_generation(engine)` before
   cloning or collecting external data.
3. Pass the complete package inventory, checked checkout SHA, and expected
   generation to `store.ingest(engine, request)`. It rejects an outdated generation
   with `GenerationConflictError`, atomically replaces upstream and status rows,
   and increments the generation. Packages removed from a successfully read
   default branch checkout disappear. An incomplete fetch must not call ingest;
   a failed database write leaves the previous inventory unchanged. A conflict
   fails the Job visibly, and the next run collects a fresh inventory. Grant the
   CronJob database access limited to tracker-owned tables; use a separate
   migration role for schema changes. The dashboard remains a read-only consumer
   of tracker data.
4. A later tracker-owned reconciliation step resolves `rolled_out_at` for a
   package/version from matching merged update MRs. The dashboard may show one
   open update MR on a package row, but the **Stuck MRs** view queries all open
   rpms update MRs independently. Obtain MR creation time from the MR-owning
   feed or GitLab when missing; MRs without a known creation time appear as
   `unknown age`, not silently filtered out.
5. Render an RPM Tracker tab with a **Rollout** view (version, status, staleness
   or rollout lag, lifecycle, blocking MR) and a **Stuck MRs** view, mirroring
   the CVE Tracker, plus JSON API endpoints. Show `checked_at`, the rpms
   checkout SHA, and the last successful collection time. Warn when collection
   exceeds eight hours without success. Show monitor scan failures and each
   project's last successful upstream check separately. `unmonitored` means
   no verified monitor project. `unknown` means a mapped project without a
   usable update target, release, or comparison. On collection failure,
   retain the last successful state and show its age.
6. Keep the existing Slack and failure-file output for update automation.
   Monitor failed or missed collector Jobs in OpenShift and show the last
   successful collection time in the dashboard.

### MR reconciliation refinements

- Add `rpm_update_mrs` in the tracker schema, keyed by `(project, iid)`, as the
  dashboard's MR read model. Reconciliation reads the Status table and stores
  package/version identity, MR state and links, explicit creation/merge times,
  and uncertainty reasons. Preserve merged identity evidence across later edits
  and retain known timestamps when later payloads omit them. Only currently
  qualifying MRs are rollout candidates.
- Extend `rpm_sync_state` with `mr_project`, `mr_target_branch`, and
  `last_mr_reconciled_at`. Fix project/branch scope on the first successful
  reconciliation and track MR freshness independently of inventory generation
  and collection freshness. Commit MR and rollout changes under the sync-row
  lock without changing the inventory generation.
- Add `unavailable_reason` to `rpm_version_rollouts` and define `resolved_at` as
  the time the resolution result last changed. Revisit historical keys after
  inventory advances and invalidate results when new ambiguity appears.
- The initial reconciliation step uses only explicit timestamps in stored
  Status webhook payloads. The planned GitLab backfill remains follow-up work;
  missing timestamps stay unavailable until reliable evidence is supplied.
- Expose the independent open-MR listing through `GET /api/rpm-update-mrs`,
  including its `stuck_only` filter, and include MR information in
  `GET /api/rpm-upstream-state`. Select the single package-row MR from the read
  model without restricting the independent listing.

## Dependencies

1. **Package registration and mapping.** Count actual package coverage by
   verifying metadata against monitor definitions. Provision missing projects
   separately. `POST /v1/projects` requires a project ID, name, homepage,
   source provider configuration, version policy, and stream policy. An
   `upstream_repo` URL alone is not sufficient to register a project. A package
   without a verified project mapping remains `unmonitored`.
2. **Monitor load.** The existing API provides paginated release history,
   project streams, and project scan status. Confirm that a complete collection
   of mapped projects fits within the four-hour schedule at a bounded request
   rate. No new monitor endpoint or cursor protocol is required for v1.
3. **Collector runtime.** The CronJob image needs Git, `rpmspec`, the Python RPM
   bindings, and the dependencies needed to use the version parsing helper
   from the rpms repo. Install the tracker package with its bundled migrations,
   run the initializer before the collector, and give the CronJob schema-scoped
   database credentials.
   Provide a read-only GitLab credential if the repo requires one.
4. **MR timestamps.** MR reconciliation needs actual `merged_at` and creation
   times from the MR-owning feed or GitLab for missing historical data. It
   cannot infer these times from mutable `updated_at`.

## Acceptance Criteria

1. The Rollout view lists every package with its status. Behind packages show
   time since the **first** newer upstream release and the blocking update MR.
   Current packages show merge-time rollout lag only when their version equals
   the selected upstream version and has a matching MR. Unsupported upstream
   streams are flagged without changing CVE policy.
2. Packages not registered with the monitor are shown as `unmonitored`, not
   omitted or assumed current. Ambiguous stream mappings are `unknown`. A
   packaged version equal to or ahead of the latest eligible upstream version
   is `current`. A package pinned with `track_upstream` ignores releases outside
   its pinned line. A package tracking `latest` considers eligible releases in
   all streams while lifecycle reflects the packaged stream.
3. The Stuck MRs view lists **every** open update MR older than 24 hours with
   links and creation-based ages, including older MRs hidden by a newer MR on
   the same package. MRs with unknown creation time remain visible as such.
4. The collector runs as a separate CronJob on OpenShift every four hours. It
   recomputes every package from the current specs and complete upstream
   history without slowing `upstream_version_update`. Failed and missed runs
   are visible through OpenShift and dashboard freshness.
5. A release arriving during a run appears in that or the next complete run.
   Failed pagination or ingest retains the last successful inventory. Later
   runs pick up corrected release dates, lifecycle changes, and newly registered
   projects without replaying a release feed. Withdrawals require a monitor
   eligibility signal rather than disappearance from upstream. An outdated ingest
   generation is rejected even when upstream releases did not change.
6. A newer upstream release does not reset staleness. An MR for a different
   version or with no reliable merge time does not acquire a rollout timestamp.
7. Failed collection retains the last successful data with visible age.
   Monitor scan failures remain visible even after a successful collector run.
   Removed packages disappear only after a complete inventory run.
8. Existing dashboard behavior, the update loop, and existing reporting are
   unchanged.

## Phase 2

Future work, contingent on the independent pipeline-timing service from !1261
providing tested lineage and sufficient data in its `pipeline_timing` schema.
The joins needed by this tracker remain to be verified. !1261 preserves the
existing `public` tables and Dashboard reads. There is no planned removal of
`public.gitlab_merge_requests` that requires this phase.

### What changes

**Rollout timestamp**: Replace the v1 merge-time proxy with completion time of
the **first successful publication** for the matched package and version. The
future model must support an exact component match, MR merge commit → build
PipelineRun → Snapshot → Release lineage, and identification of the intended
ReleasePlan/destination. A SHA alone or an arbitrary single run is insufficient.
If several builds or releases qualify, choose the earliest successful completion
for that destination. Failed releases and releases to other destinations must
not count. If the lineage or destination is ambiguous, leave `rolled_out_at`
null. Retry unresolved rows as new timeline events arrive. A version shipped
by a later repair commit needs a documented version-to-release association
rather than being attributed to the original MR by SHA alone. Use
`pipeline_timing` completion times only where the lineage is verified.

Backfill existing rollout rows and compare results with the v1 merge-time proxy
before switching the timestamp. Retain the matched run and Release identifiers
for traceability. Settle their column names against the implemented schema.

### What stays the same

The collector CronJob, upstream mirror, and `public.gitlab_merge_requests` as
the source for versioned MR matching and the Stuck MRs view. The bot-convention
matching, `rpm_version_status` columns, and UI also stay the same. Phase 2 is a
rollout timestamp upgrade with no planned UI restructure.

[rpms-repo]: https://gitlab.com/redhat/hummingbird/rpms
[hum-5820]: https://redhat.atlassian.net/browse/HUM-5820
[upstream-monitor]: https://upstream-monitor-hummingbird--factory-sandbox.apps.int.spoke.preprod.us-east-1.aws.paas.redhat.com/
