This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Hummingbird Dashboard

Web dashboard and CLI for monitoring Konflux build pipeline status.

Features

  • Web Dashboard - Real-time view of build, test, and release status
  • Commits View - Detailed pipeline status per commit with expandable releases
  • Components View - Latest status per component grouped by state
  • Merge Requests View - Track build status across MR versions
  • CLI Tool - Command-line access to pipeline data in YAML/JSON/table formats
  • Failed Releases - View and manage failed releases with LLM-powered analysis
  • Failed Push Builds - Detect and auto-retry failed on-push Konflux builds
  • Auto-Rerun - Automatically retry transient release and push build failures
  • Blocked Snapshots - Block/unblock snapshots from auto-rerun with error pattern matching
  • Smart Triggering - Application-aware rules for determining affected components
  • Error Budget Alerts - Slack notifications when CVE R-Time error budget hits 20%, 5%, or 0% remaining
  • Blame Page - /cve/blame view listing late unblamed tickets with per-category breakdown and per-ticket stage detail
  • ProdSec CVE-HUM SLA Auto-Blame - Auto-locks ProdSec blame when CVE-HUM time exceeds 6h
  • Image Pull Availability - Estimated SLI-UE1-B availability from manifest and sampled checks
  • RPM Tracker API - Read-only snapshot of tracker-owned tables in the shared database
  • RPM Tracker Views - Package currency, upstream staleness, merge based rollout lag, and stuck update MRs

Prerequisites

Installation

cd hummingbird-dashboard
pip install -e .

Usage

Local Development

cd hummingbird-dashboard

# Option 1: Port-forward to production database
./dev.sh port-forward  # In terminal 1
DATABASE_URL=postgresql://postgres@localhost:15432/events ./dev.sh start  # In terminal 2

# Option 2: Use local database (via hummingbird-status)
cd ../hummingbird-status && ./dev.sh db-start && ./dev.sh db-init
cd ../hummingbird-dashboard
DATABASE_URL=postgresql://postgres:dev@localhost:5432/events ./dev.sh start

The dashboard runs at http://localhost:8080 with live reload.

The dashboard’s Alembic migrations do not create the components table: it belongs to hummingbird-status. With a fresh local database created by dev.sh db-start alone, every page that lists applications fails with relation "components" does not exist, so run hummingbird-status/dev.sh db-init first (or hummingbird_status.db.create_schema() when the venv lacks boto3).

Web UI

Endpoint Description
/ Dashboard with all applications
/apps/{app}/commits Commits with build or promotion PipelineRuns and releases
/apps/{app}/components Component status from builds, promotions, and releases
/mrs Merge requests overview (filterable)
/mr/{project}/{iid} MR builds and the post-merge build or promotion
/releases/failed Failed releases with analysis
/releases/blocked Blocked snapshots management
/releases/analysis-log LLM analysis run history
/cve/status CVE ticket status dashboard
/cve/run-log CVE analysis run history
/cve/open Open CVE trackers with SLO status
/cve/closed Closed tickets with VEX reconciliation
/cve/r-time CVE R-Time (notified to done) metrics
/cve/blame Late unblamed tickets and per-category blame breakdown
/rpm/rollout RPM currency, staleness, lifecycle, and update MRs
/rpm/stuck-mrs Stuck RPM update MRs, including unknown ages
/availability Estimated image pull availability (SLI-UE1-B)
/health Health check endpoint
/metrics Prometheus metrics

For RPMs configured for MR artifact promotion, the post-merge PipelineRun promotes the tested MR image. The commits and MR detail views label that run as a promotion and follow its Component Snapshot to the automatic Release. Component status counts a successful promotion and Release as current. If a retry reuses the Component Snapshot, the dashboard also links it through the retry’s COMPONENT_SNAPSHOT result. Failed promotions offer the manual retrigger button: the clone keeps the merge SHA, gets a fresh git-auth secret, and the promotion Task reuses the deterministic Component Snapshot when its evidence still matches. Build throughput and failed push-build auto-rerun continue to count actual builds only.

Snapshot lookups use the indexed source_plr column or the Snapshot name from the promotion PipelineRun result. This also covers historical promotion Snapshots that were ingested before their annotation was stored in source_plr. The dashboard summary validates the newest successful run per component first, then checks older runs in one batch when a required test failed or a promotion has no successful Release.

Each CVE tracker page (/cve/status, /cve/open, /cve/closed, /cve/r-time) has a “Download CSV” button at the bottom of the page that downloads the current view (same filters as the page) as a CSV file, via a matching /cve/{tab}/csv route (for example /cve/open/csv?slo=24). A “Show CVE ID” toggle in the header reveals the CVE ID next to each HUM ticket key (HUM-1234/CVE-2026-56789); the preference persists via localStorage.

On /cve/status, a “Show no-CVE” toggle next to it reveals HUM tickets that have no assigned CVE ID in the category drawers. They are hidden by default, and the preference persists via localStorage.

/cve/r-time/csv appends one column per Stage Breakdown milestone (HUM Ticket Created, CVE Published, OSIDB Flaw Created, OSIDB Flaw Public, OSIDB Affect Created, Upstream Fix Merged, Fedora Update Available, RPM Published to Pulp, Image Rebuilt on Quay, HUM Ticket Closed, VEX Updated, CVE Embargo Lifted) after the summary columns, in the same chronological order used in the per-ticket Stage Breakdown panel. Each cell is an ISO 8601 timestamp, or blank when that milestone has no data for the ticket.

On the CVE Tracker Status, Open, and Closed pages, a compact outline icon beside a HUM ticket indicates the CVE Agent’s final Jira verdict label, not a confirmed Jira resolution. Hover or focus the ticket for the recommendation; the “Agent verdicts” help control explains the icons. One policy table per verdict drives the icon, the expected closure, the automatic review and the failure handling. It mirrors hummingbird_cve_agent/labels.py and hummingbird_cve_analysis/lib/states.py without adding a build dependency.

Closed-tab scoring

Verdict Expected resolution Treatment
needs_change, already_fixed Done-Errata Matching closure agrees; contradictory closure disagrees
not_affected Not a Bug Matching closure agrees; contradictory closure disagrees
needs_human, error — Failure
no_upstream_fix — Reviewable, excluded from closure scoring

Accuracy is matches divided by comparable closures plus failures, within the selected period. A current saved review overrides Jira closure for both the row label and accuracy, so a reviewed no_upstream_fix counts while it stays current; stale reviews fall back to Jira closure. Tickets without a verdict and incomparable resolutions without a review (such as Won't Do or a missing resolution) are excluded; no score is shown without scorable results. Closure agreement establishes neither decision correctness nor MR quality.

Reviews

Review Behavior
Automatic Derived without database writes, for failures and for not_affected closed as Not a Bug with no linked agent MR and no VEX mismatch
Saved Takes precedence over an automatic review; a non-stale decision overrules the closure comparison
No upstream fix Can confirm the historical judgment; recorded upstream_fix_merged evidence marks the verdict outdated and its review stale

The agent never re-investigates a recorded verdict, and saving another review does not clear the no-fix stale flag. Automatic re-investigation is not implemented.

RPM Tracker Views

The Rollout view supports package search, status and packaged stream lifecycle filters, and sortable columns. Summary counts describe the full inventory before filters. By default, All packages lists current and behind packages first, with unmonitored and unknown packages last. Each group is sorted alphabetically. Explicit column sorting overrides this default grouping. Unavailable sort values remain last in either direction.

Each row shows the packaged and eligible upstream versions, currency, staleness, merge based rollout lag, packaged stream lifecycle, and a selected open update MR. Staleness starts at the first newer eligible upstream release, not the most recent one. Rollout lag is available only for current packages whose version equals the selected upstream version and has a resolved merge time. Missing or inconsistent dates remain unavailable. The merge timestamp is a proxy, not proof of successful publication. A fallback MR is labeled Package candidate only, not a verified update to the selected upstream version. Unsupported streams flag support risk without changing downstream CVE policy.

The Stuck MRs view lists every open update MR older than 24 hours, plus MRs with unknown creation based ages. It includes multiple MRs for the same package and MRs whose packages are absent from the inventory. The view also supports listing all open update MRs and searching by package or MR title. Unknown ages are not shown as zero or silently excluded.

Both views read the rpm_tracker schema through this dashboard deployment’s existing DATABASE_URL configuration. The preprod dashboard reads the preprod Status database, and the production dashboard reads the production Status database. No separate tracker URL, data copying, or environment selector is required. The dashboard does not migrate or write tracker tables.

Rollout reads are limited to rollouts for currently packaged versions. The MR view and listing API do not read package inventory or rollout history. Retained historical rollouts remain available through GET /api/rpm-upstream-state.

The pages show the last successful collection and independent MR reconciliation time. Collection older than eight hours displays a warning while retaining the last successful inventory. Package rows show their check time, upstream monitor scan state, last successful scan, and scan errors separately. A missing or incompatible schema produces a readable HTTP 503 page, rather than an empty inventory. Database connection failures and pool timeouts also return HTTP 503, with a temporary unavailability message rather than a schema warning. These failures are logged without exposing connection details on the page. A schema with no successful collection or MR reconciliation shows the corresponding pending state. See RPM Tracker for collection, reconciliation, and migration details.

Examples:

/rpm/rollout?status=behind&search=openssl
/rpm/rollout?status=current&lifecycle=active&sort=rollout_lag_hours&order=desc
/rpm/stuck-mrs?stuck_only=false&search=openssl

On /competitive, click trend for an image to compare Hummingbird and Chainguard CVE counts over the past 30 days. Distributor series share a chronological time axis and value scale, with distinct colors and legend labels. Hover over a point to see its distributor, collection date and time, and CVE count. Missing observations are not filled with zero, and explicit null counts leave gaps in the line. Points with invalid collection timestamps are omitted. Opening another image clears the previous chart immediately, so a failed request cannot leave another image’s data displayed. Data collection and CVE calculations are unchanged.

Competitive CVE Diff

On /competitive, click the Delta value on any row to expand a drawer listing the CVE IDs that differ between the Hummingbird and Chainguard scans for that image. A + marks a CVE ID reported only by Hummingbird; a - marks a CVE ID reported only by Chainguard. This is a scan result difference between the two tools, not a confirmed affectedness determination — reasons a CVE shows up on only one side include differing scanner databases, matching heuristics, and false positives/negatives, not necessarily that the other side is truly unaffected. CVE IDs both sides report are shared and excluded from the diff. Each CVE ID links to its Red Hat CVE page; when a HUM Jira ticket tracks it, the ticket is shown and linked alongside it (HUM-5678/CVE-2026-1234). If a scan predates CVE ID collection, the drawer shows a notice instead of a misleading empty diff.

GET /api/competitive/comparison (the page’s main listing) returns only per-image counts and delta — not the CVE ID arrays, HUM ticket links, or even whether per-CVE detail is available — with hundreds of tracked images, computing and shipping that detail for every row up front would be wasted work for the rows no one expands. A row’s drawer content is fetched from GET /api/competitive/diff?image_name=...&tag=... the first time it’s opened, then cached client-side so reopening it doesn’t refetch; that response’s own cve_detail_available decides whether to render the diff or a “check back later” notice.

Both sides of a scan can report a native GHSA or Go vulnerability DB ID instead of a CVE for some ecosystems (e.g. Go modules in an image like cosign): Chainguard’s own Grype scan and the Hummingbird catalog API’s Grype-based scan alike. The collector resolves these against grype’s local database before storage (see chainguard-comparison.md), so by the time data reaches the dashboard, only IDs with no CVE anywhere in that database remain unresolved. These are shown as plain text with no link and no HUM ticket match attempted, rather than resolved here: that would require an OSV.dev network call per unresolved ID, which is unreliable (many environments, including this one, can’t reach it) and, done sequentially, slow enough to hang the drawer open for tens of seconds on an image with several unresolved IDs.

A ticket is indexed under every CVE ID in its cve_ids field (from hummingbird_cve_analysis’s dedicated CVE ID Jira field — see hummingbird-cve-analysis.md), since a single ticket can legitimately track more than one CVE; tickets collected before that field existed fall back to the single first CVE-format label found. The Competitive page compares whole images, but HUM tickets are filed per affected RPM. A CVE in a single bundled dependency (e.g. openssh) can affect several unrelated images (e.g. both git and curl) with no per-image ticket at all. Tickets resolved known_not_affected (VEX “Not a Bug”) are excluded from matching entirely — a ticket confirming its own package isn’t actually impacted must never be suggested as if it were tracking the CVE for some other image. Of the remaining candidates, each one’s catalog_images field (every {image}:{stream} its package actually appears in, from hummingbird_cve_analysis’s SBOM-derived catalog source map — see hummingbird-cve-analysis.md) is checked first: an exact match there is ground truth, so it’s used directly. Older tickets collected before that field existed fall back to a package-name heuristic (an exact {image}{stream} match, the Red Hat packaging convention for versioned streams like mariadb11.8, then substring overlap). When nothing disambiguates to a single ticket either way, every remaining candidate is linked rather than guessing one or hiding all of them.

The image name/tag checked against catalog_images is the catalog’s own identity (chainguard_scans.catalog_image_name / catalog_stream, populated by chainguard-comparison.md’s collector — see below), not the Competitive page’s “common” display name or parsed version: a few images are shown under a cross-vendor alias (Hummingbird’s valkey image is displayed as redis, to pair with Chainguard’s image of that name) and the page’s parsed version does not always match the catalog’s own stream label (e.g. a canonical tag 8.23.0 parses to 8.23 here while the catalog calls that stream 8). Comparing the display name/version against catalog_images would spuriously reject tickets that actually apply. Scan rows collected before catalog_image_name / catalog_stream existed fall back to the display name/tag, matching the pre-migration behavior exactly.

A candidate is only excluded by a non-matching catalog_images when that image/stream was actually checked during the ticket’s collection run. hummingbird_cve_analysis’s catalog source map silently skips an image/stream whose tag listing or SBOM fetch failed (see hummingbird-cve-analysis.md), so a nonempty catalog_images list is not necessarily complete; each ticket’s catalog_unverified_images field lists exactly which image/stream pairs that run couldn’t verify, and a candidate matching one of those is kept as plausible (falling through to the package-name heuristics) rather than excluded.

Only the most-recently-collected tag is shown per image; if a tracked image’s version changes between collection runs (e.g. curl 8.22 -> 8.23), the superseded tag’s row is dropped rather than lingering alongside the current one.

CVE Search API

GET /api/cve-runs/search — search CVE analysis run history.

Parameter Type Default Description
package string — Filter by package name (substring match)
ticket string — Filter by ticket key (substring match)
q string — Free-text search across ticket details
log string — Search CronJob log output
days int 30 Time window in days (1–365)
limit int 500 Max results (1–2000)

At least one of package, ticket, q, or log is required.

The response includes two result sets:

  • results — per-ticket matches from the JSONB details column
  • log_matches — per-run matches from CronJob log output, each with a log_excerpt showing the matching lines in context

Example:

# Find runs whose logs mention "merge train" — show excerpts
curl -s '/api/cve-runs/search?log=merge+train' | jq '.log_matches[] | {recorded_at, log_excerpt}'

# Find tickets for a package, also searching logs
curl -s '/api/cve-runs/search?package=openssl&log=advisory+failed&days=7' \
  | jq '{tickets: [.results[].detail.key], log_hits: .log_matches | length}'

# List all ticket keys matching a free-text query
curl -s '/api/cve-runs/search?q=needs-attention&days=14' | jq '[.results[].detail.key] | unique'

CLI

# View latest commit status
hummingbird-dashboard --application myapp --format table commit

# View specific commit
hummingbird-dashboard --application myapp commit abc1234

# View multiple commits
hummingbird-dashboard --application myapp --format table commit --limit 10

# Component status overview
hummingbird-dashboard --application myapp component

# JSON output
hummingbird-dashboard --format json commit | jq .

# Auto-rerun failed releases
hummingbird-dashboard auto-rerun

# Analyze failed releases via LLM
hummingbird-dashboard analyze-failures

# Probe catalog image pull availability (HUM-721)
hummingbird-dashboard check-image-pulls
hummingbird-dashboard check-image-pulls --dry-run
hummingbird-dashboard check-image-pulls --force

REST API Reference

All endpoints return JSON. Read-only GET endpoints are unauthenticated. Interactive OpenAPI/Swagger documentation is available at /docs.

RPM Tracker

GET /api/rpm-upstream-state returns the collection generation and checkout SHA with upstream, package status, rollout reasons, and all open update MRs from the independently migrated rpm_tracker schema. It also exposes the MR reconciliation scope and freshness and one selected blocking_mr per package row. GET /api/rpm-update-mrs lists every open update MR independently of that selection, including multiple MRs for one package. Add ?stuck_only=true for MRs older than 24 hours and MRs with unknown creation-based ages; unknown ages remain null rather than disappearing. The dashboard does not create or write these tables. Run the tracker initializer before deploying this API; until a compatible tracker schema is present, this endpoint returns HTTP 503. See RPM Tracker for migration and storage details.

Tier 1 — Core Pipeline Status

GET /api/apps/{app_name}/commits

Pipeline status for commits in an application.

Parameter Type Default Description
sha string — Filter by commit SHA (prefix)
component string — Filter by component name
limit int 10 Max commits to return (max 100)
curl -s 'http://localhost:8080/api/apps/rpms/commits?limit=5' | jq .
curl -s 'http://localhost:8080/api/apps/rpms/commits?sha=abc123&component=openssl' | jq .

GET /api/apps/{app_name}/components

Latest build status per component, grouped by state.

Parameter Type Default Description
search string — Filter components by name (substring)
curl -s 'http://localhost:8080/api/apps/rpms/components' | jq .
curl -s 'http://localhost:8080/api/apps/rpms/components?search=openssl' | jq .

Tier 2 — Operational Visibility

GET /api/dashboard

Full overview across all applications: aggregate build counts, failed releases, component staleness, and recent activity.

curl -s 'http://localhost:8080/api/dashboard' | jq .

GET /api/releases/failed

List of currently failed releases with analysis results.

Parameter Type Default Description
application string — Filter by application name
curl -s 'http://localhost:8080/api/releases/failed' | jq .
curl -s 'http://localhost:8080/api/releases/failed?application=rpms' | jq .

GET /api/releases/{name}/analysis

LLM failure analysis for a specific release (root cause, classification, recommendation).

curl -s 'http://localhost:8080/api/releases/my-release-abc/analysis' | jq .

GET /api/releases/auto-rerun-log

History of automatic rerun attempts.

Parameter Type Default Description
limit int 50 Max entries
curl -s 'http://localhost:8080/api/releases/auto-rerun-log?limit=10' | jq .

GET /api/releases/analysis-log

History of LLM analysis runs.

Parameter Type Default Description
limit int 50 Max entries
curl -s 'http://localhost:8080/api/releases/analysis-log?limit=10' | jq .

GET /api/releases/analysis-costs

Cumulative token usage and estimated cost for LLM analysis runs.

curl -s 'http://localhost:8080/api/releases/analysis-costs' | jq .

GET /api/mrs

Merge requests across monitored repositories.

Parameter Type Default Description
state string opened MR state: opened, merged, closed, all
project string — Filter by project (repository) name
curl -s 'http://localhost:8080/api/mrs' | jq .
curl -s 'http://localhost:8080/api/mrs?state=merged&project=rpms' | jq .

GET /api/mrs/projects

List of all projects (repositories) with tracked merge requests.

curl -s 'http://localhost:8080/api/mrs/projects' | jq .

GET /api/mr/{project}/{iid}

Detail for a single merge request, including all versions and per-version build status.

curl -s 'http://localhost:8080/api/mr/rpms/42' | jq .

GET /api/cve/status

Current CVE ticket status across all tracked packages.

curl -s 'http://localhost:8080/api/cve/status' | jq .

GET /api/cve/open

Open HUM CVE tracker tickets with SLO status. Notified is the later of HUM ticket created and fix available (upstream or Fedora). SLO is green PASS unless a fix is available and elapsed time is within 8h of the selected SLO (yellow AT RISK) or past it (red FAIL).

Parameter Type Default Description
slo float 24 SLO threshold hours (24, 72, or 168)
include_next_release bool true Include cve-next-release tickets
curl -s 'http://localhost:8080/api/cve/open?slo=24' | jq .

GET /api/cve/closed

Closed HUM CVE tracker tickets with Red Hat CSAF VEX reconciliation (HUM-5843). Analysis stores package-scoped vex_status and Jira resolution; the dashboard computes MATCH from those facts (Done-Errata → fixed, Not a Bug → known_not_affected, package_not_listed, or fixed – a downstream package rebuilt by an unrelated dependency’s fix is not a contradiction of a Not a Bug verdict). Shows Hummingbird vex_status, match state (matched / pending / mismatch), and vex_resolved (first scan time where VEX agreed with the Jira resolution). Event metadata is merged across a ticket’s rows in occurred_at order (HUM-6091): later non-empty fields overlay earlier ones, so Close/VEX facts are not stuck on the first cve_published row. The Resolution column uses Jira Closed / {resolution} when stored analysis text is not already Closed. The Closed tab shows the VEX timestamp as a link to the CVE’s Red Hat CSAF VEX document when the package is named in CSAF. package_not_listed (Not a Bug, package absent from the document) shows N/A with no link: this ticket did not produce a VEX change. Tickets without a VEX match status are counted below the summary cards. The awaiting-vex label count overlaps the matched, pending, mismatch, and unknown groups; it is not an additional category to add to the total.

The HTML page renders 100 tickets per page (page query, 1-based) while the summary cards, CVE Agent performance and CSV export cover the whole period.

Parameter Type Default Description
days int 28 Time window (0 = all time)
curl -s 'http://localhost:8080/api/cve/closed?days=30' | jq .

Entries carry the agent verdict, closure agreement, linked MRs, any recorded ticket cost (agent_cost, with a null estimated_cost_usd when the analysis predates cost reporting, and unpriced_calls when some calls had no pricing) and any saved review; agent_stats aggregates them as described above, including cost_total, cost_tickets, cost_unknown and cost_partial.

PUT /api/retrigger/cve-agent-review/{ticket_key}

Save a reviewer’s needed and mr_outcome, plus an optional note, for a closed CVE ticket. Requires login; the reviewed verdict, resolution and MR identities/head SHAs are snapshotted server-side. Returns 422 when the ticket has no agent verdict. The response returns status, ticket_key, the ticket’s updated agent_outcome_label, agent_outcome_tone and agent_outcome_note, and the reviewed_text byline, so the page updates the row and review panel in place without reloading. The save reads only the reviewed ticket, not the whole Closed tab; period statistics refresh on the next page load.

curl -s -X PUT 'http://localhost:8080/api/retrigger/cve-agent-review/HUM-123' \
  -H 'Content-Type: application/json' \
  -d '{"needed": "backport", "mr_outcome": "edits", "note": "test needed fixing"}'

GET /api/cve/r-time

CVE R-Time (notified to done) for Done-Errata tickets measures time spent on a fixable CVE, so its start never predates the upstream fix in either mode. Include CVE-HUM starts at CVE publication (NVD datePublished / cve_published), falling back to HUM ticket creation only when CVE publication is missing – CVE publication wins even when the HUM ticket predates it, so an embargoed ticket’s R-Time starts at public disclosure rather than at ticket creation (which can predate disclosure by days while the CVE is still privately embargoed). Exclude CVE-HUM starts at the later of HUM created and CVE publication – same as HUM created for a normal ticket (Exclude’s purpose is to ignore filing lag after disclosure), but an embargoed ticket’s clock must never start before public disclosure in either mode, so it also starts at CVE publication there. Whichever of the two the mode picks, Upstream Fix Merged is then applied as a floor on top: the final notified timestamp is the later of that value and Upstream Fix Merged, so neither mode ever counts time before a fix existed to backport. Delivery is the catalog image publish when the package is a catalog image source, otherwise the RPM publish. Include VEX requires delivery, VEX updated, and Closed / Done-Errata; Exclude VEX requires delivery and close only. The done timestamp is the latest of the required gates, and only when all of them exist. Incomplete tickets stay on the table with elapsed time until now; the days window filters by completion (fallback: Done-Errata close, then notified), so a period means completed in that window – the Closed tab filters by Jira close date instead. CVE-HUM is filing lag after NVD (max(HUM created − NVD, 0)); HUM-first tickets show 0.0h. ADV-VEX is Done-Errata close to VEX feed update (max(VEX − close, 0)); close is recorded when the advisory MR merges. Fedora is not start or end; Upstream Fix Merged only ever gates R-Time’s start, never its end. Pre-built (delivery before notification) is informational only. Rows without catalog_image_source in metadata still require an image until the collector restamps them. Each entry includes package onboarding timestamps and delay analysis (rpm_first_published_at, pkg_lag_hours, and our_delay_hours). Default excludes cve-next-release, CVE-HUM time, and VEX time.

Parameter Type Default Description
days int 28 Time window (0 = all time)
slo float 168 SLO threshold hours (24, 72, or 168)
include_next_release bool false Include cve-next-release tickets
include_hum_cve_time bool false Include NVD-to-HUM filing lag in R-Time
include_vex_time bool false Require VEX feed update before R-Time ends
curl -s 'http://localhost:8080/api/cve/r-time?days=10&slo=168' | jq .

GET /api/cve/run-log

History of CVE analysis CronJob runs.

Parameter Type Default Description
limit int 100 Max entries
curl -s 'http://localhost:8080/api/cve/run-log?limit=10' | jq .

POST /api/cve/agent-cost-keys

Report which of the posted tickets already have a recorded CVE agent cost, so the collector downloads only the remaining analysis attachments. The check returns keys only and uses the unique ticket-key index. Requires the same bearer token as /api/cve-report, which accepts the costs as an agent_costs list (ticket_key, input_tokens, output_tokens, cache_read_tokens, cache_creation_tokens, unpriced_calls, estimated_cost_usd, where a null cost marks a pre-cost attachment) and rejects the whole report when a row is malformed. The cache counters are subsets of input_tokens, and a non-zero unpriced_calls makes the cost a lower bound.

curl -s -X POST 'http://localhost:8080/api/cve/agent-cost-keys' \
  -H "Authorization: Bearer $CVE_REPORT_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"tickets": ["HUM-123", "HUM-124"]}' | jq .

GET /api/cve-export

Export CVE sync data as JSON for prod-to-preprod replication. Valid section values are cve_analysis_log, cve_ticket_events, package_lifecycle, and cve_ticket_blame.

Optional pagination parameters:

Parameter Type Default Description
section string — Section to export
limit int 2000 Rows per page
offset int 0 Pagination offset
curl -s -H "Authorization: Bearer $CVE_REPORT_TOKEN" \
  'http://localhost:8080/api/cve-export' | jq '.cve_ticket_events | length'

curl -s -H "Authorization: Bearer $CVE_REPORT_TOKEN" \
  'http://localhost:8080/api/cve-export?section=cve_ticket_events&limit=1000&offset=0' \
  | jq '.count,.has_more'

# Package-scoped lifecycle milestones (rpm_first_published, etc.)
curl -s -H "Authorization: Bearer $CVE_REPORT_TOKEN" \
  'http://localhost:8080/api/cve-export?section=package_lifecycle&limit=10000&offset=0' \
  | jq '.rows[] | select(.event_type == "rpm_first_published")'

Event type validation. The import path (POST /api/cve-import) validates every event_type value against a canonical allowlist defined in hummingbird_dashboard/sources.py:

  • CANONICAL_TICKET_EVENT_TYPES — values accepted for cve_ticket_events: cve_published, osidb_flaw_created, osidb_affect_created, cve_embargo_lifted, osidb_flaw_public, hum_ticket_created, hum_ticket_closed, upstream_fix_merged, fedora_update_available, rpm_fix_published_to_pulp, image_rebuilt_on_quay, vex_resolved.
  • CANONICAL_PACKAGE_EVENT_TYPES — values accepted for package_lifecycle: rpm_first_published.

Legacy names (e.g. jira_created, delivered_in_rpm, …) are mapped to their canonical equivalents via _LEGACY_EVENT_KEY_MAP. Unknown names are rejected with HTTP 400.

POST /api/cve-import

Replace local CVE sync data from a payload previously returned by /api/cve-export.

curl -s -X POST \
  -H "Authorization: Bearer $CVE_REPORT_TOKEN" \
  -H "Content-Type: application/json" \
  --data @cve-export.json \
  'http://localhost:8080/api/cve-import' | jq .

curl -s -X POST \
  -H "Authorization: Bearer $CVE_REPORT_TOKEN" \
  -H "Content-Type: application/json" \
  --data '{"section":"cve_ticket_events","rows":[],"reset":true}' \
  'http://localhost:8080/api/cve-import' | jq .

Ticket-event types such as osidb_flaw_created, osidb_affect_created, cve_embargo_lifted, and osidb_flaw_public import with the rest of cve_ticket_events because they are on CANONICAL_TICKET_EVENT_TYPES.

Analysis-log exports include run_started_at when available. Paginated imports ignore rows whose non-null identity is already present, making retries safe; legacy rows without an identity remain append-only. The ingest timestamp is exported as recorded_at; imports still accept the legacy run_at key, but a sync target older than this release cannot read the new key.

occurred_at conflict resolution on import matches the live collector path (HUM-6860): rpm_fix_published_to_pulp and image_rebuilt_on_quay are mutable delivery timestamps where a non-null incoming value wins, so a sync can update them, without letting a null incoming value erase an already-recorded delivery timestamp; every other event_type is write-once, keeping the existing timestamp once set so a sync or backfill re-run cannot overwrite an established immutable milestone.

Bulk prod-to-preprod replication of cve_analysis_log, cve_ticket_events, package_lifecycle, and cve_ticket_blame no longer goes through this API — cve_analysis_log rows can carry up to 512 KiB of log_output, so paginating that table meant hundreds of small requests that aborted entirely on the first 5xx. See copy_prod_to_preprod_db in CVE Analysis Reference for the direct pg_dump/pg_restore sync tool that replaced it. These endpoints remain in use for the smaller, targeted writes collect_rpm_first_published and backfill_osidb_timestamps.py make directly against prod or preprod.

Tier 3 — Analytics & SRE Metrics

GET /api/stats/build-throughput

Build counts over time (successful, failed, total).

Parameter Type Default Description
days int 30 Lookback window in days
curl -s 'http://localhost:8080/api/stats/build-throughput?days=7' | jq .

GET /api/stats/release-success-rate

Release success/failure ratios over time.

Parameter Type Default Description
days int 30 Lookback window in days
application string — Filter by application
curl -s 'http://localhost:8080/api/stats/release-success-rate?days=14&application=rpms' | jq .

GET /api/stats/auto-rerun-effectiveness

Success rate and time-to-resolution for automatic reruns.

Parameter Type Default Description
days int 30 Lookback window in days
curl -s 'http://localhost:8080/api/stats/auto-rerun-effectiveness?days=7' | jq .

GET /api/components/staleness

Components ranked by time since last successful build.

curl -s 'http://localhost:8080/api/components/staleness' | jq .

GET /api/releases/similar-failures

Find releases with error messages similar to a given string.

Parameter Type Default Description
error string — Error text to match against
curl -s 'http://localhost:8080/api/releases/similar-failures?error=timeout+connecting' | jq .

GET /api/apps/{app_name}/last-successful

Timestamp and SHA of the last fully successful pipeline per component.

curl -s 'http://localhost:8080/api/apps/rpms/last-successful' | jq .

Tier 4 — System

GET /api/service-status/json

Health of upstream services the dashboard depends on (database, Konflux API, KubeArchive, etc.).

curl -s 'http://localhost:8080/api/service-status/json' | jq .

GET /api/settings

Current runtime settings (auto-rerun enabled, analysis enabled, blocked patterns, etc.).

curl -s 'http://localhost:8080/api/settings' | jq .

GET /health

Enriched health check returning service version, uptime, database connectivity, and dependency status.

curl -s 'http://localhost:8080/health' | jq .

Configuration

Environment Variables

Variable Default Description
DATABASE_URL postgresql://...localhost:5432/ PostgreSQL connection URL
PORT 8080 Web server port

Authentication Variables

These control the retrigger functionality (requires OAuth proxy in production):

Variable Default Description
TRIGGER_AUTH_MODE oauth oauth (production) or local
TRIGGER_AUTH_GROUP konflux-hummingbird-admin-access OpenShift group required to retrigger
TRIGGER_LOCAL_USER local-dev Username when TRIGGER_AUTH_MODE=local
CVE_REPORT_TOKEN Bearer token for CVE API auth

The dashboard checks TRIGGER_AUTH_GROUP against the user’s groups on the OpenShift cluster hosting the dashboard. Access on the Konflux cluster alone does not grant retrigger permission. Membership in the default konflux-hummingbird-admin-access group is managed in Rover and synchronized to the dashboard cluster by the infrastructure configuration. Claiming a CVE ticket requires only a dashboard login, so the user menu can show Claim ✅ and Retrigger ❌. Successful group lookups are cached for five minutes; failed lookups are retried on the next request.

For local development with retrigger enabled:

TRIGGER_AUTH_MODE=local DATABASE_URL=... ./dev.sh start

Auto-Rerun Variables

These control the auto-rerun cronjob and failure analysis:

Variable Default Description
AUTO_RERUN_MIN_AGE_MINUTES 30 Minimum failure age before retrying
AUTO_RERUN_MAX_RETRIES 3 Max rerun attempts per snapshot+plan
KONFLUX_KUBECONFIG_PATH Path to Konflux kubeconfig file
RELEASE_NAMESPACE Namespace where releases are created
MANAGED_NAMESPACE Namespace for fetching release PLRs
KUBEARCHIVE_URL KubeArchive API URL for archived resources
GOOGLE_APPLICATION_CREDENTIALS Path to GCP SA key for Vertex AI
GOOGLE_CLOUD_PROJECT GCP project ID for Vertex AI
ANALYSIS_MODEL_API_KEY Gemini API key (fallback if no GCP creds)
ANALYSIS_MODEL gemini-2.5-flash LLM model for failure analysis
ANALYSIS_MODEL_REGION global Vertex AI region
MAX_ANALYSES_PER_CYCLE 5 Max releases to analyze per cycle
SLACK_WEBHOOK_URL Slack webhook for rerun/analysis notifications
DASHBOARD_URL Dashboard base URL for Slack links
GITLAB_SLACK_MAP_PATH /etc/hummingbird/gitlab-slack-map.yaml GitLab→Slack map for author @-mentions (infra-mounted; missing omits mention)
PUSH_RERUN_MIN_AGE_MINUTES 10 Min push build failure age before retrying
PUSH_RERUN_MAX_RETRIES 3 Max retry attempts per component+sha

Error Budget Alert Variables

The dashboard pod periodically evaluates CVE R-Time error budget (168h SLO, 30-day window) and posts to Slack on worsening transitions only:

Remaining budget Slack alert
≤20% Warning
≤5% Critical
0% Exhausted (sprint-stop)

Incoming webhooks always post to the channel they were created for; the payload cannot select #team-hummingbird. Create the webhook for that channel and set ERROR_BUDGET_SLACK_WEBHOOK_URL (or point the shared SLACK_WEBHOOK_URL at that webhook).

State is stored in dashboard_settings (error_budget_rtime_alert_state) so the same threshold is not re-alerted. The stored state is claimed with a compare-and-set before Slack is called, and rolled back if the post fails, so a transient Slack error is retried on the next interval and concurrent evaluators cannot double-post the same transition. Recovery updates the stored state silently so a later re-worsening can alert again. No message is sent when the webhook URL is unset, or when alerts are disabled via the R-Time page toggle (error_budget_alerts_enabled in dashboard_settings; default enabled).

Variable Default Description
ERROR_BUDGET_SLACK_WEBHOOK_URL SLACK_WEBHOOK_URL Incoming webhook created for the alert channel (typically #team-hummingbird)
SLACK_WEBHOOK_URL Fallback webhook if the error-budget URL is unset
DASHBOARD_URL Base URL for the R-Time deep link in alerts
ERROR_BUDGET_ALERT_INTERVAL_SECONDS 900 Seconds between background checks

Failures

The failures section is accessible via the “Failures” nav item and provides two views selectable by tab: Releases and Push Builds.

Failed Releases

The /failures/releases page shows all failed releases with:

  • LLM Analysis — Each failure is analyzed by Gemini with root cause, classification, and recommendation
  • Failure Classification — Transient, Configuration, Code, External Service, or Unknown
  • Rerun History — Past rerun attempts and outcomes per snapshot
  • Error Pattern Matching — Auto-block snapshots matching known error patterns
  • Auto-Rerun — Automatically retry transient failures (configurable via dashboard toggle)
  • Analysis Toggle — Enable/disable LLM analysis from the dashboard

CLI Subcommands

The auto-rerun subcommand retries eligible failed releases:

hummingbird-dashboard auto-rerun --application myapp

The auto-rerun-push subcommand retries eligible failed push builds:

hummingbird-dashboard auto-rerun-push --application myapp

Before creating a retry PipelineRun, the command records an atomic claim for its component and commit in push_build_rerun_claims. Later CronJob runs skip that pair until the claimed PipelineRun is recorded as failed in the dashboard database. A successful or still-running retry prevents another attempt, and PUSH_RERUN_MAX_RETRIES also applies to claims that are not yet ingested. If the create request has an uncertain outcome, the claim is retained to avoid accidentally creating a duplicate. Check the claimed plr_name in Konflux before manually clearing such a claim.

The analyze-failures subcommand runs LLM analysis on unanalyzed failures:

hummingbird-dashboard analyze-failures --managed-namespace rhtap-releng-tenant

All three are designed to run as Kubernetes CronJobs. Slack messages for auto-rerun, push-retry, analysis, and manual UI reruns include a GitLab MR link (and author @-mention when mapped) when the failure SHA resolves to an MR. Release success messages follow reruns tracked by the previous auto-rerun run, so delayed status ingestion does not skip a success. Failed Slack posts are recorded as pending in the auto-rerun log and retried on the next run.

Image Pull Availability

The /availability page tracks HUM-721 (SLI-UE1-B): can a customer actually pull the images we publish. Continuously pulling every catalog image on a tight schedule is not feasible (terabytes of registry traffic), so the check is split into three tiers, run by check-image-pulls:

  • Manifest HEAD (exhaustive) — a cheap HEAD request, no image bytes transferred, against every latest* image:variant tag in the catalog on every run. This is the headline “Estimated Availability” number. A digest cache skips the live HEAD when the tag’s digest is unchanged, the previous check succeeded, and it was confirmed within the last 6 hours.
  • Blob sample (sampled) — a real HEAD against every blob (config + layers) of a small random sample of currently-available images each run, confirming referenced layers still exist in storage.
  • Full-pull sample (sampled) — a real GET of one image’s config blob (a few KB, following the registry’s CDN redirect) each run — genuine end-to-end retrieval, not just a HEAD.

Both sample tiers prioritize tags published in the last hour over older ones, since “did the thing we just shipped actually land” is the highest-value failure mode. The resulting availability % is estimated: the manifest tier is exhaustive, but the blob/full-pull tiers are sampled, not exhaustive — the page shows all three separately rather than blending them into one number.

  • SLO: 95% target, 28-day rolling window, based on the manifest tier
  • Error Budget: manifest-check misses allowed within the window
hummingbird-dashboard check-image-pulls
hummingbird-dashboard check-image-pulls --dry-run
hummingbird-dashboard check-image-pulls --force
Option Description
--dry-run Probe without writing results to the database
--force Force live manifest HEAD checks (ignore digest cache)
--catalog-api-base Catalog API base URL (default: CATALOG_API_BASE or built-in)

Designed to run as a Kubernetes CronJob (every 15 minutes). Checks can be disabled from the dashboard via the image_pull_checks_enabled setting.

Merge Requests

The /mrs page shows merge requests across all monitored repositories with:

  • State filter - Open, merged, closed, or all MRs
  • Project filter - Filter by specific repository
  • Build status - Aggregate status across all components

The /mr/{project}/{iid} detail page shows:

  • All MR versions - Each head commit SHA that was pushed to the MR
  • Build status per version - Full pipeline status (build, snapshot, test, release)
  • Links to Konflux UI - Direct links to PipelineRuns and Snapshots

Versions are ordered by latest event timestamp, so force-pushed commits appear in the correct position even if they reuse an earlier SHA.

Component Status

The dashboard tracks component build status with these states:

Status Icon Description
Success ✅ Build passed for expected commit
Superseded 🔄 Expected commit not built, but newer succeeded
Failed ❌ Build failed for expected commit
Stale ⚠️ Build not triggered for expected commit
Running ⏳ Build in progress
Missing ❓ No build found

Components are grouped by status: Failed/Stale → Running → OK → Missing.

Trigger Rules

The dashboard uses application-specific rules to determine which components are affected by a push:

  • containers: Changes in images/{component} trigger builds
  • rpms: Changes in rpms/{component} or mock/mock.cfg trigger builds
  • tools: Changes in {component} directory trigger builds

Certain files are excluded from triggering (README, templates, etc.).

Development

See the main README for development workflows.

Reusable text filtering

hummingbird_dashboard/web/static/text-filter.mjs exports initTextFilter(root). It is independent of RPM data and can enhance other dashboard tables.

Mark the input with data-text-filter-input and each searchable row with data-filter-text. The row attribute contains the text to search, such as a package name or a combination of ticket key, package, and CVE ID. The optional count, table wrapper, and empty message use the attributes shown below. Matching ignores case and surrounding whitespace, and leaves table order intact.

<div data-text-filter>
  <form method="get">
    <label for="package-search">Package</label>
    <input
      id="package-search"
      name="search"
      type="search"
      data-text-filter-input
    />
    <button type="submit">Apply filters</button>
  </form>
  <p aria-live="polite">Showing <span data-text-filter-count>1</span></p>
  <div data-text-filter-table>
    <table>
      <tbody>
        <tr data-filter-text="example-package">
          <td>example-package</td>
        </tr>
      </tbody>
    </table>
  </div>
  <p data-text-filter-empty hidden>No matches.</p>
</div>
<script type="module">
  import { initTextFilter } from "/static/text-filter.mjs";
  document.querySelectorAll("[data-text-filter]").forEach(initTextFilter);
</script>

The input’s name is the URL query parameter. Mark sort or filter links with data-text-filter-link to preserve the typed text when navigating. An unnamed input filters rows without changing URLs. Components are scoped to their root, and a form is optional. Use a dedicated GET filter form when server controls are present. Text alone does not submit a request, but changing another form field still submits normally.

For expandable tickets, put data-filter-text on a shared row group, such as a tbody, rather than only the main row. The group hides together while its details keep their expansion state.

Render all rows allowed by the other filters, not just the initial text matches. Use hidden on initial nonmatching rows and set the initial count and empty message on the server. Paginated tables should keep server filtering instead. RPM Tracker uses this module now; adding it to CVE Tracker is separate work.

Running Tests

cd hummingbird-dashboard
pip install -e ".[dev]"
pytest
node --test tests/*.mjs

Browser logic tests use the Node.js built in test runner and are also included in make node-test-host from the repository root.

The PostgreSQL migration regression test uses the CI PostgreSQL sidecar via DATABASE_URL. Locally, set DATABASE_URL to a disposable PostgreSQL database to run it; without one, the test is skipped.

Building Container Image

cd hummingbird-dashboard
podman build -f Containerfile -t hummingbird-dashboard .

Database Migrations

The dashboard uses Alembic to manage its schema. Migrations run automatically on every startup via alembic upgrade head.

On first startup against an existing environment, the dashboard detects that the schema is already present (via the blocked_snapshots sentinel table) and automatically stamps the baseline migration as applied before upgrading. No manual intervention is required.

Local development — migrations run automatically via ./dev.sh db-seed:

./dev.sh db-start
./dev.sh db-seed   # runs alembic upgrade head then seeds test data
./dev.sh start

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

1 - Hummingbird Dashboard: RPM upstream rollout tracker

Problem

We lack a clear view of which RPMs are behind upstream, how long they have been behind, and which update MRs are stuck. Staleness is found reactively.

Background

The upstream monitor service discovers, normalizes and classifies upstream releases. It exposes a released_at timestamp per release, groups versions into streams, and carries support lifecycle state sourced from endoflife.date. The timestamp’s meaning depends on the upstream source. It currently monitors about 200 projects.

The rpms repo package metadata carries upstream_repo and sometimes track_upstream, upstream_branch, upstream_version_transform, or version_suffix_strip. The packaged version comes from the spec file. The existing ci/check_upstream_versions.py reads it with rpmspec, which resolves macros. A scheduled upstream_version_update CI job bumps specs, fetches sources, and opens update MRs.

hummingbird-status initializes the shared events PostgreSQL database and ingests GitLab MRs into its public.gitlab_merge_requests table. The dashboard reads that table and owns additional tables for its own features. The CVE Tracker uses a collector POSTing to a token-authenticated endpoint. The RPM tracker instead follows Pipeline Timing: an independent schema owner writes directly to its schema in the shared database, and the dashboard reads the resulting state.

Findings

  1. All 557 packages carry upstream_repo in metadata. Read their packaged versions from the specs with the existing parse_spec_version helper rather than treating the metadata version field as authoritative.
  2. The upstream monitor reports released_at and the release source. The tracker does not need to fetch release dates from multiple upstream APIs, although a reported date is not always the instant a source became public.
  3. The monitor has about 200 projects, not necessarily 200 of our packages. Package coverage must be counted after matching packages to projects. Its published API already exposes project definitions, paginated release history, stream lifecycle, and project scan status.
  4. Stream lifecycle (active, security, extended, unsupported) is exposed per stream. An unsupported upstream stream is a support-risk signal. It does not change our downstream CVE policy on its own.
  5. V1 can use an update MR’s actual merge time as a proxy for rollout when it is available. The planned Phase 2 definition is successful Release completion, confirmed with Benjamin Sherman (HUM-5820). The timeline data model and its resolution path are not yet implemented.
  6. A pipeline timeline alone cannot identify the target package and version of an update MR. That identity must be resolved from the rpms repo’s update MRs.

Proposal

Mirror upstream state from the monitor into the shared events database, compare it against our packaged versions, and render an RPM Tracker tab. A collector maintains the upstream and version rows through tracker-owned transactional storage. The dashboard reads these rows and resolves open update MRs from the existing Status data.

Data

RPM tracker owns and migrates its tables in a dedicated rpm_tracker schema within the shared events database. Table names below refer to this schema unless qualified otherwise. The original event tables in public, including public.gitlab_merge_requests, are owned by hummingbird-status. Keeping these use cases in distinct schemas preserves table ownership while allowing the dashboard to join tracker and MR data across schemas in one query. Run the tracker initializer before exposing the dashboard read API and before each collector rollout; the collector checks the schema revision before collecting or writing. A failed migration or revision check stops collection without changing Status ingestion. Postgres can filter and sort computed staleness alongside the existing MR data.

rpm_upstream_state is a tracker-owned read model. The monitor remains the authority for upstream releases. Anything else that needs upstream state consumes the monitor directly rather than reading our mirror.

rpm_upstream_state: Latest eligible upstream state and packaged stream lifecycle per mapped package, computed from the monitor on every collector run. Unmonitored packages have no upstream row and appear through a left join with rpm_version_status.

Column Meaning
package rpms repo package name (primary key)
project_id Verified monitor project identifier
upstream_version Latest eligible upstream version
upstream_released_at Monitor timestamp for that release
upstream_source Source reported for the upstream release
packaged_stream Stream containing the packaged version, if known
lifecycle_status Lifecycle of packaged_stream, or unknown
monitor_last_success_at Last successful upstream scan for this project
monitor_state Current upstream monitor scan state
monitor_last_error Latest upstream monitor scan error, if any
updated_at When we last wrote this row

Map each package’s upstream_repo to a monitor project using its project definition’s source and a verified repository mapping. If that mapping is not unique, require an explicit upstream_monitor_project_id in package metadata. A project ID is not derivable from upstream_repo alone.

Use track_upstream to select eligible releases as the existing update job does. A version prefix restricts candidates to that line. latest considers all streams. For a package without track_upstream, use an explicit upstream_stream metadata override or the unique stream matching its packaged version. An ambiguous choice yields unknown, not the project’s global latest. When track_upstream is set, an upstream_stream override can identify the packaged stream for lifecycle but cannot silently change the update target. Do not treat upstream_branch as a stream name without a verified mapping. Resolve packaged_stream independently of the update target so lifecycle describes the version we package even when we track a newer stream. Respect the monitor’s prerelease policy and the package’s version_suffix_strip and upstream_version_transform when comparing with RPM versions. The unrelated version_transform metadata field is for CVE version matching. If the monitor already normalized a version in the same way, do not apply a transform twice. Source artifact availability may delay update automation, but does not change whether an upstream release is visible in this tracker.

rpm_version_status: One row per package, holding our side of the comparison and the derived state. Upstream version and release date are read from rpm_upstream_state via JOIN, and rollout timing from rpm_version_rollouts, so neither is duplicated here.

Column Meaning
package Package name (primary key)
current_version Version currently packaged, from the spec file
status current, behind, unmonitored, unknown
first_behind_at Earliest newer upstream release date, if behind
checked_at When the collector last successfully refreshed this row

The dashboard resolves the blocking MR from public.gitlab_merge_requests for display; MR identity and URL are not part of the replaceable inventory. MR creation time is not stored in that table today, so the Stuck MRs view must obtain it from GitLab or the MR-owning feed.

current means the packaged spec version is equal to or ahead of the latest eligible upstream version. behind means an eligible upstream version is newer. unmonitored means there is no verified monitor project mapping. unknown means a mapped project exists, but its update target, release history, spec version, or comparison is unusable.

rpm_sync_state: One row holding the last successful collection time, the rpms checkout SHA, and a generation number. A successful full inventory replaces current tracker rows and increments the generation in the same database transaction. A failed run leaves the previous rows and generation intact.

Column Meaning
id Primary key fixed to 1 for the singleton row
generation Integer incremented after each successful ingest
rpms_checkout_sha Default branch commit inspected by the collector
last_successful_at Completion time of the last successful collection

rpm_version_rollouts: Resolved rollout times, one row per (package, version). A later tracker-owned reconciliation step can populate them from the status-owned public.gitlab_merge_requests table. Keeping these rows separate from the replaceable inventory preserves historical rollouts; the dashboard reads them but does not write them.

Column Meaning
package Package name (composite primary key)
version The version that was rolled out (composite primary key)
rolled_out_at Actual merge time of the matched MR (v1 proxy for publication)
mr_iid IID of the matched MR
mr_project Project of the matched MR
resolved_at When this entry was resolved

Match merged update MRs in public.gitlab_merge_requests by rpms project, package, target version, and target branch. The update job uses a source branch containing the package name and an MR title containing both package and version. A branch match alone does not identify a version. Use GitLab’s actual merged_at from the webhook payload when present. Obtain it from GitLab for older MRs that lack it. Do not substitute the mutable MR updated_at. If the version or merge time cannot be established, or multiple MRs still match, leave rolled_out_at null and show rollout lag as unavailable. Manual bumps without an identifiable package and version remain unresolved rather than silently matching a different MR. Re-resolve unresolved rows as MR data arrives.

Collection and reconciliation

Every four hours, collect a fresh inventory from the rpms checkout and the monitor’s existing REST API. GET /v1/projects lists projects. GET /v1/projects/{projectID}/definition supplies the source configuration used to verify repository mappings. For each distinct mapped project, GET /v1/projects/{projectID}/status reports scan health, GET /v1/projects/{projectID}/streams provides lifecycle, and GET /v1/projects/{projectID}/releases provides dated release history with X-Next-Cursor pagination. Fetch every release page and GET /v1/projects/{projectID} for snapshot releases and observations, including undated versions omitted from pagination. Merge by source and release ID, with snapshot records taking precedence, including dates corrected to unknown. Treat the monitor’s JSON null empty-stream representation as an empty list. Group releases by project and reuse the results for packages sharing a project. Choose the latest eligible version with RPM comparison rather than relying on API page order. The combined releases give the first newer upstream release date; an eligible newer release with no date makes staleness unavailable. This full read also sees changed dates, stream lifecycle changes, and new project mappings without an event cursor or a second database of historical releases. The monitor’s historical catalog preserves observed releases after they disappear upstream; a full read alone cannot identify withdrawals. Excluding withdrawn releases requires an authoritative monitor eligibility signal.

The collector is implemented in rpm-tracker/rpm_tracker/collect.py, with monitor reads in monitor.py and package selection in policy.py. It reads the expected generation before cloning the default branch, reuses the checkout’s spec parser and named transforms, and writes only through store.ingest. See RPM Tracker for runtime prerequisites and usage. Deployment as an OpenShift CronJob remains a subsequent step.

The monitor’s per-project endpoints do not promise one snapshot across all pages or projects. If any required request or page fails, do not replace the last successful inventory. If releases change during collection, the next full run reconciles them. Do not claim that each four-hour run is an atomic view of all upstream projects. Expose the monitor’s project scan state and last_success_at alongside the collector’s own freshness so a successful API read does not conceal failed upstream polling. Prevent overlapping scheduled runs and check the expected rpm_sync_state generation on ingest, including runs with no release changes.

Metrics

  1. Currency (behind / current): compare the spec version against the latest eligible upstream version with RPM semantics. behind means upstream is newer. An equal or ahead packaged version is current. If versions cannot be compared or the update target cannot be identified, use unknown rather than assuming current. These are source repository statuses, not proof of publication.
  2. Staleness (behind packages): Time since first_behind_at, the earliest monitor released_at among eligible releases newer than the spec version. A newer upstream release does not reset the clock. This is based on the monitor’s reported dates and does not measure security exposure. If history or its date is missing, show unavailable, not zero.
  3. Rollout lag (current packages whose version equals upstream_version): Time between upstream_released_at and rolled_out_at. An ahead packaged version is still current, but its lag is unavailable. In v1, rolled_out_at is the matched update MR’s merge time, not proof of successful publication. Historical rollout rows remain stored, but the v1 Rollout view stops showing their lag when a newer upstream release makes the package behind again.
  4. Lifecycle: flag an unsupported upstream stream as support risk. Whether to fix or close CVEs follows the downstream fix_status policy in package metadata, not this upstream flag.
  5. Stuck update MRs: all open update MRs older than 24 hours by GitLab MR creation time, independently of the single MR shown on a package row.

Approach

  1. Deploy the collector as a Kubernetes CronJob on OpenShift. Its batch/v1 schedule starts a Job every four hours, independently of upstream_version_update. Set concurrencyPolicy: Forbid to prevent overlapping scheduled runs. Run the tracker schema initializer before collection starts. Clone the rpms repo’s default branch read-only at the start of each run and record the checked commit SHA.
  2. Read metadata/*.json from that checkout and parse each package’s spec version with the existing parse_spec_version helper. If a spec cannot be parsed, mark that package unknown rather than trusting a possibly stale metadata version. Map packages to projects using verified monitor definitions, fetch all needed project status, streams, and release history, and compute upstream state and first_behind_at with rpm.labelCompare (provided by the collector image). Create a SQLAlchemy engine and read the generation with store.get_generation(engine) before cloning or collecting external data.
  3. Pass the complete package inventory, checked checkout SHA, and expected generation to store.ingest(engine, request). It rejects an outdated generation with GenerationConflictError, atomically replaces upstream and status rows, and increments the generation. Packages removed from a successfully read default branch checkout disappear. An incomplete fetch must not call ingest; a failed database write leaves the previous inventory unchanged. A conflict fails the Job visibly, and the next run collects a fresh inventory. Grant the CronJob database access limited to tracker-owned tables; use a separate migration role for schema changes. The dashboard remains a read-only consumer of tracker data.
  4. A later tracker-owned reconciliation step resolves rolled_out_at for a package/version from matching merged update MRs. The dashboard may show one open update MR on a package row, but the Stuck MRs view queries all open rpms update MRs independently. Obtain MR creation time from the MR-owning feed or GitLab when missing; MRs without a known creation time appear as unknown age, not silently filtered out.
  5. Render an RPM Tracker tab with a Rollout view (version, status, staleness or rollout lag, lifecycle, blocking MR) and a Stuck MRs view, mirroring the CVE Tracker, plus JSON API endpoints. Show checked_at, the rpms checkout SHA, and the last successful collection time. Warn when collection exceeds eight hours without success. Show monitor scan failures and each project’s last successful upstream check separately. unmonitored means no verified monitor project. unknown means a mapped project without a usable update target, release, or comparison. On collection failure, retain the last successful state and show its age.
  6. Keep the existing Slack and failure-file output for update automation. Monitor failed or missed collector Jobs in OpenShift and show the last successful collection time in the dashboard.

MR reconciliation refinements

  • Add rpm_update_mrs in the tracker schema, keyed by (project, iid), as the dashboard’s MR read model. Reconciliation reads the Status table and stores package/version identity, MR state and links, explicit creation/merge times, and uncertainty reasons. Preserve merged identity evidence across later edits and retain known timestamps when later payloads omit them. Only currently qualifying MRs are rollout candidates.
  • Extend rpm_sync_state with mr_project, mr_target_branch, and last_mr_reconciled_at. Fix project/branch scope on the first successful reconciliation and track MR freshness independently of inventory generation and collection freshness. Commit MR and rollout changes under the sync-row lock without changing the inventory generation.
  • Add unavailable_reason to rpm_version_rollouts and define resolved_at as the time the resolution result last changed. Revisit historical keys after inventory advances and invalidate results when new ambiguity appears.
  • The initial reconciliation step uses only explicit timestamps in stored Status webhook payloads. The planned GitLab backfill remains follow-up work; missing timestamps stay unavailable until reliable evidence is supplied.
  • Expose the independent open-MR listing through GET /api/rpm-update-mrs, including its stuck_only filter, and include MR information in GET /api/rpm-upstream-state. Select the single package-row MR from the read model without restricting the independent listing.

Dependencies

  1. Package registration and mapping. Count actual package coverage by verifying metadata against monitor definitions. Provision missing projects separately. POST /v1/projects requires a project ID, name, homepage, source provider configuration, version policy, and stream policy. An upstream_repo URL alone is not sufficient to register a project. A package without a verified project mapping remains unmonitored.
  2. Monitor load. The existing API provides paginated release history, project streams, and project scan status. Confirm that a complete collection of mapped projects fits within the four-hour schedule at a bounded request rate. No new monitor endpoint or cursor protocol is required for v1.
  3. Collector runtime. The CronJob image needs Git, rpmspec, the Python RPM bindings, and the dependencies needed to use the version parsing helper from the rpms repo. Install the tracker package with its bundled migrations, run the initializer before the collector, and give the CronJob schema-scoped database credentials. Provide a read-only GitLab credential if the repo requires one.
  4. MR timestamps. MR reconciliation needs actual merged_at and creation times from the MR-owning feed or GitLab for missing historical data. It cannot infer these times from mutable updated_at.

Acceptance Criteria

  1. The Rollout view lists every package with its status. Behind packages show time since the first newer upstream release and the blocking update MR. Current packages show merge-time rollout lag only when their version equals the selected upstream version and has a matching MR. Unsupported upstream streams are flagged without changing CVE policy.
  2. Packages not registered with the monitor are shown as unmonitored, not omitted or assumed current. Ambiguous stream mappings are unknown. A packaged version equal to or ahead of the latest eligible upstream version is current. A package pinned with track_upstream ignores releases outside its pinned line. A package tracking latest considers eligible releases in all streams while lifecycle reflects the packaged stream.
  3. The Stuck MRs view lists every open update MR older than 24 hours with links and creation-based ages, including older MRs hidden by a newer MR on the same package. MRs with unknown creation time remain visible as such.
  4. The collector runs as a separate CronJob on OpenShift every four hours. It recomputes every package from the current specs and complete upstream history without slowing upstream_version_update. Failed and missed runs are visible through OpenShift and dashboard freshness.
  5. A release arriving during a run appears in that or the next complete run. Failed pagination or ingest retains the last successful inventory. Later runs pick up corrected release dates, lifecycle changes, and newly registered projects without replaying a release feed. Withdrawals require a monitor eligibility signal rather than disappearance from upstream. An outdated ingest generation is rejected even when upstream releases did not change.
  6. A newer upstream release does not reset staleness. An MR for a different version or with no reliable merge time does not acquire a rollout timestamp.
  7. Failed collection retains the last successful data with visible age. Monitor scan failures remain visible even after a successful collector run. Removed packages disappear only after a complete inventory run.
  8. Existing dashboard behavior, the update loop, and existing reporting are unchanged.

Phase 2

Future work, contingent on the independent pipeline-timing service from !1261 providing tested lineage and sufficient data in its pipeline_timing schema. The joins needed by this tracker remain to be verified. !1261 preserves the existing public tables and Dashboard reads. There is no planned removal of public.gitlab_merge_requests that requires this phase.

What changes

Rollout timestamp: Replace the v1 merge-time proxy with completion time of the first successful publication for the matched package and version. The future model must support an exact component match, MR merge commit → build PipelineRun → Snapshot → Release lineage, and identification of the intended ReleasePlan/destination. A SHA alone or an arbitrary single run is insufficient. If several builds or releases qualify, choose the earliest successful completion for that destination. Failed releases and releases to other destinations must not count. If the lineage or destination is ambiguous, leave rolled_out_at null. Retry unresolved rows as new timeline events arrive. A version shipped by a later repair commit needs a documented version-to-release association rather than being attributed to the original MR by SHA alone. Use pipeline_timing completion times only where the lineage is verified.

Backfill existing rollout rows and compare results with the v1 merge-time proxy before switching the timestamp. Retain the matched run and Release identifiers for traceability. Settle their column names against the implemented schema.

What stays the same

The collector CronJob, upstream mirror, and public.gitlab_merge_requests as the source for versioned MR matching and the Stuck MRs view. The bot-convention matching, rpm_version_status columns, and UI also stay the same. Phase 2 is a rollout timestamp upgrade with no planned UI restructure.