Hummingbird Dashboard

Web dashboard and CLI for monitoring Konflux build pipeline status.

Features

  • Web Dashboard - Real-time view of build, test, and release status
  • Commits View - Detailed pipeline status per commit with expandable releases
  • Components View - Latest status per component grouped by state
  • Merge Requests View - Track build status across MR versions
  • CLI Tool - Command-line access to pipeline data in YAML/JSON/table formats
  • Failed Releases - View and manage failed releases with LLM-powered analysis
  • Failed Push Builds - Detect and auto-retry failed on-push Konflux builds
  • Auto-Rerun - Automatically retry transient release and push build failures
  • Blocked Snapshots - Block/unblock snapshots from auto-rerun with error pattern matching
  • Smart Triggering - Application-aware rules for determining affected components
  • Error Budget Alerts - Slack notifications when CVE R-Time error budget hits 20%, 5%, or 0% remaining
  • Blame Page - /cve/blame view listing late unblamed tickets with per-category breakdown and per-ticket stage detail
  • ProdSec CVE-HUM SLA Auto-Blame - Auto-locks ProdSec blame when CVE-HUM time exceeds 6h
  • Image Pull Availability - Estimated SLI-UE1-B availability from manifest and sampled checks
  • RPM Tracker API - Read-only snapshot of tracker-owned tables in the shared database
  • RPM Tracker Views - Package currency, upstream staleness, merge based rollout lag, and stuck update MRs

Prerequisites

Installation

cd hummingbird-dashboard
pip install -e .

Usage

Local Development

cd hummingbird-dashboard

# Option 1: Port-forward to production database
./dev.sh port-forward  # In terminal 1
DATABASE_URL=postgresql://postgres@localhost:15432/events ./dev.sh start  # In terminal 2

# Option 2: Use local database (via hummingbird-status)
cd ../hummingbird-status && ./dev.sh db-start && ./dev.sh db-init
cd ../hummingbird-dashboard
DATABASE_URL=postgresql://postgres:dev@localhost:5432/events ./dev.sh start

The dashboard runs at http://localhost:8080 with live reload.

The dashboard’s Alembic migrations do not create the components table: it belongs to hummingbird-status. With a fresh local database created by dev.sh db-start alone, every page that lists applications fails with relation "components" does not exist, so run hummingbird-status/dev.sh db-init first (or hummingbird_status.db.create_schema() when the venv lacks boto3).

Web UI

Endpoint Description
/ Dashboard with all applications
/apps/{app}/commits Commits with build or promotion PipelineRuns and releases
/apps/{app}/components Component status from builds, promotions, and releases
/mrs Merge requests overview (filterable)
/mr/{project}/{iid} MR builds and the post-merge build or promotion
/releases/failed Failed releases with analysis
/releases/blocked Blocked snapshots management
/releases/analysis-log LLM analysis run history
/cve/status CVE ticket status dashboard
/cve/run-log CVE analysis run history
/cve/open Open CVE trackers with SLO status
/cve/closed Closed tickets with VEX reconciliation
/cve/r-time CVE R-Time (notified to done) metrics
/cve/blame Late unblamed tickets and per-category blame breakdown
/rpm/rollout RPM currency, staleness, lifecycle, and update MRs
/rpm/stuck-mrs Stuck RPM update MRs, including unknown ages
/availability Estimated image pull availability (SLI-UE1-B)
/health Health check endpoint
/metrics Prometheus metrics

For RPMs configured for MR artifact promotion, the post-merge PipelineRun promotes the tested MR image. The commits and MR detail views label that run as a promotion and follow its Component Snapshot to the automatic Release. Component status counts a successful promotion and Release as current. If a retry reuses the Component Snapshot, the dashboard also links it through the retry’s COMPONENT_SNAPSHOT result. Failed promotions offer the manual retrigger button: the clone keeps the merge SHA, gets a fresh git-auth secret, and the promotion Task reuses the deterministic Component Snapshot when its evidence still matches. Build throughput and failed push-build auto-rerun continue to count actual builds only.

Snapshot lookups use the indexed source_plr column or the Snapshot name from the promotion PipelineRun result. This also covers historical promotion Snapshots that were ingested before their annotation was stored in source_plr. The dashboard summary validates the newest successful run per component first, then checks older runs in one batch when a required test failed or a promotion has no successful Release.

Each CVE tracker page (/cve/status, /cve/open, /cve/closed, /cve/r-time) has a “Download CSV” button at the bottom of the page that downloads the current view (same filters as the page) as a CSV file, via a matching /cve/{tab}/csv route (for example /cve/open/csv?slo=24). A “Show CVE ID” toggle in the header reveals the CVE ID next to each HUM ticket key (HUM-1234/CVE-2026-56789); the preference persists via localStorage.

On /cve/status, a “Show no-CVE” toggle next to it reveals HUM tickets that have no assigned CVE ID in the category drawers. They are hidden by default, and the preference persists via localStorage.

/cve/r-time/csv appends one column per Stage Breakdown milestone (HUM Ticket Created, CVE Published, OSIDB Flaw Created, OSIDB Flaw Public, OSIDB Affect Created, Upstream Fix Merged, Fedora Update Available, RPM Published to Pulp, Image Rebuilt on Quay, HUM Ticket Closed, VEX Updated, CVE Embargo Lifted) after the summary columns, in the same chronological order used in the per-ticket Stage Breakdown panel. Each cell is an ISO 8601 timestamp, or blank when that milestone has no data for the ticket.

On the CVE Tracker Status, Open, and Closed pages, a compact outline icon beside a HUM ticket indicates the CVE Agent’s final Jira verdict label, not a confirmed Jira resolution. Hover or focus the ticket for the recommendation; the “Agent verdicts” help control explains the icons. One policy table per verdict drives the icon, the expected closure, the automatic review and the failure handling. It mirrors hummingbird_cve_agent/labels.py and hummingbird_cve_analysis/lib/states.py without adding a build dependency.

Closed-tab scoring

Verdict Expected resolution Treatment
needs_change, already_fixed Done-Errata Matching closure agrees; contradictory closure disagrees
not_affected Not a Bug Matching closure agrees; contradictory closure disagrees
needs_human, error — Failure
no_upstream_fix — Reviewable, excluded from closure scoring

Accuracy is matches divided by comparable closures plus failures, within the selected period. A current saved review overrides Jira closure for both the row label and accuracy, so a reviewed no_upstream_fix counts while it stays current; stale reviews fall back to Jira closure. Tickets without a verdict and incomparable resolutions without a review (such as Won't Do or a missing resolution) are excluded; no score is shown without scorable results. Closure agreement establishes neither decision correctness nor MR quality.

Reviews

Review Behavior
Automatic Derived without database writes, for failures and for not_affected closed as Not a Bug with no linked agent MR and no VEX mismatch
Saved Takes precedence over an automatic review; a non-stale decision overrules the closure comparison
No upstream fix Can confirm the historical judgment; recorded upstream_fix_merged evidence marks the verdict outdated and its review stale

The agent never re-investigates a recorded verdict, and saving another review does not clear the no-fix stale flag. Automatic re-investigation is not implemented.

RPM Tracker Views

The Rollout view supports package search, status and packaged stream lifecycle filters, and sortable columns. Summary counts describe the full inventory before filters. By default, All packages lists current and behind packages first, with unmonitored and unknown packages last. Each group is sorted alphabetically. Explicit column sorting overrides this default grouping. Unavailable sort values remain last in either direction.

Each row shows the packaged and eligible upstream versions, currency, staleness, merge based rollout lag, packaged stream lifecycle, and a selected open update MR. Staleness starts at the first newer eligible upstream release, not the most recent one. Rollout lag is available only for current packages whose version equals the selected upstream version and has a resolved merge time. Missing or inconsistent dates remain unavailable. The merge timestamp is a proxy, not proof of successful publication. A fallback MR is labeled Package candidate only, not a verified update to the selected upstream version. Unsupported streams flag support risk without changing downstream CVE policy.

The Stuck MRs view lists every open update MR older than 24 hours, plus MRs with unknown creation based ages. It includes multiple MRs for the same package and MRs whose packages are absent from the inventory. The view also supports listing all open update MRs and searching by package or MR title. Unknown ages are not shown as zero or silently excluded.

Both views read the rpm_tracker schema through this dashboard deployment’s existing DATABASE_URL configuration. The preprod dashboard reads the preprod Status database, and the production dashboard reads the production Status database. No separate tracker URL, data copying, or environment selector is required. The dashboard does not migrate or write tracker tables.

Rollout reads are limited to rollouts for currently packaged versions. The MR view and listing API do not read package inventory or rollout history. Retained historical rollouts remain available through GET /api/rpm-upstream-state.

The pages show the last successful collection and independent MR reconciliation time. Collection older than eight hours displays a warning while retaining the last successful inventory. Package rows show their check time, upstream monitor scan state, last successful scan, and scan errors separately. A missing or incompatible schema produces a readable HTTP 503 page, rather than an empty inventory. Database connection failures and pool timeouts also return HTTP 503, with a temporary unavailability message rather than a schema warning. These failures are logged without exposing connection details on the page. A schema with no successful collection or MR reconciliation shows the corresponding pending state. See RPM Tracker for collection, reconciliation, and migration details.

Examples:

/rpm/rollout?status=behind&search=openssl
/rpm/rollout?status=current&lifecycle=active&sort=rollout_lag_hours&order=desc
/rpm/stuck-mrs?stuck_only=false&search=openssl

On /competitive, click trend for an image to compare Hummingbird and Chainguard CVE counts over the past 30 days. Distributor series share a chronological time axis and value scale, with distinct colors and legend labels. Hover over a point to see its distributor, collection date and time, and CVE count. Missing observations are not filled with zero, and explicit null counts leave gaps in the line. Points with invalid collection timestamps are omitted. Opening another image clears the previous chart immediately, so a failed request cannot leave another image’s data displayed. Data collection and CVE calculations are unchanged.

Competitive CVE Diff

On /competitive, click the Delta value on any row to expand a drawer listing the CVE IDs that differ between the Hummingbird and Chainguard scans for that image. A + marks a CVE ID reported only by Hummingbird; a - marks a CVE ID reported only by Chainguard. This is a scan result difference between the two tools, not a confirmed affectedness determination — reasons a CVE shows up on only one side include differing scanner databases, matching heuristics, and false positives/negatives, not necessarily that the other side is truly unaffected. CVE IDs both sides report are shared and excluded from the diff. Each CVE ID links to its Red Hat CVE page; when a HUM Jira ticket tracks it, the ticket is shown and linked alongside it (HUM-5678/CVE-2026-1234). If a scan predates CVE ID collection, the drawer shows a notice instead of a misleading empty diff.

GET /api/competitive/comparison (the page’s main listing) returns only per-image counts and delta — not the CVE ID arrays, HUM ticket links, or even whether per-CVE detail is available — with hundreds of tracked images, computing and shipping that detail for every row up front would be wasted work for the rows no one expands. A row’s drawer content is fetched from GET /api/competitive/diff?image_name=...&tag=... the first time it’s opened, then cached client-side so reopening it doesn’t refetch; that response’s own cve_detail_available decides whether to render the diff or a “check back later” notice.

Both sides of a scan can report a native GHSA or Go vulnerability DB ID instead of a CVE for some ecosystems (e.g. Go modules in an image like cosign): Chainguard’s own Grype scan and the Hummingbird catalog API’s Grype-based scan alike. The collector resolves these against grype’s local database before storage (see chainguard-comparison.md), so by the time data reaches the dashboard, only IDs with no CVE anywhere in that database remain unresolved. These are shown as plain text with no link and no HUM ticket match attempted, rather than resolved here: that would require an OSV.dev network call per unresolved ID, which is unreliable (many environments, including this one, can’t reach it) and, done sequentially, slow enough to hang the drawer open for tens of seconds on an image with several unresolved IDs.

A ticket is indexed under every CVE ID in its cve_ids field (from hummingbird_cve_analysis’s dedicated CVE ID Jira field — see hummingbird-cve-analysis.md), since a single ticket can legitimately track more than one CVE; tickets collected before that field existed fall back to the single first CVE-format label found. The Competitive page compares whole images, but HUM tickets are filed per affected RPM. A CVE in a single bundled dependency (e.g. openssh) can affect several unrelated images (e.g. both git and curl) with no per-image ticket at all. Tickets resolved known_not_affected (VEX “Not a Bug”) are excluded from matching entirely — a ticket confirming its own package isn’t actually impacted must never be suggested as if it were tracking the CVE for some other image. Of the remaining candidates, each one’s catalog_images field (every {image}:{stream} its package actually appears in, from hummingbird_cve_analysis’s SBOM-derived catalog source map — see hummingbird-cve-analysis.md) is checked first: an exact match there is ground truth, so it’s used directly. Older tickets collected before that field existed fall back to a package-name heuristic (an exact {image}{stream} match, the Red Hat packaging convention for versioned streams like mariadb11.8, then substring overlap). When nothing disambiguates to a single ticket either way, every remaining candidate is linked rather than guessing one or hiding all of them.

The image name/tag checked against catalog_images is the catalog’s own identity (chainguard_scans.catalog_image_name / catalog_stream, populated by chainguard-comparison.md’s collector — see below), not the Competitive page’s “common” display name or parsed version: a few images are shown under a cross-vendor alias (Hummingbird’s valkey image is displayed as redis, to pair with Chainguard’s image of that name) and the page’s parsed version does not always match the catalog’s own stream label (e.g. a canonical tag 8.23.0 parses to 8.23 here while the catalog calls that stream 8). Comparing the display name/version against catalog_images would spuriously reject tickets that actually apply. Scan rows collected before catalog_image_name / catalog_stream existed fall back to the display name/tag, matching the pre-migration behavior exactly.

A candidate is only excluded by a non-matching catalog_images when that image/stream was actually checked during the ticket’s collection run. hummingbird_cve_analysis’s catalog source map silently skips an image/stream whose tag listing or SBOM fetch failed (see hummingbird-cve-analysis.md), so a nonempty catalog_images list is not necessarily complete; each ticket’s catalog_unverified_images field lists exactly which image/stream pairs that run couldn’t verify, and a candidate matching one of those is kept as plausible (falling through to the package-name heuristics) rather than excluded.

Only the most-recently-collected tag is shown per image; if a tracked image’s version changes between collection runs (e.g. curl 8.22 -> 8.23), the superseded tag’s row is dropped rather than lingering alongside the current one.

CVE Search API

GET /api/cve-runs/search — search CVE analysis run history.

Parameter Type Default Description
package string — Filter by package name (substring match)
ticket string — Filter by ticket key (substring match)
q string — Free-text search across ticket details
log string — Search CronJob log output
days int 30 Time window in days (1–365)
limit int 500 Max results (1–2000)

At least one of package, ticket, q, or log is required.

The response includes two result sets:

  • results — per-ticket matches from the JSONB details column
  • log_matches — per-run matches from CronJob log output, each with a log_excerpt showing the matching lines in context

Example:

# Find runs whose logs mention "merge train" — show excerpts
curl -s '/api/cve-runs/search?log=merge+train' | jq '.log_matches[] | {recorded_at, log_excerpt}'

# Find tickets for a package, also searching logs
curl -s '/api/cve-runs/search?package=openssl&log=advisory+failed&days=7' \
  | jq '{tickets: [.results[].detail.key], log_hits: .log_matches | length}'

# List all ticket keys matching a free-text query
curl -s '/api/cve-runs/search?q=needs-attention&days=14' | jq '[.results[].detail.key] | unique'

CLI

# View latest commit status
hummingbird-dashboard --application myapp --format table commit

# View specific commit
hummingbird-dashboard --application myapp commit abc1234

# View multiple commits
hummingbird-dashboard --application myapp --format table commit --limit 10

# Component status overview
hummingbird-dashboard --application myapp component

# JSON output
hummingbird-dashboard --format json commit | jq .

# Auto-rerun failed releases
hummingbird-dashboard auto-rerun

# Analyze failed releases via LLM
hummingbird-dashboard analyze-failures

# Probe catalog image pull availability (HUM-721)
hummingbird-dashboard check-image-pulls
hummingbird-dashboard check-image-pulls --dry-run
hummingbird-dashboard check-image-pulls --force

REST API Reference

All endpoints return JSON. Read-only GET endpoints are unauthenticated. Interactive OpenAPI/Swagger documentation is available at /docs.

RPM Tracker

GET /api/rpm-upstream-state returns the collection generation and checkout SHA with upstream, package status, rollout reasons, and all open update MRs from the independently migrated rpm_tracker schema. It also exposes the MR reconciliation scope and freshness and one selected blocking_mr per package row. GET /api/rpm-update-mrs lists every open update MR independently of that selection, including multiple MRs for one package. Add ?stuck_only=true for MRs older than 24 hours and MRs with unknown creation-based ages; unknown ages remain null rather than disappearing. The dashboard does not create or write these tables. Run the tracker initializer before deploying this API; until a compatible tracker schema is present, this endpoint returns HTTP 503. See RPM Tracker for migration and storage details.

Tier 1 — Core Pipeline Status

GET /api/apps/{app_name}/commits

Pipeline status for commits in an application.

Parameter Type Default Description
sha string — Filter by commit SHA (prefix)
component string — Filter by component name
limit int 10 Max commits to return (max 100)
curl -s 'http://localhost:8080/api/apps/rpms/commits?limit=5' | jq .
curl -s 'http://localhost:8080/api/apps/rpms/commits?sha=abc123&component=openssl' | jq .

GET /api/apps/{app_name}/components

Latest build status per component, grouped by state.

Parameter Type Default Description
search string — Filter components by name (substring)
curl -s 'http://localhost:8080/api/apps/rpms/components' | jq .
curl -s 'http://localhost:8080/api/apps/rpms/components?search=openssl' | jq .

Tier 2 — Operational Visibility

GET /api/dashboard

Full overview across all applications: aggregate build counts, failed releases, component staleness, and recent activity.

curl -s 'http://localhost:8080/api/dashboard' | jq .

GET /api/releases/failed

List of currently failed releases with analysis results.

Parameter Type Default Description
application string — Filter by application name
curl -s 'http://localhost:8080/api/releases/failed' | jq .
curl -s 'http://localhost:8080/api/releases/failed?application=rpms' | jq .

GET /api/releases/{name}/analysis

LLM failure analysis for a specific release (root cause, classification, recommendation).

curl -s 'http://localhost:8080/api/releases/my-release-abc/analysis' | jq .

GET /api/releases/auto-rerun-log

History of automatic rerun attempts.

Parameter Type Default Description
limit int 50 Max entries
curl -s 'http://localhost:8080/api/releases/auto-rerun-log?limit=10' | jq .

GET /api/releases/analysis-log

History of LLM analysis runs.

Parameter Type Default Description
limit int 50 Max entries
curl -s 'http://localhost:8080/api/releases/analysis-log?limit=10' | jq .

GET /api/releases/analysis-costs

Cumulative token usage and estimated cost for LLM analysis runs.

curl -s 'http://localhost:8080/api/releases/analysis-costs' | jq .

GET /api/mrs

Merge requests across monitored repositories.

Parameter Type Default Description
state string opened MR state: opened, merged, closed, all
project string — Filter by project (repository) name
curl -s 'http://localhost:8080/api/mrs' | jq .
curl -s 'http://localhost:8080/api/mrs?state=merged&project=rpms' | jq .

GET /api/mrs/projects

List of all projects (repositories) with tracked merge requests.

curl -s 'http://localhost:8080/api/mrs/projects' | jq .

GET /api/mr/{project}/{iid}

Detail for a single merge request, including all versions and per-version build status.

curl -s 'http://localhost:8080/api/mr/rpms/42' | jq .

GET /api/cve/status

Current CVE ticket status across all tracked packages.

curl -s 'http://localhost:8080/api/cve/status' | jq .

GET /api/cve/open

Open HUM CVE tracker tickets with SLO status. Notified is the later of HUM ticket created and fix available (upstream or Fedora). SLO is green PASS unless a fix is available and elapsed time is within 8h of the selected SLO (yellow AT RISK) or past it (red FAIL).

Parameter Type Default Description
slo float 24 SLO threshold hours (24, 72, or 168)
include_next_release bool true Include cve-next-release tickets
curl -s 'http://localhost:8080/api/cve/open?slo=24' | jq .

GET /api/cve/closed

Closed HUM CVE tracker tickets with Red Hat CSAF VEX reconciliation (HUM-5843). Analysis stores package-scoped vex_status and Jira resolution; the dashboard computes MATCH from those facts (Done-Errata → fixed, Not a Bug → known_not_affected, package_not_listed, or fixed – a downstream package rebuilt by an unrelated dependency’s fix is not a contradiction of a Not a Bug verdict). Shows Hummingbird vex_status, match state (matched / pending / mismatch), and vex_resolved (first scan time where VEX agreed with the Jira resolution). Event metadata is merged across a ticket’s rows in occurred_at order (HUM-6091): later non-empty fields overlay earlier ones, so Close/VEX facts are not stuck on the first cve_published row. The Resolution column uses Jira Closed / {resolution} when stored analysis text is not already Closed. The Closed tab shows the VEX timestamp as a link to the CVE’s Red Hat CSAF VEX document when the package is named in CSAF. package_not_listed (Not a Bug, package absent from the document) shows N/A with no link: this ticket did not produce a VEX change. Tickets without a VEX match status are counted below the summary cards. The awaiting-vex label count overlaps the matched, pending, mismatch, and unknown groups; it is not an additional category to add to the total.

The HTML page renders 100 tickets per page (page query, 1-based) while the summary cards, CVE Agent performance and CSV export cover the whole period.

Parameter Type Default Description
days int 28 Time window (0 = all time)
curl -s 'http://localhost:8080/api/cve/closed?days=30' | jq .

Entries carry the agent verdict, closure agreement, linked MRs, any recorded ticket cost (agent_cost, with a null estimated_cost_usd when the analysis predates cost reporting, and unpriced_calls when some calls had no pricing) and any saved review; agent_stats aggregates them as described above, including cost_total, cost_tickets, cost_unknown and cost_partial.

PUT /api/retrigger/cve-agent-review/{ticket_key}

Save a reviewer’s needed and mr_outcome, plus an optional note, for a closed CVE ticket. Requires login; the reviewed verdict, resolution and MR identities/head SHAs are snapshotted server-side. Returns 422 when the ticket has no agent verdict. The response returns status, ticket_key, the ticket’s updated agent_outcome_label, agent_outcome_tone and agent_outcome_note, and the reviewed_text byline, so the page updates the row and review panel in place without reloading. The save reads only the reviewed ticket, not the whole Closed tab; period statistics refresh on the next page load.

curl -s -X PUT 'http://localhost:8080/api/retrigger/cve-agent-review/HUM-123' \
  -H 'Content-Type: application/json' \
  -d '{"needed": "backport", "mr_outcome": "edits", "note": "test needed fixing"}'

GET /api/cve/r-time

CVE R-Time (notified to done) for Done-Errata tickets measures time spent on a fixable CVE, so its start never predates the upstream fix in either mode. Include CVE-HUM starts at CVE publication (NVD datePublished / cve_published), falling back to HUM ticket creation only when CVE publication is missing – CVE publication wins even when the HUM ticket predates it, so an embargoed ticket’s R-Time starts at public disclosure rather than at ticket creation (which can predate disclosure by days while the CVE is still privately embargoed). Exclude CVE-HUM starts at the later of HUM created and CVE publication – same as HUM created for a normal ticket (Exclude’s purpose is to ignore filing lag after disclosure), but an embargoed ticket’s clock must never start before public disclosure in either mode, so it also starts at CVE publication there. Whichever of the two the mode picks, Upstream Fix Merged is then applied as a floor on top: the final notified timestamp is the later of that value and Upstream Fix Merged, so neither mode ever counts time before a fix existed to backport. Delivery is the catalog image publish when the package is a catalog image source, otherwise the RPM publish. Include VEX requires delivery, VEX updated, and Closed / Done-Errata; Exclude VEX requires delivery and close only. The done timestamp is the latest of the required gates, and only when all of them exist. Incomplete tickets stay on the table with elapsed time until now; the days window filters by completion (fallback: Done-Errata close, then notified), so a period means completed in that window – the Closed tab filters by Jira close date instead. CVE-HUM is filing lag after NVD (max(HUM created − NVD, 0)); HUM-first tickets show 0.0h. ADV-VEX is Done-Errata close to VEX feed update (max(VEX − close, 0)); close is recorded when the advisory MR merges. Fedora is not start or end; Upstream Fix Merged only ever gates R-Time’s start, never its end. Pre-built (delivery before notification) is informational only. Rows without catalog_image_source in metadata still require an image until the collector restamps them. Each entry includes package onboarding timestamps and delay analysis (rpm_first_published_at, pkg_lag_hours, and our_delay_hours). Default excludes cve-next-release, CVE-HUM time, and VEX time.

Parameter Type Default Description
days int 28 Time window (0 = all time)
slo float 168 SLO threshold hours (24, 72, or 168)
include_next_release bool false Include cve-next-release tickets
include_hum_cve_time bool false Include NVD-to-HUM filing lag in R-Time
include_vex_time bool false Require VEX feed update before R-Time ends
curl -s 'http://localhost:8080/api/cve/r-time?days=10&slo=168' | jq .

GET /api/cve/run-log

History of CVE analysis CronJob runs.

Parameter Type Default Description
limit int 100 Max entries
curl -s 'http://localhost:8080/api/cve/run-log?limit=10' | jq .

POST /api/cve/agent-cost-keys

Report which of the posted tickets already have a recorded CVE agent cost, so the collector downloads only the remaining analysis attachments. The check returns keys only and uses the unique ticket-key index. Requires the same bearer token as /api/cve-report, which accepts the costs as an agent_costs list (ticket_key, input_tokens, output_tokens, cache_read_tokens, cache_creation_tokens, unpriced_calls, estimated_cost_usd, where a null cost marks a pre-cost attachment) and rejects the whole report when a row is malformed. The cache counters are subsets of input_tokens, and a non-zero unpriced_calls makes the cost a lower bound.

curl -s -X POST 'http://localhost:8080/api/cve/agent-cost-keys' \
  -H "Authorization: Bearer $CVE_REPORT_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"tickets": ["HUM-123", "HUM-124"]}' | jq .

GET /api/cve-export

Export CVE sync data as JSON for prod-to-preprod replication. Valid section values are cve_analysis_log, cve_ticket_events, package_lifecycle, and cve_ticket_blame.

Optional pagination parameters:

Parameter Type Default Description
section string — Section to export
limit int 2000 Rows per page
offset int 0 Pagination offset
curl -s -H "Authorization: Bearer $CVE_REPORT_TOKEN" \
  'http://localhost:8080/api/cve-export' | jq '.cve_ticket_events | length'

curl -s -H "Authorization: Bearer $CVE_REPORT_TOKEN" \
  'http://localhost:8080/api/cve-export?section=cve_ticket_events&limit=1000&offset=0' \
  | jq '.count,.has_more'

# Package-scoped lifecycle milestones (rpm_first_published, etc.)
curl -s -H "Authorization: Bearer $CVE_REPORT_TOKEN" \
  'http://localhost:8080/api/cve-export?section=package_lifecycle&limit=10000&offset=0' \
  | jq '.rows[] | select(.event_type == "rpm_first_published")'

Event type validation. The import path (POST /api/cve-import) validates every event_type value against a canonical allowlist defined in hummingbird_dashboard/sources.py:

  • CANONICAL_TICKET_EVENT_TYPES — values accepted for cve_ticket_events: cve_published, osidb_flaw_created, osidb_affect_created, cve_embargo_lifted, osidb_flaw_public, hum_ticket_created, hum_ticket_closed, upstream_fix_merged, fedora_update_available, rpm_fix_published_to_pulp, image_rebuilt_on_quay, vex_resolved.
  • CANONICAL_PACKAGE_EVENT_TYPES — values accepted for package_lifecycle: rpm_first_published.

Legacy names (e.g. jira_created, delivered_in_rpm, …) are mapped to their canonical equivalents via _LEGACY_EVENT_KEY_MAP. Unknown names are rejected with HTTP 400.

POST /api/cve-import

Replace local CVE sync data from a payload previously returned by /api/cve-export.

curl -s -X POST \
  -H "Authorization: Bearer $CVE_REPORT_TOKEN" \
  -H "Content-Type: application/json" \
  --data @cve-export.json \
  'http://localhost:8080/api/cve-import' | jq .

curl -s -X POST \
  -H "Authorization: Bearer $CVE_REPORT_TOKEN" \
  -H "Content-Type: application/json" \
  --data '{"section":"cve_ticket_events","rows":[],"reset":true}' \
  'http://localhost:8080/api/cve-import' | jq .

Ticket-event types such as osidb_flaw_created, osidb_affect_created, cve_embargo_lifted, and osidb_flaw_public import with the rest of cve_ticket_events because they are on CANONICAL_TICKET_EVENT_TYPES.

Analysis-log exports include run_started_at when available. Paginated imports ignore rows whose non-null identity is already present, making retries safe; legacy rows without an identity remain append-only. The ingest timestamp is exported as recorded_at; imports still accept the legacy run_at key, but a sync target older than this release cannot read the new key.

occurred_at conflict resolution on import matches the live collector path (HUM-6860): rpm_fix_published_to_pulp and image_rebuilt_on_quay are mutable delivery timestamps where a non-null incoming value wins, so a sync can update them, without letting a null incoming value erase an already-recorded delivery timestamp; every other event_type is write-once, keeping the existing timestamp once set so a sync or backfill re-run cannot overwrite an established immutable milestone.

Bulk prod-to-preprod replication of cve_analysis_log, cve_ticket_events, package_lifecycle, and cve_ticket_blame no longer goes through this API — cve_analysis_log rows can carry up to 512 KiB of log_output, so paginating that table meant hundreds of small requests that aborted entirely on the first 5xx. See copy_prod_to_preprod_db in CVE Analysis Reference for the direct pg_dump/pg_restore sync tool that replaced it. These endpoints remain in use for the smaller, targeted writes collect_rpm_first_published and backfill_osidb_timestamps.py make directly against prod or preprod.

Tier 3 — Analytics & SRE Metrics

GET /api/stats/build-throughput

Build counts over time (successful, failed, total).

Parameter Type Default Description
days int 30 Lookback window in days
curl -s 'http://localhost:8080/api/stats/build-throughput?days=7' | jq .

GET /api/stats/release-success-rate

Release success/failure ratios over time.

Parameter Type Default Description
days int 30 Lookback window in days
application string — Filter by application
curl -s 'http://localhost:8080/api/stats/release-success-rate?days=14&application=rpms' | jq .

GET /api/stats/auto-rerun-effectiveness

Success rate and time-to-resolution for automatic reruns.

Parameter Type Default Description
days int 30 Lookback window in days
curl -s 'http://localhost:8080/api/stats/auto-rerun-effectiveness?days=7' | jq .

GET /api/components/staleness

Components ranked by time since last successful build.

curl -s 'http://localhost:8080/api/components/staleness' | jq .

GET /api/releases/similar-failures

Find releases with error messages similar to a given string.

Parameter Type Default Description
error string — Error text to match against
curl -s 'http://localhost:8080/api/releases/similar-failures?error=timeout+connecting' | jq .

GET /api/apps/{app_name}/last-successful

Timestamp and SHA of the last fully successful pipeline per component.

curl -s 'http://localhost:8080/api/apps/rpms/last-successful' | jq .

Tier 4 — System

GET /api/service-status/json

Health of upstream services the dashboard depends on (database, Konflux API, KubeArchive, etc.).

curl -s 'http://localhost:8080/api/service-status/json' | jq .

GET /api/settings

Current runtime settings (auto-rerun enabled, analysis enabled, blocked patterns, etc.).

curl -s 'http://localhost:8080/api/settings' | jq .

GET /health

Enriched health check returning service version, uptime, database connectivity, and dependency status.

curl -s 'http://localhost:8080/health' | jq .

Configuration

Environment Variables

Variable Default Description
DATABASE_URL postgresql://...localhost:5432/ PostgreSQL connection URL
PORT 8080 Web server port

Authentication Variables

These control the retrigger functionality (requires OAuth proxy in production):

Variable Default Description
TRIGGER_AUTH_MODE oauth oauth (production) or local
TRIGGER_AUTH_GROUP konflux-hummingbird-admin-access OpenShift group required to retrigger
TRIGGER_LOCAL_USER local-dev Username when TRIGGER_AUTH_MODE=local
CVE_REPORT_TOKEN Bearer token for CVE API auth

The dashboard checks TRIGGER_AUTH_GROUP against the user’s groups on the OpenShift cluster hosting the dashboard. Access on the Konflux cluster alone does not grant retrigger permission. Membership in the default konflux-hummingbird-admin-access group is managed in Rover and synchronized to the dashboard cluster by the infrastructure configuration. Claiming a CVE ticket requires only a dashboard login, so the user menu can show Claim ✅ and Retrigger ❌. Successful group lookups are cached for five minutes; failed lookups are retried on the next request.

For local development with retrigger enabled:

TRIGGER_AUTH_MODE=local DATABASE_URL=... ./dev.sh start

Auto-Rerun Variables

These control the auto-rerun cronjob and failure analysis:

Variable Default Description
AUTO_RERUN_MIN_AGE_MINUTES 30 Minimum failure age before retrying
AUTO_RERUN_MAX_RETRIES 3 Max rerun attempts per snapshot+plan
KONFLUX_KUBECONFIG_PATH Path to Konflux kubeconfig file
RELEASE_NAMESPACE Namespace where releases are created
MANAGED_NAMESPACE Namespace for fetching release PLRs
KUBEARCHIVE_URL KubeArchive API URL for archived resources
GOOGLE_APPLICATION_CREDENTIALS Path to GCP SA key for Vertex AI
GOOGLE_CLOUD_PROJECT GCP project ID for Vertex AI
ANALYSIS_MODEL_API_KEY Gemini API key (fallback if no GCP creds)
ANALYSIS_MODEL gemini-2.5-flash LLM model for failure analysis
ANALYSIS_MODEL_REGION global Vertex AI region
MAX_ANALYSES_PER_CYCLE 5 Max releases to analyze per cycle
SLACK_WEBHOOK_URL Slack webhook for rerun/analysis notifications
DASHBOARD_URL Dashboard base URL for Slack links
GITLAB_SLACK_MAP_PATH /etc/hummingbird/gitlab-slack-map.yaml GitLab→Slack map for author @-mentions (infra-mounted; missing omits mention)
PUSH_RERUN_MIN_AGE_MINUTES 10 Min push build failure age before retrying
PUSH_RERUN_MAX_RETRIES 3 Max retry attempts per component+sha

Error Budget Alert Variables

The dashboard pod periodically evaluates CVE R-Time error budget (168h SLO, 30-day window) and posts to Slack on worsening transitions only:

Remaining budget Slack alert
≤20% Warning
≤5% Critical
0% Exhausted (sprint-stop)

Incoming webhooks always post to the channel they were created for; the payload cannot select #team-hummingbird. Create the webhook for that channel and set ERROR_BUDGET_SLACK_WEBHOOK_URL (or point the shared SLACK_WEBHOOK_URL at that webhook).

State is stored in dashboard_settings (error_budget_rtime_alert_state) so the same threshold is not re-alerted. The stored state is claimed with a compare-and-set before Slack is called, and rolled back if the post fails, so a transient Slack error is retried on the next interval and concurrent evaluators cannot double-post the same transition. Recovery updates the stored state silently so a later re-worsening can alert again. No message is sent when the webhook URL is unset, or when alerts are disabled via the R-Time page toggle (error_budget_alerts_enabled in dashboard_settings; default enabled).

Variable Default Description
ERROR_BUDGET_SLACK_WEBHOOK_URL SLACK_WEBHOOK_URL Incoming webhook created for the alert channel (typically #team-hummingbird)
SLACK_WEBHOOK_URL Fallback webhook if the error-budget URL is unset
DASHBOARD_URL Base URL for the R-Time deep link in alerts
ERROR_BUDGET_ALERT_INTERVAL_SECONDS 900 Seconds between background checks

Failures

The failures section is accessible via the “Failures” nav item and provides two views selectable by tab: Releases and Push Builds.

Failed Releases

The /failures/releases page shows all failed releases with:

  • LLM Analysis — Each failure is analyzed by Gemini with root cause, classification, and recommendation
  • Failure Classification — Transient, Configuration, Code, External Service, or Unknown
  • Rerun History — Past rerun attempts and outcomes per snapshot
  • Error Pattern Matching — Auto-block snapshots matching known error patterns
  • Auto-Rerun — Automatically retry transient failures (configurable via dashboard toggle)
  • Analysis Toggle — Enable/disable LLM analysis from the dashboard

CLI Subcommands

The auto-rerun subcommand retries eligible failed releases:

hummingbird-dashboard auto-rerun --application myapp

The auto-rerun-push subcommand retries eligible failed push builds:

hummingbird-dashboard auto-rerun-push --application myapp

Before creating a retry PipelineRun, the command records an atomic claim for its component and commit in push_build_rerun_claims. Later CronJob runs skip that pair until the claimed PipelineRun is recorded as failed in the dashboard database. A successful or still-running retry prevents another attempt, and PUSH_RERUN_MAX_RETRIES also applies to claims that are not yet ingested. If the create request has an uncertain outcome, the claim is retained to avoid accidentally creating a duplicate. Check the claimed plr_name in Konflux before manually clearing such a claim.

The analyze-failures subcommand runs LLM analysis on unanalyzed failures:

hummingbird-dashboard analyze-failures --managed-namespace rhtap-releng-tenant

All three are designed to run as Kubernetes CronJobs. Slack messages for auto-rerun, push-retry, analysis, and manual UI reruns include a GitLab MR link (and author @-mention when mapped) when the failure SHA resolves to an MR. Release success messages follow reruns tracked by the previous auto-rerun run, so delayed status ingestion does not skip a success. Failed Slack posts are recorded as pending in the auto-rerun log and retried on the next run.

Image Pull Availability

The /availability page tracks HUM-721 (SLI-UE1-B): can a customer actually pull the images we publish. Continuously pulling every catalog image on a tight schedule is not feasible (terabytes of registry traffic), so the check is split into three tiers, run by check-image-pulls:

  • Manifest HEAD (exhaustive) — a cheap HEAD request, no image bytes transferred, against every latest* image:variant tag in the catalog on every run. This is the headline “Estimated Availability” number. A digest cache skips the live HEAD when the tag’s digest is unchanged, the previous check succeeded, and it was confirmed within the last 6 hours.
  • Blob sample (sampled) — a real HEAD against every blob (config + layers) of a small random sample of currently-available images each run, confirming referenced layers still exist in storage.
  • Full-pull sample (sampled) — a real GET of one image’s config blob (a few KB, following the registry’s CDN redirect) each run — genuine end-to-end retrieval, not just a HEAD.

Both sample tiers prioritize tags published in the last hour over older ones, since “did the thing we just shipped actually land” is the highest-value failure mode. The resulting availability % is estimated: the manifest tier is exhaustive, but the blob/full-pull tiers are sampled, not exhaustive — the page shows all three separately rather than blending them into one number.

  • SLO: 95% target, 28-day rolling window, based on the manifest tier
  • Error Budget: manifest-check misses allowed within the window
hummingbird-dashboard check-image-pulls
hummingbird-dashboard check-image-pulls --dry-run
hummingbird-dashboard check-image-pulls --force
Option Description
--dry-run Probe without writing results to the database
--force Force live manifest HEAD checks (ignore digest cache)
--catalog-api-base Catalog API base URL (default: CATALOG_API_BASE or built-in)

Designed to run as a Kubernetes CronJob (every 15 minutes). Checks can be disabled from the dashboard via the image_pull_checks_enabled setting.

Merge Requests

The /mrs page shows merge requests across all monitored repositories with:

  • State filter - Open, merged, closed, or all MRs
  • Project filter - Filter by specific repository
  • Build status - Aggregate status across all components

The /mr/{project}/{iid} detail page shows:

  • All MR versions - Each head commit SHA that was pushed to the MR
  • Build status per version - Full pipeline status (build, snapshot, test, release)
  • Links to Konflux UI - Direct links to PipelineRuns and Snapshots

Versions are ordered by latest event timestamp, so force-pushed commits appear in the correct position even if they reuse an earlier SHA.

Component Status

The dashboard tracks component build status with these states:

Status Icon Description
Success ✅ Build passed for expected commit
Superseded 🔄 Expected commit not built, but newer succeeded
Failed ❌ Build failed for expected commit
Stale ⚠️ Build not triggered for expected commit
Running ⏳ Build in progress
Missing ❓ No build found

Components are grouped by status: Failed/Stale → Running → OK → Missing.

Trigger Rules

The dashboard uses application-specific rules to determine which components are affected by a push:

  • containers: Changes in images/{component} trigger builds
  • rpms: Changes in rpms/{component} or mock/mock.cfg trigger builds
  • tools: Changes in {component} directory trigger builds

Certain files are excluded from triggering (README, templates, etc.).

Development

See the main README for development workflows.

Reusable text filtering

hummingbird_dashboard/web/static/text-filter.mjs exports initTextFilter(root). It is independent of RPM data and can enhance other dashboard tables.

Mark the input with data-text-filter-input and each searchable row with data-filter-text. The row attribute contains the text to search, such as a package name or a combination of ticket key, package, and CVE ID. The optional count, table wrapper, and empty message use the attributes shown below. Matching ignores case and surrounding whitespace, and leaves table order intact.

<div data-text-filter>
  <form method="get">
    <label for="package-search">Package</label>
    <input
      id="package-search"
      name="search"
      type="search"
      data-text-filter-input
    />
    <button type="submit">Apply filters</button>
  </form>
  <p aria-live="polite">Showing <span data-text-filter-count>1</span></p>
  <div data-text-filter-table>
    <table>
      <tbody>
        <tr data-filter-text="example-package">
          <td>example-package</td>
        </tr>
      </tbody>
    </table>
  </div>
  <p data-text-filter-empty hidden>No matches.</p>
</div>
<script type="module">
  import { initTextFilter } from "/static/text-filter.mjs";
  document.querySelectorAll("[data-text-filter]").forEach(initTextFilter);
</script>

The input’s name is the URL query parameter. Mark sort or filter links with data-text-filter-link to preserve the typed text when navigating. An unnamed input filters rows without changing URLs. Components are scoped to their root, and a form is optional. Use a dedicated GET filter form when server controls are present. Text alone does not submit a request, but changing another form field still submits normally.

For expandable tickets, put data-filter-text on a shared row group, such as a tbody, rather than only the main row. The group hides together while its details keep their expansion state.

Render all rows allowed by the other filters, not just the initial text matches. Use hidden on initial nonmatching rows and set the initial count and empty message on the server. Paginated tables should keep server filtering instead. RPM Tracker uses this module now; adding it to CVE Tracker is separate work.

Running Tests

cd hummingbird-dashboard
pip install -e ".[dev]"
pytest
node --test tests/*.mjs

Browser logic tests use the Node.js built in test runner and are also included in make node-test-host from the repository root.

The PostgreSQL migration regression test uses the CI PostgreSQL sidecar via DATABASE_URL. Locally, set DATABASE_URL to a disposable PostgreSQL database to run it; without one, the test is skipped.

Building Container Image

cd hummingbird-dashboard
podman build -f Containerfile -t hummingbird-dashboard .

Database Migrations

The dashboard uses Alembic to manage its schema. Migrations run automatically on every startup via alembic upgrade head.

On first startup against an existing environment, the dashboard detects that the schema is already present (via the blocked_snapshots sentinel table) and automatically stamps the baseline migration as applied before upgrading. No manual intervention is required.

Local development — migrations run automatically via ./dev.sh db-seed:

./dev.sh db-start
./dev.sh db-seed   # runs alembic upgrade head then seeds test data
./dev.sh start

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.