Web dashboard and CLI for monitoring Konflux build pipeline status.
Features
- Web Dashboard - Real-time view of build, test, and release status
- Commits View - Detailed pipeline status per commit with expandable releases
- Components View - Latest status per component grouped by state
- Merge Requests View - Track build status across MR versions
- CLI Tool - Command-line access to pipeline data in YAML/JSON/table formats
- Failed Releases - View and manage failed releases with LLM-powered analysis
- Failed Push Builds - Detect and auto-retry failed on-push Konflux builds
- Auto-Rerun - Automatically retry transient release and push build failures
- Blocked Snapshots - Block/unblock snapshots from auto-rerun with error pattern matching
- Smart Triggering - Application-aware rules for determining affected components
- Error Budget Alerts - Slack notifications when CVE R-Time error budget hits 20%, 5%, or 0% remaining
- Blame Page -
/cve/blameview listing late unblamed tickets with per-category breakdown and per-ticket stage detail - ProdSec CVE-HUM SLA Auto-Blame - Auto-locks ProdSec blame when CVE-HUM time exceeds 6h
- Image Pull Availability - Estimated SLI-UE1-B availability from manifest and sampled checks
- RPM Tracker API - Read-only snapshot of tracker-owned tables in the shared database
- RPM Tracker Views - Package currency, upstream staleness, merge based rollout lag, and stuck update MRs
Prerequisites
- Python 3.11+
- PostgreSQL database (via hummingbird-status or port-forward)
Installation
cd hummingbird-dashboard
pip install -e .
Usage
Local Development
cd hummingbird-dashboard
# Option 1: Port-forward to production database
./dev.sh port-forward # In terminal 1
DATABASE_URL=postgresql://postgres@localhost:15432/events ./dev.sh start # In terminal 2
# Option 2: Use local database (via hummingbird-status)
cd ../hummingbird-status && ./dev.sh db-start && ./dev.sh db-init
cd ../hummingbird-dashboard
DATABASE_URL=postgresql://postgres:dev@localhost:5432/events ./dev.sh start
The dashboard runs at http://localhost:8080 with live reload.
The dashboard’s Alembic migrations do not create the components table: it
belongs to hummingbird-status. With a fresh local database created by
dev.sh db-start alone, every page that lists applications fails with
relation "components" does not exist, so run
hummingbird-status/dev.sh db-init first (or hummingbird_status.db.create_schema()
when the venv lacks boto3).
Web UI
| Endpoint | Description |
|---|---|
/ |
Dashboard with all applications |
/apps/{app}/commits |
Commits with build or promotion PipelineRuns and releases |
/apps/{app}/components |
Component status from builds, promotions, and releases |
/mrs |
Merge requests overview (filterable) |
/mr/{project}/{iid} |
MR builds and the post-merge build or promotion |
/releases/failed |
Failed releases with analysis |
/releases/blocked |
Blocked snapshots management |
/releases/analysis-log |
LLM analysis run history |
/cve/status |
CVE ticket status dashboard |
/cve/run-log |
CVE analysis run history |
/cve/open |
Open CVE trackers with SLO status |
/cve/closed |
Closed tickets with VEX reconciliation |
/cve/r-time |
CVE R-Time (notified to done) metrics |
/cve/blame |
Late unblamed tickets and per-category blame breakdown |
/rpm/rollout |
RPM currency, staleness, lifecycle, and update MRs |
/rpm/stuck-mrs |
Stuck RPM update MRs, including unknown ages |
/availability |
Estimated image pull availability (SLI-UE1-B) |
/health |
Health check endpoint |
/metrics |
Prometheus metrics |
For RPMs configured for MR artifact promotion, the post-merge PipelineRun promotes
the tested MR image. The commits and MR detail views label that run as a
promotion and follow its Component Snapshot to the automatic Release. Component
status counts a successful promotion and Release as current. If a retry reuses
the Component Snapshot, the dashboard also links it through the retry’s
COMPONENT_SNAPSHOT result. Failed promotions offer the manual retrigger
button: the clone keeps the merge SHA, gets a fresh git-auth secret, and the
promotion Task reuses the deterministic Component Snapshot when its evidence
still matches. Build throughput and failed push-build auto-rerun continue to
count actual builds only.
Snapshot lookups use the indexed source_plr column or the Snapshot name from
the promotion PipelineRun result. This also covers historical promotion
Snapshots that were ingested before their annotation was stored in source_plr.
The dashboard summary validates the newest successful run per component first,
then checks older runs in one batch when a required test failed or a promotion
has no successful Release.
Each CVE tracker page (/cve/status, /cve/open, /cve/closed,
/cve/r-time) has a “Download CSV” button at the bottom of the page that downloads
the current view (same filters as the page) as a CSV file, via a matching
/cve/{tab}/csv route (for example /cve/open/csv?slo=24). A “Show CVE ID”
toggle in the header reveals the CVE ID next to each HUM ticket key
(HUM-1234/CVE-2026-56789); the preference persists via localStorage.
On /cve/status, a “Show no-CVE” toggle next to it reveals HUM tickets that
have no assigned CVE ID in the category drawers. They are hidden by default,
and the preference persists via localStorage.
/cve/r-time/csv appends one column per Stage Breakdown milestone (HUM Ticket Created, CVE Published, OSIDB Flaw Created, OSIDB Flaw Public,
OSIDB Affect Created, Upstream Fix Merged, Fedora Update Available,
RPM Published to Pulp, Image Rebuilt on Quay, HUM Ticket Closed, VEX Updated, CVE Embargo Lifted) after the summary columns, in the same
chronological order used in the per-ticket Stage Breakdown panel. Each cell
is an ISO 8601 timestamp, or blank when that milestone has no data for the
ticket.
On the CVE Tracker Status, Open, and Closed pages, a compact outline icon beside
a HUM ticket indicates the CVE Agent’s final Jira verdict label, not a confirmed
Jira resolution. Hover or focus the ticket for the recommendation; the “Agent
verdicts” help control explains the icons. One policy table per verdict drives
the icon, the expected closure, the automatic review and the failure handling. It
mirrors hummingbird_cve_agent/labels.py and
hummingbird_cve_analysis/lib/states.py without adding a build dependency.
Closed-tab scoring
| Verdict | Expected resolution | Treatment |
|---|---|---|
needs_change, already_fixed |
Done-Errata |
Matching closure agrees; contradictory closure disagrees |
not_affected |
Not a Bug |
Matching closure agrees; contradictory closure disagrees |
needs_human, error |
— | Failure |
no_upstream_fix |
— | Reviewable, excluded from closure scoring |
Accuracy is matches divided by comparable closures plus failures, within the
selected period. A current saved review overrides Jira closure for both the row
label and accuracy, so a reviewed no_upstream_fix counts while it stays
current; stale reviews fall back to Jira closure. Tickets without a verdict and
incomparable resolutions without a review (such as Won't Do or a missing
resolution) are excluded; no score is shown without scorable results. Closure
agreement establishes neither decision correctness nor MR quality.
Reviews
| Review | Behavior |
|---|---|
| Automatic | Derived without database writes, for failures and for not_affected closed as Not a Bug with no linked agent MR and no VEX mismatch |
| Saved | Takes precedence over an automatic review; a non-stale decision overrules the closure comparison |
| No upstream fix | Can confirm the historical judgment; recorded upstream_fix_merged evidence marks the verdict outdated and its review stale |
The agent never re-investigates a recorded verdict, and saving another review does not clear the no-fix stale flag. Automatic re-investigation is not implemented.
RPM Tracker Views
The Rollout view supports package search, status and packaged stream lifecycle filters, and sortable columns. Summary counts describe the full inventory before filters. By default, All packages lists current and behind packages first, with unmonitored and unknown packages last. Each group is sorted alphabetically. Explicit column sorting overrides this default grouping. Unavailable sort values remain last in either direction.
Each row shows the packaged and eligible upstream versions, currency, staleness, merge based rollout lag, packaged stream lifecycle, and a selected open update MR. Staleness starts at the first newer eligible upstream release, not the most recent one. Rollout lag is available only for current packages whose version equals the selected upstream version and has a resolved merge time. Missing or inconsistent dates remain unavailable. The merge timestamp is a proxy, not proof of successful publication. A fallback MR is labeled Package candidate only, not a verified update to the selected upstream version. Unsupported streams flag support risk without changing downstream CVE policy.
The Stuck MRs view lists every open update MR older than 24 hours, plus MRs with unknown creation based ages. It includes multiple MRs for the same package and MRs whose packages are absent from the inventory. The view also supports listing all open update MRs and searching by package or MR title. Unknown ages are not shown as zero or silently excluded.
Both views read the rpm_tracker schema through this dashboard deployment’s
existing DATABASE_URL configuration. The preprod dashboard reads the preprod
Status database, and the production dashboard reads the production Status
database. No separate tracker URL, data copying, or environment selector is
required. The dashboard does not migrate or write tracker tables.
Rollout reads are limited to rollouts for currently packaged versions. The MR
view and listing API do not read package inventory or rollout history. Retained
historical rollouts remain available through GET /api/rpm-upstream-state.
The pages show the last successful collection and independent MR reconciliation time. Collection older than eight hours displays a warning while retaining the last successful inventory. Package rows show their check time, upstream monitor scan state, last successful scan, and scan errors separately. A missing or incompatible schema produces a readable HTTP 503 page, rather than an empty inventory. Database connection failures and pool timeouts also return HTTP 503, with a temporary unavailability message rather than a schema warning. These failures are logged without exposing connection details on the page. A schema with no successful collection or MR reconciliation shows the corresponding pending state. See RPM Tracker for collection, reconciliation, and migration details.
Examples:
/rpm/rollout?status=behind&search=openssl
/rpm/rollout?status=current&lifecycle=active&sort=rollout_lag_hours&order=desc
/rpm/stuck-mrs?stuck_only=false&search=openssl
Competitive Trends
On /competitive, click trend for an image to compare Hummingbird and
Chainguard CVE counts over the past 30 days. Distributor series share a
chronological time axis and value scale, with distinct colors and legend labels.
Hover over a point to see its distributor, collection date and time, and CVE count.
Missing observations are not filled with zero, and explicit null counts leave
gaps in the line. Points with invalid collection timestamps are omitted. Opening
another image clears the previous chart immediately, so a failed request cannot
leave another image’s data displayed. Data collection and CVE calculations are
unchanged.
Competitive CVE Diff
On /competitive, click the Delta value on any row to expand a drawer
listing the CVE IDs that differ between the Hummingbird and Chainguard scans
for that image. A + marks a CVE ID reported only by Hummingbird; a -
marks a CVE ID reported only by Chainguard. This is a scan result
difference between the two tools, not a confirmed affectedness
determination — reasons a CVE shows up on only one side include differing
scanner databases, matching heuristics, and false positives/negatives, not
necessarily that the other side is truly unaffected. CVE IDs both sides
report are shared and excluded from the diff. Each CVE ID links to its Red
Hat CVE page; when a
HUM Jira ticket tracks it, the ticket is shown and linked alongside it
(HUM-5678/CVE-2026-1234). If a scan predates CVE ID collection, the drawer
shows a notice instead of a misleading empty diff.
GET /api/competitive/comparison (the page’s main listing) returns only
per-image counts and delta — not the CVE ID arrays, HUM ticket links, or
even whether per-CVE detail is available — with hundreds of tracked images,
computing and shipping that detail for every row up front would be wasted
work for the rows no one expands. A row’s drawer content is fetched from
GET /api/competitive/diff?image_name=...&tag=... the first time it’s
opened, then cached client-side so reopening it doesn’t refetch; that
response’s own cve_detail_available decides whether to render the diff or
a “check back later” notice.
Both sides of a scan can report a native GHSA or Go vulnerability DB ID
instead of a CVE for some ecosystems (e.g. Go modules in an image like
cosign): Chainguard’s own Grype scan and the Hummingbird catalog API’s
Grype-based scan alike. The collector resolves these against grype’s local
database before storage (see chainguard-comparison.md), so by the time data
reaches the dashboard, only IDs with no CVE anywhere in that database
remain unresolved. These are shown as plain text with no link and no HUM
ticket match attempted, rather than resolved here: that would require an
OSV.dev network call per unresolved ID, which is unreliable (many
environments, including this one, can’t reach it) and, done sequentially,
slow enough to hang the drawer open for tens of seconds on an image with
several unresolved IDs.
A ticket is indexed under every CVE ID in its cve_ids field (from
hummingbird_cve_analysis’s dedicated CVE ID Jira field — see
hummingbird-cve-analysis.md), since a single ticket can legitimately track
more than one CVE; tickets collected before that field existed fall back to
the single first CVE-format label found. The Competitive page compares
whole images, but HUM tickets are filed per affected RPM. A CVE in a single
bundled dependency (e.g. openssh) can affect several unrelated images
(e.g. both git and curl) with no per-image ticket at all. Tickets
resolved known_not_affected (VEX “Not a Bug”) are excluded from matching
entirely — a ticket confirming its own
package isn’t actually impacted must never be suggested as if it were
tracking the CVE for some other image. Of the remaining candidates, each
one’s catalog_images field (every {image}:{stream} its package actually
appears in, from hummingbird_cve_analysis’s SBOM-derived catalog source
map — see hummingbird-cve-analysis.md) is checked first: an exact match
there is ground truth, so it’s used directly. Older tickets collected before
that field existed fall back to a package-name heuristic (an exact
{image}{stream} match, the Red Hat packaging convention for versioned
streams like mariadb11.8, then substring overlap). When nothing
disambiguates to a single ticket either way, every remaining candidate is
linked rather than guessing one or hiding all of them.
The image name/tag checked against catalog_images is the catalog’s own
identity (chainguard_scans.catalog_image_name /
catalog_stream, populated by chainguard-comparison.md’s collector — see
below), not the Competitive page’s “common” display name or parsed
version: a few images are shown under a cross-vendor alias (Hummingbird’s
valkey image is displayed as redis, to pair with Chainguard’s image of
that name) and the page’s parsed version does not always match the
catalog’s own stream label (e.g. a canonical tag 8.23.0 parses to 8.23
here while the catalog calls that stream 8). Comparing the display
name/version against catalog_images would spuriously reject tickets that
actually apply. Scan rows collected before catalog_image_name /
catalog_stream existed fall back to the display name/tag, matching the
pre-migration behavior exactly.
A candidate is only excluded by a non-matching catalog_images when that
image/stream was actually checked during the ticket’s collection run.
hummingbird_cve_analysis’s catalog source map silently skips an
image/stream whose tag listing or SBOM fetch failed (see
hummingbird-cve-analysis.md), so a nonempty catalog_images list is not
necessarily complete; each ticket’s catalog_unverified_images field lists
exactly which image/stream pairs that run couldn’t verify, and a candidate
matching one of those is kept as plausible (falling through to the
package-name heuristics) rather than excluded.
Only the most-recently-collected tag is shown per image; if a tracked
image’s version changes between collection runs (e.g. curl 8.22 -> 8.23),
the superseded tag’s row is dropped rather than lingering alongside the
current one.
CVE Search API
GET /api/cve-runs/search — search CVE analysis run history.
| Parameter | Type | Default | Description |
|---|---|---|---|
package |
string | — | Filter by package name (substring match) |
ticket |
string | — | Filter by ticket key (substring match) |
q |
string | — | Free-text search across ticket details |
log |
string | — | Search CronJob log output |
days |
int | 30 | Time window in days (1–365) |
limit |
int | 500 | Max results (1–2000) |
At least one of package, ticket, q, or log is required.
The response includes two result sets:
results— per-ticket matches from the JSONB details columnlog_matches— per-run matches from CronJob log output, each with alog_excerptshowing the matching lines in context
Example:
# Find runs whose logs mention "merge train" — show excerpts
curl -s '/api/cve-runs/search?log=merge+train' | jq '.log_matches[] | {recorded_at, log_excerpt}'
# Find tickets for a package, also searching logs
curl -s '/api/cve-runs/search?package=openssl&log=advisory+failed&days=7' \
| jq '{tickets: [.results[].detail.key], log_hits: .log_matches | length}'
# List all ticket keys matching a free-text query
curl -s '/api/cve-runs/search?q=needs-attention&days=14' | jq '[.results[].detail.key] | unique'
CLI
# View latest commit status
hummingbird-dashboard --application myapp --format table commit
# View specific commit
hummingbird-dashboard --application myapp commit abc1234
# View multiple commits
hummingbird-dashboard --application myapp --format table commit --limit 10
# Component status overview
hummingbird-dashboard --application myapp component
# JSON output
hummingbird-dashboard --format json commit | jq .
# Auto-rerun failed releases
hummingbird-dashboard auto-rerun
# Analyze failed releases via LLM
hummingbird-dashboard analyze-failures
# Probe catalog image pull availability (HUM-721)
hummingbird-dashboard check-image-pulls
hummingbird-dashboard check-image-pulls --dry-run
hummingbird-dashboard check-image-pulls --force
REST API Reference
All endpoints return JSON. Read-only GET endpoints are unauthenticated.
Interactive OpenAPI/Swagger documentation is available at /docs.
RPM Tracker
GET /api/rpm-upstream-state returns the collection generation and checkout
SHA with upstream, package status, rollout reasons, and all open update MRs
from the independently migrated rpm_tracker schema. It also exposes the MR
reconciliation scope and freshness and one selected blocking_mr per package
row. GET /api/rpm-update-mrs lists every open update MR independently of that
selection, including multiple MRs for one package. Add ?stuck_only=true for
MRs older than 24 hours and MRs with unknown creation-based ages; unknown ages
remain null rather than disappearing. The dashboard does not create or write these
tables. Run the tracker initializer before deploying this API; until a compatible
tracker schema is present, this endpoint returns HTTP 503. See
RPM Tracker for migration and storage details.
Tier 1 — Core Pipeline Status
GET /api/apps/{app_name}/commits
Pipeline status for commits in an application.
| Parameter | Type | Default | Description |
|---|---|---|---|
sha |
string | — | Filter by commit SHA (prefix) |
component |
string | — | Filter by component name |
limit |
int | 10 | Max commits to return (max 100) |
curl -s 'http://localhost:8080/api/apps/rpms/commits?limit=5' | jq .
curl -s 'http://localhost:8080/api/apps/rpms/commits?sha=abc123&component=openssl' | jq .
GET /api/apps/{app_name}/components
Latest build status per component, grouped by state.
| Parameter | Type | Default | Description |
|---|---|---|---|
search |
string | — | Filter components by name (substring) |
curl -s 'http://localhost:8080/api/apps/rpms/components' | jq .
curl -s 'http://localhost:8080/api/apps/rpms/components?search=openssl' | jq .
Tier 2 — Operational Visibility
GET /api/dashboard
Full overview across all applications: aggregate build counts, failed releases, component staleness, and recent activity.
curl -s 'http://localhost:8080/api/dashboard' | jq .
GET /api/releases/failed
List of currently failed releases with analysis results.
| Parameter | Type | Default | Description |
|---|---|---|---|
application |
string | — | Filter by application name |
curl -s 'http://localhost:8080/api/releases/failed' | jq .
curl -s 'http://localhost:8080/api/releases/failed?application=rpms' | jq .
GET /api/releases/{name}/analysis
LLM failure analysis for a specific release (root cause, classification, recommendation).
curl -s 'http://localhost:8080/api/releases/my-release-abc/analysis' | jq .
GET /api/releases/auto-rerun-log
History of automatic rerun attempts.
| Parameter | Type | Default | Description |
|---|---|---|---|
limit |
int | 50 | Max entries |
curl -s 'http://localhost:8080/api/releases/auto-rerun-log?limit=10' | jq .
GET /api/releases/analysis-log
History of LLM analysis runs.
| Parameter | Type | Default | Description |
|---|---|---|---|
limit |
int | 50 | Max entries |
curl -s 'http://localhost:8080/api/releases/analysis-log?limit=10' | jq .
GET /api/releases/analysis-costs
Cumulative token usage and estimated cost for LLM analysis runs.
curl -s 'http://localhost:8080/api/releases/analysis-costs' | jq .
GET /api/mrs
Merge requests across monitored repositories.
| Parameter | Type | Default | Description |
|---|---|---|---|
state |
string | opened |
MR state: opened, merged, closed, all |
project |
string | — | Filter by project (repository) name |
curl -s 'http://localhost:8080/api/mrs' | jq .
curl -s 'http://localhost:8080/api/mrs?state=merged&project=rpms' | jq .
GET /api/mrs/projects
List of all projects (repositories) with tracked merge requests.
curl -s 'http://localhost:8080/api/mrs/projects' | jq .
GET /api/mr/{project}/{iid}
Detail for a single merge request, including all versions and per-version build status.
curl -s 'http://localhost:8080/api/mr/rpms/42' | jq .
GET /api/cve/status
Current CVE ticket status across all tracked packages.
curl -s 'http://localhost:8080/api/cve/status' | jq .
GET /api/cve/open
Open HUM CVE tracker tickets with SLO status. Notified is the later of HUM ticket created and fix available (upstream or Fedora). SLO is green PASS unless a fix is available and elapsed time is within 8h of the selected SLO (yellow AT RISK) or past it (red FAIL).
| Parameter | Type | Default | Description |
|---|---|---|---|
slo |
float | 24 | SLO threshold hours (24, 72, or 168) |
include_next_release |
bool | true | Include cve-next-release tickets |
curl -s 'http://localhost:8080/api/cve/open?slo=24' | jq .
GET /api/cve/closed
Closed HUM CVE tracker tickets with Red Hat CSAF VEX reconciliation
(HUM-5843). Analysis stores package-scoped vex_status and Jira
resolution; the dashboard computes MATCH from those facts
(Done-Errata → fixed, Not a Bug → known_not_affected,
package_not_listed, or fixed – a downstream package rebuilt by an
unrelated dependency’s fix is not a contradiction of a Not a Bug verdict).
Shows Hummingbird vex_status, match state
(matched / pending / mismatch), and vex_resolved (first
scan time where VEX agreed with the Jira resolution). Event metadata is
merged across a ticket’s rows in occurred_at order (HUM-6091): later
non-empty fields overlay earlier ones, so Close/VEX facts are not stuck
on the first cve_published row. The Resolution column uses Jira
Closed / {resolution} when stored analysis text is not already Closed.
The Closed tab shows the VEX timestamp as a link to the CVE’s Red Hat
CSAF VEX document when the package is named in CSAF. package_not_listed
(Not a Bug, package absent from the document) shows N/A with no link:
this ticket did not produce a VEX change.
Tickets without a VEX match status are counted below the summary cards. The
awaiting-vex label count overlaps the matched, pending, mismatch, and unknown
groups; it is not an additional category to add to the total.
The HTML page renders 100 tickets per page (page query, 1-based) while the
summary cards, CVE Agent performance and CSV export cover the whole period.
| Parameter | Type | Default | Description |
|---|---|---|---|
days |
int | 28 | Time window (0 = all time) |
curl -s 'http://localhost:8080/api/cve/closed?days=30' | jq .
Entries carry the agent verdict, closure agreement, linked MRs, any recorded
ticket cost (agent_cost, with a null estimated_cost_usd when the analysis
predates cost reporting, and unpriced_calls when some calls had no pricing)
and any saved review; agent_stats aggregates them as described above,
including cost_total, cost_tickets, cost_unknown and cost_partial.
PUT /api/retrigger/cve-agent-review/{ticket_key}
Save a reviewer’s needed and mr_outcome, plus an optional note, for a
closed CVE ticket. Requires login; the reviewed verdict, resolution and MR
identities/head SHAs are snapshotted server-side. Returns 422 when the ticket
has no agent verdict. The response returns status, ticket_key, the ticket’s
updated agent_outcome_label, agent_outcome_tone and agent_outcome_note,
and the reviewed_text byline, so the page updates the row and review panel in
place without reloading. The save reads only the reviewed ticket, not the whole
Closed tab; period statistics refresh on the next page load.
curl -s -X PUT 'http://localhost:8080/api/retrigger/cve-agent-review/HUM-123' \
-H 'Content-Type: application/json' \
-d '{"needed": "backport", "mr_outcome": "edits", "note": "test needed fixing"}'
GET /api/cve/r-time
CVE R-Time (notified to done) for Done-Errata tickets measures time spent on a
fixable CVE, so its start never predates the upstream fix in either mode.
Include CVE-HUM starts at CVE publication (NVD datePublished /
cve_published), falling back to HUM ticket creation only when CVE
publication is missing – CVE publication wins even when the HUM ticket
predates it, so an embargoed ticket’s R-Time starts at public disclosure
rather than at ticket creation (which can predate disclosure by days while
the CVE is still privately embargoed). Exclude CVE-HUM starts at the later of
HUM created and CVE publication – same as HUM created for a normal ticket
(Exclude’s purpose is to ignore filing lag after disclosure), but an
embargoed ticket’s clock must never start before public disclosure in either
mode, so it also starts at CVE publication there. Whichever of the two the
mode picks, Upstream Fix Merged is then applied as a floor on top: the final
notified timestamp is the later of that value and Upstream Fix Merged, so
neither mode ever counts time before a fix existed to backport. Delivery is
the catalog image publish when the package is a catalog image source,
otherwise the RPM publish. Include VEX requires delivery, VEX updated, and
Closed / Done-Errata; Exclude VEX requires delivery and close only. The done
timestamp is the latest of the required gates, and only when all of them
exist. Incomplete tickets stay on the table with elapsed time until now; the
days window filters by completion (fallback: Done-Errata close, then
notified), so a period means completed in that window – the Closed tab
filters by Jira close date instead. CVE-HUM is filing lag after NVD
(max(HUM created − NVD, 0)); HUM-first tickets show 0.0h. ADV-VEX is
Done-Errata close to VEX feed update (max(VEX − close, 0)); close is
recorded when the advisory MR merges. Fedora is not start or end; Upstream
Fix Merged only ever gates R-Time’s start, never its end. Pre-built (delivery
before notification) is informational only. Rows without
catalog_image_source in metadata still require an image until the collector
restamps them. Each entry includes package onboarding timestamps and delay
analysis (rpm_first_published_at, pkg_lag_hours, and our_delay_hours).
Default excludes cve-next-release, CVE-HUM time, and VEX time.
| Parameter | Type | Default | Description |
|---|---|---|---|
days |
int | 28 | Time window (0 = all time) |
slo |
float | 168 | SLO threshold hours (24, 72, or 168) |
include_next_release |
bool | false | Include cve-next-release tickets |
include_hum_cve_time |
bool | false | Include NVD-to-HUM filing lag in R-Time |
include_vex_time |
bool | false | Require VEX feed update before R-Time ends |
curl -s 'http://localhost:8080/api/cve/r-time?days=10&slo=168' | jq .
GET /api/cve/run-log
History of CVE analysis CronJob runs.
| Parameter | Type | Default | Description |
|---|---|---|---|
limit |
int | 100 | Max entries |
curl -s 'http://localhost:8080/api/cve/run-log?limit=10' | jq .
POST /api/cve/agent-cost-keys
Report which of the posted tickets already have a recorded CVE agent cost, so
the collector downloads only the remaining analysis attachments. The check
returns keys only and uses the unique ticket-key index. Requires the same
bearer token as /api/cve-report, which accepts the costs as an agent_costs
list (ticket_key, input_tokens, output_tokens, cache_read_tokens,
cache_creation_tokens, unpriced_calls, estimated_cost_usd, where a null
cost marks a pre-cost attachment) and rejects the whole report when a row is
malformed. The cache counters are subsets of input_tokens, and a non-zero
unpriced_calls makes the cost a lower bound.
curl -s -X POST 'http://localhost:8080/api/cve/agent-cost-keys' \
-H "Authorization: Bearer $CVE_REPORT_TOKEN" \
-H 'Content-Type: application/json' \
-d '{"tickets": ["HUM-123", "HUM-124"]}' | jq .
GET /api/cve-export
Export CVE sync data as JSON for prod-to-preprod replication. Valid section
values are cve_analysis_log, cve_ticket_events, package_lifecycle, and
cve_ticket_blame.
Optional pagination parameters:
| Parameter | Type | Default | Description |
|---|---|---|---|
section |
string | — | Section to export |
limit |
int | 2000 | Rows per page |
offset |
int | 0 | Pagination offset |
curl -s -H "Authorization: Bearer $CVE_REPORT_TOKEN" \
'http://localhost:8080/api/cve-export' | jq '.cve_ticket_events | length'
curl -s -H "Authorization: Bearer $CVE_REPORT_TOKEN" \
'http://localhost:8080/api/cve-export?section=cve_ticket_events&limit=1000&offset=0' \
| jq '.count,.has_more'
# Package-scoped lifecycle milestones (rpm_first_published, etc.)
curl -s -H "Authorization: Bearer $CVE_REPORT_TOKEN" \
'http://localhost:8080/api/cve-export?section=package_lifecycle&limit=10000&offset=0' \
| jq '.rows[] | select(.event_type == "rpm_first_published")'
Event type validation. The import path (POST /api/cve-import) validates
every event_type value against a canonical allowlist defined in
hummingbird_dashboard/sources.py:
CANONICAL_TICKET_EVENT_TYPES— values accepted forcve_ticket_events:cve_published,osidb_flaw_created,osidb_affect_created,cve_embargo_lifted,osidb_flaw_public,hum_ticket_created,hum_ticket_closed,upstream_fix_merged,fedora_update_available,rpm_fix_published_to_pulp,image_rebuilt_on_quay,vex_resolved.CANONICAL_PACKAGE_EVENT_TYPES— values accepted forpackage_lifecycle:rpm_first_published.
Legacy names (e.g. jira_created, delivered_in_rpm, …) are
mapped to their canonical equivalents via _LEGACY_EVENT_KEY_MAP.
Unknown names are rejected with HTTP 400.
POST /api/cve-import
Replace local CVE sync data from a payload previously returned by
/api/cve-export.
curl -s -X POST \
-H "Authorization: Bearer $CVE_REPORT_TOKEN" \
-H "Content-Type: application/json" \
--data @cve-export.json \
'http://localhost:8080/api/cve-import' | jq .
curl -s -X POST \
-H "Authorization: Bearer $CVE_REPORT_TOKEN" \
-H "Content-Type: application/json" \
--data '{"section":"cve_ticket_events","rows":[],"reset":true}' \
'http://localhost:8080/api/cve-import' | jq .
Ticket-event types such as osidb_flaw_created, osidb_affect_created,
cve_embargo_lifted, and osidb_flaw_public import with the rest of
cve_ticket_events because they are on CANONICAL_TICKET_EVENT_TYPES.
Analysis-log exports include run_started_at when available. Paginated imports
ignore rows whose non-null identity is already present, making retries safe;
legacy rows without an identity remain append-only. The ingest timestamp is
exported as recorded_at; imports still accept the legacy run_at key, but a
sync target older than this release cannot read the new key.
occurred_at conflict resolution on import matches the live collector path
(HUM-6860): rpm_fix_published_to_pulp and image_rebuilt_on_quay are
mutable delivery timestamps where a non-null incoming value wins, so a sync
can update them, without letting a null incoming value erase an
already-recorded delivery timestamp; every other event_type is write-once,
keeping the existing timestamp once set so a sync or backfill re-run cannot
overwrite an established immutable milestone.
Bulk prod-to-preprod replication of cve_analysis_log, cve_ticket_events,
package_lifecycle, and cve_ticket_blame no longer goes through this API —
cve_analysis_log rows can carry up to 512 KiB of log_output, so paginating
that table meant hundreds of small requests that aborted entirely on the
first 5xx. See copy_prod_to_preprod_db in
CVE Analysis
Reference
for the
direct pg_dump/pg_restore sync tool that replaced it. These endpoints
remain in use for the smaller, targeted writes collect_rpm_first_published
and backfill_osidb_timestamps.py make directly against prod or preprod.
Tier 3 — Analytics & SRE Metrics
GET /api/stats/build-throughput
Build counts over time (successful, failed, total).
| Parameter | Type | Default | Description |
|---|---|---|---|
days |
int | 30 | Lookback window in days |
curl -s 'http://localhost:8080/api/stats/build-throughput?days=7' | jq .
GET /api/stats/release-success-rate
Release success/failure ratios over time.
| Parameter | Type | Default | Description |
|---|---|---|---|
days |
int | 30 | Lookback window in days |
application |
string | — | Filter by application |
curl -s 'http://localhost:8080/api/stats/release-success-rate?days=14&application=rpms' | jq .
GET /api/stats/auto-rerun-effectiveness
Success rate and time-to-resolution for automatic reruns.
| Parameter | Type | Default | Description |
|---|---|---|---|
days |
int | 30 | Lookback window in days |
curl -s 'http://localhost:8080/api/stats/auto-rerun-effectiveness?days=7' | jq .
GET /api/components/staleness
Components ranked by time since last successful build.
curl -s 'http://localhost:8080/api/components/staleness' | jq .
GET /api/releases/similar-failures
Find releases with error messages similar to a given string.
| Parameter | Type | Default | Description |
|---|---|---|---|
error |
string | — | Error text to match against |
curl -s 'http://localhost:8080/api/releases/similar-failures?error=timeout+connecting' | jq .
GET /api/apps/{app_name}/last-successful
Timestamp and SHA of the last fully successful pipeline per component.
curl -s 'http://localhost:8080/api/apps/rpms/last-successful' | jq .
Tier 4 — System
GET /api/service-status/json
Health of upstream services the dashboard depends on (database, Konflux API, KubeArchive, etc.).
curl -s 'http://localhost:8080/api/service-status/json' | jq .
GET /api/settings
Current runtime settings (auto-rerun enabled, analysis enabled, blocked patterns, etc.).
curl -s 'http://localhost:8080/api/settings' | jq .
GET /health
Enriched health check returning service version, uptime, database connectivity, and dependency status.
curl -s 'http://localhost:8080/health' | jq .
Configuration
Environment Variables
| Variable | Default | Description |
|---|---|---|
DATABASE_URL |
postgresql://...localhost:5432/ |
PostgreSQL connection URL |
PORT |
8080 |
Web server port |
Authentication Variables
These control the retrigger functionality (requires OAuth proxy in production):
| Variable | Default | Description |
|---|---|---|
TRIGGER_AUTH_MODE |
oauth |
oauth (production) or local |
TRIGGER_AUTH_GROUP |
konflux-hummingbird-admin-access |
OpenShift group required to retrigger |
TRIGGER_LOCAL_USER |
local-dev |
Username when TRIGGER_AUTH_MODE=local |
CVE_REPORT_TOKEN |
Bearer token for CVE API auth |
The dashboard checks TRIGGER_AUTH_GROUP against the user’s groups on the
OpenShift cluster hosting the dashboard. Access on the Konflux cluster alone
does not grant retrigger permission. Membership in the default
konflux-hummingbird-admin-access group is managed in Rover and synchronized
to the dashboard cluster by the infrastructure configuration. Claiming a CVE
ticket requires only a dashboard login, so the user menu can show Claim ✅ and
Retrigger ❌. Successful group lookups are cached for five minutes; failed
lookups are retried on the next request.
For local development with retrigger enabled:
TRIGGER_AUTH_MODE=local DATABASE_URL=... ./dev.sh start
Auto-Rerun Variables
These control the auto-rerun cronjob and failure analysis:
| Variable | Default | Description |
|---|---|---|
AUTO_RERUN_MIN_AGE_MINUTES |
30 |
Minimum failure age before retrying |
AUTO_RERUN_MAX_RETRIES |
3 |
Max rerun attempts per snapshot+plan |
KONFLUX_KUBECONFIG_PATH |
Path to Konflux kubeconfig file | |
RELEASE_NAMESPACE |
Namespace where releases are created | |
MANAGED_NAMESPACE |
Namespace for fetching release PLRs | |
KUBEARCHIVE_URL |
KubeArchive API URL for archived resources | |
GOOGLE_APPLICATION_CREDENTIALS |
Path to GCP SA key for Vertex AI | |
GOOGLE_CLOUD_PROJECT |
GCP project ID for Vertex AI | |
ANALYSIS_MODEL_API_KEY |
Gemini API key (fallback if no GCP creds) | |
ANALYSIS_MODEL |
gemini-2.5-flash |
LLM model for failure analysis |
ANALYSIS_MODEL_REGION |
global |
Vertex AI region |
MAX_ANALYSES_PER_CYCLE |
5 |
Max releases to analyze per cycle |
SLACK_WEBHOOK_URL |
Slack webhook for rerun/analysis notifications | |
DASHBOARD_URL |
Dashboard base URL for Slack links | |
GITLAB_SLACK_MAP_PATH |
/etc/hummingbird/gitlab-slack-map.yaml |
GitLab→Slack map for author @-mentions (infra-mounted; missing omits mention) |
PUSH_RERUN_MIN_AGE_MINUTES |
10 |
Min push build failure age before retrying |
PUSH_RERUN_MAX_RETRIES |
3 |
Max retry attempts per component+sha |
Error Budget Alert Variables
The dashboard pod periodically evaluates CVE R-Time error budget (168h SLO, 30-day window) and posts to Slack on worsening transitions only:
| Remaining budget | Slack alert |
|---|---|
| ≤20% | Warning |
| ≤5% | Critical |
| 0% | Exhausted (sprint-stop) |
Incoming webhooks always post to the channel they were created for; the
payload cannot select #team-hummingbird. Create the webhook for that
channel and set ERROR_BUDGET_SLACK_WEBHOOK_URL (or point the shared
SLACK_WEBHOOK_URL at that webhook).
State is stored in dashboard_settings (error_budget_rtime_alert_state) so
the same threshold is not re-alerted. The stored state is claimed with a
compare-and-set before Slack is called, and rolled back if the post fails, so
a transient Slack error is retried on the next interval and concurrent
evaluators cannot double-post the same transition. Recovery updates the
stored state silently so a later re-worsening can alert again. No message is
sent when the webhook URL is unset, or when alerts are disabled via the
R-Time page toggle (error_budget_alerts_enabled in dashboard_settings;
default enabled).
| Variable | Default | Description |
|---|---|---|
ERROR_BUDGET_SLACK_WEBHOOK_URL |
SLACK_WEBHOOK_URL |
Incoming webhook created for the alert channel (typically #team-hummingbird) |
SLACK_WEBHOOK_URL |
Fallback webhook if the error-budget URL is unset | |
DASHBOARD_URL |
Base URL for the R-Time deep link in alerts | |
ERROR_BUDGET_ALERT_INTERVAL_SECONDS |
900 |
Seconds between background checks |
Failures
The failures section is accessible via the “Failures” nav item and provides two views selectable by tab: Releases and Push Builds.
Failed Releases
The /failures/releases page shows all failed releases with:
- LLM Analysis — Each failure is analyzed by Gemini with root cause, classification, and recommendation
- Failure Classification — Transient, Configuration, Code, External Service, or Unknown
- Rerun History — Past rerun attempts and outcomes per snapshot
- Error Pattern Matching — Auto-block snapshots matching known error patterns
- Auto-Rerun — Automatically retry transient failures (configurable via dashboard toggle)
- Analysis Toggle — Enable/disable LLM analysis from the dashboard
CLI Subcommands
The auto-rerun subcommand retries eligible failed releases:
hummingbird-dashboard auto-rerun --application myapp
The auto-rerun-push subcommand retries eligible failed push builds:
hummingbird-dashboard auto-rerun-push --application myapp
Before creating a retry PipelineRun, the command records an atomic claim for
its component and commit in push_build_rerun_claims. Later CronJob runs skip
that pair until the claimed PipelineRun is recorded as failed in the dashboard
database. A successful or still-running retry prevents another attempt, and
PUSH_RERUN_MAX_RETRIES also applies to claims that are not yet ingested.
If the create request has an uncertain outcome, the claim is retained to avoid
accidentally creating a duplicate. Check the claimed plr_name in Konflux
before manually clearing such a claim.
The analyze-failures subcommand runs LLM analysis on unanalyzed failures:
hummingbird-dashboard analyze-failures --managed-namespace rhtap-releng-tenant
All three are designed to run as Kubernetes CronJobs. Slack messages for auto-rerun, push-retry, analysis, and manual UI reruns include a GitLab MR link (and author @-mention when mapped) when the failure SHA resolves to an MR. Release success messages follow reruns tracked by the previous auto-rerun run, so delayed status ingestion does not skip a success. Failed Slack posts are recorded as pending in the auto-rerun log and retried on the next run.
Image Pull Availability
The /availability page tracks HUM-721 (SLI-UE1-B): can a customer actually
pull the images we publish. Continuously pulling every catalog image on a
tight schedule is not feasible (terabytes of registry traffic), so the check
is split into three tiers, run by check-image-pulls:
- Manifest HEAD (exhaustive) — a cheap HEAD request, no image bytes
transferred, against every
latest*image:variant tag in the catalog on every run. This is the headline “Estimated Availability” number. A digest cache skips the live HEAD when the tag’s digest is unchanged, the previous check succeeded, and it was confirmed within the last 6 hours. - Blob sample (sampled) — a real HEAD against every blob (config + layers) of a small random sample of currently-available images each run, confirming referenced layers still exist in storage.
- Full-pull sample (sampled) — a real GET of one image’s config blob (a few KB, following the registry’s CDN redirect) each run — genuine end-to-end retrieval, not just a HEAD.
Both sample tiers prioritize tags published in the last hour over older ones, since “did the thing we just shipped actually land” is the highest-value failure mode. The resulting availability % is estimated: the manifest tier is exhaustive, but the blob/full-pull tiers are sampled, not exhaustive — the page shows all three separately rather than blending them into one number.
- SLO: 95% target, 28-day rolling window, based on the manifest tier
- Error Budget: manifest-check misses allowed within the window
hummingbird-dashboard check-image-pulls
hummingbird-dashboard check-image-pulls --dry-run
hummingbird-dashboard check-image-pulls --force
| Option | Description |
|---|---|
--dry-run |
Probe without writing results to the database |
--force |
Force live manifest HEAD checks (ignore digest cache) |
--catalog-api-base |
Catalog API base URL (default: CATALOG_API_BASE or built-in) |
Designed to run as a Kubernetes CronJob (every 15 minutes). Checks can be
disabled from the dashboard via the image_pull_checks_enabled setting.
Merge Requests
The /mrs page shows merge requests across all monitored repositories with:
- State filter - Open, merged, closed, or all MRs
- Project filter - Filter by specific repository
- Build status - Aggregate status across all components
The /mr/{project}/{iid} detail page shows:
- All MR versions - Each head commit SHA that was pushed to the MR
- Build status per version - Full pipeline status (build, snapshot, test, release)
- Links to Konflux UI - Direct links to PipelineRuns and Snapshots
Versions are ordered by latest event timestamp, so force-pushed commits appear in the correct position even if they reuse an earlier SHA.
Component Status
The dashboard tracks component build status with these states:
| Status | Icon | Description |
|---|---|---|
| Success | ✅ | Build passed for expected commit |
| Superseded | 🔄 | Expected commit not built, but newer succeeded |
| Failed | ❌ | Build failed for expected commit |
| Stale | ⚠️ | Build not triggered for expected commit |
| Running | ⏳ | Build in progress |
| Missing | ❓ | No build found |
Components are grouped by status: Failed/Stale → Running → OK → Missing.
Trigger Rules
The dashboard uses application-specific rules to determine which components are affected by a push:
- containers: Changes in
images/{component}trigger builds - rpms: Changes in
rpms/{component}ormock/mock.cfgtrigger builds - tools: Changes in
{component}directory trigger builds
Certain files are excluded from triggering (README, templates, etc.).
Development
See the main README for development workflows.
Reusable text filtering
hummingbird_dashboard/web/static/text-filter.mjs exports initTextFilter(root).
It is independent of RPM data and can enhance other dashboard tables.
Mark the input with data-text-filter-input and each searchable row with
data-filter-text. The row attribute contains the text to search, such as a
package name or a combination of ticket key, package, and CVE ID. The optional
count, table wrapper, and empty message use the attributes shown below.
Matching ignores case and surrounding whitespace, and leaves table order intact.
<div data-text-filter>
<form method="get">
<label for="package-search">Package</label>
<input
id="package-search"
name="search"
type="search"
data-text-filter-input
/>
<button type="submit">Apply filters</button>
</form>
<p aria-live="polite">Showing <span data-text-filter-count>1</span></p>
<div data-text-filter-table>
<table>
<tbody>
<tr data-filter-text="example-package">
<td>example-package</td>
</tr>
</tbody>
</table>
</div>
<p data-text-filter-empty hidden>No matches.</p>
</div>
<script type="module">
import { initTextFilter } from "/static/text-filter.mjs";
document.querySelectorAll("[data-text-filter]").forEach(initTextFilter);
</script>
The input’s name is the URL query parameter. Mark sort or filter links with
data-text-filter-link to preserve the typed text when navigating. An unnamed
input filters rows without changing URLs. Components are scoped to their root,
and a form is optional. Use a dedicated GET filter form when server controls are
present. Text alone does not submit a request, but changing another form field
still submits normally.
For expandable tickets, put data-filter-text on a shared row group, such as a
tbody, rather than only the main row. The group hides together while its
details keep their expansion state.
Render all rows allowed by the other filters, not just the initial text matches.
Use hidden on initial nonmatching rows and set the initial count and empty
message on the server. Paginated tables should keep server filtering instead.
RPM Tracker uses this module now; adding it to CVE Tracker is separate work.
Running Tests
cd hummingbird-dashboard
pip install -e ".[dev]"
pytest
node --test tests/*.mjs
Browser logic tests use the Node.js built in test runner and are also included
in make node-test-host from the repository root.
The PostgreSQL migration regression test uses the CI PostgreSQL sidecar via
DATABASE_URL. Locally, set DATABASE_URL to a disposable PostgreSQL database
to run it; without one, the test is skipped.
Building Container Image
cd hummingbird-dashboard
podman build -f Containerfile -t hummingbird-dashboard .
Database Migrations
The dashboard uses Alembic to manage its schema.
Migrations run automatically on every startup via alembic upgrade head.
On first startup against an existing environment, the dashboard detects that the
schema is already present (via the blocked_snapshots sentinel table) and
automatically stamps the baseline migration as applied before upgrading. No
manual intervention is required.
Local development — migrations run automatically via ./dev.sh db-seed:
./dev.sh db-start
./dev.sh db-seed # runs alembic upgrade head then seeds test data
./dev.sh start
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.