Hummingbird Data Flow

Overview of all data sources, what is stored where, and how data moves between systems.

Data sources

PostgreSQL databases

There are two Postgres databases.

1. hummingbird-status — Konflux pipeline events

Fed by SNS/SQS events via the hummingbird-status ingestor.

The legacy tables above remain in public. The independent pipeline-timing service consumes a second SQS copy of each SNS envelope and writes only to its Alembic-managed pipeline_timing schema in the same database. The pipeline-timing Pod’s init container creates that schema and runs migrations before its worker container starts. The single-replica Deployment uses Recreate for upgrades. Scaling it to zero leaves the queue untouched without affecting legacy ingestion.

These tables store selected fields and event references, not raw event JSON. The archive retains the raw SNS envelope. Canonical HTTPS repository URL plus SHA links MR commits, GitLab pipelines, and Konflux PipelineRuns; project and pipeline IDs link GitLab jobs. Kubernetes owner UIDs link TaskRuns to PipelineRuns, while the Snapshot label and source run name connect build to test, and a Release names its Snapshot and managed PipelineRun. Distinct run UIDs and job IDs preserve retries. Consumers calculate elapsed intervals from the stored timestamps. The Dashboard continues to read only the public tables.

2. hummingbird-dashboard — CVE lifecycle + dashboard overlays

cve_ticket_events lifecycle milestone event_type values

These are the canonical names after the HUM-5918 migration:

rpm_fix_published_to_pulp and image_rebuilt_on_quay are mutable — they are updated when a newer delivery event supersedes the previous one. All other event types are write-once: once set, occurred_at is never overwritten.

cve_ticket_events.metadata JSONB fields

Each row carries a metadata blob that reflects the ticket’s state at the time of the most recent analysis run:

Jira whiteboard (customfield_10841)

The CVE analysis tool uses the Jira whiteboard field as a per-ticket timestamp cache. This is being phased out in favour of Postgres as the durable store (HUM-5917), but is still the source for a subset of fields.

The whiteboard holds compact JSON. The relevant sub-key is cve_cycle:

The whiteboard is read and written only by the CVE analysis tool. The dashboard never reads it directly; its authoritative source is always cve_ticket_events.

Jira fields read by the CVE analysis tool

On each run the analysis tool fetches every open HUM CVE tracker ticket and reads the following fields from the Jira REST API:

Data flow

┌─────────────────────────────────────────────────────────────────┐
│  External data sources                                          │
│                                                                 │
│  Jira REST API ─────────────────────────────────────┐           │
│  NVD / CVE list ────────────────────────────────────┤           │
│  GitHub / GitLab (upstream fix commits) ────────────┤           │
│  Fedora updates ────────────────────────────────────┤           │
│  Red Hat Pulp (RPM publish times) ─────────────────►│           │
│  Hummingbird catalog API (image rebuild times) ─────┤           │
│  Red Hat CSAF VEX feed ─────────────────────────────┤           │
│  OSIDB (subpackage affectedness) ───────────────────┘           │
└──────────────────────────────┬──────────────────────────────────┘
                               │
                               ▼
┌──────────────────────────────────────────────────────────────────┐
│  CVE analysis tool (cron, hummingbird-cve-analysis)              │
│                                                                  │
│  • Reads Jira ticket fields + whiteboard cache                   │
│  • Resolves fix SRPM from Pulp repodata                          │
│  • Resolves image rebuild timestamp from catalog API             │
│  • Checks upstream PR/release dates (GitHub/GitLab)              │
│  • Fetches CVE publish date (NVD/CVE list)                       │
│  • Reconciles VEX status (CSAF feed)                             │
│  • Checks subpackage affectedness (OSIDB)                        │
│                                                                  │
│  Writes back to:                                                 │
│  ├── Jira whiteboard (cve_cycle timestamp cache)                 │
│  ├── Jira labels / Fixed-in-Build field                          │
│  ├── Jira comments                                               │
│  └── dashboard DB → cve_ticket_events (one row per milestone)    │
│                      cve_analysis_log (run record)               │
└──────────────────────────────┬───────────────────────────────────┘
                               │
           ┌───────────────────┴───────────────────┐
           │                                       │
           ▼                                       ▼
┌──────────────────────────┐      ┌───────────────────────────────┐
│  Jira whiteboard         │      │  dashboard DB                 │
│  (per-ticket JSON cache) │      │  cve_ticket_events table      │
│  cve_cycle timestamps    │      │  one row per (ticket,         │
│  ← being phased out      │      │    event_type) milestone      │
│    (HUM-5917)            │      └───────────────┬───────────────┘
└──────────────────────────┘                      │
                                                  ▼
                               ┌──────────────────────────────────┐
                               │  hummingbird-dashboard API       │
                               │                                  │
                               │  Pivots cve_ticket_events rows   │
                               │  into per-ticket dicts, computes:│
                               │  • entry-level R-Time fields     │
                               │    (HUM-5920)                    │
                               │  • duration legs inside stages   │
                               │    (HUM-5921)                    │
                               │                                  │
                               │  Serves JSON + Jinja templates   │
                               └──────────────────────────────────┘

flowchart LR
    EVENTS[GitLab pushes, pipelines, jobs, MRs<br/>Kubernetes PipelineRuns, TaskRuns, Snapshots, Releases]
    SNS[(SNS topic)]
    ARCHIVER[S3 archiver]
    S3[(Deterministic archive)]
    STATUS[hummingbird-status ingestor]
    STATUS_DB[(hummingbird-status DB)]
    DASHBOARD[Dashboard API]
    TIMELINE[Pipeline timing queue and worker]
    TIMELINE_DB[(pipeline_timing schema)]

    EVENTS --> SNS
    SNS --> ARCHIVER --> S3
    SNS --> STATUS --> STATUS_DB --> DASHBOARD
    SNS --> TIMELINE --> TIMELINE_DB

Kubernetes TaskRun transport

The Kubernetes event forwarder applies optional resource labelSelector and fieldSelector values to both its initial LIST and subsequent Watch requests. Field-selector support depends on the Kubernetes resource and API server; TaskRun collection should use a confirmed stable RPM application label instead of a status field selector.

The forwarder publishes the logical kubeconfig context.cluster value, rather than the API server URL, with each Kubernetes event. For TaskRuns, it removes only metadata.managedFields before gzip/base64 encoding; identity, owner references, labels, annotations, task specification, conditions, results, pod name, and timing remain in the published and archived payload.

API field names

The dashboard API exposes these computed fields per R-Time entry:

Entry-level fields

Duration legs (stages dict)

Aggregate stats

Package lifecycle data

rpm_first_published is the earliest timestamp at which any SRPM for a given package appeared in the Hummingbird Pulp repo. It is package-scoped — one row per package — as opposed to cve_ticket_events which is ticket-scoped.

Collection

scripts/collect_rpm_first_published --prod --rpms-repo /path/to/rpms

Flow:

1. Loads the package list from rpms_repo.load_package_map_from_metadata().
2. For each package: calls pulp.fetch_earliest_srpm_time(), which browses packages.redhat.com/.../source/Packages/{letter}/, parses the HTML listing already used by the analysis tool, and returns the earliest upload timestamp across all SRPMs for that package.
3. POSTs all results in one request to /api/cve-import (same endpoint used for cve_ticket_events sync). The dashboard upserts with COALESCE(LEAST(existing, incoming), existing, incoming) — re-runs only update occurred_at if the incoming timestamp is earlier, and NULLs are never stored over a real value.

Surfaced in dump_lifecycle

scripts/dump_lifecycle --prod HUM-1234

After fetching ticket milestones from cve_ticket_events, the script resolves the package name from those rows and makes a second call to /api/cve-export?section=package_lifecycle to fetch rpm_first_published for that package. It is shown at the bottom of the text output and under rpm_first_published in the JSON output.