This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

CVE Analysis

Automated analysis of HUM Jira Security (CVE) tickets. For each ticket, the tool fetches vulnerability data from MITRE and NVD, compares it against the versions shipped in the Hummingbird package repository, searches for upstream and Fedora fixes, and computes whether the ticket should be closed, moved to In Progress, or flagged for manual investigation.

CVE Lifecycle & State Vocabulary

A CVE tracker moves through a shared lifecycle that both this tool and the CVE agent reason about. The deterministic analysis decides a target resolution for each ticket; the residual it cannot resolve (cve-needs-attention) is handed to the agent; closed tickets are later reconciled against the VEX feed.

stateDiagram-v2
    direction TB

    [*] --> New

    New --> InProgress: affected / needs investigation
    New --> DoneErrata: fixed by analysis
    New --> NotABug: component not present
    New --> WontDo: package EOL

    InProgress --> Agent: upstream-fix-available / cve-needs-attention

    Agent: CVE Agent
    Agent --> Review: fix MR opened
    Agent --> DoneErrata: already_fixed
    Agent --> NotABug: not_affected
    Agent --> InProgress: no_upstream_fix / needs_human / error

    Review --> DoneErrata: MR merged
    Review --> Review: MR conflicts / needs human

    state "Closed (JiraStatus)" as Closed {
        DoneErrata: Done-Errata
        NotABug: Not a Bug
        WontDo: Won't Do
    }

    DoneErrata --> AwaitingVEX: enqueue awaiting-vex
    NotABug --> AwaitingVEX: enqueue awaiting-vex
    WontDo --> [*]: no VEX reconciliation
    AwaitingVEX --> [*]: VEX matches resolution
    AwaitingVEX --> AwaitingVEX: pending / mismatch

    classDef jiraStatus fill:#e5e7e9,stroke:#27ae60,color:#1f2937
    classDef closedStatus fill:#e5e7e9,stroke:#1f5f8b,color:#fff
    classDef resolution fill:#d5f5e3,stroke:#27ae60,color:#1f2937
    classDef automation fill:#e5e7e9,stroke:#7f8c8d,color:#1f2937
    class New,InProgress,Review jiraStatus
    class Closed closedStatus
    class DoneErrata,NotABug,WontDo resolution
    class Agent,AwaitingVEX automation

Notes on the diagram:

  • Agent is the CVE agent. See that tool’s doc for the full internals and automated sweep deployment.
  • Blue states are Jira statuses, green states are resolutions, and gray states represent supporting automation.
  • Won’t Do (package EOL) closes without VEX reconciliation; Done-Errata and Not a Bug enqueue awaiting-vex and leave AwaitingVEX only once the VEX feed matches the resolution.

These state names live in exactly one place, hummingbird_cve_analysis/lib/states.py, imported by both tools:

  • JiraStatus — New, In Progress, Review, Closed.
  • Resolution — Done-Errata, Not a Bug, Won't Do (set when closing).
  • Disposition — the second field of a computed_resolution line: the three resolutions above plus the open-ticket pseudo-dispositions affected and needs investigation.
  • Verdict — the CVE agent’s per-ticket outcome (needs_change, not_affected, already_fixed, no_upstream_fix, needs_human, error), with VERDICT_OUTCOME mapping each verdict to the Jira status/resolution it implies.

How It Works

1. Ticket Selection

The tool queries Jira for HUM project tickets with component Security. Tickets can be filtered by creation period (--show-since "2 weeks"), limited to specific keys (HUM-796 HUM-518), or scanned in bulk. Closed tickets are skipped by default unless --include-closed is specified or specific ticket keys are given on the command line.

Embargoed tickets (security level “Embargoed Security Issue” or embargo status field set to true) are always skipped.

Trackers filed against a Hummingbird image rather than an RPM (a pscomponent:hi/<name> label, or CVE-YYYY-NNNN hi/<name>: in the Summary) are filed in error and are always skipped. Like other automation, this respects the human override: trackers assigned to someone other than the bot only get the skip-assignee notice. Otherwise each run logs a warning with the alert text and ticket link, and with --resolve open trackers get a tracker comment, the cve-needs-attention label, and a one-time Slack alert via --slack-webhook-url (":alert-siren: HUM CVE TRACKER FILED IN ERROR – TRACKER FILED AGAINST IMAGE AND NOT AN RPM", with the ticket link and an @-mention of prarit). A separate comment marker records successful Slack delivery. The notice, the delivery marker, and the label are checked independently, so a failed webhook call or Jira write is retried on the next run.

Tickets with no valid CVE-YYYY-NNNN in the CVE ID field or Summary (for example GHSA/OSV-only trackers) are postponed: analysis is skipped, an INFO message is logged, and with --resolve a one-time tracker comment is posted and the cve-needs-attention label is applied (open tickets only). Use --skip-no-cve to omit those tickets without posting the postpone notice or adding the label.

Jira mutations driven by ticket selection itself (the image tracker alert, skip-assignee notice, and no-CVE postponement above) run serially, before the per-ticket pool starts. Per-ticket analysis (sections 2-7) is read-only. With --resolve, each ticket’s decide step (analyze plus the Jira/advisory mutations in section 9) also runs on that same pool. Both use --max-workers threads (default: 8; also CVE_ANALYSIS_MAX_WORKERS). The advisory batch MR apply (section 8, push/create) stays serial after the pool drains. Results are collected as each ticket finishes but reported — both on stdout and in the --handoff-file payload — back in the same order Jira returned the tickets, regardless of which ticket finished first.

2. CVE Data Collection

For each ticket the tool:

  1. Extracts the CVE ID from the Jira CVE ID field (customfield_10667) when populated, otherwise from the Summary via a regex match for CVE-YYYY-NNNN patterns. If neither source has a CVE ID, analysis is postponed (see above).

  2. Loads the CVE record from a local clone of the cvelistV5 repository and parses the CVE 5.0 affected block, following the CVE 5.0 Product and Version Encodings specification. This includes:

    • Parsing lessThan and lessThanOrEqual version ranges
    • Handling lessThan: "*" (no upper bound) and lessThan: "4.*" (end-of-series, all versions in the 4.x series) wildcards per the spec
    • Processing changes lists that subdivide ranges into affected and unaffected segments
    • Respecting defaultStatus at the product level
    • Filtering out versionType: "git" entries that use commit hashes instead of numeric versions (these are preserved as informational data but not used for version comparison)
    • Detecting CNA data errors where git commit hashes are used in version fields without versionType: "git", including hashes wrapped in operator syntax (e.g., < 6374ae0bcdfe...)
    • Parsing inline comparison operators (< 12.3.0, >= 4.0, <= 2.5) that are not part of the CVE 5.0 schema but are widely used by CNAs in practice
    • Parsing compound range bands (>= 2.0, < 2.2.26 and > 1.32.3, < 1.34.6) used by some CNAs to express a closed-open range in a single version string
  3. Looks up version ranges from the NVD data feeds and merges them with the MITRE data. NVD data is obtained from the JSON 2.0 data feeds at nvd.nist.gov/feeds/json/cve/2.0/ (per-year .json.gz tarballs updated daily, plus a CVE-Modified overlay updated every 2 hours). Feeds are cached locally (--nvd-cache-dir) and only re-downloaded when the NVD .meta file SHA256 indicates newer data is available. When MITRE has no vendor-specific data (all vendors are “n/a”), NVD ranges replace the MITRE data entirely. NVD references tagged “Patch” or matching commit URL patterns are also extracted and used for upstream fix detection (see below).

  4. Looks up the Hummingbird package version by scraping the Pulp repository index at packages.redhat.com for the latest RPM matching the package name extracted from the ticket summary.

Pulp directory listings are parsed once and shared by version and SRPM lookups. The parsed filenames and upload times are cached by directory URL until clear_listing_cache() clears the cache between analysis cycles, so repository updates become visible after the cache is cleared.

3. Resolution Computation

The tool compares the shipped Hummingbird version against the affected version ranges to compute a recommended resolution:

  • Closed / Done-Errata: The repo version is explicitly unaffected (an exact version, an unaffected range, or product defaultStatus: unaffected), or NVD-only data establishes that it is past the fix/outside affected ranges. An exact unaffected version does not cover later versions. For CVE 5 data, when no entry matches, omitted defaultStatus means unknown, not safe. Overlapping affected ranges from multiple streams prevent a version-based close unless the version is explicitly unaffected.
  • In Progress / affected: The repo version falls within an affected range.
  • In Progress / affected (no version data): Neither MITRE nor NVD has affected version information. The ticket is moved to In Progress for manual review but does not receive the cve-needs-attention label.
  • In Progress / needs investigation: Version data was available but could not be compared (e.g., CVE only provides git commit hashes), or multiple distinct products with different versioning schemes are listed and no cve_product override is configured. These cases receive the cve-needs-attention label.

4. Product Mismatch Detection

The tool detects when a CVE was filed against the wrong Hummingbird package. It compares the CVE vendor/product names against the Hummingbird package name and upstream repo URL from the package map. For example, a CVE for isaacs/node-tar (an npm package) filed against the tar RPM (GNU tar) is flagged as a mismatch. This detection works even without a repo URL by comparing normalized product names.

Mismatch triage uses a two-phase evidence gate:

  1. Identity mismatch — CVE product/vendor does not match the package (cve_product override or heuristic name/repo matching)
  2. SBOM (+ binary) evidence — only then decide misfiled vs vendored vs needs investigation

Phase 2 fetches the latest package SBOM from the Hummingbird Pulp repository and records the artifact used (NVR/URL) plus match evidence. SBOM JSON is parsed incrementally; an invalid document is treated as unavailable. Go module matching accepts the module root and major-version paths (github.com/vendor/product or .../v5). Deeper import paths such as .../api or .../daemon only count as a hit when a CVE path hint (from malformed/version-path fields) aligns with that subpackage, so a client/API module does not satisfy a daemon-only CVE. If the CVE product is found as a vendored dependency, the tool runs binary confirmation in this order:

  1. SPDX SBOM evidence — sourceInfo pointing at shipped paths (e.g. Go buildinfo under /usr/bin/...) or CONTAINS from binary RPM roots (*.x86_64, *.aarch64, …) marks present_in_binary without Syft. Evidence only under .src / go.mod is treated as source-only for the next step.
  2. RPM Provides: bundled(...) — downloaded non-debug binary RPMs are queried with rpm -qp --provides. A matching Fedora bundled Provide (npm bundled(npm(name)), Go bundled(golang(IMPORT_PATH)), Node core bundled(nodejs-undici), etc.) marks present_in_binary with evidence_method=rpm_bundled_provides. Ecosystem wrappers such as npm(...) are unwrapped so a CVE/SBOM search for nanoid matches bundled(npm(nanoid)). This catches dependencies embedded into binaries (including minified frontend JS) that Syft cannot inventory as on-disk packages.
  3. Syft on binary RPMs — Syft scans the same RPMs, even after a Provides or SPDX hit, to find additional shipped copies. When OSIDB reports subpackage names that match RPMs for the NVR, only those RPMs are scanned for absence checks; vendored-version checks scan the full NVR. If OSIDB data is missing or the names do not match this package’s RPMs (for example a Redis CVE filed against boost), the full NVR set is used. Pulp download URLs use the arch from the RPM filename (aarch64 files under aarch64/, x86_64 under x86_64/; noarch stays on the x86_64 listing).

Those scans are reused. The process keeps provides tuples and Syft artifact lists in memory, and when RPM_SCAN_CACHE_S3_BUCKET is set it also stores that JSON in S3, keyed by RPM URL. An object is written only after both rpm --provides and Syft ran, so a cycle that could not run Syft is not later reused as a confirmed absence. --rpm-scan-cache-max-bytes (default 1.5 GiB) is a soft LRU budget on the bucket. Each analysis cycle clears the in-memory cache and leaves the S3 objects in place. The client is configured with RPM_SCAN_CACHE_S3_REGION, RPM_SCAN_CACHE_S3_ACCESS_KEY_ID, and RPM_SCAN_CACHE_S3_SECRET_ACCESS_KEY from the RPM scan cache ObjectBucketClaim.

Syft absence only becomes absent_in_binary_confirmed when SBOM evidence is source-only (or no SBOM document was available) and Syft actually ran. A miss with ambiguous SBOM evidence stays unknown so embedded Go/Rust modules are not closed as source-only when SPDX already ties them to a binary. A Provides-only scan (Syft unavailable) never confirms absence. Per-RPM download or scan failures are skipped so sibling RPMs can still confirm presence; a failed scan never proves absence or that all vendored copies are fixed. A Provides query failure falls through to Syft for that same RPM but leaves the version evidence incomplete for auto-closure.

  • present_in_binary (mismatch_vendored): legitimate vendored ticket; compare the confirmed binary/Provide version (falling back to the SBOM vendored dep version) against the CVE ranges (parent latest_repo_version / Hummingbird repo version stay the package), and apply cve-needs-attention so operators can review the binary hit. Matching versions across binary RPMs, Syft artifacts, SPDX binary packages, and bundled Provides are all collected. A binary scan must complete before the versions can justify auto-closure; a confirmed presence with an incomplete scan remains open for investigation. Resolution is worst-of / all-must-be-fixed: the ticket stays affected if any copy is still affected; Done-Errata only when every copy is not affected. The tool does not invent multi-major fix mappings from a single CVE fixed line — Fixed in Build remains the override when CVE version data only describes one release line. Placeholder dep versions such as cargo std@0.0.0 are not comparable: fall back to the parent package version when available; otherwise set non_comparable_vendored_version, recommend New / needs investigation, and apply cve-needs-attention — never Done-Errata from “0.0.0 is not in any affected range”
  • absent_in_binary_confirmed (mismatch_source_only): source-only; eligible to close as Not a Bug / Component not Present. Must never set Fixed in Build or take the advisory / Done-Errata path.
  • unknown (mismatch_binary_unknown): leave for investigation; do not auto-close or call the ticket misfiled

Stable reason codes are emitted in analysis output (Reason codes:) and on each CVE entry (reason_code / mismatch_gate) for automation:

Reason code Meaning
mismatch_sbom_miss Identity mismatch + SBOM miss → misfiled candidate
mismatch_sbom_unavailable Identity mismatch but SBOM could not be checked
mismatch_binary_unknown SBOM hit but binary presence unknown
mismatch_source_only SBOM hit confirmed absent from binaries
mismatch_vendored SBOM hit confirmed present in binaries → label for review
non_comparable_vendored_version Vendored version is a placeholder (e.g. 0.0.0) with no parent fallback → needs attention

A misfiled recommendation requires identity mismatch + SBOM miss (no vendored hit). Ambiguous checks and confirmed binary RPM matches keep cve-needs-attention with the reason code above. When present in binaries, normal resolution flow still proceeds (In Progress / Close) in addition to the attention label. The upstream fix search targets the parent package’s repo. Each analysis cycle re-fetches the SBOM and re-runs binary confirmation.

5. Upstream Fix Detection

Fix search uses a precedence model so tickets are not dual-labeled from both Fedora and upstream when a definitive path exists:

  1. CVE/NVD fix links (preferred): NVD references tagged “Patch” or matching commit URL patterns, plus CVE/NVD PR/MR references that match the package’s upstream repo. When these yield a fix label, Bodhi and forge search are skipped.
  2. Otherwise, repo-aware search from metadata/*.json upstream_repo:
    • Fedora-based (src.fedoraproject.org, pagure.io, or missing repo URL): Bodhi + DistGit only (see §6)
    • Non-Fedora forge (GitHub/GitLab/cgit): forge search only

Forge Search (non-Fedora fallback):

  • GitHub: Searches PRs via the GitHub API (/search/issues) for PRs mentioning the CVE ID. Fetches commit counts from the pulls API.
  • GitLab: Searches merge requests via the GitLab API on supported instances (gitlab.com, gitlab.gnome.org, gitlab.freedesktop.org, etc.). The --gitlab-token is only sent to gitlab.com to avoid 401 errors on other instances.
  • cgit: Scrapes commit log pages on cgit hosts (Savannah, Sourceware, kernel.org, busybox.net, etc.) for commits mentioning the CVE ID. Handles URL rewrites for Savannah hosts (e.g., git.savannah.gnu.org/git/ to cgit.git.savannah.gnu.org/cgit/).

Fix status is classified as:

  • upstream-fix-available: At least one merged/closed PR, committed fix, or NVD patch reference
  • upstream-fix-in-progress: At least one open PR with no merged fixes

6. Fedora Fix Detection

Used only for Fedora-based packages (or when no upstream repo is configured) and only when CVE/NVD fix links did not already produce a label. Hits are classified with the same upstream-fix-* labels as forge/NVD evidence (Bodhi update links are still shown in analysis output):

  • Bodhi: Queries the Fedora Bodhi API for updates matching the package name that reference the CVE ID (in the cves list or notes field). Supports multi-page results. Stable → upstream-fix-available; testing/pending → upstream-fix-in-progress.
  • DistGit Spec Scan: When Bodhi has no matches, fetches the Fedora DistGit spec file and scans for CVE references in patch filenames, changelog entries, and comments to detect backported fixes (upstream-fix-available).

7. Fixed Build Detection

The tool automatically detects whether the current Hummingbird build fixed a CVE by checking if all CVE IDs are mentioned anywhere in the package directory (patch names, changelogs, comments in the .spec file and so on). If found, it returns the current source RPM name. Assuming that we run the analysis frequently enough, this is precise enough, otherwise it errs on the side of caution (i.e. it possibly marks a higher version as fixed even if an earlier one already carried the fix).

The detected build is shown in the output as Detected Fix: package-version-release.hum1.src.rpm.

When --resolve is specified and a fixed build is detected, the tool automatically populates the “Fixed in Build” Jira field with the detected SRPM name, but only if:

  • The field is not already set (manual values take precedence)
  • The issue is not in Closed status (respects human closure decisions)
  • The assessment is not mismatch_source_only / source_only_confirmed (Component not Present closes as Not a Bug and clears any Fixed in Build)
  • The computed resolution is already Closed / Done-Errata or every ticket CVE ID appears on an uncommented PatchN: line (HUM-6182). A spec comment or leftover unapplied file that merely mentions the CVE is not enough. Product-mismatch and EOL tickets never take this PatchN override.

Version-range analysis can still say “affected” after a backport that does not bump Version (for example an unbounded range such as 1.1.1+ on popt 1.19). In that case an applied CVE-YYYY-NNNN.patch listed as PatchN: is allowed to set Fixed in Build. The SRPM must already be in Pulp (the existing fix_committed_not_built gate is unchanged). The ticket is not closed here; advisory review remains the backstop.

The RPMs repository can be provided via --rpms-repo, pointing to a local clone. This is preferable with multiple runs to avoid the git clone operation. When not given, the repository will be (shallow) cloned into a temporary directory.

Similarly, the cvelistV5 repository (used for CVE record lookups) can be provided via --cve-repo. When not given, a shallow clone is performed into a temporary directory. For production use, pre-clone and update the repo externally (e.g. via a CronJob) to avoid the clone overhead on each run. The tool does not attempt to update the provided repo itself.

8. Advisory Integration (with --resolve)

When a CVE is resolved as “Closed / Done-Errata”, the tool integrates with the CEE GitLab advisories repo to document the fix:

  1. Clone: The bot fork of the advisories repo (default upstream: releng/advisories) is cloned blobless (--filter=blob:none), synced to upstream main, and force-pushed to the fork. A blobless clone keeps the commit history that pushes to a stale fork need while downloading only the checked-out files; a --depth 1 clone would be rejected by the fork. The token is embedded in the git URL and redacted from logs.
  2. Modify: For each resolved ticket, the advisory YAML is updated: cves.fixed entries are added, the type is changed from RHBA to RHSA, and CVE references are appended.
  3. Batch MR: Per-ticket advisory YAML edits run inside the decide worker pool (serialized by an advisory write lock). Changes accumulate as individual commits on a single branch (cve-analysis/batch). One merge request is created after the pool drains, covering all tickets.
  4. Review state: Tickets are transitioned to “Review” (not directly closed) and the batch MR URL is posted to each affected ticket. The MR is approved and queued onto the GitLab merge train; the run does not wait for merge or the post-merge pipeline (HUM-7562).
  5. Auto-close: On subsequent runs, tickets in “Review” are checked: if the advisory MR has been merged, the ticket is closed as “Done-Errata”. The MR URL is taken from the ticket comment, or, if that comment is missing, from an advisories search by ticket key. If the MR has unresolvable rebase conflicts, a comment with manual resolution steps is posted and the advisory-mr-failed label is added.
  6. Slack notifications: When --slack-webhook-url (or the SLACK_WEBHOOK_URL env var) is set, the tool sends a Slack message when batch MR creation fails. Failure messages @-mention prarit and jstibran. It also alerts, with an @-mention of prarit, when a tracker is filed against an image instead of an RPM (see above).

The advisory flow is skipped when --advisories-project points to a non-production URL, allowing safe testing without modifying Jira.

The clone lives in a temporary directory and is removed at the end of the run. It is intentionally left on disk when the batch MR push fails or --keep-advisory-repo is passed, so an engineer can inspect it; a leftover clone from a prior run is automatically removed once it is a few hours old to prevent repeated failures from exhausting disk/inode space.

8.1 Recovery for failed advisory MRs

Troubleshoot an advisory MR

An advisory MR updates release metadata for a published fixed build.

What changes Advisory field Purpose
CVE associations spec.content.artifacts[].cves.fixed Records which CVEs the applicable artifacts fix.
Tracker links spec.issues.fixed Links the HUM trackers covered by the advisory.
Security references spec.references Adds links to CVE details and security information.
Advisory type spec.type Changes RHBA to RHSA when the update includes security fixes.
Advisory text, when needed spec.synopsis, spec.topic, spec.description Describes the security fixes and packages covered.

If a retry is not enough, the fix depends on the failure: resolve merge conflicts against the current advisory, or check that the CVE, tracker, Fixed in Build and artifact entries refer to the right package/build. Correct advisory text and references where needed, preserving unrelated fixes, artifact provenance, advisory IDs and ship dates. Permission failures need an authorized author/workflow rather than another retry.

  1. Open the Advisory MR linked in the Jira tracker comments or the failure notification. Check its pipeline, failed job logs and merge status.
  2. For a transient failure, retry the failed job or rerun the pipeline in GitLab. This is often enough; no advisory edits are needed.
  3. If retrying does not help, or the MR has conflicts, permission failures or needs an advisory correction, ask ProdSec Team for help. Include the MR URL, affected HUM tracker(s), failed job/error and what you retried. For manual changes, see the advisories repository’s contribution guide and current field permissions. Note: changes to spec.type require an allowlisted author and must be committed through the Hummingbird CVE Analysis <hummingbird-jira-bot-admins@redhat.com> bot.
  4. Confirm that the MR merges and its default-branch publication pipeline succeeds. Leave Done-Errata closure to advisory automation. If the tracker or dashboard remains out of sync, use the recovery steps below.

Recover lifecycle events after advisory delivery

In some failure cases (for example, advisory MR conflicts followed by manual ticket closure), Jira status can be corrected to Done-Errata while dashboard lifecycle metadata remains incomplete. This leaves R-Time rows PENDING (elapsed-to-now) until image, VEX, and close all exist.

Use the recovery script to backfill delivery events:

  • rpm_fix_published_to_pulp (hb_rpm_fix) from packages.redhat.com source RPM timestamps
  • image_rebuilt_on_quay (hb_image_fix) from catalog/quay image history
  • hum_ticket_closed from Jira resolutiondate

Only tickets that are Closed / Done-Errata are used for event derivation and dashboard import. Tickets that remain in Review (or any non-closed state) are reported and skipped for event backfill.

For tickets still in Review, the script also attempts advisory reconciliation before deriving events:

  • Read advisory MR URL from Jira comments (Advisory MR: https://...)
  • If no MR URL is present in comments, search merged advisories MRs for the HUM ticket key in MR description/title
  • When a merged MR is found and --apply is used, transition the ticket to Closed / Done-Errata and then derive hum_ticket_closed

Script location:

  • hummingbird-cve-analysis/scripts/cleanup_advisory_mr_failure_events.py

Run in dry-run mode first (default):

cd /path/to/tools
export JIRA_TOKEN=your_jira_token
export CEE_GITLAB_TOKEN=your_cee_gitlab_token
export CVE_REPORT_TOKEN=your_dashboard_token

python3 hummingbird-cve-analysis/scripts/cleanup_advisory_mr_failure_events.py \
  --jira-user your-jira-user@example.com \
  --cee-gitlab-token your_cee_gitlab_token \
  --output-json /tmp/cleanup-advisory-derived.json

Apply the backfill to dashboard event storage:

python3 hummingbird-cve-analysis/scripts/cleanup_advisory_mr_failure_events.py \
  --jira-user your-jira-user@example.com \
  --cee-gitlab-token your_cee_gitlab_token \
  --output-json /tmp/cleanup-advisory-derived.json \
  --apply

Target specific tickets (instead of default JQL):

python3 hummingbird-cve-analysis/scripts/cleanup_advisory_mr_failure_events.py \
  --jira-user your-jira-user@example.com \
  --cee-gitlab-token your_cee_gitlab_token \
  --apply \
  HUM-4884 HUM-4871 HUM-4844

Verification example:

# Replace with the actual dashboard URL (same as --dashboard-url default)
DASHBOARD_URL=https://hummingbird-dashboard-hummingbird--internal.apps.int.spoke.prod.us-east-1.aws.paas.redhat.com
curl -s "${DASHBOARD_URL}/api/cve/r-time?days=30" | jq -r '
  .entries[]
  | select(.key=="HUM-4884")
  | [.key, (.fix_delivered_at // "-"), (.stages.hb_rpm_fix // "-"), (.stages.hb_image_fix // "-")]
  | @tsv
'

If hb_rpm_fix and hb_image_fix remain empty in the derived JSON output, the issue is source-data availability (no resolvable RPM/image publish signal), not dashboard ingestion.

If a ticket is in Review and no advisory MR can be found in Jira comments or merged advisories MRs, it is skipped and reported for manual follow-up. By default, the script targets advisory-mr-failed tickets (for both Closed / Done-Errata and Review states); override with --jql when needed.

8.2 Dashboard collection ticket selection

Dashboard lifecycle collection (HUM-5603) uses a separate ticket-selection path from cve_analysis mutations. Helpers live in hummingbird_cve_analysis/dashboard_selection.py.

Watermark (start of previous run): read the newest entry from the dashboard Run Log (GET /api/cve/run-log?limit=1). Use run_started_at when available; older entries estimate the start as recorded_at - duration_seconds (recorded_at is ingest time). Subtract a small buffer (default 2 minutes) for clock skew. The overlap avoids missing tickets updated while the previous collector run was in progress and is safe because /api/cve-report event upserts are idempotent.

JQL selection:

  • All still-open HUM Security tickets (rescan for updates)
  • Closed tickets with Jira updated >= watermark
  • Bootstrap (empty Run Log): open tickets only
  • Explicit ticket keys: key in (...) (any status)

Use --start-time (ISO-8601, e.g. 2026-08-01T00:00:00Z) to override the Run Log watermark and recollect Closed tickets with updated >= that start, so dashboard data can be overwritten/backfilled.

Inspect watermark and JQL without modifying Jira or the dashboard. From the tools repo root, either install the package (make hummingbird-cve-analysis/setup) or set PYTHONPATH:

cd /path/to/tools
export PYTHONPATH=hummingbird-cve-analysis
export CVE_REPORT_TOKEN=your_dashboard_token

python3 -m hummingbird_cve_analysis.dashboard_selection --prod

# Optional: also search Jira and list matching keys
export JIRA_TOKEN=your_jira_token
python3 -m hummingbird_cve_analysis.dashboard_selection \
  --prod \
  --jira-user user@example.com \
  --fetch-issues -o json-pretty
Option Environment Variable Description
--prod / --preprod Required. Hardcoded production or preprod dashboard URL
--cve-report-token CVE_REPORT_TOKEN Bearer token for /api/cve/run-log
--start-time Explicit UTC collection start (ISO-8601); skips Run Log fetch
--watermark Alias for --start-time
--watermark-buffer-minutes Minutes subtracted from previous-run start estimate (default: 2)
--fetch-issues Also search Jira with the constructed JQL
--jira-token / --jira-user JIRA_TOKEN Jira credentials (only with --fetch-issues)
--jira-url JIRA_URL Jira base URL (only with --fetch-issues)
--max-results Max issues to fetch with --fetch-issues (default: 2000)
--output, -o human, json, or json-pretty

8.3 Analysis → collector handoff file

cve_analysis writes a small JSON handoff file for collect_cve_dashboard (HUM-5799 / HUM-5800). This carries Run Log counters, captured log_output, and per-ticket fields that are expensive to recompute (upstream_fix, fedora_fix) plus human_text for Run Log details. Delivery timestamps (hb_rpm_fix / hb_image_fix) and hum_ticket_closed are gathered by the collector, not this file. Analysis stdout is human text only; it does not emit /api/cve-report JSON.

PYTHONPATH=hummingbird-cve-analysis python -u -m hummingbird_cve_analysis.cve_analysis \
  --jira-user "$BOT_USER" --jira-token "$BOT_JIRA_TOKEN" \
  --handoff-file /tmp/cve_analysis_handoff.json \
  ...

Handoff shape (schema_version: 1):

  • Top level: run_started_at, duration_seconds, mutation counters (labels_changed, advisories_created / advisories_failed, tickets_closed, comments_posted, attachments_uploaded, fatal_errors), log_output
  • tickets[]: key, upstream_fix, fedora_fix, human_text
  • vex_updates[] (HUM-5843): key, vex_status, vex_match_state, vex_resolved, labels from the awaiting-vex reconcile pass

8.4 collect_cve_dashboard

collect_cve_dashboard (HUM-5798 / HUM-5800) owns dashboard JSON emission. cve_analysis no longer prints /api/cve-report JSON on stdout; it prints human analysis text and optionally writes --handoff-file for the collector.

  1. Resolve Run Log watermark + open-ticket rescan (section 8.2). Selection always includes Closed tickets labeled awaiting-vex (HUM-5843), awaiting-cve-published, or awaiting-image-rebuild (HUM-9975), then any vex_updates[] keys missing from that JQL
  2. For each ticket, gather slim lifecycle fields from Jira / cvelistV5 / Fixed-in-Build → Pulp (cve_published, hum_ticket_created, hum_ticket_closed, hb_rpm_fix) and catalog image history (hb_image_fix), plus cve_ids – every CVE ID the ticket tracks, from its dedicated CVE ID Jira field (falling back to Summary text), not just the first found in labels; a ticket can legitimately track more than one CVE, and consumers keyed by CVE ID (e.g. the dashboard’s Competitive page HUM ticket matching) index it under every entry here, not just one (see hummingbird-dashboard.md), plus OSIDB osidb_flaw_created / osidb_affect_created from the public OSIDB API (omitted when OSIDB has no data), cve_embargo_lifted – the OSIDB flaw unembargo_dt, included only when it postdates the flaw’s created_dt (i.e. the CVE was genuinely embargoed; omitted otherwise) – and osidb_flaw_public – the same unembargo_dt, but included unconditionally whenever OSIDB reports one, regardless of whether the flaw was ever embargoed – computed_resolution from Jira status/resolution (R-Time gates on Done-Errata in that string), catalog_image_source from the catalog source map (R-Time delivery is image publish when true, RPM publish when false), and catalog_images – every {image}:{stream} the ticket’s package actually appears in, per the same catalog source map’s SBOM data (used by the dashboard’s Competitive page to match a ticket to a specific image precisely instead of guessing from the package name; see hummingbird-dashboard.md). Timestamp helpers live in lib/lifecycle.py and are shared with analyze_issue
  3. Merge --handoff-file counters, log_output, per-ticket upstream_fix / fedora_fix / human_text, and vex_updates
  4. Collect agent costs: every ticket carrying a cve-agent-* label is found with one paginated key-only Jira search, independent of this run’s watermark selection – an unchanged Closed ticket would otherwise never be retried after a failed download. Unrecorded keys (checked in batches against /api/cve/agent-cost-keys) have their <TICKET>-cve-agent-analysis.json attachment downloaded for its token_cost block, and the result travels as the report’s agent_costs list. A ticket without an attachment yet is retried on the next run; an attachment predating cost reporting is recorded once with a null estimated_cost_usd so it is not downloaded again.
  5. Dry-run prints /api/cve-report JSON; --apply POSTs it

The collector builds the catalog source map once per run for hb_image_fix, catalog_image_source, and catalog_images. When a delivery timestamp is omitted, the dashboard keeps any existing rpm_fix_published_to_pulp / image_rebuilt_on_quay value rather than clearing it. An empty or failed catalog map omits catalog_image_source and catalog_images so the dashboard keeps requiring an image / falls back to its own package-name heuristics.

Catalog fetches (an image’s tag listing, or a specific stream’s SBOM) are not retried; a failure is logged and that image/stream is silently skipped when building the source map, so catalog_images’s absence of an image/stream is not always confirmed ground truth – it can also mean that image/stream’s data simply failed to collect this run. build_catalog_source_map() returns the set of (image, stream) pairs it couldn’t verify alongside the map (stream is None when the image’s own tag listing failed, so every one of its streams is unknown). Every ticket in the run carries this set as catalog_unverified_images (rendered as {image}:{stream}, or {image}:* for a whole unverified image) so a downstream consumer – the dashboard’s _match_cve_tickets – can tell “confirmed absent” from “unknown” before excluding a ticket candidate based on catalog_images alone (see hummingbird-dashboard.md).

PYTHONPATH=hummingbird-cve-analysis python -u -m hummingbird_cve_analysis.collect_cve_dashboard \
  --preprod \
  --cve-report-token "$CVE_REPORT_TOKEN" \
  --jira-user "$BOT_USER" --jira-token "$BOT_JIRA_TOKEN" \
  --cve-repo /tmp/cvelistV5 \
  --handoff-file /tmp/cve_analysis_handoff.json

# POST to preprod dashboard
PYTHONPATH=hummingbird-cve-analysis python -u -m hummingbird_cve_analysis.collect_cve_dashboard \
  --preprod \
  --cve-report-token "$CVE_REPORT_TOKEN" \
  --jira-user "$BOT_USER" --jira-token "$BOT_JIRA_TOKEN" \
  --cve-repo /tmp/cvelistV5 \
  --handoff-file /tmp/cve_analysis_handoff.json \
  --apply
Option Environment Variable Description
--prod / --preprod Required. Hardcoded production or preprod dashboard URL
--cve-report-token CVE_REPORT_TOKEN Bearer token for Run Log + /api/cve-report
--handoff-file CVE_ANALYSIS_HANDOFF_FILE Analysis handoff JSON from cve_analysis --handoff-file
--apply POST report to /api/cve-report (default: dry-run JSON on stdout)
--start-time Explicit UTC collection start (ISO-8601); skips Run Log fetch
--watermark Alias for --start-time
--cve-repo Local cvelistV5 repo for cve_published
--jira-token / --jira-user JIRA_TOKEN Jira credentials
--output, -o human, json, or json-pretty

8.5 verify_image_publish

verify_image_publish (HUM-721) catches CVE tickets that claim a fix via image rebuild (hb_image_fix in the whiteboard cve_cycle) but whose image was never actually published to the registry — hb_image_fix is set from catalog release-history metadata (oldest_created), which can lag or be wrong; this re-verifies with a live check.

  1. Search Jira for Closed HUM Security tickets updated within the last --window-minutes (default 60)
  2. For each, read hb_rpm_fix / hb_image_fix from the whiteboard; skip tickets where hb_image_fix falls outside the window
  3. Re-derive the source package name from the summary/labels (no SBOM / network calls) and re-run the same catalog-history lookup (find_image_publish_release) that originally produced hb_image_fix, recovering the specific image/stream/variant/digest
  4. HEAD the image manifest by digest directly against the registry (image_manifest_exists) — this is the part hb_image_fix itself never did
  5. Post a Slack alert for each unverified claim; log a summary either way

Read-only: never modifies Jira, the dashboard, or the CVE analysis pipeline.

PYTHONPATH=hummingbird-cve-analysis python3 -m hummingbird_cve_analysis.verify_image_publish \
  --jira-user "$BOT_USER" --jira-token "$BOT_JIRA_TOKEN" \
  --window-minutes 60
Option Environment Variable Description
--window-minutes Look back this many minutes for hb_image_fix (default: 60)
--slack-webhook-url SLACK_WEBHOOK_URL Webhook for unverified-claim alerts
--dry-run Check and log, but do not post to Slack
--catalog-api-base Catalog API base URL
--jira-token / --jira-user JIRA_TOKEN Jira credentials
--jira-url JIRA_URL Jira base URL

9. Jira Actions (with --resolve)

When --resolve is specified, the tool modifies Jira tickets. Those per-ticket mutations run concurrently with analysis on the --max-workers pool (HUM-11537); only the advisory batch MR push/create (section 8) runs once after the pool finishes. Shared advisory-repo writes are serialized by an internal lock; Jira API GET/PUT retries cover transient conflicts.

The deployed analysis CronJob also uses --publish-cve-agent-sweep with --resolve. After a completed cycle it publishes a small SNS event (source=cve-analysis, event_type=cve_agent_sweep_requested) to wake the CVE-agent SQS worker. The event contains only the run timestamp; the agent searches Jira for eligible tickets itself. SNS publishing failures are logged without undoing or failing completed Jira analysis; the next scheduled run publishes another request. See the message bus contract for the exact message body and attributes.

  • Fixed in Build: When a fixed build is detected and the field is not already set, the tool populates it with the detected SRPM name (only for non-closed issues). Not-affected Closed / Done-Errata closes also set Fixed in Build from detected_fixed_build or latest_srpm when empty, so R-Time image timestamps can be resolved.
  • Closed / Done-Errata: Tickets where the shipped version is not affected are transitioned to Review with an advisory MR (see above). After the MR merges, they are closed as Done-Errata on the next run.
  • Move to In Progress: Tickets where the shipped version is affected are transitioned to In Progress with a comment including affected version ranges, upstream fix status, and Fedora update links.
  • Needs Investigation: Tickets with incomplete version data get a comment explaining why automatic resolution was not possible.
  • Product Mismatch: A comment is posted explaining the mismatch (with SBOM artifact evidence) and the cve-needs-attention label is applied. No transition is performed. Auto-detected Fixed-build / delivery timestamps (hb_rpm_fix, hb_image_fix) are cleared so R-Time does not treat misfiled tickets as done. Human Fixed in Build overrides are kept. When --resolve runs against the production advisories project, the analyzed package SBOM JSON is also attached to the ticket as {nvr}.sbom.json (skipped if that filename is already present). The same attachment is applied for other SBOM-backed review paths (vendored binary hits, source-only closes, binary-unknown).
  • Package Not Present: If the package is missing from the rpms repo, the tool still runs an SBOM-first check against the CVE product(s). An SBOM hit triggers Syft binary confirmation: absent_in_binary_confirmed closes as Not a Bug / Component not Present; presence or unknown leaves the ticket for investigation. An SBOM miss/unavailable closes as Not a Bug with VEX Component not Present. The close timestamp is persisted as hum_ticket_closed for dashboard ingestion.
  • Package EOL (fix_status: 0): Tickets for packages marked End-Of-Life in rpms metadata are closed as Won’t Do with no VEX Justification. This takes precedence over Done-Errata and Fixed-in-Build / advisory paths. Automation labels (upstream-fix-*, legacy fedora-fix-*, cve-needs-attention, cve-next-release, advisory-mr-failed) are removed; fedora-bz-filed is kept as an audit record. The close timestamp is persisted as hum_ticket_closed for dashboard ingestion.
  • Label Management: Labels are applied and updated:
    • upstream-fix-available / upstream-fix-in-progress (including Bodhi/DistGit evidence for Fedora-based packages)
    • cve-needs-attention (applied when human review is needed; removed when resolved)
    • Legacy fedora-fix-* labels are stripped on subsequent runs
    • Labels are upgraded (in-progress to available) and stale labels are cleaned up on ticket closure.

Every Jira action (label add/remove, status transition, product mismatch) includes a single comment with a bold action summary heading followed by the full analysis output in a preformatted code block, giving the reader the same detail they would see on the CLI.

10. Assignee-Based Automation Control

When --resolve is active, the tool checks each ticket’s assignee. If the assignee is not the bot account (--jira-user), the tool skips all automated actions (comments, labels, transitions) for that ticket. A one-time comment is posted explaining that automation is skipped. This allows humans to take ownership of a ticket by assigning it to themselves, preventing the bot from interfering with manual work. Reassigning back to the bot account re-enables automation.

For testing purposes, use --skip-assignee-check to process all tickets regardless of assignee.

11. Human Closure Protection

When the tool encounters a Closed ticket whose analysis recommends a non-Closed resolution (e.g., In Progress), it checks the Jira changelog to determine who performed the last status transition. If a human (not the bot) closed the ticket, the tool skips the transition to respect the human’s decision. This prevents the tool from reopening tickets where a fix was backported without a version bump or where a human determined the CVE does not apply.

12. Stale Closure Detection (with --include-closed)

When --include-closed is used, the tool also analyzes Closed tickets. If the current analysis recommends a non-Closed resolution (e.g., the package version changed and is now affected), a warning is emitted flagging the ticket for review.

13. Sprint and Epic Assignment

When --resolve is active, the tool automatically adds CVE tickets to the current Hummingbird sprint and links them to a per-sprint CVE tracking epic when human interaction is detected. This ensures sprint metrics capture human work on CVE tickets.

The tool queries the Jira Agile API for the active sprint on the Hummingbird board (--board-id, default 1489) and filters by name prefix (--sprint-prefix, default “Hum S”) to identify the correct sprint among multiple teams sharing the HUM project.

Sprint and epic assignment is triggered in two cases:

  • Self-assigned tickets: When a ticket is assigned to someone other than the bot, the ticket is added to the current sprint and linked to the sprint’s CVE epic.
  • Human-set Fixed in Build: When the Fixed in Build field was set by a human (not the bot), as determined by the issue changelog, the ticket is added to the current sprint.

If no matching epic exists for the current sprint, one is created automatically with the summary “CVE tickets for sprint <name>”. The epic key is cached for the duration of the run. A Jira comment is posted on the ticket recording the sprint and epic assignment.

Tickets that already have a sprint assigned are skipped. Tickets closed manually without going through either automated path should be added to the sprint by hand (see the manual process guide).

14. Continuous Operation

When --time-between-runs N is set (N > 0), the tool runs in a loop, re-executing the full analysis every N minutes. The default is 0 (single run). Shutdown is graceful: SIGINT/SIGTERM finishes the current cycle before exiting.

15. Feature Flag

When --feature-flag-name is set (default: cve_analysis_enabled), the tool checks a GitLab feature flag on the project specified by --feature-flag-project-id (default: 73447720, i.e. redhat/hummingbird/rpms) at the start of each cycle. If the flag is inactive, the cycle is skipped and the tool sleeps for 60 seconds before checking again.

The check uses the --gitlab-token / GITLAB_TOKEN credential, which must have Developer role or higher on the target project. The check is fail-open: if GitLab is unreachable or the token lacks permissions, the tool assumes the flag is enabled and proceeds.

Set --feature-flag-name "" to disable the check entirely.

16. Timing Instrumentation

A @timed(label) decorator (hummingbird_cve_analysis.lib.timing) wraps the heavier per-ticket network calls – NVD (fetch_nvd_ranges, get_nvd_references), Pulp (fetch_hummingbird_versions, fetch_hummingbird_latest_srpm_with_time, fetch_hummingbird_latest_srpm_as_of, fetch_srpm_upload_time), forge search (search_forge), Fedora (fetch_bodhi_updates), and SBOM/Syft (fetch_latest_sbom, confirm_component_in_binary_rpms) – timing each call with time.perf_counter().

Two levels of output:

  • Per ticket: _analyze_one_issue() (read-only) logs <KEY> analysis timing: ...; with --resolve, _process_one_ticket() logs <KEY> process timing: ... covering analyze plus decide. Both include the per-label breakdown for that ticket and are logged via finally, so they still appear when the ticket raised.
  • Per cycle: after the ticket loop, one Cycle sub-step timing: ... line aggregates every label’s total seconds and call count across the whole cycle (all tickets, all workers).

The per-ticket breakdown is isolated per ticket even under the --max-workers thread pool (HUM-10139 / HUM-11537): it’s tracked with a contextvars.ContextVar, which each worker thread gets its own copy of, rather than a single shared dict two concurrent tickets could stomp on. The cycle-level totals are a lock-guarded dict shared across all workers, the same pattern as resolve.increment_run_stat() (HUM-10136).

Before/after: validating the HUM-10139 parallelization

Three rounds of paired runs (sequential baseline at the commit just before HUM-10139, vs. the parallelized version, same frozen ticket list and shared NVD/cvelistV5/rpms-repo caches for both runs in a pair, run back-to-back) measured the actual wall-clock effect of parallelizing Phase 1:

Round Pairs Tickets/run Mean speedup Aggregate speedup Worst single run
1: most-recent 150 tickets 10 150 1.72x 1.47x 0.43x
2: random sample of 200 tickets 10 200 2.77x 2.74x 1.84x
3: all currently-open tickets 10 309 1.43x 1.41x 0.95x

Across all three rounds (30 pairs, ~5,400 ticket-analyses total), resolution output matched between the sequential and parallel run of each pair with zero correctness regressions; every mismatch traced to a live external cause (a Hummingbird build publishing to Pulp between the two runs of a pair, or the real production bot mutating a ticket’s Jira fields concurrently), not to the code.

Run-to-run variance (both within and across rounds) is dominated by a small number of tickets needing large, multi-architecture RPM downloads for SBOM/Syft binary confirmation – real network/CDN time that varies independent of sequential vs. parallel execution, and that a bounded thread pool can’t finish faster than its slowest in-flight ticket. Round 2’s random sample, less weighted toward any one such cluster, shows both the highest and the most consistent speedup of the three.

See Also

  • CVE Analysis Reference — Package map, vendored dependency detection, prerequisites, usage, configuration, managed labels, output, logs, library layout, development, license

1 - CVE Analysis Reference

Reference and operational documentation for the CVE Analysis tool. This page covers package map, vendored dependency detection, prerequisites, usage, configuration, managed labels, output format, logs, library layout, development, and license. For conceptual documentation (CVE lifecycle, how the analysis pipeline works), see CVE Analysis.

Package Map

The package map is loaded from the rpms repo’s per-package metadata files (metadata/<package>.json). Each file contains an upstream_repo field pointing to the canonical upstream git repository, plus optional fields:

  • upstream_branch – upstream branch (for versioned packages sharing a repo)
  • cve_product – CVE vendor/product override
  • version_transform – version transform rule
  • fix_status – package fix policy (0 = End-Of-Life / will not fix CVEs; close as Won’t Do)

If fix_status is 0, analysis always resolves to Closed / Won’t Do (including when the shipped version is outside the CVE affected range or a product mismatch would otherwise need investigation). No VEX Justification is set. If fix_status is missing or set to 1, behavior is unchanged from the default analysis flow.

When an upstream fix is available, Jira backport guidance keeps the NEXT STEPS: Backport upstream patches to Hummingbird marker and reminds maintainers to keep independent packages’ modification_status as independent and omit modification_reason, mark Fedora-imported packages modified with a modification_reason, and leave metadata release unchanged. This guidance does not affect the CVE assessment or fix decision.

The cve_product field is an optional human-curated value that specifies which CVE vendor/product entry maps to this package. It supports these formats:

  • Vendor / Product (exact match): matches a specific vendor and product pair. Example: "F5 / NGINX Open Source" for the nginx package, which excludes NGINX Plus entries that use incompatible R-versioning.
  • Vendor (vendor prefix match): matches any product whose vendor starts with the given string. Example: "Go " for golang packages, where the vendor varies (Go standard library, Go toolchain, etc.) and the product varies by module (net/url, os, crypto/x509).
  • Multiple selectors: provide more than one acceptable value either as a JSON list in metadata (preferred) or as a semicolon-delimited string. Example: ["vda-linux / busybox_mirror", "BusyBox / BusyBox"] or "vda-linux / busybox_mirror; BusyBox / BusyBox". The tool treats this as “match any selector”.

When cve_product is set, the tool uses it for exact product matching in both resolution computation (filtering to the correct product in multi-product CVEs) and mismatch detection. When empty, the tool falls back to heuristic name matching.

The rpms repo is cloned automatically at startup (or provided via --rpms-repo). A CSV override can be passed via --package-map for backward compatibility.

Vendored Dependency Detection

Vendored dependency detection uses live SBOM lookups from the Hummingbird Pulp repository. When a CVE product mismatch is detected, the tool fetches the SPDX SBOM for the ticket’s package from packages.redhat.com/.../metadata/sboms/{package}-main/ (same public index used by the rpms CVE skill), selects the newest dated sha256-….sbom entry from that listing, and searches that document for the CVE product(s) only (via purl references), stopping on the first product hit. Binary evidence then collects all matching copies of that product. The analysis output always records the SBOM artifact used and whether the lookup hit, missed, or was unavailable.

The same SBOM-first check also gates Component not Present closures for packages missing from the rpms repo. An SBOM hit alone no longer blocks that closure: binary confirmation must reach absent_in_binary_confirmed (no bundled Provide match and Syft miss on source-only SPDX evidence) before Not a Bug / Component not Present is recommended. A binary hit or unknown result keeps the ticket open for investigation.

Go subpackages are matched by progressively stripping path components (e.g., github.com/jackc/pgx/v5/pgproto3 matches github.com/jackc/pgx/v5). When a match is found and confirmed in binaries, the confirmed binary/Provide version is preferred for resolution over the SBOM lockfile version. The SBOM and binary confirmation are re-run each analysis cycle.

Requires the rpm and/or syft CLI on PATH (both are available in the analysis container image via rpm-build and the Syft install). Without either tool, binary confirmation returns unknown. With only rpm, a bundled Provide can still confirm present_in_binary, but absence is never auto-confirmed without Syft.

The generate_vendored_map_sbom.py script in package_maps/ can still be used for auditing vendored dependencies across all packages, but is no longer required at runtime.

Prerequisites

  • Python 3.11 or later
  • Jira API token (Bearer or Basic auth)
  • Network access to redhat.atlassian.net, github.com, nvd.nist.gov, bodhi.fedoraproject.org, src.fedoraproject.org, and packages.redhat.com (note: github.com is only needed when --cve-repo is not provided and the tool must clone cvelistV5 itself)
  • rpm and syft CLIs on PATH (for binary RPM confirmation after SBOM hits: bundled Provides via rpm, component inventory via Syft)
  • GitHub API token (optional, for upstream PR search)
  • GitLab API token (optional, for gitlab.com MR search)
  • CEE GitLab API token (required with --resolve, for advisory repo)

Usage

# Basic usage with Jira token from environment
export JIRA_TOKEN=your_token
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com

# Analyze specific tickets
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com HUM-796 HUM-518

# Show only tickets from the last 2 weeks
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com --show-since "2 weeks"

# Analyze with upstream fix detection (report findings without modifying Jira)
export GITHUB_TOKEN=your_github_token
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com

# Resolve tickets and apply labels
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com --resolve

# Include closed tickets and check for stale closures
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com --include-closed

# Include closed tickets but exclude specific ones
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com \
  --include-closed --exclude "HUM-555,HUM-552"

# Skip issues without CVE links; write collector handoff
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com \
  --skip-no-cve --handoff-file /tmp/cve_analysis_handoff.json

# Continuous mode: resolve tickets every 30 minutes
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com \
  --resolve --time-between-runs 30

Dashboard /api/cve-report JSON is produced by collect_cve_dashboard (section 8.4), not by cve_analysis stdout (HUM-5800).

Inspecting lifecycle timing with dump_lifecycle

scripts/dump_lifecycle reads lifecycle milestones and package data from the dashboard HTTP API and prints them for one or more HUM tickets. CVE_REPORT_TOKEN is needed.

# Text output for one ticket against prod
scripts/dump_lifecycle --prod HUM-1234

# JSON output for multiple tickets against preprod
scripts/dump_lifecycle --preprod --json HUM-1234 HUM-5678

# Custom dashboard URL
scripts/dump_lifecycle --dashboard-url https://... HUM-1234

Output includes all milestone timestamps (cve_published, osidb_flaw_created, osidb_affect_created, cve_embargo_lifted, hum_ticket_created, …), computed duration legs, and rpm_first_published (package) sourced from package_lifecycle. The --json flag emits the output as a JSON array instead of human-readable text.

Backfilling OSIDB timestamps with backfill_osidb_timestamps.py

scripts/backfill_osidb_timestamps.py is a one-time helper that fills osidb_flaw_created and osidb_affect_created for tickets already stored in cve_ticket_events. It reads existing rows from /api/cve-export, looks up timestamps from the public OSIDB API, and POSTs new rows to /api/cve-import with reset=false (write-once; existing timestamps are kept). Tickets that already have both OSIDB events are skipped unless --force is set. CVE_REPORT_TOKEN is needed.

# Backfill prod
scripts/backfill_osidb_timestamps.py --prod

# Dry run against preprod
scripts/backfill_osidb_timestamps.py --preprod --dry-run

# Limit to specific tickets
scripts/backfill_osidb_timestamps.py --prod HUM-1234 HUM-5678

Ongoing collection is handled by collect_cve_dashboard. After a prod backfill, copy_prod_to_preprod_db copies the new event types with the rest of cve_ticket_events (they are on the dashboard import allowlist).

Syncing prod CVE data to preprod with copy_prod_to_preprod_db

scripts/copy_prod_to_preprod_db copies cve_analysis_log, cve_ticket_events, package_lifecycle, and cve_ticket_blame from prod to preprod via a direct pg_dump/pg_restore over two kubectl port-forward tunnels, instead of paginating them through /api/cve-export -> /api/cve-import. cve_analysis_log rows can carry up to 512 KiB of log_output, so the HTTP path needs hundreds of small paginated requests and aborts entirely on the first 5xx; the direct-DB path does the whole sync in one dump/restore pair, typically well under a minute end to end.

The script manages its own port-forwards — killing anything already bound to the local ports first — and retries pg_dump/pg_restore if a tunnel goes stale mid-transfer (a kubectl port-forward can keep its local listener up while silently dropping all traffic). It also checks kubectl auth whoami on both contexts before doing anything, printing the OpenShift console login URL if a token has expired.

Prerequisites: kubectl, pg_dump, pg_restore, and pg_isready on PATH, and an active oc login to both the prod and preprod OpenShift clusters.

# Check auth, start tunnels, and preview the planned pg_dump/pg_restore
# commands; no data is transferred
scripts/copy_prod_to_preprod_db --dry-run

# Run the sync (destructive on preprod; --yes is required)
scripts/copy_prod_to_preprod_db --yes

# Sync a subset of tables only
scripts/copy_prod_to_preprod_db --yes --table cve_analysis_log

A stalled tunnel is detected by watching for no progress (growing dump file size on export, pg_isready on restore) for --stall-timeout seconds (default 300s — preprod’s Postgres pod runs under a 500m CPU limit and has been observed taking 150s+ to respond during a checkpoint, so the default gives 2x margin above that; a shorter timeout causes false-positive “dead tunnel” restarts). On a stall, the affected tunnel is restarted and the step retried up to --max-attempts times (default 3); pg_restore --clean --if-exists makes every retry idempotent. A timing summary (per phase and total) prints at the end.

Collecting rpm_first_published with collect_rpm_first_published

scripts/collect_rpm_first_published sweeps the Hummingbird Pulp repo for all packages, finds the earliest SRPM upload timestamp for each, and writes the results to the package_lifecycle table via POST /api/cve-export - no direct DB access.

# Sweep all packages from the rpms repo and write to prod
scripts/collect_rpm_first_published --prod --rpms-repo /path/to/rpms

# Only process specific packages
scripts/collect_rpm_first_published --prod --rpms-repo /path/to/rpms --package curl wget

# Dry run — log what would be written without posting
scripts/collect_rpm_first_published --prod --rpms-repo /path/to/rpms --dry-run

Re-runs are safe: the dashboard upsert only updates occurred_at if the incoming timestamp is earlier than the stored one.

Analyzing CVE Response Delays with analyze_rpm_first_published

scripts/analyze_rpm_first_published inspects the relation between CVE publication, package onboarding (rpm_first_published), and HUM ticket creation:

# Analyze all tickets on prod
scripts/analyze_rpm_first_published --prod --token $CVE_REPORT_TOKEN

# Filter to new packages onboarded after CVE publication
scripts/analyze_rpm_first_published --prod --filter new-pkg

# Filter to pre-existing packages with at least 30 days actionable delay
scripts/analyze_rpm_first_published --prod --filter pre-existing --min-days 30 --sort actionable

CLI options:

  • --filter {all,new-pkg,pre-existing}: filter by onboarding timing relative to CVE.
  • --min-days N: minimum delay threshold in days (default: 0).
  • --sort {cve_to_hum,actionable,cve_to_rpm}: sort column (default: actionable).
  • --limit N: max rows to display (default: 50).

Configuration

Option Environment Variable Description
--jira-token JIRA_TOKEN Jira API or Bearer token
--jira-url JIRA_URL Jira base URL (default: https://redhat.atlassian.net)
--jira-user Username for Basic auth
--output, -o Stdout format: human only (json / json-pretty removed in HUM-5800)
--show-since Filter by creation period (e.g. 2 weeks, 3 hours)
--resolve Transition Jira tickets and apply labels
--skip-assignee-check Skip assignee validation (for testing)
--include-closed Include Closed tickets in analysis; warn on stale closures
--exclude Comma-separated ticket keys to skip (e.g. HUM-555,HUM-552)
--skip-no-cve Omit issues with no CVE ID (CVE ID field or Summary); no postpone comment/label
--max-results Max number of issues to fetch (default: 2000)
--max-workers CVE_ANALYSIS_MAX_WORKERS Worker threads for per-ticket analysis and (with --resolve) decide mutations (default: 8)
--github-token GITHUB_TOKEN GitHub API token for upstream PR search
--gitlab-token GITLAB_TOKEN GitLab API token for gitlab.com MR search
--nvd-cache-dir NVD_CACHE_DIR Directory for caching NVD data feed files; avoids re-downloading unchanged feeds
--rpm-scan-cache-s3-bucket RPM_SCAN_CACHE_S3_BUCKET S3 bucket for durable RPM provides and Syft artifact JSON; disabled if unset
--rpm-scan-cache-s3-region RPM_SCAN_CACHE_S3_REGION Region for the RPM scan cache bucket; required when the bucket is set
--rpm-scan-cache-s3-access-key-id RPM_SCAN_CACHE_S3_ACCESS_KEY_ID Access key for the RPM scan cache bucket; required when the bucket is set
--rpm-scan-cache-s3-secret-access-key RPM_SCAN_CACHE_S3_SECRET_ACCESS_KEY Secret key for the RPM scan cache bucket; required when the bucket is set
--rpm-scan-cache-max-bytes RPM_SCAN_CACHE_MAX_BYTES Soft LRU size budget in bytes for the RPM scan S3 cache (default 1.5 GiB)
--cee-gitlab-token CEE_GITLAB_TOKEN CEE GitLab token for advisory repo operations
--advisories-project ADVISORIES_REPO Advisories repo URL (default: releng/advisories)
--advisories-fork ADVISORIES_FORK Bot’s fork URL for advisory MR creation
--slack-webhook-url SLACK_WEBHOOK_URL Slack webhook URL for failure and image-tracker alerts; empty or unset disables Slack
--no-merge-request Skip advisory MR creation during --resolve
--keep-advisory-repo Do not delete the cloned advisory repo after the run (useful for debugging)
--test-advisory Create advisory MR then immediately close it (for testing)
--package-map Path to CSV override; by default uses rpms repo metadata
--rpms-repo Path to RPMs git repo for fixed build detection
--cve-repo Path to local cvelistV5 git repo for CVE record lookups
--time-between-runs Re-run every N minutes; 0 = single run (default)
--feature-flag-project-id GitLab project ID for feature flag lookup (default: 73447720, rpms project)
--feature-flag-name Feature flag name to check each cycle (default: cve_analysis_enabled; empty = disabled)
--board-id Jira Agile board ID for sprint lookup (default: 1489)
--sprint-prefix Sprint name prefix to identify Hummingbird sprints (default: Hum S)
--handoff-file CVE_ANALYSIS_HANDOFF_FILE Write analysis→collector handoff JSON (counters, log, fix times, human_text)
SENTRY_DSN Optional Sentry DSN for error tracking

Managed Labels

The tool manages the following labels on Jira tickets. These are automatically applied, upgraded, and cleaned up:

Label Meaning
upstream-fix-available A fix exists (forge PR/commit, NVD/CVE link, or Fedora Bodhi/DistGit)
upstream-fix-in-progress A fix is in progress (open forge PR, or testing/pending Bodhi update)
fedora-bz-filed A Fedora Bugzilla has been filed for this CVE
cve-needs-attention Analysis needs human attention (see below)
advisory-mr-failed Advisory MR has unresolvable rebase conflicts
cve-next-release Fix will arrive via the next upstream release (set manually; see below)
awaiting-vex Closed ticket awaiting VEX agreement with Jira resolution (HUM-5843 work queue)
awaiting-cve-published Ticket (open or closed) awaiting a cve_published date from cvelistV5
awaiting-image-rebuild Closed Done-Errata ticket awaiting its image rebuild on Quay (HUM-9975)

Legacy fedora-fix-available / fedora-fix-in-progress labels are no longer applied; the bot removes them on subsequent runs (Bodhi/DistGit evidence now uses upstream-fix-*).

Labels are upgraded automatically (e.g., upstream-fix-in-progress is replaced by upstream-fix-available when a fix is merged). Stale labels are removed when tickets are closed, except fedora-bz-filed which is preserved as an audit record and awaiting-vex which is managed by the VEX reconcile pass. The cve-needs-attention label is removed automatically when the warning condition no longer applies.

awaiting-vex (HUM-5843)

When a ticket is closed as Done-Errata or Not a Bug in production, analysis adds awaiting-vex. Each --resolve cycle then:

  1. Queries Closed tickets labeled awaiting-vex, plus Closed Done-Errata / Not a Bug tickets resolved in the last 7 days (so human/agent closes that skipped the close-time enqueue still enter the queue)
  2. Fetches Hummingbird status from the Red Hat CSAF VEX feed for every CVE ID in the ticket summary, scoped to that ticket’s package (not a CVE-wide worst-case across other Hummingbird products)
  3. Compares to the Jira resolution for the awaiting-vex work queue (Done-Errata → fixed, Not a Bug → known_not_affected, package_not_listed when CSAF omits the package, or fixed – CSAF’s known_not_affected/fixed split tracks dependency-version presence, not code-path reachability, so a Not a Bug verdict is compatible with either, e.g. a downstream package swept into an unrelated dependency’s rebuild-and-fix); match requires all CVE IDs to agree
  4. On match: records vex_status + vex_resolved (scan time) in the collector handoff and removes the label if present
  5. On pending/mismatch: adds awaiting-vex if missing, otherwise keeps it, and still emits current vex_status. Catch-up closes with no CVE ID in the summary are not labeled: MATCH is impossible, and adding the label would re-select the ticket forever.

The Closed tab computes MATCH from stored vex_status plus Jira resolution; it does not use stored vex_match_state as source of truth. Dashboard reads merged event metadata (later rows overlay earlier ones) so a ticket closed after cve_published does not keep leftover New / In Progress analysis text as Resolution (HUM-6091).

The collector also fetches any vex_updates[] keys that the watermark JQL missed, so a same-cycle catch-up match still lands on the Closed tab.

Won’t Do closures do not enqueue awaiting-vex. Open-ticket VEX mismatches remain a vex-checker reconcile concern.

awaiting-image-rebuild (HUM-9975)

A ticket closes Done-Errata the moment the RPM-level fix ships – resolution is a version comparison with no dependency on whether the Hummingbird image has rebuilt with that fix yet, and image rebuilds routinely lag RPM publish by hours. collect_cve_dashboard’s watermark JQL only re-examines a Closed ticket when it was updated within the current watermark window or it carries an awaiting-* workflow label, so without a dedicated label a ticket that closes before its image finishes rebuilding falls out of the watermark window on the very next run and image_rebuilt_on_quay is never captured.

_sync_awaiting_image_rebuild_labels (collect_cve_dashboard.py) adds awaiting-image-rebuild to a Closed Done-Errata ticket once it has an hb_rpm_fix but no hb_image_fix yet, so the JQL keeps re-selecting it every run; the label is removed once hb_image_fix resolves or the ticket is no longer Done-Errata. This mirrors the awaiting-vex / awaiting-cve-published pattern.

cve-next-release

The cve-next-release label is used for CVEs where no immediate action is possible and the fix will arrive via the next upstream release. It is set manually. Common scenarios include:

  • Vendored dependencies: A fix exists in a vendored package (e.g. ws inside dotnet) but cannot be consumed until the parent package updates its vendored copy.
  • No backport path: The fix cannot be backported to the current release and must wait for a future upstream version.
  • Upstream fix pending: A fix is expected upstream but has not landed yet, and no interim mitigation is available.

Package metadata fix_status: 0 (EOL) does not use this label; those tickets are closed as Won’t Do instead.

When this label is present on a ticket:

  • The automation skips applying upstream-fix-available and upstream-fix-in-progress labels, since they would be misleading, and removes them if already present
  • The SBOM version check continues each analysis cycle, so when a new upstream release containing the fix is consumed, the ticket is closed normally
  • Fedora labels and cve-needs-attention are still managed normally
  • The label is not removed automatically; it must be removed manually when no longer applicable

cve-next-release tickets are not closed just because Hummingbird shipped a new build. They close when analysis concludes the CVE is no longer affected; for vendored dependencies, that means the SBOM shows the fixed upstream version, not merely a new parent NVR.

cve-needs-attention conditions

The cve-needs-attention label is applied when any of these conditions are detected:

  • Tracker filed against an image: the tracker component is a Hummingbird image (hi/<name>) instead of an RPM; a Slack alert is also sent
  • Product mismatch: CVE vendor/product does not match the Hummingbird package (e.g. node-tar CVE filed against GNU tar) after an SBOM-first check finds no vendored dependency hit
  • Package missing from rpms but present in SBOM/binaries: package directory is absent from the rpms repo, yet the CVE product appears in the package SBOM and binary confirmation is present or unknown
  • Multiple products: CVE lists multiple distinct products with different versioning schemes and no cve_product override is configured (e.g. NGINX Open Source + NGINX Plus)
  • Git-only version data: CVE version data uses commit hashes instead of numeric versions (e.g. libsodium with lessThan: ad3004ec...)
  • CNA data error: A git commit hash is used where a version number is expected without setting versionType: "git", including hashes embedded in operator syntax (e.g., "version": "< 6374ae0bcdfe..."). The warning includes a link to the CVE 5.0 source control versions spec
  • Malformed version field: The version field contains syntax that could not be parsed (e.g., compound operator+hash ranges like >= hash1, < hash2)
  • CVE record not found: the CVE record file was not found in the cvelistV5 repo
  • No repo version: the Hummingbird package was not found in the Pulp repository (package name mismatch or missing SRPM)
  • Stale closure: ticket is Closed in Jira but current analysis recommends a different resolution

Human override workflow

When a human wants to take over a ticket from the bot:

  1. Assign the ticket to yourself – the bot skips all automation on tickets not assigned to the bot account. Reassign to the bot to re-enable automation.

  2. Set “Fixed in Build” – if you know the fix is in a specific SRPM, set the “Fixed in Build” field to the SRPM name (e.g. libarchive-3.8.7-1.hum1.src.rpm). The bot will use this to create the advisory MR and close the ticket, bypassing its own analysis.

Migration note: The cve-analysis-okay label is deprecated and no longer suppresses automation. Existing tickets with this label will be re-processed by the bot on its next run. To keep the bot from touching a specific ticket, reassign it to yourself before the next run. The cve-analysis-okay label can then be removed manually.

Output

Human-Readable

Project: HUM  Component: Security

  HUM-796  CVE-2026-2673 openssl: buffer overflow [hummingbird-1]
    Open since:       7 days (Mar 24 2026)
    Labels resolution: upstream-fix-available
    OpenSSL / OpenSSL:
      Affected versions: 3.5.0 < 3.5.6
      Fixed in:          3.5.6
      Hummingbird repo (latest):  3.5.5 / openssl-3.5.5-1.hum1.src.rpm
    Repo:             https://github.com/openssl/openssl
    CVE-2026-2673: upstream-fix-available (3 PRs)
      [CLOSED, 3 commits] Fix group tuple handling in DEFAULT expansion (3.5)
              https://github.com/openssl/openssl/pull/30110
    Fedora update: FEDORA-2026-abc123 openssl-3.5.6-1.fc44 (stable, security)
              https://bodhi.fedoraproject.org/updates/FEDORA-2026-abc123
    Jira current:     In Progress / (none)
    Jira resolution:  In Progress / affected (repo 3.5.5 is in affected range 3.5.0 < 3.5.6)

JSON

Each issue includes: key, summary, labels, open_since, cve_ids, cves (with version and resolution data), jira_current (status/resolution), computed_resolution, and upstream (with PR/MR search results, Bodhi updates, and Fedora version data).

Viewing Logs

The CVE analysis tool runs as a pod in the hummingbird--internal namespace on mpp-prod. See the OpenShift MP+ internal docs page for general log access instructions.

For CVE analysis specifically:

  • Live: Open the mpp-prod pods list in the OpenShift console, filter for cve-analysis, and open the Logs tab.
  • Grafana/Loki: Open the Hummingbird Grafana logs dashboard, select cluster mpp-prod and namespace hummingbird--internal, then filter for hummingbird-cve-analysis pods.

Library layout

Reusable library code lives under hummingbird_cve_analysis/lib/. The CLI entry point remains cve_analysis.py at the package root.

Module Responsibility
lib/github_client.py GitHub PR and release API reads
lib/gitlab_client.py gitlab.com merge request, release, and feature-flag reads
lib/fedora.py Fedora Bodhi, DistGit spec parsing, and Bugzilla helpers
lib/upstream.py Upstream forge search, cgit commits, and fix-status analysis
lib/net.py Shared network error tuple used by library HTTP callers
lib/nvd.py NVD JSON 2.0 feed cache, download, and reference parsing
lib/cvelist.py Local cvelistV5 repository access and CVE 5.0 affected parsing
lib/jira_client.py Jira REST read/write primitives and ADF comment helpers
lib/versions.py Pure version comparison and range helpers
lib/analysis.py CVE decision pipeline, product matching, and lifecycle analysis
lib/formatting.py Human and Jira comment/output formatting
lib/resolve.py Jira/advisory mutation helpers (labels, close, attach SBOM)
lib/ticket.py Shared one-ticket analyze/resolve API
lib/advisory_handler.py CEE advisories repo clone/edit/MR helpers
lib/osidb_client.py OSIDB subpackage PURL lookups and flaw/affect created timestamps
lib/pulp.py Hummingbird Pulp repo RPM/SRPM listings and repodata lookups
lib/catalog.py Container catalog API for image publish times
lib/rpms_repo.py Local rpms git repo, package map loading, fixed-build detection
lib/rpm_scan_cache.py S3 LRU cache of RPM provides and Syft artifact JSON
lib/sbom.py SBOM fetch, vendored dependency lookup, binary RPM confirmation
lib/slack.py Slack webhook helper
lib/version_transforms.py Named version-transform helpers

cve_analysis.py is the CLI entry point only (argument parsing, JQL building, signal handling, and the main orchestration loop). Library code lives under lib/; import and patch those modules directly. New cron jobs, webhooks, and other integrations should prefer ticket.process_ticket for analyzing (and optionally resolving) a single Jira issue:

from hummingbird_cve_analysis.lib import ticket

result = ticket.process_ticket(
    issue,
    pkg_map,
    catalog_source_map,
    github_token,
    gitlab_token,
    base_url=base_url,
    token=token,
    resolve=False,
)

Development

See the main README for development workflows.

make hummingbird-cve-analysis/setup  # Install dependencies
make check                            # Lint code (ruff)
make test                             # Run unit tests

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.