CVE Analysis

Automated analysis of HUM Jira Security (CVE) tickets. For each ticket, the tool fetches vulnerability data from MITRE and NVD, compares it against the versions shipped in the Hummingbird package repository, searches for upstream and Fedora fixes, and computes whether the ticket should be closed, moved to In Progress, or flagged for manual investigation.

How It Works

1. Ticket Selection

The tool queries Jira for HUM project tickets with component Security. Tickets can be filtered by creation period (--show-since "2 weeks"), limited to specific keys (HUM-796 HUM-518), or scanned in bulk. Closed tickets are skipped by default unless --include-closed is specified or specific ticket keys are given on the command line.

Embargoed tickets (security level “Embargoed Security Issue” or embargo status field set to true) are always skipped.

Tickets with no valid CVE-YYYY-NNNN in the CVE ID field or Summary (for example GHSA/OSV-only trackers) are postponed: analysis is skipped, an INFO message is logged, and with --resolve a one-time tracker comment is posted and the cve-needs-attention label is applied (open tickets only). Use --skip-no-cve to omit those tickets without posting the postpone notice or adding the label.

2. CVE Data Collection

For each ticket the tool:

1. Extracts the CVE ID from the Jira CVE ID field (customfield_10667) when populated, otherwise from the Summary via a regex match for CVE-YYYY-NNNN patterns. If neither source has a CVE ID, analysis is postponed (see above).

2. Loads the CVE record from a local clone of the cvelistV5 repository and parses the CVE 5.0 affected block, following the CVE 5.0 Product and Version Encodings specification. This includes:

   • Parsing lessThan and lessThanOrEqual version ranges
   • Handling lessThan: "*" (no upper bound) and lessThan: "4.*" (end-of-series, all versions in the 4.x series) wildcards per the spec
   • Processing changes lists that subdivide ranges into affected and unaffected segments
   • Respecting defaultStatus at the product level
   • Filtering out versionType: "git" entries that use commit hashes instead of numeric versions (these are preserved as informational data but not used for version comparison)
   • Detecting CNA data errors where git commit hashes are used in version fields without versionType: "git", including hashes wrapped in operator syntax (e.g., < 6374ae0bcdfe...)
   • Parsing inline comparison operators (< 12.3.0, >= 4.0, <= 2.5) that are not part of the CVE 5.0 schema but are widely used by CNAs in practice
   • Parsing compound range bands (>= 2.0, < 2.2.26 and > 1.32.3, < 1.34.6) used by some CNAs to express a closed-open range in a single version string

3. Looks up version ranges from the NVD data feeds and merges them with the MITRE data. NVD data is obtained from the JSON 2.0 data feeds at nvd.nist.gov/feeds/json/cve/2.0/ (per-year .json.gz tarballs updated daily, plus a CVE-Modified overlay updated every 2 hours). Feeds are cached locally (--nvd-cache-dir) and only re-downloaded when the NVD .meta file SHA256 indicates newer data is available. When MITRE has no vendor-specific data (all vendors are “n/a”), NVD ranges replace the MITRE data entirely. NVD references tagged “Patch” or matching commit URL patterns are also extracted and used for upstream fix detection (see below).

4. Looks up the Hummingbird package version by scraping the Pulp repository index at packages.redhat.com for the latest RPM matching the package name extracted from the ticket summary.

3. Resolution Computation

The tool compares the shipped Hummingbird version against the affected version ranges to compute a recommended resolution:

• Closed / Done-Errata: The repo version is not in any affected range, or the repo version is >= the fix version, or the repo version appears in the “not affected” list.
• In Progress / affected: The repo version falls within an affected range.
• In Progress / affected (no version data): Neither MITRE nor NVD has affected version information. The ticket is moved to In Progress for manual review but does not receive the cve-needs-attention label.
• In Progress / needs investigation: Version data was available but could not be compared (e.g., CVE only provides git commit hashes), or multiple distinct products with different versioning schemes are listed and no cve_product override is configured. These cases receive the cve-needs-attention label.

4. Product Mismatch Detection

The tool detects when a CVE was filed against the wrong Hummingbird package. It compares the CVE vendor/product names against the Hummingbird package name and upstream repo URL from the package map. For example, a CVE for isaacs/node-tar (an npm package) filed against the tar RPM (GNU tar) is flagged as a mismatch. This detection works even without a repo URL by comparing normalized product names.

Mismatch triage uses a two-phase evidence gate:

1. Identity mismatch — CVE product/vendor does not match the package (cve_product override or heuristic name/repo matching)
2. SBOM (+ binary) evidence — only then decide misfiled vs vendored vs needs investigation

Phase 2 fetches the latest package SBOM from the Hummingbird Pulp repository and records the artifact used (NVR/URL) plus match evidence. Go module matching accepts the module root and major-version paths (github.com/vendor/product or .../v5). Deeper import paths such as .../api or .../daemon only count as a hit when a CVE path hint (from malformed/version-path fields) aligns with that subpackage, so a client/API module does not satisfy a daemon-only CVE. If the CVE product is found as a vendored dependency, the tool runs binary confirmation in this order:

1. SPDX SBOM evidence — sourceInfo pointing at shipped paths (e.g. Go buildinfo under /usr/bin/...) or CONTAINS from binary RPM roots (*.x86_64, *.aarch64, …) marks present_in_binary without Syft. Evidence only under .src / go.mod is treated as source-only for the next step.
2. RPM Provides: bundled(...) — when SBOM evidence is source-only or ambiguous, downloaded non-debug binary RPMs are queried with rpm -qp --provides. A matching Fedora bundled Provide (npm bundled(npm(name)), Go bundled(golang(IMPORT_PATH)), Node core bundled(nodejs-undici), etc.) marks present_in_binary with evidence_method=rpm_bundled_provides. Ecosystem wrappers such as npm(...) are unwrapped so a CVE/SBOM search for nanoid matches bundled(npm(nanoid)). This catches dependencies embedded into binaries (including minified frontend JS) that Syft cannot inventory as on-disk packages.
3. Syft on binary RPMs — when neither SPDX nor bundled Provides confirm presence, Syft scans those same RPMs. When OSIDB reports subpackage names that match RPMs for the NVR, only those RPMs are scanned; if OSIDB data is missing or the names do not match this package’s RPMs (for example a Redis CVE filed against boost), the full NVR set is used. Pulp download URLs use the arch from the RPM filename (aarch64 files under aarch64/, x86_64 under x86_64/; noarch stays on the x86_64 listing).

Syft absence only becomes absent_in_binary_confirmed when SBOM evidence is source-only (or no SBOM document was available) and Syft actually ran. A miss with ambiguous SBOM evidence stays unknown so embedded Go/Rust modules are not closed as source-only when SPDX already ties them to a binary. A Provides-only scan (Syft unavailable) never confirms absence. Per-RPM download or scan failures are skipped so sibling RPMs can still confirm presence; if any RPM failed and no hit was found, the result stays unknown (absence is never confirmed on a partial scan). A Provides query failure soft-misses and falls through to Syft for that same RPM.

• present_in_binary (mismatch_vendored): legitimate vendored ticket; compare the confirmed binary/Provide version (falling back to the SBOM vendored dep version) against the CVE ranges (parent latest_repo_version / Hummingbird repo version stay the package), and apply cve-needs-attention so operators can review the binary hit. When several bundled Provides match the same dependency (for example npm(nanoid) at 3.3.16 and 5.1.16), every copy is collected. Resolution is worst-of / all-must-be-fixed: the ticket stays affected if any copy is still affected; Done-Errata only when every copy is not affected. The tool does not invent multi-major fix mappings from a single CVE fixed line — Fixed in Build remains the override when CVE version data only describes one release line. Placeholder dep versions such as cargo std@0.0.0 are not comparable: fall back to the parent package version when available; otherwise set non_comparable_vendored_version, recommend New / needs investigation, and apply cve-needs-attention — never Done-Errata from “0.0.0 is not in any affected range”
• absent_in_binary_confirmed (mismatch_source_only): source-only; eligible to close as Not a Bug / Component not Present. Must never set Fixed in Build or take the advisory / Done-Errata path.
• unknown (mismatch_binary_unknown): leave for investigation; do not auto-close or call the ticket misfiled

Stable reason codes are emitted in analysis output (Reason codes:) and on each CVE entry (reason_code / mismatch_gate) for automation:

A misfiled recommendation requires identity mismatch + SBOM miss (no vendored hit). Ambiguous checks and confirmed binary RPM matches keep cve-needs-attention with the reason code above. When present in binaries, normal resolution flow still proceeds (In Progress / Close) in addition to the attention label. The upstream fix search targets the parent package’s repo. Each analysis cycle re-fetches the SBOM and re-runs binary confirmation.

5. Upstream Fix Detection

Fix search uses a precedence model so tickets are not dual-labeled from both Fedora and upstream when a definitive path exists:

1. CVE/NVD fix links (preferred): NVD references tagged “Patch” or matching commit URL patterns, plus CVE/NVD PR/MR references that match the package’s upstream repo. When these yield a fix label, Bodhi and forge search are skipped.
2. Otherwise, repo-aware search from metadata/*.json upstream_repo:
   • Fedora-based (src.fedoraproject.org, pagure.io, or missing repo URL): Bodhi + DistGit only (see §6)
   • Non-Fedora forge (GitHub/GitLab/cgit): forge search only

Forge Search (non-Fedora fallback):

• GitHub: Searches PRs via the GitHub API (/search/issues) for PRs mentioning the CVE ID. Fetches commit counts from the pulls API.
• GitLab: Searches merge requests via the GitLab API on supported instances (gitlab.com, gitlab.gnome.org, gitlab.freedesktop.org, etc.). The --gitlab-token is only sent to gitlab.com to avoid 401 errors on other instances.
• cgit: Scrapes commit log pages on cgit hosts (Savannah, Sourceware, kernel.org, busybox.net, etc.) for commits mentioning the CVE ID. Handles URL rewrites for Savannah hosts (e.g., git.savannah.gnu.org/git/ to cgit.git.savannah.gnu.org/cgit/).

Fix status is classified as:

• upstream-fix-available: At least one merged/closed PR, committed fix, or NVD patch reference
• upstream-fix-in-progress: At least one open PR with no merged fixes

6. Fedora Fix Detection

Used only for Fedora-based packages (or when no upstream repo is configured) and only when CVE/NVD fix links did not already produce a label. Hits are classified with the same upstream-fix-* labels as forge/NVD evidence (Bodhi update links are still shown in analysis output):

• Bodhi: Queries the Fedora Bodhi API for updates matching the package name that reference the CVE ID (in the cves list or notes field). Supports multi-page results. Stable → upstream-fix-available; testing/pending → upstream-fix-in-progress.
• DistGit Spec Scan: When Bodhi has no matches, fetches the Fedora DistGit spec file and scans for CVE references in patch filenames, changelog entries, and comments to detect backported fixes (upstream-fix-available).

7. Fixed Build Detection

The tool automatically detects whether the current Hummingbird build fixed a CVE by checking if all CVE IDs are mentioned anywhere in the package directory (patch names, changelogs, comments in the .spec file and so on). If found, it returns the current source RPM name. Assuming that we run the analysis frequently enough, this is precise enough, otherwise it errs on the side of caution (i.e. it possibly marks a higher version as fixed even if an earlier one already carried the fix).

The detected build is shown in the output as Detected Fix: package-version-release.hum1.src.rpm.

When --resolve is specified and a fixed build is detected, the tool automatically populates the “Fixed in Build” Jira field with the detected SRPM name, but only if:

• The field is not already set (manual values take precedence)
• The issue is not in Closed status (respects human closure decisions)
• The assessment is not mismatch_source_only / source_only_confirmed (Component not Present closes as Not a Bug and clears any Fixed in Build)
• The computed resolution is already Closed / Done-Errata or every ticket CVE ID appears on an uncommented PatchN: line (HUM-6182). A spec comment or leftover unapplied file that merely mentions the CVE is not enough. Product-mismatch and EOL tickets never take this PatchN override.

Version-range analysis can still say “affected” after a backport that does not bump Version (for example an unbounded range such as 1.1.1+ on popt 1.19). In that case an applied CVE-YYYY-NNNN.patch listed as PatchN: is allowed to set Fixed in Build. The SRPM must already be in Pulp (the existing fix_committed_not_built gate is unchanged). The ticket is not closed here; advisory review remains the backstop.

The RPMs repository can be provided via --rpms-repo, pointing to a local clone. This is preferable with multiple runs to avoid the git clone operation. When not given, the repository will be (shallow) cloned into a temporary directory.

Similarly, the cvelistV5 repository (used for CVE record lookups) can be provided via --cve-repo. When not given, a shallow clone is performed into a temporary directory. For production use, pre-clone and update the repo externally (e.g. via a CronJob) to avoid the clone overhead on each run. The tool does not attempt to update the provided repo itself.

8. Advisory Integration (with --resolve)

When a CVE is resolved as “Closed / Done-Errata”, the tool integrates with the CEE GitLab advisories repo to document the fix:

1. Clone: The advisories repo (default: releng/advisories) is shallow-cloned via a bot fork using netrc-based authentication (no tokens in process arguments or logs).
2. Modify: For each resolved ticket, the advisory YAML is updated: cves.fixed entries are added, the type is changed from RHBA to RHSA, and CVE references are appended.
3. Batch MR: All advisory changes are accumulated as individual commits on a single branch (cve-analysis/batch). One merge request is created at the end of the run covering all tickets.
4. Review state: Tickets are transitioned to “Review” (not directly closed) and the batch MR URL is posted to each affected ticket.
5. Auto-close: On subsequent runs, tickets in “Review” are checked: if the advisory MR has been merged, the ticket is closed as “Done-Errata”. If the MR has unresolvable rebase conflicts, a comment with manual resolution steps is posted and the advisory-mr-failed label is added.
6. Slack notifications: When --slack-webhook-url (or the SLACK_WEBHOOK_URL env var) is set, the tool sends a Slack message when the batch MR fails to merge or the post-merge pipeline fails. Merge-failure messages @-mention prarit and jstibran.

The advisory flow is skipped when --advisories-project points to a non-production URL, allowing safe testing without modifying Jira.

8.1 Recovery for failed advisory MRs

In some failure cases (for example, advisory MR conflicts followed by manual ticket closure), Jira status can be corrected to Done-Errata while dashboard lifecycle metadata remains incomplete. This leaves R-Time rows PENDING (elapsed-to-now) until image, VEX, and close all exist.

Use the recovery script to backfill delivery events:

• rpm_fix_published_to_pulp (hb_rpm_fix) from packages.redhat.com source RPM timestamps
• image_rebuilt_on_quay (hb_image_fix) from catalog/quay image history
• hum_ticket_closed from Jira resolutiondate

Only tickets that are Closed / Done-Errata are used for event derivation and dashboard import. Tickets that remain in Review (or any non-closed state) are reported and skipped for event backfill.

For tickets still in Review, the script also attempts advisory reconciliation before deriving events:

• Read advisory MR URL from Jira comments (Advisory MR: https://...)
• If no MR URL is present in comments, search merged advisories MRs for the HUM ticket key in MR description/title
• When a merged MR is found and --apply is used, transition the ticket to Closed / Done-Errata and then derive hum_ticket_closed

Script location:

• hummingbird-cve-analysis/scripts/cleanup_advisory_mr_failure_events.py

Run in dry-run mode first (default):

cd /path/to/tools
export JIRA_TOKEN=your_jira_token
export CEE_GITLAB_TOKEN=your_cee_gitlab_token
export CVE_REPORT_TOKEN=your_dashboard_token

python3 hummingbird-cve-analysis/scripts/cleanup_advisory_mr_failure_events.py \
  --jira-user your-jira-user@example.com \
  --cee-gitlab-token your_cee_gitlab_token \
  --output-json /tmp/cleanup-advisory-derived.json

Apply the backfill to dashboard event storage:

python3 hummingbird-cve-analysis/scripts/cleanup_advisory_mr_failure_events.py \
  --jira-user your-jira-user@example.com \
  --cee-gitlab-token your_cee_gitlab_token \
  --output-json /tmp/cleanup-advisory-derived.json \
  --apply

Target specific tickets (instead of default JQL):

python3 hummingbird-cve-analysis/scripts/cleanup_advisory_mr_failure_events.py \
  --jira-user your-jira-user@example.com \
  --cee-gitlab-token your_cee_gitlab_token \
  --apply \
  HUM-4884 HUM-4871 HUM-4844

Verification example:

# Replace with the actual dashboard URL (same as --dashboard-url default)
DASHBOARD_URL=https://hummingbird-dashboard-hummingbird--internal.apps.int.spoke.prod.us-east-1.aws.paas.redhat.com
curl -s "${DASHBOARD_URL}/api/cve/r-time?days=30" | jq -r '
  .entries[]
  | select(.key=="HUM-4884")
  | [.key, (.fix_delivered_at // "-"), (.stages.hb_rpm_fix // "-"), (.stages.hb_image_fix // "-")]
  | @tsv
'

If hb_rpm_fix and hb_image_fix remain empty in the derived JSON output, the issue is source-data availability (no resolvable RPM/image publish signal), not dashboard ingestion.

If a ticket is in Review and no advisory MR can be found in Jira comments or merged advisories MRs, it is skipped and reported for manual follow-up. By default, the script targets advisory-mr-failed tickets (for both Closed / Done-Errata and Review states); override with --jql when needed.

8.2 Dashboard collection ticket selection

Dashboard lifecycle collection (HUM-5603) uses a separate ticket-selection path from cve_analysis mutations. Helpers live in hummingbird_cve_analysis/dashboard_selection.py.

Watermark (start of previous run): read the newest entry from the dashboard Run Log (GET /api/cve/run-log?limit=1). Approximate previous-run start as run_at - duration_seconds (Run Log run_at is ingest time), then subtract a small buffer (default 2 minutes) for clock skew. Using start — not end — avoids missing tickets updated while the previous collector run was in progress. Overlap is intentional; /api/cve-report event upserts are idempotent.

JQL selection:

• All still-open HUM Security tickets (rescan for updates)
• Closed tickets with Jira updated >= watermark
• Bootstrap (empty Run Log): open tickets only
• Explicit ticket keys: key in (...) (any status)

Use --start-time (ISO-8601, e.g. 2026-08-01T00:00:00Z) to override the Run Log watermark and recollect Closed tickets with updated >= that start, so dashboard data can be overwritten/backfilled.

Inspect watermark and JQL without modifying Jira or the dashboard. From the tools repo root, either install the package (make hummingbird-cve-analysis/setup) or set PYTHONPATH:

cd /path/to/tools
export PYTHONPATH=hummingbird-cve-analysis
export CVE_REPORT_TOKEN=your_dashboard_token

python3 -m hummingbird_cve_analysis.dashboard_selection --prod

# Optional: also search Jira and list matching keys
export JIRA_TOKEN=your_jira_token
python3 -m hummingbird_cve_analysis.dashboard_selection \
  --prod \
  --jira-user user@example.com \
  --fetch-issues -o json-pretty

8.3 Analysis → collector handoff file

cve_analysis writes a small JSON handoff file for collect_cve_dashboard (HUM-5799 / HUM-5800). This carries Run Log counters, captured log_output, and per-ticket fields that are expensive to recompute (upstream_fix, fedora_fix) plus human_text for Run Log details. Delivery timestamps (hb_rpm_fix / hb_image_fix) and hum_ticket_closed are gathered by the collector, not this file. Analysis stdout is human text only; it does not emit /api/cve-report JSON.

PYTHONPATH=hummingbird-cve-analysis python -u -m hummingbird_cve_analysis.cve_analysis \
  --jira-user "$BOT_USER" --jira-token "$BOT_JIRA_TOKEN" \
  --handoff-file /tmp/cve_analysis_handoff.json \
  ...

Handoff shape (schema_version: 1):

• Top level: run_started_at, duration_seconds, mutation counters (labels_changed, advisories_created / advisories_failed, tickets_closed, comments_posted, attachments_uploaded, fatal_errors), log_output
• tickets[]: key, upstream_fix, fedora_fix, human_text
• vex_updates[] (HUM-5843): key, vex_status, vex_match_state, vex_resolved, labels from the awaiting-vex reconcile pass

8.4 collect_cve_dashboard

collect_cve_dashboard (HUM-5798 / HUM-5800) owns dashboard JSON emission. cve_analysis no longer prints /api/cve-report JSON on stdout; it prints human analysis text and optionally writes --handoff-file for the collector.

1. Resolve Run Log watermark + open-ticket rescan (section 8.2). Selection always includes Closed tickets labeled awaiting-vex (HUM-5843), then any vex_updates[] keys missing from that JQL
2. For each ticket, gather slim lifecycle fields from Jira / cvelistV5 / Fixed-in-Build → Pulp (cve_published, hum_ticket_created, hum_ticket_closed, hb_rpm_fix) and catalog image history (hb_image_fix), plus OSIDB osidb_flaw_created / osidb_affect_created from the public OSIDB API (omitted when OSIDB has no data), computed_resolution from Jira status/resolution (R-Time gates on Done-Errata in that string) and catalog_image_source from the catalog source map (R-Time delivery is image publish when true, RPM publish when false). Timestamp helpers live in lib/lifecycle.py and are shared with analyze_issue
3. Merge --handoff-file counters, log_output, per-ticket upstream_fix / fedora_fix / human_text, and vex_updates
4. Dry-run prints /api/cve-report JSON; --apply POSTs it

The collector builds the catalog source map once per run for hb_image_fix and catalog_image_source. When a delivery timestamp is omitted, the dashboard keeps any existing rpm_fix_published_to_pulp / image_rebuilt_on_quay value rather than clearing it. An empty or failed catalog map omits catalog_image_source so the dashboard keeps requiring an image.

PYTHONPATH=hummingbird-cve-analysis python -u -m hummingbird_cve_analysis.collect_cve_dashboard \
  --preprod \
  --cve-report-token "$CVE_REPORT_TOKEN" \
  --jira-user "$BOT_USER" --jira-token "$BOT_JIRA_TOKEN" \
  --cve-repo /tmp/cvelistV5 \
  --handoff-file /tmp/cve_analysis_handoff.json

# POST to preprod dashboard
PYTHONPATH=hummingbird-cve-analysis python -u -m hummingbird_cve_analysis.collect_cve_dashboard \
  --preprod \
  --cve-report-token "$CVE_REPORT_TOKEN" \
  --jira-user "$BOT_USER" --jira-token "$BOT_JIRA_TOKEN" \
  --cve-repo /tmp/cvelistV5 \
  --handoff-file /tmp/cve_analysis_handoff.json \
  --apply

9. Jira Actions (with --resolve)

When --resolve is specified, the tool modifies Jira tickets:

• Fixed in Build: When a fixed build is detected and the field is not already set, the tool populates it with the detected SRPM name (only for non-closed issues). Not-affected Closed / Done-Errata closes also set Fixed in Build from detected_fixed_build or latest_srpm when empty, so R-Time image timestamps can be resolved.
• Closed / Done-Errata: Tickets where the shipped version is not affected are transitioned to Review with an advisory MR (see above). After the MR merges, they are closed as Done-Errata on the next run.
• Move to In Progress: Tickets where the shipped version is affected are transitioned to In Progress with a comment including affected version ranges, upstream fix status, and Fedora update links.
• Needs Investigation: Tickets with incomplete version data get a comment explaining why automatic resolution was not possible.
• Product Mismatch: A comment is posted explaining the mismatch (with SBOM artifact evidence) and the cve-needs-attention label is applied. No transition is performed. Auto-detected Fixed-build / delivery timestamps (hb_rpm_fix, hb_image_fix) are cleared so R-Time does not treat misfiled tickets as done. Human Fixed in Build overrides are kept. When --resolve runs against the production advisories project, the analyzed package SBOM JSON is also attached to the ticket as {nvr}.sbom.json (skipped if that filename is already present). The same attachment is applied for other SBOM-backed review paths (vendored binary hits, source-only closes, binary-unknown).
• Package Not Present: If the package is missing from the rpms repo, the tool still runs an SBOM-first check against the CVE product(s). An SBOM hit triggers Syft binary confirmation: absent_in_binary_confirmed closes as Not a Bug / Component not Present; presence or unknown leaves the ticket for investigation. An SBOM miss/unavailable closes as Not a Bug with VEX Component not Present. The close timestamp is persisted as hum_ticket_closed for dashboard ingestion.
• Package EOL (fix_status: 0): Tickets for packages marked End-Of-Life in rpms metadata are closed as Won’t Do with no VEX Justification. This takes precedence over Done-Errata and Fixed-in-Build / advisory paths. Automation labels (upstream-fix-*, legacy fedora-fix-*, cve-needs-attention, cve-next-release, advisory-mr-failed) are removed; fedora-bz-filed is kept as an audit record. The close timestamp is persisted as hum_ticket_closed for dashboard ingestion.
• Label Management: Labels are applied and updated:
  • upstream-fix-available / upstream-fix-in-progress (including Bodhi/DistGit evidence for Fedora-based packages)
  • cve-needs-attention (applied when human review is needed; removed when resolved)
  • Legacy fedora-fix-* labels are stripped on subsequent runs
  • Labels are upgraded (in-progress to available) and stale labels are cleaned up on ticket closure.

Every Jira action (label add/remove, status transition, product mismatch) includes a single comment with a bold action summary heading followed by the full analysis output in a preformatted code block, giving the reader the same detail they would see on the CLI.

10. Assignee-Based Automation Control

When --resolve is active, the tool checks each ticket’s assignee. If the assignee is not the bot account (--jira-user), the tool skips all automated actions (comments, labels, transitions) for that ticket. A one-time comment is posted explaining that automation is skipped. This allows humans to take ownership of a ticket by assigning it to themselves, preventing the bot from interfering with manual work. Reassigning back to the bot account re-enables automation.

For testing purposes, use --skip-assignee-check to process all tickets regardless of assignee.

11. Human Closure Protection

When the tool encounters a Closed ticket whose analysis recommends a non-Closed resolution (e.g., In Progress), it checks the Jira changelog to determine who performed the last status transition. If a human (not the bot) closed the ticket, the tool skips the transition to respect the human’s decision. This prevents the tool from reopening tickets where a fix was backported without a version bump or where a human determined the CVE does not apply.

12. Stale Closure Detection (with --include-closed)

When --include-closed is used, the tool also analyzes Closed tickets. If the current analysis recommends a non-Closed resolution (e.g., the package version changed and is now affected), a warning is emitted flagging the ticket for review.

13. Sprint and Epic Assignment

When --resolve is active, the tool automatically adds CVE tickets to the current Hummingbird sprint and links them to a per-sprint CVE tracking epic when human interaction is detected. This ensures sprint metrics capture human work on CVE tickets.

The tool queries the Jira Agile API for the active sprint on the Hummingbird board (--board-id, default 1489) and filters by name prefix (--sprint-prefix, default “Hum S”) to identify the correct sprint among multiple teams sharing the HUM project.

Sprint and epic assignment is triggered in two cases:

• Self-assigned tickets: When a ticket is assigned to someone other than the bot, the ticket is added to the current sprint and linked to the sprint’s CVE epic.
• Human-set Fixed in Build: When the Fixed in Build field was set by a human (not the bot), as determined by the issue changelog, the ticket is added to the current sprint.

If no matching epic exists for the current sprint, one is created automatically with the summary “CVE tickets for sprint <name>”. The epic key is cached for the duration of the run. A Jira comment is posted on the ticket recording the sprint and epic assignment.

Tickets that already have a sprint assigned are skipped. Tickets closed manually without going through either automated path should be added to the sprint by hand (see the manual process guide).

14. Continuous Operation

When --time-between-runs N is set (N > 0), the tool runs in a loop, re-executing the full analysis every N minutes. The default is 0 (single run). Shutdown is graceful: SIGINT/SIGTERM finishes the current cycle before exiting.

15. Feature Flag

When --feature-flag-name is set (default: cve_analysis_enabled), the tool checks a GitLab feature flag on the project specified by --feature-flag-project-id (default: 73447720, i.e. redhat/hummingbird/rpms) at the start of each cycle. If the flag is inactive, the cycle is skipped and the tool sleeps for 60 seconds before checking again.

The check uses the --gitlab-token / GITLAB_TOKEN credential, which must have Developer role or higher on the target project. The check is fail-open: if GitLab is unreachable or the token lacks permissions, the tool assumes the flag is enabled and proceeds.

Set --feature-flag-name "" to disable the check entirely.

Package Map

The package map is loaded from the rpms repo’s per-package metadata files (metadata/<package>.json). Each file contains an upstream_repo field pointing to the canonical upstream git repository, plus optional fields:

• upstream_branch – upstream branch (for versioned packages sharing a repo)
• cve_product – CVE vendor/product override
• version_transform – version transform rule
• fix_status – package fix policy (0 = End-Of-Life / will not fix CVEs; close as Won’t Do)

If fix_status is 0, analysis always resolves to Closed / Won’t Do (including when the shipped version is outside the CVE affected range or a product mismatch would otherwise need investigation). No VEX Justification is set. If fix_status is missing or set to 1, behavior is unchanged from the default analysis flow.

The cve_product field is an optional human-curated value that specifies which CVE vendor/product entry maps to this package. It supports these formats:

• Vendor / Product (exact match): matches a specific vendor and product pair. Example: "F5 / NGINX Open Source" for the nginx package, which excludes NGINX Plus entries that use incompatible R-versioning.
• Vendor (vendor prefix match): matches any product whose vendor starts with the given string. Example: "Go " for golang packages, where the vendor varies (Go standard library, Go toolchain, etc.) and the product varies by module (net/url, os, crypto/x509).
• Multiple selectors: provide more than one acceptable value either as a JSON list in metadata (preferred) or as a semicolon-delimited string. Example: ["vda-linux / busybox_mirror", "BusyBox / BusyBox"] or "vda-linux / busybox_mirror; BusyBox / BusyBox". The tool treats this as “match any selector”.

When cve_product is set, the tool uses it for exact product matching in both resolution computation (filtering to the correct product in multi-product CVEs) and mismatch detection. When empty, the tool falls back to heuristic name matching.

The rpms repo is cloned automatically at startup (or provided via --rpms-repo). A CSV override can be passed via --package-map for backward compatibility.

Vendored Dependency Detection

Vendored dependency detection uses live SBOM lookups from the Hummingbird Pulp repository. When a CVE product mismatch is detected, the tool fetches the SPDX SBOM for the ticket’s package from packages.redhat.com/.../metadata/sboms/{package}-main/ (same public index used by the rpms CVE skill), selects the newest dated sha256-….sbom entry from that listing, and searches that document for the CVE product(s) only (via purl references), stopping on the first hit. The analysis output always records the SBOM artifact used and whether the lookup hit, missed, or was unavailable.

The same SBOM-first check also gates Component not Present closures for packages missing from the rpms repo. An SBOM hit alone no longer blocks that closure: binary confirmation must reach absent_in_binary_confirmed (no bundled Provide match and Syft miss on source-only SPDX evidence) before Not a Bug / Component not Present is recommended. A binary hit or unknown result keeps the ticket open for investigation.

Go subpackages are matched by progressively stripping path components (e.g., github.com/jackc/pgx/v5/pgproto3 matches github.com/jackc/pgx/v5). When a match is found and confirmed in binaries, the confirmed binary/Provide version is preferred for resolution over the SBOM lockfile version. The SBOM and binary confirmation are re-run each analysis cycle.

Requires the rpm and/or syft CLI on PATH (both are available in the analysis container image via rpm-build and the Syft install). Without either tool, binary confirmation returns unknown. With only rpm, a bundled Provide can still confirm present_in_binary, but absence is never auto-confirmed without Syft.

The generate_vendored_map_sbom.py script in package_maps/ can still be used for auditing vendored dependencies across all packages, but is no longer required at runtime.

Prerequisites

• Python 3.11 or later
• Jira API token (Bearer or Basic auth)
• Network access to redhat.atlassian.net, github.com, nvd.nist.gov, bodhi.fedoraproject.org, src.fedoraproject.org, and packages.redhat.com (note: github.com is only needed when --cve-repo is not provided and the tool must clone cvelistV5 itself)
• rpm and syft CLIs on PATH (for binary RPM confirmation after SBOM hits: bundled Provides via rpm, component inventory via Syft)
• GitHub API token (optional, for upstream PR search)
• GitLab API token (optional, for gitlab.com MR search)
• CEE GitLab API token (required with --resolve, for advisory repo)

Usage

# Basic usage with Jira token from environment
export JIRA_TOKEN=your_token
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com

# Analyze specific tickets
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com HUM-796 HUM-518

# Show only tickets from the last 2 weeks
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com --show-since "2 weeks"

# Analyze with upstream fix detection (report findings without modifying Jira)
export GITHUB_TOKEN=your_github_token
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com

# Resolve tickets and apply labels
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com --resolve

# Include closed tickets and check for stale closures
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com --include-closed

# Include closed tickets but exclude specific ones
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com \
  --include-closed --exclude "HUM-555,HUM-552"

# Skip issues without CVE links; write collector handoff
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com \
  --skip-no-cve --handoff-file /tmp/cve_analysis_handoff.json

# Continuous mode: resolve tickets every 30 minutes
python3 -m hummingbird_cve_analysis.cve_analysis --jira-user user@example.com \
  --resolve --time-between-runs 30

Dashboard /api/cve-report JSON is produced by collect_cve_dashboard (section 8.4), not by cve_analysis stdout (HUM-5800).

Inspecting lifecycle timing with dump_lifecycle

scripts/dump_lifecycle reads lifecycle milestones and package data from the dashboard HTTP API and prints them for one or more HUM tickets. CVE_REPORT_TOKEN is needed.

# Text output for one ticket against prod
scripts/dump_lifecycle --prod HUM-1234

# JSON output for multiple tickets against preprod
scripts/dump_lifecycle --preprod --json HUM-1234 HUM-5678

# Custom dashboard URL
scripts/dump_lifecycle --dashboard-url https://... HUM-1234

Output includes all milestone timestamps (cve_published, osidb_flaw_created, osidb_affect_created, hum_ticket_created, …), computed duration legs, and rpm_first_published (package) sourced from package_lifecycle. The --json flag emits the output as a JSON array instead of human-readable text.

Backfilling OSIDB timestamps with backfill_osidb_timestamps.py

scripts/backfill_osidb_timestamps.py is a one-time helper that fills osidb_flaw_created and osidb_affect_created for tickets already stored in cve_ticket_events. It reads existing rows from /api/cve-export, looks up timestamps from the public OSIDB API, and POSTs new rows to /api/cve-import with reset=false (write-once; existing timestamps are kept). Tickets that already have both OSIDB events are skipped unless --force is set. CVE_REPORT_TOKEN is needed.

# Backfill prod
scripts/backfill_osidb_timestamps.py --prod

# Dry run against preprod
scripts/backfill_osidb_timestamps.py --preprod --dry-run

# Limit to specific tickets
scripts/backfill_osidb_timestamps.py --prod HUM-1234 HUM-5678

Ongoing collection is handled by collect_cve_dashboard. After a prod backfill, copy_prod_to_preprod.sh copies the new event types with the rest of cve_ticket_events (they are on the dashboard import allowlist).

Collecting rpm_first_published with collect_rpm_first_published

scripts/collect_rpm_first_published sweeps the Hummingbird Pulp repo for all packages, finds the earliest SRPM upload timestamp for each, and writes the results to the package_lifecycle table via POST /api/cve-export - no direct DB access.

# Sweep all packages from the rpms repo and write to prod
scripts/collect_rpm_first_published --prod --rpms-repo /path/to/rpms

# Only process specific packages
scripts/collect_rpm_first_published --prod --rpms-repo /path/to/rpms --package curl wget

# Dry run — log what would be written without posting
scripts/collect_rpm_first_published --prod --rpms-repo /path/to/rpms --dry-run

Re-runs are safe: the dashboard upsert only updates occurred_at if the incoming timestamp is earlier than the stored one.

Analyzing CVE Response Delays with analyze_rpm_first_published

scripts/analyze_rpm_first_published inspects the relation between CVE publication, package onboarding (rpm_first_published), and HUM ticket creation:

# Analyze all tickets on prod
scripts/analyze_rpm_first_published --prod --token $CVE_REPORT_TOKEN

# Filter to new packages onboarded after CVE publication
scripts/analyze_rpm_first_published --prod --filter new-pkg

# Filter to pre-existing packages with at least 30 days actionable delay
scripts/analyze_rpm_first_published --prod --filter pre-existing --min-days 30 --sort actionable

CLI options:

• --filter {all,new-pkg,pre-existing}: filter by onboarding timing relative to CVE.
• --min-days N: minimum delay threshold in days (default: 0).
• --sort {cve_to_hum,actionable,cve_to_rpm}: sort column (default: actionable).
• --limit N: max rows to display (default: 50).

Configuration

Managed Labels

The tool manages the following labels on Jira tickets. These are automatically applied, upgraded, and cleaned up:

Legacy fedora-fix-available / fedora-fix-in-progress labels are no longer applied; the bot removes them on subsequent runs (Bodhi/DistGit evidence now uses upstream-fix-*).

Labels are upgraded automatically (e.g., upstream-fix-in-progress is replaced by upstream-fix-available when a fix is merged). Stale labels are removed when tickets are closed, except fedora-bz-filed which is preserved as an audit record and awaiting-vex which is managed by the VEX reconcile pass. The cve-needs-attention label is removed automatically when the warning condition no longer applies.

awaiting-vex (HUM-5843)

When a ticket is closed as Done-Errata or Not a Bug in production, analysis adds awaiting-vex. Each --resolve cycle then:

1. Queries Closed tickets labeled awaiting-vex, plus Closed Done-Errata / Not a Bug tickets resolved in the last 7 days (so human/agent closes that skipped the close-time enqueue still enter the queue)
2. Fetches Hummingbird status from the Red Hat CSAF VEX feed for every CVE ID in the ticket summary, scoped to that ticket’s package (not a CVE-wide worst-case across other Hummingbird products)
3. Compares to the Jira resolution for the awaiting-vex work queue (Done-Errata → fixed, Not a Bug → known_not_affected or package_not_listed when CSAF omits the package); match requires all CVE IDs to agree
4. On match: records vex_status + vex_resolved (scan time) in the collector handoff and removes the label if present
5. On pending/mismatch: adds awaiting-vex if missing, otherwise keeps it, and still emits current vex_status. Catch-up closes with no CVE ID in the summary are not labeled: MATCH is impossible, and adding the label would re-select the ticket forever.

The Closed tab computes MATCH from stored vex_status plus Jira resolution; it does not use stored vex_match_state as source of truth. Dashboard reads merged event metadata (later rows overlay earlier ones) so a ticket closed after cve_published does not keep leftover New / In Progress analysis text as Resolution (HUM-6091).

The collector also fetches any vex_updates[] keys that the watermark JQL missed, so a same-cycle catch-up match still lands on the Closed tab.

Won’t Do closures do not enqueue awaiting-vex. Open-ticket VEX mismatches remain a vex-checker reconcile concern.

cve-next-release

The cve-next-release label is used for CVEs where no immediate action is possible and the fix will arrive via the next upstream release. It is set manually. Common scenarios include:

• Vendored dependencies: A fix exists in a vendored package (e.g. ws inside dotnet) but cannot be consumed until the parent package updates its vendored copy.
• No backport path: The fix cannot be backported to the current release and must wait for a future upstream version.
• Upstream fix pending: A fix is expected upstream but has not landed yet, and no interim mitigation is available.

Package metadata fix_status: 0 (EOL) does not use this label; those tickets are closed as Won’t Do instead.

When this label is present on a ticket:

• The automation skips applying upstream-fix-available and upstream-fix-in-progress labels, since they would be misleading, and removes them if already present
• The SBOM version check continues each analysis cycle, so when a new upstream release containing the fix is consumed, the ticket is closed normally
• Fedora labels and cve-needs-attention are still managed normally
• The label is not removed automatically; it must be removed manually when no longer applicable

cve-next-release tickets are not closed just because Hummingbird shipped a new build. They close when analysis concludes the CVE is no longer affected; for vendored dependencies, that means the SBOM shows the fixed upstream version, not merely a new parent NVR.

cve-needs-attention conditions

The cve-needs-attention label is applied when any of these conditions are detected:

• Product mismatch: CVE vendor/product does not match the Hummingbird package (e.g. node-tar CVE filed against GNU tar) after an SBOM-first check finds no vendored dependency hit
• Package missing from rpms but present in SBOM/binaries: package directory is absent from the rpms repo, yet the CVE product appears in the package SBOM and binary confirmation is present or unknown
• Multiple products: CVE lists multiple distinct products with different versioning schemes and no cve_product override is configured (e.g. NGINX Open Source + NGINX Plus)
• Git-only version data: CVE version data uses commit hashes instead of numeric versions (e.g. libsodium with lessThan: ad3004ec...)
• CNA data error: A git commit hash is used where a version number is expected without setting versionType: "git", including hashes embedded in operator syntax (e.g., "version": "< 6374ae0bcdfe..."). The warning includes a link to the CVE 5.0 source control versions spec
• Malformed version field: The version field contains syntax that could not be parsed (e.g., compound operator+hash ranges like >= hash1, < hash2)
• CVE record not found: the CVE record file was not found in the cvelistV5 repo
• No repo version: the Hummingbird package was not found in the Pulp repository (package name mismatch or missing SRPM)
• Stale closure: ticket is Closed in Jira but current analysis recommends a different resolution

Human override workflow

When a human wants to take over a ticket from the bot:

1. Assign the ticket to yourself – the bot skips all automation on tickets not assigned to the bot account. Reassign to the bot to re-enable automation.

2. Set “Fixed in Build” – if you know the fix is in a specific SRPM, set the “Fixed in Build” field to the SRPM name (e.g. libarchive-3.8.7-1.hum1.src.rpm). The bot will use this to create the advisory MR and close the ticket, bypassing its own analysis.

Migration note: The cve-analysis-okay label is deprecated and no longer suppresses automation. Existing tickets with this label will be re-processed by the bot on its next run. To keep the bot from touching a specific ticket, reassign it to yourself before the next run. The cve-analysis-okay label can then be removed manually.

Output

Human-Readable

Project: HUM  Component: Security

  HUM-796  CVE-2026-2673 openssl: buffer overflow [hummingbird-1]
    Open since:       7 days (Mar 24 2026)
    Labels resolution: upstream-fix-available
    OpenSSL / OpenSSL:
      Affected versions: 3.5.0 < 3.5.6
      Fixed in:          3.5.6
      Hummingbird repo (latest):  3.5.5 / openssl-3.5.5-1.hum1.src.rpm
    Repo:             https://github.com/openssl/openssl
    CVE-2026-2673: upstream-fix-available (3 PRs)
      [CLOSED, 3 commits] Fix group tuple handling in DEFAULT expansion (3.5)
              https://github.com/openssl/openssl/pull/30110
    Fedora update: FEDORA-2026-abc123 openssl-3.5.6-1.fc44 (stable, security)
              https://bodhi.fedoraproject.org/updates/FEDORA-2026-abc123
    Jira current:     In Progress / (none)
    Jira resolution:  In Progress / affected (repo 3.5.5 is in affected range 3.5.0 < 3.5.6)

JSON

Each issue includes: key, summary, labels, open_since, cve_ids, cves (with version and resolution data), jira_current (status/resolution), computed_resolution, and upstream (with PR/MR search results, Bodhi updates, and Fedora version data).

Viewing Logs

The CVE analysis tool runs as a pod in the hummingbird--internal namespace on mpp-prod. See the OpenShift MP+ internal docs page for general log access instructions.

For CVE analysis specifically:

• Live: Open the mpp-prod pods list in the OpenShift console, filter for cve-analysis, and open the Logs tab.
• Grafana/Loki: Open the Hummingbird Grafana logs dashboard, select cluster mpp-prod and namespace hummingbird--internal, then filter for hummingbird-cve-analysis pods.

Library layout

Reusable library code lives under hummingbird_cve_analysis/lib/. The CLI entry point remains cve_analysis.py at the package root.

cve_analysis.py is the CLI entry point only (argument parsing, JQL building, signal handling, and the main orchestration loop). Library code lives under lib/; import and patch those modules directly. New cron jobs, webhooks, and other integrations should prefer ticket.process_ticket for analyzing (and optionally resolving) a single Jira issue:

from hummingbird_cve_analysis.lib import ticket

result = ticket.process_ticket(
    issue,
    pkg_map,
    catalog_source_map,
    github_token,
    gitlab_token,
    base_url=base_url,
    token=token,
    resolve=False,
)

Development

See the main README for development workflows.

make hummingbird-cve-analysis/setup  # Install dependencies
make check                            # Lint code (ruff)
make test                             # Run unit tests

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.