Hummingbird CVE agent

CLI that investigates open HUM Security tickets labeled cve-needs-attention or upstream-fix-available, using AI where deterministic analysis could not resolve the CVE. Fixes are handed to workqueue-service as mr work items.

Diagram

flowchart TB

  TICKET["HUM ticket key"] --> RUN["run HUM-XXXX"]
  SWEEP["sweep [--limit N] [--workers N]"] --> SEARCH["Paginated Jira search<br/>open HUM Security tickets"]
  SEARCH --> FILTER{"Either scope label?<br/>Not embargoed?<br/>No agent analysis JSON?"}
  FILTER -->|eligible| POOL["Worker ticket pool"]
  FILTER -->|skipped| REPORT["Sweep report JSON<br/>counts + outcomes"]

  subgraph AGENT["CVE Agent"]
      direction TB

      RUN --> COL["Collect facts<br/>Jira · Pulp · SBOM"] --> CAND{"Backport<br/>candidate?"}
      POOL --> COL

      subgraph SANDBOX["Ephemeral rpms sandbox · one clone per ticket"]
          direction TB
          AIINV["AI investigation<br/>read-only SBOM · file tools"]
          BACKPORT["Secret-free sparse clone<br/>patches → file_actions"]
      end

      CAND -->|"affected + fix, current"| BACKPORT
      CAND -->|"no · or stale"| AIINV
      AIINV -->|"needs_change + fix"| BACKPORT
      AIINV -->|verdict| RESULT["InvestigationResult"]
      BACKPORT --> RESULT
      RESULT --> OUT["Result JSON<br/>stdout · --output"]
      RESULT -->|"--post-comment"| COMMENT["Jira comment<br/>+ result attachment"]
      RESULT --> REPORT
  end

  MR["workqueue-service<br/>mr work item · draft RPM MR"]
  KONFLUX["Konflux build"]

  RESULT -->|"--submit-mr"| MR --> KONFLUX

  classDef agent fill:#dbeafe,stroke:#2563eb,color:#111827
  classDef sandbox fill:#dcfce7,stroke:#16a34a,color:#111827
  classDef external fill:#f3f4f6,stroke:#6b7280,color:#111827
  class RUN,SWEEP,SEARCH,FILTER,POOL,COL,CAND,RESULT,OUT,COMMENT,AIINV,REPORT agent
  class BACKPORT sandbox
  class TICKET,MR,KONFLUX external

The agent runs as a CLI and is stateless: run handles one ticket, while sweep pages through all eligible tickets and investigates them concurrently. By default it only prints or persists results locally; Jira comments and MR submissions are opt-in (see Writes).

Critical evidence network, download, or parse failures abort the investigation; their local diagnostic report leaves the ticket eligible for a later retry. Unsupported or ambiguous evidence becomes needs_human. The ticket-level state machine lives in the CVE Lifecycle & State Vocabulary.

Tl;Dr

• Scoped to CVE Security tickets - investigates open HUM Security tickets with either cve-needs-attention or upstream-fix-available; these are the deterministic-analysis residual and upstream-fix populations respectively. Sweeps skip embargoed tickets and tickets already carrying their <ticket>-cve-agent-analysis.json attachment.

• AI in one place - every LLM call goes through ai.AgentAiClient, which uses the shared hummingbird_ai.models library directly, so its token/cost metrics feed the shared GCP AI-costs dashboard. AI judges the fuzzy residual the deterministic path does not own; see What the model sees for the exact context. The model can use bounded read-only tools to search the SBOM, inspect package files, inspect checksum-pinned package sources, and look up exact npm release metadata. Each ticket makes at most five tool-enabled model calls plus one final-answer call, whose request carries no tool definitions so the model must conclude. SBOM searches return at most 20 components with binary-presence evidence (present, source_only, or unknown) from the shared SBOM classifier; a name/version match or source-only hit does not prove the component ships. Package file listings return at most 500 paths. Source archive searches scan at most 48 MiB and 50,000 members; a single file is capped at 8 MiB. Source reads, package-file reads, and upstream patch excerpts share a limit of 12,000 characters each and 48,000 total. npm metadata responses are capped at 16 MiB, and package lookups share a three-call budget. When limits are exhausted, the agent must return its best supported answer or needs_human. Tool names, result keys, safe counts/provenance, and errors are logged without logging source contents.

• Structured results - every investigation produces a validated, versioned result contract. The result status is a Verdict from the shared lifecycle vocabulary (hummingbird_cve_analysis/lib/states.py), so the agent and cve_analysis speak the same state language. See the CVE Lifecycle & State Vocabulary for the full state machine. Every result also carries the ticket’s collection records: each evidence step’s outcome (success, empty, partial, failure, or limitation), its retryability, and its provenance. The caller fills them; the model neither sees nor writes them.

• Deterministic backports - for an unambiguous single-CVE ticket whose analysis names upstream commits, the agent fetches those commits as patches and mechanically emits MR-compatible patch, spec, and metadata file_actions. No AI call is needed when the analysis asserted affectedness and still matches the collected package version and NVR; when it punted but still named a fix, AI judges affectedness and the same preparation attaches the file_actions. AI can also cite an exact upstream commit as structured evidence; deterministic validation still fetches the patch and generates all file actions, so the model never authors a patch. Vendored dependency fixes retain their upstream findings but usually require human adaptation; an AI-selected commit still must pass the same source preflight. PR-only analysis is resolved to its commits first, in apply order (GitHub returns PR commits oldest-first, GitLab newest-first). If a GitHub PR’s original patch does not apply, the agent checks for one merged backport PR linked in the title and targeting the shipped major.minor release line. Its complete PR diff must pass the same source preflight before file_actions are emitted; --submit-mr then submits those actions as a draft MR work item. If GitHub rate-limits the optional lookup, the original manual-backport result is retained. When source preflight cannot verify a backport for a fast-path ticket, the result is needs_human and has no file_actions. Two rpms-repo conventions are applied:

  • Patch files are named <NNNNN>-<cve-id>-<slug>.patch, where the slug is up to five words taken from the Jira summary’s description (for example 00003-cve-2026-82474-policy-bypass-allows-unauthorized-program.patch). When one CVE needs several commits, the patch number keeps the names unique. Tickets with no usable summary fall back to the short commit ID.
  • Release gets a trailing micro bump rather than an increment of the base release, which stays owned by rebases: 4 becomes 4.1, 4.1 becomes 4.2, and 16.p2%{?dist} becomes 16.p2.1%{?dist}. Date-shaped releases are refused and left for human review.
  • Clean Fedora-imported packages are marked modified, with the ticket/CVE appended to any existing modification_reason. Independent packages stay independent and omit modification_reason; other metadata, including release, is preserved.

  Branch-tagged upstream commits must match the shipped release line, even when the analysis names only one patch. Mismatches require human review.

• Ephemeral rpms sandbox - one throwaway, uncredentialed clone per ticket, shared by the AI’s read-only tools and deterministic backports, and opened only when needed. Simple %autosetup specs using patch or git_am and either PatchN: declarations or a literal %patchlist, optionally preceded by %{gpgverify} or %gpgverify -kN -sN -dN and followed by build-input commands, are checked against checksum-pinned source with GNU patch at zero fuzz or Git’s patch engine. Simple %setup specs with unconditional %patch N -p1 or %patch -P N -p1 commands applying every declared patch in numeric order are also supported, including legacy patch headers with backup filename suffixes. Setup -b N and -a N options load checksum-pinned secondary archives before or after Source0, preserving their locations relative to the build tree. Source is fetched from the rpms repo’s lookaside or the original dist-git’s; upstream monorepo patch paths must match the packaged source. A single shared upstream directory prefix is stripped only when every stripped path is present in the pinned tarball and none of the original paths are; the adapted patch still must pass source preflight. Unsupported %autosetup errors show the offending line. Full patch-stack replay rejects conditional patch declarations; the existing non-overlapping direct-preflight bypass remains available for supported conditional or filtered %autopatch preparation.

Prerequisites

• Python 3.11+
• GNU patch for automatic backports (gpatch on macOS); unsupported or unavailable patch tools leave the ticket for human review.
• Git for %autosetup -S git_am backports.
• GITHUB_TOKEN for GitHub backport lookup when unauthenticated API requests hit GitHub’s rate limit.
• Jira API credentials (JIRA_TOKEN, and JIRA_USER where basic auth applies)
• Vertex AI access for the AI path: GOOGLE_CLOUD_PROJECT, or GOOGLE_API_KEY for the direct Gemini API. The deterministic fast path needs neither.
• For --submit-mr only: WORKQUEUE_SERVICE_URL and a WORKQUEUE_SERVICE_TOKEN bearing submit permission on the mr work type. submit comes from membership in the konflux-hummingbird-admin-access group. Check with curl -H "Authorization: Bearer $WORKQUEUE_SERVICE_TOKEN" "$WORKQUEUE_SERVICE_URL/api/whoami".

Installation

make hummingbird-cve-agent/setup

The target installs the in-repo dependencies (hummingbird-ai, hummingbird-cve-analysis) editable alongside this package. Equivalent by hand:

cd hummingbird-cve-agent
pip install -e ../hummingbird-ai -e ../hummingbird-cve-analysis -e ".[dev]"

For the environment, copy hummingbird-cve-agent/.env.example to .env and source it for a quick way to set up all necessary variables. If the workqueue endpoint uses an internal CA, export REQUESTS_CA_BUNDLE in the same shell as --submit-mr. Its PEM file must contain both the public roots (for Google OAuth) and the internal root (for the workqueue endpoint).

Usage

hummingbird-cve-agent run HUM-1234
hummingbird-cve-agent run HUM-1234 --output results/HUM-1234.json
hummingbird-cve-agent run HUM-1234 --post-comment
hummingbird-cve-agent run HUM-1234 --submit-mr
hummingbird-cve-agent sweep
hummingbird-cve-agent sweep --limit 25 --workers 6 --output sweep.json
hummingbird-cve-agent sweep --post-comment --submit-mr

This gathers facts for the ticket, runs the investigation, and prints the result as JSON or persists it to --output. Eligible backports include base64-encoded file_actions and a commit_message ready for an mr work item; everything else is judged by the AI client. A backport that cannot be prepared is not retried by AI — a patch that fails preflight needs human adaptation before submission. Failing to prepare one does not weaken the verdict: the finding and its confidence stand, the summary records why the automation stopped, and a needs_change result without file_actions already reads as a human-owned backport.

CLI logs include ticket and package context, e.g. - INFO (HUM-1234 openssl): collected facts for HUM-1234; the package is - until known, and unscoped logs show - -.

sweep defaults to all eligible tickets and four concurrent workers. --limit caps attempted investigations, including failed attempts; analyzed, embargoed, and out-of-scope search results do not consume the limit. The aggregate JSON report lists scan/skip/attempt/success/failure counts and each attempted ticket’s result or error. The command exits nonzero if any ticket fails, while continuing other investigations. Without --post-comment, the sweep does not add the attachment used to mark a ticket as investigated, so a later sweep can retry it. Critical collection failures never post a recommendation or analysis attachment, even when the write flags are enabled. The CLI and sweep retain bounded failure records locally; transient failures are marked retryable and are not cached.

Writes: Jira comment and MR submission

Both writes are off by default; each is enabled solely by its own flag, so a bare run never touches Jira or GitLab. The two flags are independent — a comment can be posted without submitting an MR and vice versa — and a failed write exits non-zero.

• --post-comment posts a deterministic RECOMMENDATION: ... comment rendered from the validated InvestigationResult — no AI call at comment time — through cve_analysis’s jira_client, which handles ADF formatting and duplicate suppression, so re-running a command is idempotent. The full result JSON is attached to the ticket as <TICKET>-cve-agent-analysis.json: the comment carries the verdict, summary, cited evidence, and the next review step when human review is needed, including when a necessary package fix has no automated backport. The attachment carries the full evidence, file actions and commit message it was derived from. The attachment is skipped alongside a duplicate comment, so an unchanged re-run adds nothing.
• --submit-mr submits a draft MR work item to a running workqueue-service instance (work items default to auto_merge=false); Konflux performs the package build verification. It is a no-op unless the result carries a prepared backport (file_actions + commit_message). Submission POSTs to WORKQUEUE_SERVICE_URL with the WORKQUEUE_SERVICE_TOKEN service-account token; the agent is headless, so there is no interactive login fallback. The submitted work item’s description includes the package, CVE, built NVR when available, selected upstream commits, investigation reasoning, and pending Konflux and human checks. Currently, workqueue-service only applies this description when updating an existing MR.

Eval harness

eval scores investigations against completed human /cve sessions. The eval set, eval-tickets.txt, holds Jira ticket keys and nothing else — ticket content is internal data and is never committed — so facts are fetched live and running the eval needs Jira access.

hummingbird-cve-agent eval eval-tickets.txt --output eval-report.json

Read-only: every Jira request is a GET (issue, comments, changelog); test_evaluate asserts it.

A resolved ticket gives the answer away — fixed_in_build set, labels moved to awaiting-vex, status Closed, newest analysis comment post-fix — so each ticket is rewound through its changelog, dropping later comments.

The rewind cutoff is the handoff, not the close: cve_analysis records Fixed in Build and posts its assessment before it transitions the ticket, so the cutoff is the earlier of the resolution and the first recorded fix build. A reopened ticket rewinds to its first build, which is deliberately conservative — rewinding too far only costs context, rewinding too little hands the agent the conclusion. The fix build is additionally blanked outright, so a build recorded outside the changelog cannot leak either.

What is scored

Two axes, because one is not enough:

• Disposition — where each side says the ticket ends up. The verdict is mapped to its eventual disposition, not through states.VERDICT_OUTCOME: that shared table answers “what does the agent do next”, so needs_change is In Progress — but a ticket the human backported still ends Done-Errata, and scoring against the immediate state would mark a correct needs_change as a miss and reward a wrong already_fixed.
• Route — which branch the orchestrator would take, since already_fixed (Fixed in Build) and needs_change (MR creation) share the Done-Errata ending and are indistinguishable on the first axis. The human’s branch is derived from Pulp upload times: a fix build published after the handoff means the package had to change (needs_change), an earlier one means the shipped package already carried the fix (already_fixed); a Not a Bug closure is not_affected. A build Pulp no longer lists is reported as n/a and excluded from the route denominator rather than guessed.

Verdicts that reach no conclusion (needs_human, error) and conclusions no verdict can produce (unknown resolutions) never match. Package EOL Won't Do closures are deterministic cve_analysis outcomes and excluded before the AI eval runs. A ticket that cannot be rewound or investigated scores error without aborting the run.

HUM-7554     agree     verdict=already_fixed  expected=Closed / Done-Errata  actual=Closed / Done-Errata  route=!=needs_change
HUM-7537     DISAGREE  verdict=needs_human    expected=Closed / Won't Do     actual=                      route=n/a
disposition agreement: 1/2 (50%)
route agreement: 0/1 (0%), 1 undetermined
note: package facts (SRPM, SBOM, rpms repo) are current state, not the state at decision time, so agreement is retrospective

Retrospective, not historical replay

Only the Jira side is rewound. Package facts — the latest SRPM, the latest SBOM, and the rpms repo the sandbox clones — are read live, so an old ticket is re-investigated against today’s package state rather than the state that existed when the human decided. Where the fix has since shipped, already_fixed is partly handed to the agent, so the score is retrospective current-state agreement, not a historical replay; every report prints that caveat. True replay needs as-of-cutoff SRPM and SBOM selection plus a date-pinned rpms checkout, and is deliberately left to a follow-up.

Configuration

Operational settings come from the YAML config file; secrets and model credentials come from the environment.

YAML settings:

Neither forge token grants write access: they only authenticate the read-only API calls that resolve a pull/merge request to its commits, and are needed only for private or rate-limited repositories. The sandbox never holds a push token.

What the model sees

Only non-candidates reach the model.

The prompt is a static system instruction encoding the /cve resolution process, plus the validated TicketFacts as JSON: ticket key, package, CVEs, Jira summary/description/labels, status/resolution/fixed_in_build, cve_analysis_block, discussion, and the built NVR and SBOM identity.

• The SBOM document is excluded. Only its identity (sbom_nvr, sbom_url) is in the prompt; the multi-megabyte SPDX body is reachable only through the bounded search_sbom tool. Binary-presence evidence comes from SPDX sourceInfo and CONTAINS relationships to RPM roots; unknown requires further investigation. RPM repo reads inspect packaging files and patches, not necessarily the built upstream source or installed binary payload.
• Package-directory search. A bounded literal search_package_files grep locates evidence spread across packaging files — whether a build config enables the vulnerable applet, which patch names a CVE — without spending the per-file read budget. An empty result is not proof of absence.
• Other-package evidence. lookup_shipped_package resolves a candidate name to its latest source-package SBOM and reports the NVR, component version, and binary presence (present, source_only, or unknown). Only present confirms the named component ships; source-only or missing SBOM data remains inconclusive. Lookups are capped per ticket.
• Package source and npm evidence. Source reads/searches use the checksum-pinned archives declared in the current package spec, replaying supported downstream patches without running spec scriptlets. Results report the spec version/release, rpms-repo revision, archive filename/checksum, and ticket NVR, with built_nvr_match: unverified. This is evidence about the inspected packaging revision; it does not prove that revision built the ticket NVR unless the match is established. Source inspection refuses conditional patch declarations and patch_direct preparation rather than guessing which patches RPM applies. read_package_source and read_package_file accept optional start_line (1-based) and max_lines arguments and report start_line, end_line, and total_lines. An empty window has end_line: null. The existing per-read and shared character budgets still apply; truncated indicates character clipping within the requested window. SBOM search includes SPDX package URLs, including scoped npm identifiers. Matches with multiple versions of the same package name report binary presence as unknown, including PURL matches, to avoid mixing evidence across versions. PURL-based classification requires its embedded version to match the package’s versionInfo; missing or mismatched PURL versions use the name-based path. Positive PURL evidence reporting a different version is treated as unknown. Full npm registry metadata can provide exact-version publication time, gitHead, repository, and integrity, but neither a version date nor metadata alone proves that the dependency ships or contains a fix.
• Known fix against a release archive. check_upstream_fix fetches a known commit patch from a supported HTTPS forge (2 MiB maximum, redirects disabled) and checks it against the prepared, spec-pinned source after downstream patches. The result records source provenance and a patch_diff excerpt, capped at 12,000 characters and charged against the shared 48,000-character source/file read budget. patch_truncated identifies an incomplete excerpt. An exhausted budget returns an error before source preparation or patch fetching and does not consume the fix-check attempt. Tool logs report patch applicability and affected-path counts without logging the diff content. A patch that applies shows that exact diff is absent from the inspected source snapshot, not that the vulnerable code is reachable or that the snapshot built the ticket NVR. A patch that does not apply is inconclusive. Failed source checks, an absent CVE-named patch, %files entries, and missing disable flags alone do not establish affectedness, shipped presence, or fix absence; unresolved cases require needs_human.
• Upstream source probe. Once per ticket, the model may inspect an official upstream HTTPS Git repository. It searches shallow, blobless, tagless history and advisory/security metadata for the CVE or a supplied fix commit. When an exact base commit is known, it also checks fix ancestry and runs git apply --check; without one it reports candidates and changed paths only. A candidate with resolved: false could not be fetched and is not evidence. Other candidate results are retained as partial evidence; network failures abort the run while preserving that partial result locally. If base_commit_resolved is false, ancestry and applicability are unknown. Short SHAs that Git cannot fetch are reported unresolved rather than expanded through a forge API. GitHub and GitLab web repository URLs are normalized to .git clone endpoints; HTTP redirects remain disabled. An applicable or non-applicable patch alone does not establish affectedness or fix inclusion. No candidate in this bounded search requires needs_human, not no_upstream_fix. Collection failures abort the run; unresolved refs remain partial evidence. The Job running the agent must enforce CPU, memory, and ephemeral-storage limits for this external fetch.
• Non-retryable tool results. A result with retryable: false indicates a cached capability failure, consumed one-shot probe, or exhausted per-ticket budget. Repeating the call will not change it; the model records the evidence gap and moves on.
• The model cannot author a patch. The response schema is InvestigationResult minus CALLER_FILLED, so it returns only status, confidence, summary, evidence and vex_justification; file_actions, commit_message, ticket, package and cves are caller-owned and rejected if set. It can cite an upstream commit as evidence — never write one.

Security

The agent processes untrusted input — Jira ticket content, upstream commits, and package spec/patch files — so the sandbox is built so that no untrusted code is ever executed and untrusted input cannot reach secrets or a shared host.

• No untrusted code execution. The sandbox clones the rpms repo, reads files, resolves upstream PRs, downloads patches, and checks them with GNU patch or Git’s patch engine before staging. It never runs rpmbuild, a spec’s %prep scriptlet, or any code supplied by a spec or patch. Verifying the fix by building the package — the step that would execute spec-controlled code — is deliberately delegated to Konflux, which builds the resulting MR in its own hardened pipeline. This keeps a whole class of remote-code-execution surface (and slow, resource-heavy builds of large packages) out of the agent entirely.
• Deployment isolation. The agent ships as a container image and is meant to run as a Kubernetes Job or inside workqueue-service; it is not deployed yet. The rpms fork lives in a private temp dir inside that boundary and is removed on exit, so git operations run inside the deployment sandbox rather than on a shared host — no separate nested container is needed, because nothing here executes untrusted code.
• Secret-free subprocesses. git runs with a minimal, allowlisted environment. The service process env (Jira and model tokens) is never inherited by child processes, so a subprocess cannot read or exfiltrate them.
• Uncredentialed, hardened git. The fork is a public, read-only clone with no push token. Credential helpers and interactive prompts are disabled, system/global git config is neutralized (an attacker-controlled GIT_CONFIG_GLOBAL cannot change behaviour), and submodule recursion is off.
• Input validation. Package names are restricted to a safe charset (no path traversal or absolute paths), file reads are confined to the package directory, saved patch names must be *.patch and cannot overwrite existing files, and patch URLs must use HTTPS on a recognized upstream forge.

Development

See the main README for development workflows.

make check                         # lint
make test                          # run tests

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.