Hummingbird CVE agent

CLI and SQS worker that investigate open HUM Security tickets labeled cve-needs-attention or upstream-fix-available, using AI where deterministic analysis could not resolve the CVE. Fixes are handed to workqueue-service as mr work items.

System overview

flowchart TB

  TICKET["run HUM-XXXX"] --> COL
  SNS["Successful cve_analysis --resolve<br/>SNS → SQS wake-up"] --> SWEEP["Worker: fresh Jira sweep"]
  CLI["sweep [--limit N] [--workers N]"] --> SEARCH
  SWEEP --> SEARCH["Open HUM Security tickets<br/>in-scope labels · no cve-next-release<br/>not embargoed · no prev analysis"]
  SEARCH --> PREFLIGHT["Start and preflight all worker backends<br/>before ticket collection · skip if no tickets"]
  PREFLIGHT --> THREAD

  subgraph WORKER["Trusted worker · network and credentials"]
      direction TB
      THREAD["Sweep worker thread"] --> COL
      COL["Collect facts<br/>Jira · Pulp · SBOM"] --> CAND{"Deterministic candidate?<br/>affected + backport + fix label/reference<br/>not stale"}
      CAND -->|yes| ID{"Exact current version<br/>and SRPM NVR identified?"}
      ID -->|no| HUMAN["needs_human · no actions"]
      CAND -->|"no / stale"| AIINV["AI investigation<br/>bounded read-only tools"]
      ID -->|yes| FETCH
      AIINV -->|"source tools / needs_change + fix"| FETCH["Fetch pinned inputs<br/>hardened no-checkout Git transport<br/>checksummed sources · upstream patches"]
      AIINV -->|verdict| RESULT["InvestigationResult"]
      HUMAN --> RESULT
      EVIDENCE["Bounded evidence + verified file_actions"] --> RESULT
      RESULT --> OUT["Write result JSON<br/>stdout / --output before remote writes"]
      OUT --> SUBMIT{"--submit-mr<br/>and prepared actions?"}
       SUBMIT -->|yes| MR["Submit draft mr work item<br/>to workqueue-service"]
      SUBMIT -->|no| POST
      MR --> POST{"--post-to-jira?"}
       POST -->|yes| COMMENT["Post Jira comment → verdict label<br/>→ analysis attachment"]
  end

  subgraph SANDBOX["Offline execution · no network or credentials"]
      direction TB
      BACKEND["WorkerSandbox owns backend + package tools<br/>one per worker thread"] --> FILES["per-ticket WORK workspace<br/>reset state/files between tickets"]
      FILES --> PREP["RPM %prep<br/>existing downstream patches"]
      PREP --> INSPECT["Source inspection<br/>+ disposable candidate checks"]
       INSPECT -->|"backport path"| STAGE["Patch staging + spec edits"]
      STAGE --> REPLAY["Final-spec replay from clean inputs"]
  end
  THREAD -->|owns| BACKEND
  FETCH -->|"transfer each input once"| FILES
  INSPECT -->|"AI tools: bounded evidence"| AIINV
  REPLAY --> EVIDENCE
  MR --> REVIEW["Konflux build + human review"]

  classDef agent fill:#dbeafe,stroke:#2563eb,color:#111827
  classDef sandbox fill:#dcfce7,stroke:#16a34a,color:#111827
  class THREAD,COL,CAND,ID,HUMAN,AIINV,FETCH,EVIDENCE,RESULT,OUT,SUBMIT,POST,COMMENT agent
  class BACKEND,FILES,PREP,INSPECT,STAGE,REPLAY sandbox

The agent runs as a CLI and is stateless: run handles one ticket, while sweep pages through all eligible tickets and investigates them concurrently. The deployed worker consumes sweep requests from SQS and invokes that same bounded sweep; the event is only a wake-up signal, not a ticket snapshot. CLI writes are opt-in; the deployed worker enables both Jira posting and MR submission (see Writes).

Critical evidence network, download, or parse failures abort the investigation; their local diagnostic report leaves the ticket eligible for a later retry. Unsupported or ambiguous evidence becomes needs_human. The ticket-level state machine lives in the CVE Lifecycle & State Vocabulary.

Tl;Dr

  • Scoped to CVE Security tickets - investigates open HUM Security tickets with either cve-needs-attention or upstream-fix-available; these are the deterministic-analysis residual and upstream-fix populations respectively. Sweeps skip cve-next-release, embargoed tickets, and tickets already carrying their <ticket>-cve-agent-analysis.json attachment.
  • Backporting - verifies upstream patches against pinned package source and prepares RPM changes; see Backporting.
  • AI in one place - every LLM call goes through ai.AgentAiClient, which uses the shared hummingbird_ai.models library directly, so its token/cost metrics feed the shared GCP AI-costs dashboard. AI judges the fuzzy residual the deterministic path does not own; see What the model sees for the exact context. The model can use bounded read-only tools to search the SBOM, inspect package files, inspect checksum-pinned package sources, and look up exact npm release metadata. Each ticket has at most five tool-enabled turns plus one final-answer turn, whose request carries no tool definitions and explicitly asks for a JSON verdict using the available evidence. Invalid final replies receive a corrective request containing the result schema within the existing bounded retry budget. Exhausted retries fail the ticket without posting a result; no verdict is inferred from malformed output. Diagnostics log response length and finish reason, not response contents. Retries can make multiple model requests within a turn; six turns is not a six-request cap. SBOM searches return at most 20 components with binary-presence evidence (present, source_only, or unknown) from the shared SBOM classifier; a name/version match or source-only hit does not prove the component ships. When a matching SBOM result is unknown, the agent can scan published RPMs for that component. Pulp listings and SBOM downloads are limited to 64 MiB each.
  • Structured results - every investigation produces a validated result contract (there is no schema-version field). The result status is a Verdict from the shared lifecycle vocabulary (hummingbird_cve_analysis/lib/states.py), so the agent and cve_analysis speak the same state language. See the CVE Lifecycle & State Vocabulary for the full state machine. Every result also carries the ticket’s collection records: each evidence step’s outcome (success, empty, partial, failure, or limitation), its retryability, and its provenance. Initial collection records can appear in the model’s input facts; the caller owns the final collection records, including later tool outcomes.

Prerequisites

  • Python 3.11+
  • Podman with a running Linux VM for local source preparation, or Kubernetes sandbox access in deployment.
  • A sandbox image with RPM build tools, Python RPM bindings, python3-specfile, Git, GNU patch, macros/helpers… required by the package.
  • GITHUB_TOKEN for GitHub backport lookup when unauthenticated API requests hit GitHub’s rate limit.
  • Jira API credentials (JIRA_TOKEN, and JIRA_USER where basic auth applies)
  • Vertex AI access for the AI path: GOOGLE_CLOUD_PROJECT, or GOOGLE_API_KEY for the direct Gemini API. The deterministic fast path needs neither.
  • For --submit-mr only: WORKQUEUE_SERVICE_URL and either WORKQUEUE_SERVICE_TOKEN_FILE or WORKQUEUE_SERVICE_TOKEN, with submit permission on the mr work type. The deployed worker uses a projected token file; local development can use the environment token described in Writes: Jira comment and MR submission.

Installation

make hummingbird-cve-agent/setup

The target installs hummingbird-agent, hummingbird-ai, and hummingbird-cve-analysis editable alongside this package. Equivalent by hand:

cd hummingbird-cve-agent
pip install -e ../hummingbird-ai -e ../hummingbird-agent -e ../hummingbird-cve-analysis -e ".[dev]"

For the environment, copy hummingbird-cve-agent/.env.example to .env and source it for a quick way to set up all necessary variables. For local CA setup, activate the Python environment used to run the CLI before sourcing .env, so python -m certifi resolves that environment’s public CA bundle. If the workqueue endpoint uses an internal CA, export REQUESTS_CA_BUNDLE in the same shell as --submit-mr. Its PEM file must contain both the public roots (for Google OAuth) and the internal root (for the workqueue endpoint).

Configuration

Operational settings come from the YAML config file; secrets and model credentials come from the environment.

Variable Purpose
HUMMINGBIRD_CVE_AGENT_CONFIG Path to a YAML config file (optional)
JIRA_TOKEN Jira API token
JIRA_USER Jira basic-auth user (optional)
SENTRY_DSN Sentry error reporting for the deployed worker
GITHUB_TOKEN Read-only GitHub token for PR commit lookups (optional)
GITLAB_TOKEN Read-only GitLab.com token for MR commit lookups (optional)
GOOGLE_API_KEY Gemini API key (direct API mode)
GOOGLE_CLOUD_PROJECT GCP project for Vertex AI mode
WORKQUEUE_SERVICE_URL Base URL of a running workqueue-service instance (for --submit-mr)
WORKQUEUE_SERVICE_TOKEN_FILE Path to projected bearer token (preferred; re-read on each submission)
WORKQUEUE_SERVICE_TOKEN Bearer token fallback for local --submit-mr use
SQS_QUEUE_URL Dedicated queue consumed by the deployed worker command

YAML settings:

Key Default Purpose
jira_url Red Hat Jira Jira base URL
sandbox_type podman Sandbox backend; deployments use k8s
sandbox Published, digest-pinned CI image Override image locally to use the CVE RPM-preparation image
model claude-sonnet-4-6 Model name passed to build_model
model_regions Gemini and Claude region prefixes Model-name prefix to Vertex region

Neither forge token grants write access: they only authenticate the read-only API calls that resolve a pull/merge request to its commits, and are needed only for private or rate-limited repositories. The sandbox never holds a push token.

Usage

hummingbird-cve-agent run HUM-1234
hummingbird-cve-agent run HUM-1234 --output results/HUM-1234.json
hummingbird-cve-agent run HUM-1234 --post-to-jira
hummingbird-cve-agent run HUM-1234 --submit-mr
hummingbird-cve-agent sweep
hummingbird-cve-agent sweep --limit 25 --workers 6 --output sweep.json
hummingbird-cve-agent sweep --post-to-jira --submit-mr

This gathers facts for the ticket, runs the investigation, and prints the result as JSON or persists it to --output. See Backporting for how upstream fixes become package changes and what happens when preparation fails.

CLI logs include ticket and package context, e.g. - INFO (HUM-1234 openssl): collected facts for HUM-1234; the package is - until known, and unscoped logs show - -.

sweep defaults to all eligible tickets and two concurrent workers. --limit caps attempted investigations, including failed attempts; analyzed, embargoed, and out-of-scope search results do not consume the limit. A sweep searches once and then investigates for a long time, so a ticket can also lose scope after it was selected, by human re-triage to cve-next-release or by an overlapping sweep closing it. That is reported as out of scope, not as a failure: it does not fail the run or change the exit code. Standard output shows counts, verdicts, token usage, estimated cost, and compact per-ticket summaries. --output FILE also saves the complete JSON report. The command exits nonzero if any ticket fails, while continuing after handled ticket-level failures. A source-initialization TypeError propagates out of investigate, so that ticket produces no investigation-result JSON and makes no Jira/MR writes. The sweep includes TypeError in its existing ticket-error handling, records the failed ticket in its report, and continues other tickets normally; this is not a catch-all for unexpected exceptions. Without --post-to-jira, the sweep does not add the attachment used to mark a ticket as investigated, so a later sweep can retry it. Critical collection failures never post a recommendation or analysis attachment, even when the write flags are enabled. The CLI and sweep retain bounded failure records locally; transient failures are marked retryable and are not cached.

After finding eligible tickets, the sweep starts and preflights all min(workers, eligible tickets) execution backends before any ticket collection, investigation, or model calls. No eligible tickets means no backends. A thread pool runs that many worker loops consuming a ticket queue, each with its own backend; it does not create a backend per ticket or package. Preflight failure aborts the sweep before any ticket starts or remote writes occur and cleans up started backends. Backends are cleaned up when the sweep exits; a handled ticket failure affects only that ticket, not other workers. Transport/collection failures discard its backend so timed-out child processes cannot reach the next ticket; the next ticket starts a fresh backend. Other handled failures reset ticket state and workspace for reuse.

Backporting

The agent separates deciding that a package needs a fix from preparing the RPM changes. AI may judge affectedness or identify an upstream commit; the backporting code fetches and verifies the upstream patch, then prepares RPM changes. It does not rewrite or generate patch content (yet, see Planned work).

flowchart TD
    C{"Affected + backport + fix reference<br/>+ upstream-fix-available?"} -->|Yes| S{"Analysis stale?"}
    C -->|No| I["AI investigates"]
    S -->|Yes| I
    S -->|No| ID{"Exact current version<br/>and SRPM NVR identified?"}
    ID -->|No| H["needs_human · no file_actions"]
    ID -->|Yes| U["Resolve fix and select release patches"]
    I -->|Needs change + fix| U
    I -->|Otherwise| R["Return investigation result"]
    U --> V["Per-ticket workspace on worker-owned backend<br/>RPM prep → disposable candidate checks"]
    V --> W["Stage patches · edit spec/metadata<br/>replay final spec from clean inputs"]
    V -->|Unsupported / rejected| F["Deterministic: needs_human<br/>AI: retain finding/confidence<br/>both: no file_actions"]
    W -->|Unsupported / rejected| F
    V -->|Critical collection failure| A["Abort · local diagnostics<br/>no remote writes"]
    W -->|Verified| R
    F --> R
    H --> R
    R --> J["Agent's result processing"]

1. Identify and select the upstream fix

The fast path skips AI only when the existing CVE analysis says the package is affected, names a backport as the next step, the ticket has the upstream-fix-available label, and the analysis still matches the package version collected now and has not already reached a recorded fixed version on its release line. Stale analysis goes to AI. A candidate that is not stale but does not explicitly identify both the current version and exact SRPM NVR returns needs_human, not an AI fallback. A needs_change result with a usable fix reference enters the same preparation path; otherwise, the agent returns the investigation result only.

Commit URLs are fetched directly. A merged PR URL is first resolved to its commits in application order. Branch-tagged patches must target the release line of the shipped package; a mismatch requires human review. Mixed tagged and untagged commits are rejected as ambiguous because untagged commits might be prerequisites rather than alternative backports. Supply an explicit release-line series before retrying. Commit IDs are compared in full; abbreviated and full references are fetched separately. After selecting the line, only identical diff bodies are removed, retaining hunk positions, context, and file metadata in the comparison so distinct fixes are not silently discarded.

2. Prepare and verify against package source

Each sweep worker thread owns one WorkerSandbox, which owns the execution backend and package tools: a Podman container locally or an ephemeral Kubernetes pod in deployment. Its .ticket(package) scope resets per-ticket state and files in the WORK workspace. Sequential tickets reuse the physical container/pod, but each gets fresh Preparation state, checksum pins, caches, and candidates; none carry over to the next ticket. This avoids startup per investigation without sharing ticket state. .open() lazily imports the package into the existing worker backend. Standalone run uses its own WorkerSandbox and may avoid backend startup entirely for an SBOM-only investigation; any backend it starts is cleaned up when the run ends.

Within a ticket, the offline package workspace is shared by file/source inspection, disposable candidate checks, patch staging, spec edits, and final replay. Preparation performs RPM operations on that backend; it does not own the backend or its lifecycle. Verified base trees and the disposable candidate copy are released before the edited-spec replay, so a ticket never holds both tree sets at once. RPM macro expansion and rpmbuild -bp --nodeps run only inside that sandbox. RPM owns architecture conditions, source selection, Go macros, patch ordering, archive extraction, and shell commands; there is no handwritten prep interpreter or static fallback.

The worker has no package checkout. A temporary, hardened no-checkout Git object transport pins one repository commit and exports only rpms/<package>/, metadata/<package>.json and mock/dist-git-client.ini from that same commit. The worker brokers source/lookaside downloads, validates checksums, and fetches upstream patches; the selected inputs are validated and then transferred once as a single archive stream, preserving executable package scripts. Every RPM preparation uses fresh sandbox-local copies of those inputs, so a script changing its sources cannot contaminate the next architecture or final replay. Downloads are limited to 128 MiB each, with a shared 256 MiB input budget. The execution sandbox has no network or worker credentials, so missing inputs, macros, signatures, or preparation helpers block preparation instead of triggering dependency installation or guessed behavior.

The worker retains immutable metadata, checksum pins, and lookaside routing facts from that Git snapshot. It never uses sandbox-modified copies to choose download destinations or checksums; these acquisition facts are not a second working tree.

Inspection reads and searches the real prepared tree, cached per architecture. Reads reject escaping paths and symlinks and are limited to 8 MiB per file; searches are limited to 48 MiB and 50,000 filesystem entries. Commands have timeouts, bounded diagnostics, and a 256 MiB individual-file limit. Podman keeps its original shared startup behavior and writable filesystem. CPU, memory, storage, process-count, and container-lifetime policy belong to deployment infrastructure; the CVE agent does not impose them on shared Podman users. Infrastructure enforcement is deferred, so these application-level checks do not provide a whole-sandbox resource bound for local runs.

Candidates are checked on disposable copies using GNU patch with fuzz zero. Before each candidate is applied, a reverse dry-run on the clean copy separates source that already carries the fix from source that needs a rebased patch: the first blocks backporting as CVE already applied, the second as a patch that does not apply. Mismatched context is rejected without retrying with greater fuzz. Exact-context hunks that land at a line offset still pass, but they are recorded and reported to the MR reviewer. Original source paths take precedence. When they are missing, deterministic adaptation can remove a nested monorepo prefix, add a downstream prefix such as src/ or vendor/<package>/, or replace one shared directory prefix with another. It preserves relative paths and requires a unique mapping supported by every initially existing file touched by the ordered series. Added files must not overwrite existing destinations. A bounded search ignores symlink directories; ambiguous or incomplete searches are rejected, and addition-only patches do not guess a target directory. Only diff headers change, never source hunks or commit messages; individual file renames and source-level rewrites are not inferred. Automatic backporting must use the same adapted patch on every applicable supported target (x86_64, aarch64, and s390x). Source tools accept an explicit arch; when multiple targets exist, they require it rather than guessing which published binary the model is investigating.

The installed specfile library performs structural edits. The edited spec is then prepared again from clean inputs, using RPM’s own patch backend and options. RPM’s default patch fuzz is explicitly set to zero for every preparation. Every candidate-touched file must match its verified final content, including deletions; a later source overlay cannot silently erase the fix. If necessary, explicit architecture-guarded patch applications are appended and verified again.

The CVE-agent sandbox image supplies RPM configuration, macro packages, and prep helpers. Preparation does not establish buildroot equivalence, native execution on another architecture, compilation, runtime behavior, or that a fix resolves the CVE. Konflux builds and human review remain required.

3. Review the generated changes

Only verified patch text is staged. One file is created for each selected upstream patch; the spec and package metadata are updated as follows:

Output Change
rpms/<package>/<NNNNN>-<cve>-<slug>.patch New numbered patch file; the slug uses up to five words from the ticket summary, or the short commit ID if no usable summary exists.
Package .spec Adds patch declarations and arranges their application. Bumps Release by a trailing micro version (4.1 → 4.2), including when it comes from a %global macro, so local changes stay out of Fedora’s integer namespace; date-shaped releases require human review.
metadata/<package>.json Marks imported packages modified and adds the ticket/CVE to modification_reason. Independent packages remain independent; unrelated metadata is preserved.
Why preparation stopped Result
Supported %prep, path, or patch checks reject the fast-path fix needs_human, no file_actions.
AI investigation says needs_change, but automatic preparation fails Keep the finding and confidence; add the preparation failure to the summary; no file_actions.
Critical source fetch, checksum, or other evidence collection fails Abort with a local diagnostic report; retry behavior follows the failure classification.

4. Hand off the prepared changes

Use --submit-mr to hand off prepared actions; see Writes for submission and review details.

Planned work

HUM-8512 tracks one bounded AI attempt to adapt a patch after a hunk conflict. The planned flow replays the existing package patch stack, then verifies the adapted candidate from a clean tree with zero fuzz. It also proposes regenerating the candidate as a clean downstream diff and verifying that diff before staging. Source-level AI adaptation and patch regeneration are not implemented today; deterministic directory-prefix mapping is tried automatically during source checks and backport verification.

Writes: Jira comment and MR submission

Both writes are off by default in CLI use; each is enabled solely by its own flag, so a bare run never touches Jira or GitLab. The two flags are independent — a comment can be posted without submitting an MR and vice versa — and a failed write exits non-zero. The result writer runs first, before either remote write. When both are enabled, the MR is submitted before the Jira comment and analysis attachment, so a failed MR submission leaves the ticket eligible for a later sweep.

  • --post-to-jira posts a deterministic RECOMMENDATION: ... comment rendered from the validated InvestigationResult — no AI call at comment time — through cve_analysis’s jira_client, which handles ADF formatting and duplicate suppression, so re-running a command is idempotent. It also synchronizes one cve-agent-* Jira label to the current verdict, replacing any prior CVE Agent verdict label and preserving unrelated labels. A label update failure stops the write before the analysis attachment is uploaded. The comment carries the verdict, summary, cited evidence, and the next review step when human review is needed, including when a necessary package fix has no automated backport. The full result JSON is attached as <TICKET>-cve-agent-analysis.json with the supporting evidence, file actions, and commit message. A duplicate comment does not prevent the agent from synchronizing the label or uploading a missing analysis attachment. Existing tickets are not backfilled; a later --post-to-jira run synchronizes their verdict label.

    Verdict Jira label
    needs_change cve-agent-needs-change
    not_affected cve-agent-not-affected
    already_fixed cve-agent-already-fixed
    no_upstream_fix cve-agent-no-upstream-fix
    needs_human cve-agent-needs-human
    error cve-agent-error
  • --submit-mr submits a draft MR work item to a running workqueue-service instance (work items default to auto_merge=false). It is a no-op unless the result carries a prepared backport (file_actions + commit_message). Submission POSTs to WORKQUEUE_SERVICE_URL with the current token from WORKQUEUE_SERVICE_TOKEN_FILE when configured, otherwise WORKQUEUE_SERVICE_TOKEN; the file is re-read for each submission so projected token rotation is honored. The agent is headless, so there is no interactive login fallback. The submitted work item’s description includes the package, CVE, built NVR when available, selected upstream commits, investigation reasoning, and pending Konflux and human checks. Currently, workqueue-service only applies this description when updating an existing MR.

Automated sweep deployment

After a successful cve_analysis --resolve cycle, the analysis CronJob publishes a small SNS event requesting a sweep. A dedicated SQS queue filters for that event, and one long-running agent Deployment runs the existing Jira search and investigates eligible tickets. The message is a wake-up signal; it contains no ticket data. Every request starts a fresh Jira search. The deployed child sweep uses the CLI default of two workers, attempts at most forty tickets, and enables both --post-to-jira and --submit-mr.

The consumer renews message visibility while the sweep runs, enforces a one-hour deadline, and acknowledges only a completed sweep report. A completed pass is acknowledged even if some individual tickets failed; those tickets stay eligible for a later sweep. Failed or interrupted sweeps are retried by SQS. Repeated requests are handled sequentially by the single worker replica.

Before rollout, infrastructure must budget for two concurrent sandbox pods plus the controller pod, enforce per-pod resource limits, and set sandbox deadlines to cover the full one-hour sweep (the shared 1,800-second default is insufficient). Reap orphan sandbox pods before deployment. These are infrastructure requirements; this change adds no resource-policy flags or shared backend changes.

The analysis publisher has permission to publish to its environment’s SNS topic; the agent has permission to consume only its dedicated SQS queue. Neither needs cluster-wide Kubernetes API access; the agent needs pod management only in the sandbox namespace. The worker uses a projected short-lived token for the workqueue-service audience, where its identity can only submit mr work items.

Both environment switches start disabled in infrastructure/vars.d/hummingbird-cve-agent.yml. Deploy the SAM stack and provision each environment’s publisher and worker IAM keys in Vault before enabling its switch:

Variable Purpose
HUMMINGBIRD_CVE_AGENT_SWEEP_ENABLED_STAGING Enable staging event publishing and the worker replica.
HUMMINGBIRD_CVE_AGENT_SWEEP_ENABLED_PRODUCTION Enable production event publishing and the worker replica.

Eval harness

eval scores investigations against completed human /cve sessions. The eval set, eval-tickets.txt, holds Jira ticket keys and nothing else — ticket content is internal data and is never committed — so facts are fetched live and running the eval needs Jira access.

hummingbird-cve-agent eval eval-tickets.txt --output eval-report.json

Read-only: every Jira request is a GET (issue, comments, changelog); test_evaluate asserts it.

A resolved ticket gives the answer away — fixed_in_build set, labels moved to awaiting-vex, status Closed, newest analysis comment post-fix — so each ticket is rewound through its changelog, dropping later comments.

The rewind cutoff is the last scope-label handoff plus five minutes, allowing the assessment comment from the same triage burst to remain visible. Resolution time is the fallback when no handoff was recorded or it occurred after resolution; the fix-build timestamp is not the cutoff. fixed_in_build is also blanked so the recorded answer cannot leak through that field.

What is scored

Two axes, because one is not enough:

  • Disposition — where each side says the ticket ends up. The verdict is mapped to its eventual disposition, not through states.VERDICT_OUTCOME: that shared table answers “what does the agent do next”, so needs_change is In Progress — but a ticket the human backported still ends Done-Errata, and scoring against the immediate state would mark a correct needs_change as a miss and reward a wrong already_fixed.
  • Route — which branch the orchestrator would take, since already_fixed (Fixed in Build) and needs_change (MR creation) share the Done-Errata ending and are indistinguishable on the first axis. The human’s branch is derived from Pulp upload times: a fix build published after the handoff means the package had to change (needs_change), an earlier one means the shipped package already carried the fix (already_fixed); a Not a Bug closure is not_affected. A build Pulp no longer lists is reported as n/a and excluded from the route denominator rather than guessed.

Verdicts that reach no conclusion (needs_human, error) and conclusions no verdict can produce (unknown resolutions) never match. Package EOL Won't Do closures are deterministic cve_analysis outcomes and excluded before the AI eval runs. A ticket that cannot be rewound or investigated scores error without aborting the run.

HUM-7554     agree     verdict=already_fixed  expected=Closed / Done-Errata  actual=Closed / Done-Errata  route=!=needs_change
HUM-7537     DISAGREE  verdict=needs_human    expected=Closed / Won't Do     actual=                      route=n/a
disposition agreement: 1/2 (50%)
route agreement: 0/1 (0%), 1 undetermined
note: package facts (SRPM, SBOM, rpms repo) are current state, not the state at decision time, so agreement is retrospective

Only the Jira side is rewound. Package facts — the latest SRPM, the latest SBOM, and the pinned rpms repository inputs — are read live, so an old ticket is re-investigated against today’s package state rather than the state that existed when the human decided. Where the fix has since shipped, already_fixed is partly handed to the agent, so the score is retrospective current-state agreement, not a historical replay; every report prints that caveat. True replay needs as-of-cutoff SRPM and SBOM selection plus a date-pinned rpms revision, and is deliberately left to a follow-up.

What the model sees

Only non-candidates reach the model. The prompt combines static /cve instructions with validated TicketFacts: ticket, package and CVE details, Jira content, analysis/discussion, built NVR, and SBOM identity.

The full SBOM is not included. Evidence is collected through bounded tools:

Evidence Tool What it establishes
SBOM contents search_sbom Searches SPDX metadata, including npm PURLs. Only present confirms shipping; ambiguous versions or unknown require investigation.
Packaging files search_package_files, read_package_file Finds build settings, patches and packaging evidence. An empty search or missing CVE-named patch proves nothing by itself.
Other packages lookup_shipped_package Returns the latest source-package SBOM, NVR, component version and binary presence. source_only or missing data is inconclusive.
Prepared source read_package_source and source searches Inspects sandboxed, spec-pinned source and records provenance. built_nvr_match: unverified means it is not proven to match the ticket build. Preparation failures are not bypassed.
npm metadata Registry lookup Provides publication time, gitHead, repository and integrity. Metadata alone proves neither shipping nor fixedness.
CVE advisories lookup_cve_advisory Returns affected packages, vulnerable ranges and patched versions from GitHub advisories. Missing, truncated or mismatched results are inconclusive.
Known upstream fix check_upstream_fix Checks a commit patch against prepared source. Clean application means that exact diff is absent; already_applied means it reverse-applies cleanly. Other failures are inconclusive.
Upstream history Upstream source probe Searches an official repository once per ticket. Ancestry and applicability require a resolved exact base commit; unresolved candidates are not evidence. No candidate found means needs_human, not no_upstream_fix.

Evidence rules

  • Keep conclusions narrow. Patch applicability, %files entries, missing disable flags and failed searches alone do not prove affectedness, shipping or fix inclusion. Unresolved conclusions require needs_human.
  • Respect bounds. Tools enforce request, response and shared read budgets. Reads support line windows; excerpts may be truncated. Upstream access uses approved HTTPS hosts without redirect fallback.
  • Do not repeat permanent failures. retryable: false means move on and record the gap. Source tools permit retrying with arch when architecture selection is required.
  • Preserve partial evidence. Upstream collection failures abort the run; unresolved refs remain partial evidence.

Model output

The model returns only status, confidence, summary, evidence and vex_justification.

The caller supplies file_actions, commit_message, ai_generated, ticket, package and cves, and replaces collection records with complete evidence.

Security

The agent processes untrusted input, including executable RPM macros and prep commands. The credential-bearing worker never executes or imports RPM specs.

  • Isolated source execution. Spec parsing, structural edits, preparation, and candidate application run in an ephemeral, uncredentialed sandbox. There are no host mounts or worker secrets, and command environments are cleared. Sandbox backends come from the shared Hummingbird Sandbox library. Podman disables networking; Kubernetes requires deny-all egress policy in the dedicated sandbox namespace. Privileged mock, full builds, and runtime dependency installation are not used. Reusing a container with a reset workspace is not a fresh security boundary: use controlled images and packaging, not mutually hostile tenants.
  • Deployment isolation. Repository Git operations use private, temporary no-checkout object transport; package files exist only in the offline sandbox. The worker may manage execution pods only in the sandbox namespace once explicit API credentials are available. Those pods run non-root, drop capabilities, disable privilege escalation and token mounts, and use infrastructure-managed pod deadlines and resource limits sufficient for the sweep. Normal context-manager exit deletes sandbox pods; forced termination can leave pods until their active deadline, so operators must remove leftovers. Standalone investigations clean up at run end. Sandbox-produced source trees are never extracted into the worker.
  • Secret-free repository subprocesses. Worker Git transport operations use a minimal, allowlisted environment and do not inherit the Jira or model tokens used by the trusted worker and sweep CLI.
  • Uncredentialed, hardened git. Repository transport is public and read-only with no push token. Credential helpers and interactive prompts are disabled, system/global git config is neutralized (an attacker-controlled GIT_CONFIG_GLOBAL cannot change behaviour), and submodule recursion is off.
  • Input validation. Package names are restricted to a safe charset (no path traversal or absolute paths), file reads are confined to the package directory, saved patch names must be *.patch and cannot overwrite existing files, and patch URLs must use HTTPS on a recognized upstream forge.

Development

See the main README for development workflows.

Build the local CVE sandbox image for explicit local configuration overrides:

podman build -f hummingbird-sandbox/images/cve/Containerfile \
  -t localhost/hb-cve-sandbox:latest hummingbird-sandbox/
make hummingbird-cve-agent/setup
cd hummingbird-cve-agent
PYTHONPATH=.. python3 -m unittest discover -s tests -t .
# Opt into real offline RPM tests using an image with the required macros:
PYTHONPATH=.. HUMMINGBIRD_RPM_TEST_IMAGE=registry.example.com/rpm-prep:dev \
  python3 -m unittest tests.test_rpm_integration

The integration suite uses Podman without network access. Only source downloads are mocked; RPM parsing, Go preparation, architecture conditions, source overlays, patch checks, and final-spec replay execute for real. Standard unit discovery skips these tests unless the image variable is set.

The Containerfile pins the RPM repository commit supplying hum in HUM_COMMIT and records it in the io.hummingbird-project.hum-revision image label. Update that pin explicitly when refreshing the metadata library. Its fetch layer comes before Python dependencies, so dependency-only changes reuse the pinned checkout.

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.