# Hummingbird CVE agent

LLMS index: [llms.txt](https://hummingbird-project.io/llms.txt) | Full content: [llms-full.txt](https://hummingbird-project.io/llms-full.txt)

---

CLI and SQS worker that investigate open HUM Security tickets labeled
`cve-needs-attention` or `upstream-fix-available`, using AI where deterministic
analysis could not resolve the CVE. Fixes are handed to `workqueue-service` as
`mr` work items.

## System overview

```mermaid
flowchart TB

  TICKET["run HUM-XXXX"] --> COL
  SNS["Successful cve_analysis --resolve<br/>SNS → SQS wake-up"] --> SWEEP["Worker: fresh Jira sweep"]
  CLI["sweep [--limit N] [--workers N]"] --> SEARCH
  SWEEP --> SEARCH["Open HUM Security tickets<br/>in-scope labels · no cve-next-release<br/>not embargoed · no prev analysis"]
  SEARCH --> PREFLIGHT["Start and preflight all worker backends<br/>before ticket collection · skip if no tickets"]
  PREFLIGHT --> THREAD

  subgraph WORKER["Trusted worker · network and credentials"]
      direction TB
      THREAD["Sweep worker thread"] --> COL
      COL["Collect facts<br/>Jira · Pulp · SBOM"] --> CAND{"Deterministic candidate?<br/>affected + backport + fix label/reference<br/>not stale"}
      CAND -->|yes| ID{"Exact current version<br/>and SRPM NVR identified?"}
      ID -->|no| HUMAN["needs_human · no actions"]
      CAND -->|"no / stale"| AIINV["AI investigation<br/>bounded read-only tools"]
      ID -->|yes| FETCH
      AIINV -->|"source tools / needs_change + fix"| FETCH["Fetch pinned inputs<br/>hardened no-checkout Git transport<br/>checksummed sources · upstream patches"]
      AIINV -->|verdict| RESULT["InvestigationResult"]
      HUMAN --> RESULT
      EVIDENCE["Bounded evidence + verified file_actions"] --> RESULT
      RESULT --> OUT["Write result JSON<br/>stdout / --output before remote writes"]
      OUT --> SUBMIT{"--submit-mr<br/>and prepared actions?"}
       SUBMIT -->|yes| MR["Submit draft mr work item<br/>to workqueue-service"]
      SUBMIT -->|no| POST
      MR --> POST{"--post-to-jira?"}
       POST -->|yes| COMMENT["Post Jira comment → verdict label<br/>→ analysis attachment"]
  end

  subgraph SANDBOX["Offline execution · no network or credentials"]
      direction TB
      BACKEND["WorkerSandbox owns backend + package tools<br/>one per worker thread"] --> FILES["per-ticket WORK workspace<br/>reset state/files between tickets"]
      FILES --> PREP["RPM %prep<br/>existing downstream patches"]
      PREP --> INSPECT["Source inspection<br/>+ disposable candidate checks"]
       INSPECT -->|"backport path"| STAGE["Patch staging + spec edits"]
      STAGE --> REPLAY["Final-spec replay from clean inputs"]
  end
  THREAD -->|owns| BACKEND
  FETCH -->|"transfer each input once"| FILES
  INSPECT -->|"AI tools: bounded evidence"| AIINV
  REPLAY --> EVIDENCE
  MR --> REVIEW["Konflux build + human review"]

  classDef agent fill:#dbeafe,stroke:#2563eb,color:#111827
  classDef sandbox fill:#dcfce7,stroke:#16a34a,color:#111827
  class THREAD,COL,CAND,ID,HUMAN,AIINV,FETCH,EVIDENCE,RESULT,OUT,SUBMIT,POST,COMMENT agent
  class BACKEND,FILES,PREP,INSPECT,STAGE,REPLAY sandbox
```

The agent runs as a CLI and is stateless: `run` handles one ticket, while
`sweep` pages through all eligible tickets and investigates them concurrently.
The deployed `worker` consumes sweep requests from SQS and invokes that same
bounded sweep; the event is only a wake-up signal, not a ticket snapshot.
CLI writes are opt-in; the deployed worker enables both Jira posting and MR
submission (see [Writes](#writes-jira-comment-and-mr-submission)).

Critical evidence network, download, or parse failures abort the investigation;
their local diagnostic report leaves the ticket eligible for a later retry.
Unsupported or ambiguous evidence becomes `needs_human`. The ticket-level state machine lives in the
[CVE Lifecycle & State Vocabulary](https://hummingbird-project.io/docs/background/tools/hummingbird-cve-analysis/#cve-lifecycle--state-vocabulary).

## Tl;Dr

- **Scoped to CVE Security tickets** - investigates open HUM Security tickets
  with either `cve-needs-attention` or `upstream-fix-available`; these are the
  deterministic-analysis residual and upstream-fix populations respectively.
  Sweeps skip `cve-next-release`, embargoed tickets, and tickets already carrying their
  `<ticket>-cve-agent-analysis.json` attachment.
- **Backporting** - verifies upstream patches against pinned package source and
  prepares RPM changes; see [Backporting](#backporting).
- **AI in one place** - every LLM call goes through `ai.AgentAiClient`, which
  uses the shared `hummingbird_ai.models` library directly, so its
  token/cost metrics feed the shared GCP AI-costs dashboard. AI judges the
  fuzzy residual the deterministic path does not own; see
  [What the model sees](#what-the-model-sees) for the exact context. The model
  can use bounded read-only tools to search the SBOM, inspect package files,
  inspect checksum-pinned package sources, and look up exact npm release metadata.
  Each ticket has at most five tool-enabled turns plus one final-answer turn,
  whose request carries no tool definitions and explicitly asks for a JSON verdict
  using the available evidence. Invalid final replies receive a corrective request
  containing the result schema within the existing bounded retry budget. Exhausted
  retries fail the ticket without posting a result; no verdict is inferred from
  malformed output. Diagnostics log response length and finish reason, not response
  contents. Retries can make multiple model requests within a turn; six turns is
  not a six-request cap.
  SBOM searches return at most 20 components with binary-presence evidence
  (`present`, `source_only`, or `unknown`) from the shared SBOM classifier;
  a name/version match or source-only hit does not prove the component ships.
  When a matching SBOM result is unknown, the agent can scan published RPMs
  for that component. SBOM files are limited to 256 MiB.
  SBOM JSON is parsed incrementally; the process-local LRU retains at most eight
  full documents, so its entry limit is not a byte or process memory limit.
- **Structured results** - every investigation produces a validated result
  contract (there is no schema-version field). The result `status` is a `Verdict` from the
  shared lifecycle vocabulary (`hummingbird_cve_analysis/lib/states.py`), so
  the agent and `cve_analysis` speak the same state language. See the
  [CVE Lifecycle & State Vocabulary](https://hummingbird-project.io/docs/background/tools/hummingbird-cve-analysis/#cve-lifecycle--state-vocabulary)
  for the full state machine. Every result also carries the ticket's
  collection records: each evidence step's outcome (`success`, `empty`,
  `partial`, `failure`, or `limitation`), its retryability, and its
  provenance. Initial collection records can appear in the model's input facts;
  the caller owns the final collection records, including later tool outcomes.

## Prerequisites

- Python 3.11+
- Podman with a running Linux VM for local source preparation, or Kubernetes
  sandbox access in deployment.
- A sandbox image with RPM build tools, Python RPM bindings, `python3-specfile`,
  Git, GNU `patch`, macros/helpers... required by the package.
- `GITHUB_TOKEN` for GitHub backport lookup when unauthenticated API requests
  hit GitHub's rate limit.
- Jira API credentials (`JIRA_TOKEN`, and `JIRA_USER` where basic auth applies)
- Vertex AI access for the AI path: `GOOGLE_CLOUD_PROJECT`, or `GOOGLE_API_KEY`
  for the direct Gemini API. The deterministic fast path needs neither.
- For `--submit-mr` only: `WORKQUEUE_SERVICE_URL` and either
  `WORKQUEUE_SERVICE_TOKEN_FILE` or `WORKQUEUE_SERVICE_TOKEN`, with **`submit`**
  permission on the `mr` work type. The deployed worker uses a projected token file;
  local development can use the environment token described in
  [Writes: Jira comment and MR submission](#writes-jira-comment-and-mr-submission).

## Installation

```bash
make hummingbird-cve-agent/setup
```

The target installs `hummingbird-agent`, `hummingbird-ai`, and
`hummingbird-cve-analysis` editable alongside this package. Equivalent by hand:

```bash
cd hummingbird-cve-agent
pip install -e ../hummingbird-ai -e ../hummingbird-agent -e ../hummingbird-cve-analysis -e ".[dev]"
```

For the environment, copy `hummingbird-cve-agent/.env.example` to `.env` and
source it for a quick way to set up all necessary variables. For local CA setup,
activate the Python environment used to run the CLI before sourcing `.env`, so
`python -m certifi` resolves that environment's public CA bundle.
If the workqueue endpoint uses an internal CA, export `REQUESTS_CA_BUNDLE`
in the same shell as `--submit-mr`. Its PEM file must contain both the public
roots (for Google OAuth) and the internal root (for the workqueue endpoint).

## Configuration

Operational settings come from the YAML config file; secrets and model
credentials come from the environment.

| Variable                        | Purpose                                                                 |
| ------------------------------- | ----------------------------------------------------------------------- |
| `HUMMINGBIRD_CVE_AGENT_CONFIG`  | Path to a YAML config file (optional)                                   |
| `HUMMINGBIRD_CVE_AGENT_WORKERS` | Worker concurrency default (2; must be positive; `--workers` overrides) |
| `JIRA_TOKEN`                    | Jira API token                                                          |
| `JIRA_USER`                     | Jira basic-auth user (optional)                                         |
| `SENTRY_DSN`                    | Sentry error reporting for the deployed worker                          |
| `GITHUB_TOKEN`                  | Read-only GitHub token for PR commit lookups (optional)                 |
| `GITLAB_TOKEN`                  | Read-only GitLab.com token for MR commit lookups (optional)             |
| `GOOGLE_API_KEY`                | Gemini API key (direct API mode)                                        |
| `GOOGLE_CLOUD_PROJECT`          | GCP project for Vertex AI mode                                          |
| `WORKQUEUE_SERVICE_URL`         | Base URL of a running `workqueue-service` instance (for `--submit-mr`)  |
| `WORKQUEUE_SERVICE_TOKEN_FILE`  | Path to projected bearer token (preferred; re-read on each submission)  |
| `WORKQUEUE_SERVICE_TOKEN`       | Bearer token fallback for local `--submit-mr` use                       |
| `SQS_QUEUE_URL`                 | Dedicated queue consumed by the deployed `worker` command               |

YAML settings:

| Key             | Default                           | Purpose                                                       |
| --------------- | --------------------------------- | ------------------------------------------------------------- |
| `jira_url`      | Red Hat Jira                      | Jira base URL                                                 |
| `sandbox_type`  | `podman`                          | Sandbox backend; deployments use `k8s`                        |
| `sandbox`       | Published, digest-pinned CI image | Override `image` locally to use the CVE RPM-preparation image |
| `model`         | `claude-sonnet-4-6`               | Model name passed to `build_model`                            |
| `model_regions` | Gemini and Claude region prefixes | Model-name prefix to Vertex region                            |

Neither forge token grants write access: they only authenticate the read-only
API calls that resolve a pull/merge request to its commits, and are needed only
for private or rate-limited repositories. The sandbox never holds a push token.

## Usage

```bash
hummingbird-cve-agent run HUM-1234
hummingbird-cve-agent run HUM-1234 --output results/HUM-1234.json
hummingbird-cve-agent run HUM-1234 --post-to-jira
hummingbird-cve-agent run HUM-1234 --submit-mr
hummingbird-cve-agent sweep
hummingbird-cve-agent sweep --limit 25 --workers 6 --output sweep.json
hummingbird-cve-agent sweep --post-to-jira --submit-mr
```

This gathers facts for the ticket, runs the investigation, and prints the
result as JSON or persists it to `--output`. See [Backporting](#backporting)
for how upstream fixes become package changes and what happens when preparation
fails.

CLI logs include ticket and package context, e.g. `- INFO (HUM-1234 openssl):
collected facts for HUM-1234`; the package is `-` until known, and unscoped logs
show `- -`.

`sweep` defaults to all eligible tickets and two concurrent workers. `--limit`
caps attempted investigations, including failed attempts; analyzed, embargoed,
and out-of-scope search results do not consume the limit. A sweep searches once
and then investigates for a long time, so a ticket can also lose scope after it
was selected, by human re-triage to `cve-next-release` or by an overlapping
sweep closing it. That is reported as out of scope, not as a failure: it does
not fail the run or change the exit code. Standard output shows
counts, verdicts, token usage, estimated cost, and compact per-ticket summaries.
`--output FILE` also saves the complete JSON report. The command exits nonzero if
any ticket fails, while continuing after handled ticket-level failures.
A source-initialization `TypeError` propagates out of `investigate`, so that
ticket produces no investigation-result JSON and makes no Jira/MR writes.
The sweep includes `TypeError` in its existing ticket-error handling, records
the failed ticket in its report, and continues other tickets normally; this is
not a catch-all for unexpected exceptions. Without
`--post-to-jira`, the sweep does not add the attachment used to mark a ticket
as investigated, so a later sweep can retry it.
Critical collection failures never post a recommendation or analysis attachment,
even when the write flags are enabled. The CLI and sweep retain bounded failure
records locally; transient failures are marked retryable and are not cached.

After finding eligible tickets, the sweep starts and preflights all
`min(workers, eligible tickets)` execution backends before any ticket collection,
investigation, or model calls. No eligible tickets means no backends. A thread
pool runs that many worker loops consuming a ticket queue, each with its own
backend; it does not create a backend per ticket or package. Preflight failure
aborts the sweep before any ticket starts or remote writes occur and cleans up
started backends. Backends are cleaned up when the sweep exits; a handled ticket
failure affects only that ticket, not other workers. Transport/collection failures
discard its backend so timed-out child processes cannot reach the next ticket;
the next ticket starts a fresh backend. Other handled failures reset ticket
state and workspace for reuse.

## Backporting

The agent separates deciding that a package needs a fix from preparing the RPM
changes. AI may judge affectedness or identify an upstream commit; the
backporting code fetches and verifies the upstream patch, then prepares RPM
changes. It does not rewrite or generate patch content (yet, see
[Planned work](#planned-work)).

```mermaid
flowchart TD
    C{"Affected + backport + fix reference<br/>+ upstream-fix-available?"} -->|Yes| S{"Analysis stale?"}
    C -->|No| I["AI investigates"]
    S -->|Yes| I
    S -->|No| ID{"Exact current version<br/>and SRPM NVR identified?"}
    ID -->|No| H["needs_human · no file_actions"]
    ID -->|Yes| U["Resolve fix and select release patches"]
    I -->|Needs change + fix| U
    I -->|Otherwise| R["Return investigation result"]
    U --> V["Per-ticket workspace on worker-owned backend<br/>RPM prep → disposable candidate checks"]
    V --> W["Stage patches · edit spec/metadata<br/>replay final spec from clean inputs"]
    V -->|Unsupported / rejected| F["Deterministic: needs_human<br/>AI: retain finding/confidence<br/>both: no file_actions"]
    W -->|Unsupported / rejected| F
    V -->|Critical collection failure| A["Abort · local diagnostics<br/>no remote writes"]
    W -->|Verified| R
    F --> R
    H --> R
    R --> J["Agent's result processing"]
```

### 1. Identify and select the upstream fix

The fast path skips AI only when the existing CVE analysis says the package is
affected, names a backport as the next step, the ticket has the
`upstream-fix-available` label, and the analysis still matches the package
version collected now and has not already reached a recorded fixed version on
its release line. Stale analysis goes to AI. A candidate that is not stale but
does not explicitly identify both the current version and exact SRPM NVR returns
`needs_human`, not an AI fallback.
A `needs_change` result with a usable fix reference enters the same preparation
path; otherwise, the agent returns the investigation result only.

Commit URLs are fetched directly. A merged PR URL is first resolved to its
commits in application order. A commit is accepted only when it is reachable
from the upstream default branch or from the exact release branch named in
`metadata/<package>.json`; unmerged feature branches and open PRs are refused,
and an unknown release branch stays unverified. A failed fetch is retried
instead: it says nothing about the commit, so it is never reported as
unmerged. A review-tool URL such as a Qt
Gerrit change names no commit and is refused before any fetch; cite the landed
commit from the upstream mirror instead. A squash-merged change has no
landed SHA in common with the branch commit, so such a URL stays unverified
rather than being assumed merged. Branch-tagged patches must target the release
line of the shipped package; a mismatch requires human review. Mixed tagged and
untagged commits are rejected as ambiguous because untagged commits might be
prerequisites rather than alternative backports. After selecting the line, only
identical diff bodies are removed, retaining hunk positions, context, and file
metadata in the comparison so distinct fixes are not silently discarded.

### 2. Prepare and verify against package source

Each sweep worker thread owns one `WorkerSandbox`, which owns the execution
backend and package tools: a Podman container locally or an ephemeral Kubernetes
pod in deployment. Its `.ticket(package)` scope resets per-ticket state and
files in the `WORK` workspace. Sequential tickets reuse the physical container/pod,
but each gets fresh `Preparation` state, checksum pins, caches, and candidates;
none carry over to the next ticket. This avoids startup per investigation
without sharing ticket state. `.open()` lazily imports the package into the
existing worker backend. Standalone `run` uses its own `WorkerSandbox` and may
avoid backend startup entirely for an SBOM-only investigation; any backend it
starts is cleaned up when the run ends.

Within a ticket, the offline package workspace is shared by file/source
inspection, disposable candidate checks, patch staging, spec edits, and final
replay. `Preparation` performs RPM operations on that backend; it does not own
the backend or its lifecycle. Verified base trees and the disposable candidate
copy are released before the edited-spec replay, so a ticket never holds both
tree sets at once.
RPM macro expansion and
`rpmbuild -bp --nodeps` run only inside that sandbox. RPM owns architecture
conditions, source selection, Go macros, patch ordering, archive extraction,
and shell commands; there is no handwritten prep interpreter or static fallback.

The worker has no package checkout. A temporary, hardened no-checkout Git object
transport pins one repository commit and exports only `rpms/<package>/`,
`metadata/<package>.json` and `mock/dist-git-client.ini` from that
same commit. The worker brokers source/lookaside downloads, validates checksums,
and fetches upstream patches; the selected inputs are validated and then
transferred once as a single archive stream, preserving executable package
scripts. Every RPM preparation uses fresh
sandbox-local copies of those inputs, so a script changing its sources cannot contaminate
the next architecture or final replay. Downloads are limited to 128 MiB each, with a shared 256 MiB
input budget. The execution sandbox has no network or worker credentials, so
missing inputs, macros, signatures, or preparation helpers block preparation
instead of triggering dependency installation or guessed behavior.

The worker retains immutable metadata, checksum pins, and lookaside routing facts
from that Git snapshot. It never uses sandbox-modified copies to choose download
destinations or checksums; these acquisition facts are not a second working tree.

Inspection reads and searches the real prepared tree, cached per architecture.
Reads reject escaping paths and symlinks and are limited to 8 MiB per file;
searches are limited to 48 MiB and 50,000 filesystem entries. Commands have
timeouts, bounded diagnostics, and a 256 MiB individual-file limit. Podman keeps
its original shared startup behavior and writable filesystem. CPU, memory,
storage, process-count, and container-lifetime policy belong to deployment
infrastructure; the CVE agent does not impose them on shared Podman users.
Infrastructure enforcement is deferred, so these application-level checks do
not provide a whole-sandbox resource bound for local runs.

Candidates are checked on disposable copies using GNU `patch` with fuzz zero.
Before each candidate is applied, a reverse dry-run on the clean copy separates
source that already carries the fix from source that needs a rebased patch: the
first blocks backporting as `CVE already applied`, the second as a patch that
does not apply. Mismatched context is rejected without retrying with greater
fuzz. Exact-context hunks that land at a line offset still pass, but they are
recorded and reported to the MR reviewer.
Original source paths take precedence. When they are missing, deterministic
adaptation can remove a nested monorepo prefix, add a downstream prefix such as
`src/` or `vendor/<package>/`, or replace one shared directory prefix with another.
It preserves relative paths and requires a unique mapping supported by every
initially existing file touched by the ordered series. Added files must not
overwrite existing destinations. A bounded search ignores symlink directories;
ambiguous or incomplete searches are rejected, and addition-only patches do not
guess a target directory. Only diff headers change, never source hunks or commit
messages; individual file renames and source-level rewrites are not inferred.
Automatic backporting must use the same adapted patch on every applicable
supported target (`x86_64`, `aarch64`, and `s390x`). Source tools accept an explicit
`arch`; when multiple targets exist, they require it rather than guessing which
published binary the model is investigating.

The installed `specfile` library performs structural edits. The edited spec is
then prepared again from clean inputs, using RPM's own patch backend and options.
RPM's default patch fuzz is explicitly set to zero for every preparation.
Every candidate-touched file must match its verified final content, including
deletions; a later source overlay cannot silently erase the fix. If necessary,
explicit architecture-guarded patch applications are appended and verified again.

The CVE-agent sandbox image supplies RPM configuration, macro packages, and prep
helpers. Preparation does not establish buildroot equivalence, native execution on
another architecture, compilation, runtime behavior, or that a fix resolves the CVE.
Konflux builds and human review remain required.

### 3. Review the generated changes

Only verified patch text is staged. One file is created for each selected
upstream patch; the spec and package metadata are updated as follows:

| Output                                      | Change                                                                                                                                                                                                                                                                   |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `rpms/<package>/<NNNNN>-<cve>-<slug>.patch` | New numbered patch file; the slug uses up to five words from the ticket summary, or the short commit ID if no usable summary exists.                                                                                                                                     |
| Package `.spec`                             | Adds patch declarations and arranges their application. Bumps `Release` by a trailing micro version (`4.1` → `4.2`), including when it comes from a `%global` macro, so local changes stay out of Fedora's integer namespace; date-shaped releases require human review. |
| `metadata/<package>.json`                   | Marks imported packages `modified` and adds the ticket/CVE to `modification_reason`. Independent packages remain `independent`; unrelated metadata is preserved.                                                                                                         |

| Why preparation stopped                                               | Result                                                                                          |
| --------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| Supported `%prep`, path, or patch checks reject the fast-path fix     | `needs_human`, no `file_actions`.                                                               |
| AI investigation says `needs_change`, but automatic preparation fails | Keep the finding and confidence; add the preparation failure to the summary; no `file_actions`. |
| Critical source fetch, checksum, or other evidence collection fails   | Abort with a local diagnostic report; retry behavior follows the failure classification.        |

### 4. Hand off the prepared changes

Use `--submit-mr` to hand off prepared actions; see
[Writes](#writes-jira-comment-and-mr-submission) for submission and review
details.

### Planned work

[HUM-8512](https://redhat.atlassian.net/browse/HUM-8512) tracks one bounded AI
attempt to adapt a patch after a hunk conflict. The planned flow replays the
existing package patch stack, then verifies the adapted candidate from a clean
tree with zero fuzz. It also proposes regenerating the candidate as a clean
downstream diff and verifying that diff before staging. Source-level AI adaptation
and patch regeneration are not implemented today; deterministic directory-prefix
mapping is tried automatically during source checks and backport verification.

## Writes: Jira comment and MR submission

Both writes are off by default in CLI use; each is enabled solely by its own flag, so a
bare `run` never touches Jira or GitLab. The two flags are independent — a
comment can be posted without submitting an MR and vice versa — and a failed
write exits non-zero. The result writer runs first, before either remote write.
When both are enabled, the MR is submitted before the Jira
comment and analysis attachment, so a failed MR submission leaves the ticket
eligible for a later sweep.

- `--post-to-jira` posts a deterministic `RECOMMENDATION: ...` comment
  rendered from the validated `InvestigationResult` — no AI call at comment
  time — through `cve_analysis`'s `jira_client`, which handles ADF formatting
  and duplicate suppression, so re-running a command is idempotent. It also
  synchronizes one `cve-agent-*` Jira label to the current verdict, replacing
  any prior CVE Agent verdict label and preserving unrelated labels. A label
  update failure stops the write before the analysis attachment is uploaded.
  The comment carries the verdict, summary, cited evidence, and the next review
  step when human review is needed, including when a necessary package fix
  has no automated backport. The full result JSON is attached as
  `<TICKET>-cve-agent-analysis.json` with the supporting evidence, file actions,
  and commit message. A duplicate comment
  does not prevent the agent from synchronizing the label or uploading a
  missing analysis attachment. Existing tickets are not backfilled; a later
  `--post-to-jira` run synchronizes their verdict label.

  | Verdict           | Jira label                  |
  | ----------------- | --------------------------- |
  | `needs_change`    | `cve-agent-needs-change`    |
  | `not_affected`    | `cve-agent-not-affected`    |
  | `already_fixed`   | `cve-agent-already-fixed`   |
  | `no_upstream_fix` | `cve-agent-no-upstream-fix` |
  | `needs_human`     | `cve-agent-needs-human`     |
  | `error`           | `cve-agent-error`           |

- `--submit-mr` submits a **draft** MR work item to a running
  `workqueue-service` instance (work items default to `auto_merge=false`). It is
  a no-op unless the result carries a prepared backport (`file_actions` +
  `commit_message`). Submission POSTs to `WORKQUEUE_SERVICE_URL` with the
  current token from `WORKQUEUE_SERVICE_TOKEN_FILE` when configured, otherwise
  `WORKQUEUE_SERVICE_TOKEN`; the file is re-read for each submission so projected
  token rotation is honored. The agent is headless, so there is no interactive
  login fallback. The submitted work item's description includes the package,
  CVE, built NVR when available, selected upstream commits, investigation
  reasoning, and pending Konflux and human checks. Currently,
  `workqueue-service` only applies this description when updating an existing MR.

## Automated sweep deployment

After a successful `cve_analysis --resolve` cycle, the analysis CronJob publishes
a small SNS event requesting a sweep. A dedicated SQS queue filters for that
event, and one long-running agent Deployment runs the existing Jira search and
investigates eligible tickets. The message is a wake-up signal; it contains no
ticket data. Every request starts a fresh Jira search. The deployed child sweep
uses two workers by default; `HUMMINGBIRD_CVE_AGENT_WORKERS` sets that default,
and the worker's `--workers` option overrides it. The value must be positive.
The sweep attempts at most forty tickets and enables both `--post-to-jira` and
`--submit-mr`.

The consumer renews message visibility while the sweep runs, enforces a
one-hour deadline, and acknowledges only a completed sweep report. A completed
pass is acknowledged even if some individual tickets failed; those tickets stay
eligible for a later sweep. Failed or interrupted sweeps are retried by SQS.
Repeated requests are handled sequentially by the single worker replica.

Before rollout, infrastructure must budget for the configured number of concurrent
sandbox pods plus the controller pod, enforce per-pod resource limits, and set
sandbox deadlines to cover the full one-hour sweep (the shared 1,800-second default
is insufficient). Reap orphan sandbox pods before deployment. These are
infrastructure requirements; this change adds no resource-policy flags or shared
backend changes.

The analysis publisher has permission to publish to its environment's SNS topic;
the agent has permission to consume only its dedicated SQS queue. Neither needs
cluster-wide Kubernetes API access; the agent needs pod management only in the
sandbox namespace. The worker uses a projected short-lived token for the
`workqueue-service` audience, where its identity can only submit `mr` work items.

Both environment switches start disabled in
`infrastructure/vars.d/hummingbird-cve-agent.yml`. Deploy the SAM stack and
provision each environment's publisher and worker IAM keys in Vault before
enabling its switch:

| Variable                                         | Purpose                                                    |
| ------------------------------------------------ | ---------------------------------------------------------- |
| `HUMMINGBIRD_CVE_AGENT_SWEEP_ENABLED_STAGING`    | Enable staging event publishing and the worker replica.    |
| `HUMMINGBIRD_CVE_AGENT_SWEEP_ENABLED_PRODUCTION` | Enable production event publishing and the worker replica. |

## Eval harness

`eval` scores investigations against completed human `/cve` sessions. The eval
set, `eval-tickets.txt`, holds **Jira ticket keys and nothing else** — ticket
content is internal data and is never committed — so facts are fetched live and
running the eval needs Jira access.

```bash
hummingbird-cve-agent eval eval-tickets.txt --output eval-report.json
```

**Read-only:** every Jira request is a GET (issue, comments, changelog);
`test_evaluate` asserts it.

A resolved ticket gives the answer away — `fixed_in_build` set, labels moved to
`awaiting-vex`, status Closed, newest analysis comment post-fix — so each ticket
is rewound through its changelog, dropping later comments.

The rewind cutoff is the **last scope-label handoff plus five minutes**, allowing
the assessment comment from the same triage burst to remain visible. Resolution
time is the fallback when no handoff was recorded or it occurred after resolution;
the fix-build timestamp is not the cutoff. `fixed_in_build` is also blanked so
the recorded answer cannot leak through that field.

### What is scored

Two axes, because one is not enough:

- **Disposition** — where each side says the ticket _ends up_. The verdict is
  mapped to its eventual disposition, not through `states.VERDICT_OUTCOME`: that
  shared table answers "what does the agent do next", so `needs_change` is In
  Progress — but a ticket the human backported still ends `Done-Errata`, and
  scoring against the immediate state would mark a correct `needs_change` as a
  miss and reward a wrong `already_fixed`.
- **Route** — which branch the orchestrator would take, since `already_fixed`
  (Fixed in Build) and `needs_change` (MR creation) share the `Done-Errata`
  ending and are indistinguishable on the first axis. The human's branch is
  derived from Pulp upload times: a fix build published after the handoff means
  the package had to change (`needs_change`), an earlier one means the shipped
  package already carried the fix (`already_fixed`); a `Not a Bug` closure is
  `not_affected`. A build Pulp no longer lists is reported as `n/a` and excluded
  from the route denominator rather than guessed.

Verdicts that reach no conclusion (`needs_human`, `error`) and conclusions no
verdict can produce (unknown resolutions) never match. Package EOL `Won't Do`
closures are deterministic `cve_analysis` outcomes and excluded before the AI
eval runs. A ticket that cannot be rewound or investigated scores `error`
without aborting the run.

```text
HUM-7554     agree     verdict=already_fixed  expected=Closed / Done-Errata  actual=Closed / Done-Errata  route=!=needs_change
HUM-7537     DISAGREE  verdict=needs_human    expected=Closed / Won't Do     actual=                      route=n/a
disposition agreement: 1/2 (50%)
route agreement: 0/1 (0%), 1 undetermined
note: package facts (SRPM, SBOM, rpms repo) are current state, not the state at decision time, so agreement is retrospective
```

Only the Jira side is rewound. Package facts — the latest SRPM, the latest SBOM,
and the pinned rpms repository inputs — are read **live**, so an old ticket is
re-investigated against today's package state rather than the state that existed
when the human decided. Where the fix has since shipped, `already_fixed` is
partly handed to the agent, so the score is _retrospective current-state
agreement_, not a historical replay; every report prints that caveat. True
replay needs as-of-cutoff SRPM and SBOM selection plus a date-pinned rpms
revision, and is deliberately left to a follow-up.

## What the model sees

Only non-candidates reach the model. The prompt combines static `/cve`
instructions with validated `TicketFacts`: ticket, package and CVE details,
Jira content, analysis/discussion, built NVR, and SBOM identity.

The full SBOM is **not** included. Evidence is collected through bounded tools:

| Evidence           | Tool                                        | What it establishes                                                                                                                                                                                                                                                                         |
| ------------------ | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| SBOM contents      | `search_sbom`                               | Searches SPDX metadata, including npm PURLs. Only `present` confirms shipping; ambiguous versions or `unknown` require investigation.                                                                                                                                                       |
| Packaging files    | `search_package_files`, `read_package_file` | Finds build settings, patches and packaging evidence. An empty search or missing CVE-named patch proves nothing by itself.                                                                                                                                                                  |
| Other packages     | `lookup_shipped_package`                    | Returns the latest source-package SBOM, NVR, component version and binary presence. `source_only` or missing data is inconclusive.                                                                                                                                                          |
| Prepared source    | `read_package_source` and source searches   | Inspects sandboxed, spec-pinned source and records provenance. `built_nvr_match: unverified` means it is not proven to match the ticket build. Preparation failures are not bypassed.                                                                                                       |
| npm metadata       | Registry lookup                             | Provides publication time, `gitHead`, repository and integrity. Metadata alone proves neither shipping nor fixedness.                                                                                                                                                                       |
| CVE advisories     | `lookup_cve_advisory`                       | Returns affected packages, vulnerable ranges and patched versions from GitHub advisories. Missing, truncated or mismatched results are inconclusive.                                                                                                                                        |
| Known upstream fix | `check_upstream_fix`                        | Checks a commit patch against prepared source. The commit must be merged into the package's upstream default branch or its declared release branch. Clean application means that exact diff is absent; `already_applied` means it reverse-applies cleanly. Other failures are inconclusive. |
| Upstream history   | Upstream source probe                       | Searches an official repository once per ticket. Ancestry and applicability require a resolved exact base commit; unresolved candidates are not evidence. No candidate found means `needs_human`, not `no_upstream_fix`.                                                                    |

### Evidence rules

- **Keep conclusions narrow.** Patch applicability, `%files` entries, missing
  disable flags and failed searches alone do not prove affectedness, shipping
  or fix inclusion. Unresolved conclusions require `needs_human`.
- **Respect bounds.** Tools enforce request, response and shared read budgets.
  Reads support line windows; excerpts may be truncated. Upstream access uses
  approved HTTPS hosts without redirect fallback.
- **Do not repeat permanent failures.** `retryable: false` means move on and
  record the gap. Source tools permit retrying with `arch` when architecture
  selection is required.
- **Preserve partial evidence.** Upstream collection failures abort the run;
  unresolved refs remain partial evidence.

### Model output

The model returns only `status`, `confidence`, `summary`, `evidence` and
`vex_justification`. `vex_justification` is accepted only with `not_affected`
and only as `Component not Present` or `Vulnerable Code not Present`.

The caller supplies `file_actions`, `commit_message`, `ai_generated`, `ticket`,
`package` and `cves`, and replaces collection records with complete evidence.

## Security

The agent processes untrusted input, including executable RPM macros and prep
commands. The credential-bearing worker never executes or imports RPM specs.

- **Isolated source execution.** Spec parsing, structural edits, preparation,
  and candidate application run in an ephemeral, uncredentialed sandbox. There
  are no host mounts or worker secrets, and command environments are cleared.
  Sandbox backends come from the shared [Hummingbird Sandbox library][sandbox-docs].
  Podman disables networking; Kubernetes requires deny-all egress policy in the
  dedicated sandbox namespace. Privileged `mock`, full builds, and runtime
  dependency installation are not used.
  Reusing a container with a reset workspace is not a fresh security boundary:
  use controlled images and packaging, not mutually hostile tenants.
- **Deployment isolation.** Repository Git operations use private, temporary
  no-checkout object transport; package files exist only in the offline sandbox.
  The worker may manage execution pods only in the sandbox
  namespace once explicit API credentials are available. Those pods run
  non-root, drop capabilities, disable privilege escalation and token mounts,
  and use infrastructure-managed pod deadlines and resource limits sufficient
  for the sweep. Normal context-manager exit deletes sandbox pods; forced
  termination can leave pods until their active deadline, so operators must
  remove leftovers. Standalone investigations clean up at run end.
  Sandbox-produced source trees are never extracted into the worker.
- **Secret-free repository subprocesses.** Worker Git transport operations use a
  minimal, allowlisted environment and do not inherit the Jira or model tokens
  used by the trusted worker and sweep CLI.
- **Uncredentialed, hardened git.** Repository transport is public and read-only with
  no push token. Credential helpers and interactive prompts are disabled,
  system/global git config is neutralized (an attacker-controlled
  `GIT_CONFIG_GLOBAL` cannot change behaviour), and submodule recursion is off.
- **Input validation.** Package names are restricted to a safe charset (no path
  traversal or absolute paths), file reads are confined to the package
  directory, saved patch names must be `*.patch` and cannot overwrite existing
  files, and patch URLs must use HTTPS on a recognized upstream forge.

## Development

See the main [README][readme] for development workflows.

Build the local CVE sandbox image for explicit local configuration overrides:

```bash
podman build -f hummingbird-sandbox/images/cve/Containerfile \
  -t localhost/hb-cve-sandbox:latest hummingbird-sandbox/
```

```bash
make hummingbird-cve-agent/setup
cd hummingbird-cve-agent
PYTHONPATH=.. python3 -m unittest discover -s tests -t .
# Opt into real offline RPM tests using an image with the required macros:
PYTHONPATH=.. HUMMINGBIRD_RPM_TEST_IMAGE=registry.example.com/rpm-prep:dev \
  python3 -m unittest tests.test_rpm_integration
```

The integration suite uses Podman without network access. Only source downloads
are mocked; RPM parsing, Go preparation, architecture conditions, source overlays,
patch checks, and final-spec replay execute for real. Standard unit discovery
skips these tests unless the image variable is set.

The Containerfile pins the RPM repository commit supplying `hum` in `HUM_COMMIT`
and records it in the `io.hummingbird-project.hum-revision` image label. Update
that pin explicitly when refreshing the metadata library. Its fetch layer comes
before Python dependencies, so dependency-only changes reuse the pinned checkout.

## License

This project is licensed under the GNU General Public License v3.0 or later -
see the [LICENSE][license] file for details.

[readme]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/README.md
[license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE
[sandbox-docs]: https://hummingbird-project.io/docs/background/tools/hummingbird-sandbox/
