Hummingbird CVE agent
CLI and SQS worker that investigate open HUM Security tickets labeled
cve-needs-attention or upstream-fix-available, using AI where deterministic
analysis could not resolve the CVE. Fixes are handed to workqueue-service as
mr work items.
System overview
flowchart TB
TICKET["run HUM-XXXX"] --> COL
SNS["Successful cve_analysis --resolve<br/>SNS → SQS wake-up"] --> SWEEP["Worker: fresh Jira sweep"]
CLI["sweep [--limit N] [--workers N]"] --> SEARCH
SWEEP --> SEARCH["Open HUM Security tickets<br/>in-scope labels · no cve-next-release<br/>not embargoed · no prev analysis"]
SEARCH --> PREFLIGHT["Start and preflight all worker backends<br/>before ticket collection · skip if no tickets"]
PREFLIGHT --> THREAD
subgraph WORKER["Trusted worker · network and credentials"]
direction TB
THREAD["Sweep worker thread"] --> COL
COL["Collect facts<br/>Jira · Pulp · SBOM"] --> CAND{"Deterministic candidate?<br/>affected + backport + fix label/reference<br/>not stale"}
CAND -->|yes| ID{"Exact current version<br/>and SRPM NVR identified?"}
ID -->|no| HUMAN["needs_human · no actions"]
CAND -->|"no / stale"| AIINV["AI investigation<br/>bounded read-only tools"]
ID -->|yes| FETCH
AIINV -->|"source tools / needs_change + fix"| FETCH["Fetch pinned inputs<br/>hardened no-checkout Git transport<br/>checksummed sources · upstream patches"]
AIINV -->|verdict| RESULT["InvestigationResult"]
HUMAN --> RESULT
EVIDENCE["Bounded evidence + verified file_actions"] --> RESULT
RESULT --> OUT["Write result JSON<br/>stdout / --output before remote writes"]
OUT --> SUBMIT{"--submit-mr<br/>and prepared actions?"}
SUBMIT -->|yes| MR["Submit draft mr work item<br/>to workqueue-service"]
SUBMIT -->|no| POST
MR --> POST{"--post-to-jira?"}
POST -->|yes| COMMENT["Post Jira comment → verdict label<br/>→ analysis attachment"]
end
subgraph SANDBOX["Offline execution · no network or credentials"]
direction TB
BACKEND["WorkerSandbox owns backend + package tools<br/>one per worker thread"] --> FILES["per-ticket WORK workspace<br/>reset state/files between tickets"]
FILES --> PREP["RPM %prep<br/>existing downstream patches"]
PREP --> INSPECT["Source inspection<br/>+ disposable candidate checks"]
INSPECT -->|"backport path"| STAGE["Patch staging + spec edits"]
STAGE --> REPLAY["Final-spec replay from clean inputs"]
end
THREAD -->|owns| BACKEND
FETCH -->|"transfer each input once"| FILES
INSPECT -->|"AI tools: bounded evidence"| AIINV
REPLAY --> EVIDENCE
MR --> REVIEW["Konflux build + human review"]
classDef agent fill:#dbeafe,stroke:#2563eb,color:#111827
classDef sandbox fill:#dcfce7,stroke:#16a34a,color:#111827
class THREAD,COL,CAND,ID,HUMAN,AIINV,FETCH,EVIDENCE,RESULT,OUT,SUBMIT,POST,COMMENT agent
class BACKEND,FILES,PREP,INSPECT,STAGE,REPLAY sandbox
The agent runs as a CLI and is stateless: run handles one ticket, while
sweep pages through all eligible tickets and investigates them concurrently.
The deployed worker consumes sweep requests from SQS and invokes that same
bounded sweep; the event is only a wake-up signal, not a ticket snapshot.
CLI writes are opt-in; the deployed worker enables both Jira posting and MR
submission (see Writes).
Critical evidence network, download, or parse failures abort the investigation;
their local diagnostic report leaves the ticket eligible for a later retry.
Unsupported or ambiguous evidence becomes needs_human. The ticket-level state machine lives in the
CVE Lifecycle & State Vocabulary.
Tl;Dr
- Scoped to CVE Security tickets - investigates open HUM Security tickets
with either
cve-needs-attentionorupstream-fix-available; these are the deterministic-analysis residual and upstream-fix populations respectively. Sweeps skipcve-next-release, embargoed tickets, and tickets already carrying their<ticket>-cve-agent-analysis.jsonattachment. - Backporting - verifies upstream patches against pinned package source and prepares RPM changes; see Backporting.
- AI in one place - every LLM call goes through
ai.AgentAiClient, which uses the sharedhummingbird_ai.modelslibrary directly, so its token/cost metrics feed the shared GCP AI-costs dashboard. AI judges the fuzzy residual the deterministic path does not own; see What the model sees for the exact context. The model can use bounded read-only tools to search the SBOM, inspect package files, inspect checksum-pinned package sources, and look up exact npm release metadata. Each ticket has at most five tool-enabled turns plus one final-answer turn, whose request carries no tool definitions and explicitly asks for a JSON verdict using the available evidence. Invalid final replies receive a corrective request containing the result schema within the existing bounded retry budget. Exhausted retries fail the ticket without posting a result; no verdict is inferred from malformed output. Diagnostics log response length and finish reason, not response contents. Retries can make multiple model requests within a turn; six turns is not a six-request cap. SBOM searches return at most 20 components with binary-presence evidence (present,source_only, orunknown) from the shared SBOM classifier; a name/version match or source-only hit does not prove the component ships. When a matching SBOM result is unknown, the agent can scan published RPMs for that component. Pulp listings and SBOM downloads are limited to 64 MiB each. - Structured results - every investigation produces a validated result
contract (there is no schema-version field). The result
statusis aVerdictfrom the shared lifecycle vocabulary (hummingbird_cve_analysis/lib/states.py), so the agent andcve_analysisspeak the same state language. See the CVE Lifecycle & State Vocabulary for the full state machine. Every result also carries the ticket’s collection records: each evidence step’s outcome (success,empty,partial,failure, orlimitation), its retryability, and its provenance. Initial collection records can appear in the model’s input facts; the caller owns the final collection records, including later tool outcomes.
Prerequisites
- Python 3.11+
- Podman with a running Linux VM for local source preparation, or Kubernetes sandbox access in deployment.
- A sandbox image with RPM build tools, Python RPM bindings,
python3-specfile, Git, GNUpatch, macros/helpers… required by the package. GITHUB_TOKENfor GitHub backport lookup when unauthenticated API requests hit GitHub’s rate limit.- Jira API credentials (
JIRA_TOKEN, andJIRA_USERwhere basic auth applies) - Vertex AI access for the AI path:
GOOGLE_CLOUD_PROJECT, orGOOGLE_API_KEYfor the direct Gemini API. The deterministic fast path needs neither. - For
--submit-mronly:WORKQUEUE_SERVICE_URLand eitherWORKQUEUE_SERVICE_TOKEN_FILEorWORKQUEUE_SERVICE_TOKEN, withsubmitpermission on themrwork type. The deployed worker uses a projected token file; local development can use the environment token described in Writes: Jira comment and MR submission.
Installation
make hummingbird-cve-agent/setup
The target installs hummingbird-agent, hummingbird-ai, and
hummingbird-cve-analysis editable alongside this package. Equivalent by hand:
cd hummingbird-cve-agent
pip install -e ../hummingbird-ai -e ../hummingbird-agent -e ../hummingbird-cve-analysis -e ".[dev]"
For the environment, copy hummingbird-cve-agent/.env.example to .env and
source it for a quick way to set up all necessary variables. For local CA setup,
activate the Python environment used to run the CLI before sourcing .env, so
python -m certifi resolves that environment’s public CA bundle.
If the workqueue endpoint uses an internal CA, export REQUESTS_CA_BUNDLE
in the same shell as --submit-mr. Its PEM file must contain both the public
roots (for Google OAuth) and the internal root (for the workqueue endpoint).
Configuration
Operational settings come from the YAML config file; secrets and model credentials come from the environment.
| Variable | Purpose |
|---|---|
HUMMINGBIRD_CVE_AGENT_CONFIG |
Path to a YAML config file (optional) |
JIRA_TOKEN |
Jira API token |
JIRA_USER |
Jira basic-auth user (optional) |
SENTRY_DSN |
Sentry error reporting for the deployed worker |
GITHUB_TOKEN |
Read-only GitHub token for PR commit lookups (optional) |
GITLAB_TOKEN |
Read-only GitLab.com token for MR commit lookups (optional) |
GOOGLE_API_KEY |
Gemini API key (direct API mode) |
GOOGLE_CLOUD_PROJECT |
GCP project for Vertex AI mode |
WORKQUEUE_SERVICE_URL |
Base URL of a running workqueue-service instance (for --submit-mr) |
WORKQUEUE_SERVICE_TOKEN_FILE |
Path to projected bearer token (preferred; re-read on each submission) |
WORKQUEUE_SERVICE_TOKEN |
Bearer token fallback for local --submit-mr use |
SQS_QUEUE_URL |
Dedicated queue consumed by the deployed worker command |
YAML settings:
| Key | Default | Purpose |
|---|---|---|
jira_url |
Red Hat Jira | Jira base URL |
sandbox_type |
podman |
Sandbox backend; deployments use k8s |
sandbox |
Published, digest-pinned CI image | Override image locally to use the CVE RPM-preparation image |
model |
claude-sonnet-4-6 |
Model name passed to build_model |
model_regions |
Gemini and Claude region prefixes | Model-name prefix to Vertex region |
Neither forge token grants write access: they only authenticate the read-only API calls that resolve a pull/merge request to its commits, and are needed only for private or rate-limited repositories. The sandbox never holds a push token.
Usage
hummingbird-cve-agent run HUM-1234
hummingbird-cve-agent run HUM-1234 --output results/HUM-1234.json
hummingbird-cve-agent run HUM-1234 --post-to-jira
hummingbird-cve-agent run HUM-1234 --submit-mr
hummingbird-cve-agent sweep
hummingbird-cve-agent sweep --limit 25 --workers 6 --output sweep.json
hummingbird-cve-agent sweep --post-to-jira --submit-mr
This gathers facts for the ticket, runs the investigation, and prints the
result as JSON or persists it to --output. See Backporting
for how upstream fixes become package changes and what happens when preparation
fails.
CLI logs include ticket and package context, e.g. - INFO (HUM-1234 openssl): collected facts for HUM-1234; the package is - until known, and unscoped logs
show - -.
sweep defaults to all eligible tickets and two concurrent workers. --limit
caps attempted investigations, including failed attempts; analyzed, embargoed,
and out-of-scope search results do not consume the limit. A sweep searches once
and then investigates for a long time, so a ticket can also lose scope after it
was selected, by human re-triage to cve-next-release or by an overlapping
sweep closing it. That is reported as out of scope, not as a failure: it does
not fail the run or change the exit code. Standard output shows
counts, verdicts, token usage, estimated cost, and compact per-ticket summaries.
--output FILE also saves the complete JSON report. The command exits nonzero if
any ticket fails, while continuing after handled ticket-level failures.
A source-initialization TypeError propagates out of investigate, so that
ticket produces no investigation-result JSON and makes no Jira/MR writes.
The sweep includes TypeError in its existing ticket-error handling, records
the failed ticket in its report, and continues other tickets normally; this is
not a catch-all for unexpected exceptions. Without
--post-to-jira, the sweep does not add the attachment used to mark a ticket
as investigated, so a later sweep can retry it.
Critical collection failures never post a recommendation or analysis attachment,
even when the write flags are enabled. The CLI and sweep retain bounded failure
records locally; transient failures are marked retryable and are not cached.
After finding eligible tickets, the sweep starts and preflights all
min(workers, eligible tickets) execution backends before any ticket collection,
investigation, or model calls. No eligible tickets means no backends. A thread
pool runs that many worker loops consuming a ticket queue, each with its own
backend; it does not create a backend per ticket or package. Preflight failure
aborts the sweep before any ticket starts or remote writes occur and cleans up
started backends. Backends are cleaned up when the sweep exits; a handled ticket
failure affects only that ticket, not other workers. Transport/collection failures
discard its backend so timed-out child processes cannot reach the next ticket;
the next ticket starts a fresh backend. Other handled failures reset ticket
state and workspace for reuse.
Backporting
The agent separates deciding that a package needs a fix from preparing the RPM changes. AI may judge affectedness or identify an upstream commit; the backporting code fetches and verifies the upstream patch, then prepares RPM changes. It does not rewrite or generate patch content (yet, see Planned work).
flowchart TD
C{"Affected + backport + fix reference<br/>+ upstream-fix-available?"} -->|Yes| S{"Analysis stale?"}
C -->|No| I["AI investigates"]
S -->|Yes| I
S -->|No| ID{"Exact current version<br/>and SRPM NVR identified?"}
ID -->|No| H["needs_human · no file_actions"]
ID -->|Yes| U["Resolve fix and select release patches"]
I -->|Needs change + fix| U
I -->|Otherwise| R["Return investigation result"]
U --> V["Per-ticket workspace on worker-owned backend<br/>RPM prep → disposable candidate checks"]
V --> W["Stage patches · edit spec/metadata<br/>replay final spec from clean inputs"]
V -->|Unsupported / rejected| F["Deterministic: needs_human<br/>AI: retain finding/confidence<br/>both: no file_actions"]
W -->|Unsupported / rejected| F
V -->|Critical collection failure| A["Abort · local diagnostics<br/>no remote writes"]
W -->|Verified| R
F --> R
H --> R
R --> J["Agent's result processing"]
1. Identify and select the upstream fix
The fast path skips AI only when the existing CVE analysis says the package is
affected, names a backport as the next step, the ticket has the
upstream-fix-available label, and the analysis still matches the package
version collected now and has not already reached a recorded fixed version on
its release line. Stale analysis goes to AI. A candidate that is not stale but
does not explicitly identify both the current version and exact SRPM NVR returns
needs_human, not an AI fallback.
A needs_change result with a usable fix reference enters the same preparation
path; otherwise, the agent returns the investigation result only.
Commit URLs are fetched directly. A merged PR URL is first resolved to its commits in application order. Branch-tagged patches must target the release line of the shipped package; a mismatch requires human review. Mixed tagged and untagged commits are rejected as ambiguous because untagged commits might be prerequisites rather than alternative backports. Supply an explicit release-line series before retrying. Commit IDs are compared in full; abbreviated and full references are fetched separately. After selecting the line, only identical diff bodies are removed, retaining hunk positions, context, and file metadata in the comparison so distinct fixes are not silently discarded.
2. Prepare and verify against package source
Each sweep worker thread owns one WorkerSandbox, which owns the execution
backend and package tools: a Podman container locally or an ephemeral Kubernetes
pod in deployment. Its .ticket(package) scope resets per-ticket state and
files in the WORK workspace. Sequential tickets reuse the physical container/pod,
but each gets fresh Preparation state, checksum pins, caches, and candidates;
none carry over to the next ticket. This avoids startup per investigation
without sharing ticket state. .open() lazily imports the package into the
existing worker backend. Standalone run uses its own WorkerSandbox and may
avoid backend startup entirely for an SBOM-only investigation; any backend it
starts is cleaned up when the run ends.
Within a ticket, the offline package workspace is shared by file/source
inspection, disposable candidate checks, patch staging, spec edits, and final
replay. Preparation performs RPM operations on that backend; it does not own
the backend or its lifecycle. Verified base trees and the disposable candidate
copy are released before the edited-spec replay, so a ticket never holds both
tree sets at once.
RPM macro expansion and
rpmbuild -bp --nodeps run only inside that sandbox. RPM owns architecture
conditions, source selection, Go macros, patch ordering, archive extraction,
and shell commands; there is no handwritten prep interpreter or static fallback.
The worker has no package checkout. A temporary, hardened no-checkout Git object
transport pins one repository commit and exports only rpms/<package>/,
metadata/<package>.json and mock/dist-git-client.ini from that
same commit. The worker brokers source/lookaside downloads, validates checksums,
and fetches upstream patches; the selected inputs are validated and then
transferred once as a single archive stream, preserving executable package
scripts. Every RPM preparation uses fresh
sandbox-local copies of those inputs, so a script changing its sources cannot contaminate
the next architecture or final replay. Downloads are limited to 128 MiB each, with a shared 256 MiB
input budget. The execution sandbox has no network or worker credentials, so
missing inputs, macros, signatures, or preparation helpers block preparation
instead of triggering dependency installation or guessed behavior.
The worker retains immutable metadata, checksum pins, and lookaside routing facts from that Git snapshot. It never uses sandbox-modified copies to choose download destinations or checksums; these acquisition facts are not a second working tree.
Inspection reads and searches the real prepared tree, cached per architecture. Reads reject escaping paths and symlinks and are limited to 8 MiB per file; searches are limited to 48 MiB and 50,000 filesystem entries. Commands have timeouts, bounded diagnostics, and a 256 MiB individual-file limit. Podman keeps its original shared startup behavior and writable filesystem. CPU, memory, storage, process-count, and container-lifetime policy belong to deployment infrastructure; the CVE agent does not impose them on shared Podman users. Infrastructure enforcement is deferred, so these application-level checks do not provide a whole-sandbox resource bound for local runs.
Candidates are checked on disposable copies using GNU patch with fuzz zero.
Before each candidate is applied, a reverse dry-run on the clean copy separates
source that already carries the fix from source that needs a rebased patch: the
first blocks backporting as CVE already applied, the second as a patch that
does not apply. Mismatched context is rejected without retrying with greater
fuzz. Exact-context hunks that land at a line offset still pass, but they are
recorded and reported to the MR reviewer.
Original source paths take precedence. When they are missing, deterministic
adaptation can remove a nested monorepo prefix, add a downstream prefix such as
src/ or vendor/<package>/, or replace one shared directory prefix with another.
It preserves relative paths and requires a unique mapping supported by every
initially existing file touched by the ordered series. Added files must not
overwrite existing destinations. A bounded search ignores symlink directories;
ambiguous or incomplete searches are rejected, and addition-only patches do not
guess a target directory. Only diff headers change, never source hunks or commit
messages; individual file renames and source-level rewrites are not inferred.
Automatic backporting must use the same adapted patch on every applicable
supported target (x86_64, aarch64, and s390x). Source tools accept an explicit
arch; when multiple targets exist, they require it rather than guessing which
published binary the model is investigating.
The installed specfile library performs structural edits. The edited spec is
then prepared again from clean inputs, using RPM’s own patch backend and options.
RPM’s default patch fuzz is explicitly set to zero for every preparation.
Every candidate-touched file must match its verified final content, including
deletions; a later source overlay cannot silently erase the fix. If necessary,
explicit architecture-guarded patch applications are appended and verified again.
The CVE-agent sandbox image supplies RPM configuration, macro packages, and prep helpers. Preparation does not establish buildroot equivalence, native execution on another architecture, compilation, runtime behavior, or that a fix resolves the CVE. Konflux builds and human review remain required.
3. Review the generated changes
Only verified patch text is staged. One file is created for each selected upstream patch; the spec and package metadata are updated as follows:
| Output | Change |
|---|---|
rpms/<package>/<NNNNN>-<cve>-<slug>.patch |
New numbered patch file; the slug uses up to five words from the ticket summary, or the short commit ID if no usable summary exists. |
Package .spec |
Adds patch declarations and arranges their application. Bumps Release by a trailing micro version (4.1 → 4.2), including when it comes from a %global macro, so local changes stay out of Fedora’s integer namespace; date-shaped releases require human review. |
metadata/<package>.json |
Marks imported packages modified and adds the ticket/CVE to modification_reason. Independent packages remain independent; unrelated metadata is preserved. |
| Why preparation stopped | Result |
|---|---|
Supported %prep, path, or patch checks reject the fast-path fix |
needs_human, no file_actions. |
AI investigation says needs_change, but automatic preparation fails |
Keep the finding and confidence; add the preparation failure to the summary; no file_actions. |
| Critical source fetch, checksum, or other evidence collection fails | Abort with a local diagnostic report; retry behavior follows the failure classification. |
4. Hand off the prepared changes
Use --submit-mr to hand off prepared actions; see
Writes for submission and review
details.
Planned work
HUM-8512 tracks one bounded AI attempt to adapt a patch after a hunk conflict. The planned flow replays the existing package patch stack, then verifies the adapted candidate from a clean tree with zero fuzz. It also proposes regenerating the candidate as a clean downstream diff and verifying that diff before staging. Source-level AI adaptation and patch regeneration are not implemented today; deterministic directory-prefix mapping is tried automatically during source checks and backport verification.
Writes: Jira comment and MR submission
Both writes are off by default in CLI use; each is enabled solely by its own flag, so a
bare run never touches Jira or GitLab. The two flags are independent — a
comment can be posted without submitting an MR and vice versa — and a failed
write exits non-zero. The result writer runs first, before either remote write.
When both are enabled, the MR is submitted before the Jira
comment and analysis attachment, so a failed MR submission leaves the ticket
eligible for a later sweep.
-
--post-to-jiraposts a deterministicRECOMMENDATION: ...comment rendered from the validatedInvestigationResult— no AI call at comment time — throughcve_analysis’sjira_client, which handles ADF formatting and duplicate suppression, so re-running a command is idempotent. It also synchronizes onecve-agent-*Jira label to the current verdict, replacing any prior CVE Agent verdict label and preserving unrelated labels. A label update failure stops the write before the analysis attachment is uploaded. The comment carries the verdict, summary, cited evidence, and the next review step when human review is needed, including when a necessary package fix has no automated backport. The full result JSON is attached as<TICKET>-cve-agent-analysis.jsonwith the supporting evidence, file actions, and commit message. A duplicate comment does not prevent the agent from synchronizing the label or uploading a missing analysis attachment. Existing tickets are not backfilled; a later--post-to-jirarun synchronizes their verdict label.Verdict Jira label needs_changecve-agent-needs-changenot_affectedcve-agent-not-affectedalready_fixedcve-agent-already-fixedno_upstream_fixcve-agent-no-upstream-fixneeds_humancve-agent-needs-humanerrorcve-agent-error -
--submit-mrsubmits a draft MR work item to a runningworkqueue-serviceinstance (work items default toauto_merge=false). It is a no-op unless the result carries a prepared backport (file_actions+commit_message). Submission POSTs toWORKQUEUE_SERVICE_URLwith the current token fromWORKQUEUE_SERVICE_TOKEN_FILEwhen configured, otherwiseWORKQUEUE_SERVICE_TOKEN; the file is re-read for each submission so projected token rotation is honored. The agent is headless, so there is no interactive login fallback. The submitted work item’s description includes the package, CVE, built NVR when available, selected upstream commits, investigation reasoning, and pending Konflux and human checks. Currently,workqueue-serviceonly applies this description when updating an existing MR.
Automated sweep deployment
After a successful cve_analysis --resolve cycle, the analysis CronJob publishes
a small SNS event requesting a sweep. A dedicated SQS queue filters for that
event, and one long-running agent Deployment runs the existing Jira search and
investigates eligible tickets. The message is a wake-up signal; it contains no
ticket data. Every request starts a fresh Jira search. The deployed child sweep
uses the CLI default of two workers, attempts at most forty tickets, and enables
both --post-to-jira and --submit-mr.
The consumer renews message visibility while the sweep runs, enforces a one-hour deadline, and acknowledges only a completed sweep report. A completed pass is acknowledged even if some individual tickets failed; those tickets stay eligible for a later sweep. Failed or interrupted sweeps are retried by SQS. Repeated requests are handled sequentially by the single worker replica.
Before rollout, infrastructure must budget for two concurrent sandbox pods plus the controller pod, enforce per-pod resource limits, and set sandbox deadlines to cover the full one-hour sweep (the shared 1,800-second default is insufficient). Reap orphan sandbox pods before deployment. These are infrastructure requirements; this change adds no resource-policy flags or shared backend changes.
The analysis publisher has permission to publish to its environment’s SNS topic;
the agent has permission to consume only its dedicated SQS queue. Neither needs
cluster-wide Kubernetes API access; the agent needs pod management only in the
sandbox namespace. The worker uses a projected short-lived token for the
workqueue-service audience, where its identity can only submit mr work items.
Both environment switches start disabled in
infrastructure/vars.d/hummingbird-cve-agent.yml. Deploy the SAM stack and
provision each environment’s publisher and worker IAM keys in Vault before
enabling its switch:
| Variable | Purpose |
|---|---|
HUMMINGBIRD_CVE_AGENT_SWEEP_ENABLED_STAGING |
Enable staging event publishing and the worker replica. |
HUMMINGBIRD_CVE_AGENT_SWEEP_ENABLED_PRODUCTION |
Enable production event publishing and the worker replica. |
Eval harness
eval scores investigations against completed human /cve sessions. The eval
set, eval-tickets.txt, holds Jira ticket keys and nothing else — ticket
content is internal data and is never committed — so facts are fetched live and
running the eval needs Jira access.
hummingbird-cve-agent eval eval-tickets.txt --output eval-report.json
Read-only: every Jira request is a GET (issue, comments, changelog);
test_evaluate asserts it.
A resolved ticket gives the answer away — fixed_in_build set, labels moved to
awaiting-vex, status Closed, newest analysis comment post-fix — so each ticket
is rewound through its changelog, dropping later comments.
The rewind cutoff is the last scope-label handoff plus five minutes, allowing
the assessment comment from the same triage burst to remain visible. Resolution
time is the fallback when no handoff was recorded or it occurred after resolution;
the fix-build timestamp is not the cutoff. fixed_in_build is also blanked so
the recorded answer cannot leak through that field.
What is scored
Two axes, because one is not enough:
- Disposition — where each side says the ticket ends up. The verdict is
mapped to its eventual disposition, not through
states.VERDICT_OUTCOME: that shared table answers “what does the agent do next”, soneeds_changeis In Progress — but a ticket the human backported still endsDone-Errata, and scoring against the immediate state would mark a correctneeds_changeas a miss and reward a wrongalready_fixed. - Route — which branch the orchestrator would take, since
already_fixed(Fixed in Build) andneeds_change(MR creation) share theDone-Errataending and are indistinguishable on the first axis. The human’s branch is derived from Pulp upload times: a fix build published after the handoff means the package had to change (needs_change), an earlier one means the shipped package already carried the fix (already_fixed); aNot a Bugclosure isnot_affected. A build Pulp no longer lists is reported asn/aand excluded from the route denominator rather than guessed.
Verdicts that reach no conclusion (needs_human, error) and conclusions no
verdict can produce (unknown resolutions) never match. Package EOL Won't Do
closures are deterministic cve_analysis outcomes and excluded before the AI
eval runs. A ticket that cannot be rewound or investigated scores error
without aborting the run.
HUM-7554 agree verdict=already_fixed expected=Closed / Done-Errata actual=Closed / Done-Errata route=!=needs_change
HUM-7537 DISAGREE verdict=needs_human expected=Closed / Won't Do actual= route=n/a
disposition agreement: 1/2 (50%)
route agreement: 0/1 (0%), 1 undetermined
note: package facts (SRPM, SBOM, rpms repo) are current state, not the state at decision time, so agreement is retrospective
Only the Jira side is rewound. Package facts — the latest SRPM, the latest SBOM,
and the pinned rpms repository inputs — are read live, so an old ticket is
re-investigated against today’s package state rather than the state that existed
when the human decided. Where the fix has since shipped, already_fixed is
partly handed to the agent, so the score is retrospective current-state
agreement, not a historical replay; every report prints that caveat. True
replay needs as-of-cutoff SRPM and SBOM selection plus a date-pinned rpms
revision, and is deliberately left to a follow-up.
What the model sees
Only non-candidates reach the model. The prompt combines static /cve
instructions with validated TicketFacts: ticket, package and CVE details,
Jira content, analysis/discussion, built NVR, and SBOM identity.
The full SBOM is not included. Evidence is collected through bounded tools:
| Evidence | Tool | What it establishes |
|---|---|---|
| SBOM contents | search_sbom |
Searches SPDX metadata, including npm PURLs. Only present confirms shipping; ambiguous versions or unknown require investigation. |
| Packaging files | search_package_files, read_package_file |
Finds build settings, patches and packaging evidence. An empty search or missing CVE-named patch proves nothing by itself. |
| Other packages | lookup_shipped_package |
Returns the latest source-package SBOM, NVR, component version and binary presence. source_only or missing data is inconclusive. |
| Prepared source | read_package_source and source searches |
Inspects sandboxed, spec-pinned source and records provenance. built_nvr_match: unverified means it is not proven to match the ticket build. Preparation failures are not bypassed. |
| npm metadata | Registry lookup | Provides publication time, gitHead, repository and integrity. Metadata alone proves neither shipping nor fixedness. |
| CVE advisories | lookup_cve_advisory |
Returns affected packages, vulnerable ranges and patched versions from GitHub advisories. Missing, truncated or mismatched results are inconclusive. |
| Known upstream fix | check_upstream_fix |
Checks a commit patch against prepared source. Clean application means that exact diff is absent; already_applied means it reverse-applies cleanly. Other failures are inconclusive. |
| Upstream history | Upstream source probe | Searches an official repository once per ticket. Ancestry and applicability require a resolved exact base commit; unresolved candidates are not evidence. No candidate found means needs_human, not no_upstream_fix. |
Evidence rules
- Keep conclusions narrow. Patch applicability,
%filesentries, missing disable flags and failed searches alone do not prove affectedness, shipping or fix inclusion. Unresolved conclusions requireneeds_human. - Respect bounds. Tools enforce request, response and shared read budgets. Reads support line windows; excerpts may be truncated. Upstream access uses approved HTTPS hosts without redirect fallback.
- Do not repeat permanent failures.
retryable: falsemeans move on and record the gap. Source tools permit retrying witharchwhen architecture selection is required. - Preserve partial evidence. Upstream collection failures abort the run; unresolved refs remain partial evidence.
Model output
The model returns only status, confidence, summary, evidence and
vex_justification.
The caller supplies file_actions, commit_message, ai_generated, ticket,
package and cves, and replaces collection records with complete evidence.
Security
The agent processes untrusted input, including executable RPM macros and prep commands. The credential-bearing worker never executes or imports RPM specs.
- Isolated source execution. Spec parsing, structural edits, preparation,
and candidate application run in an ephemeral, uncredentialed sandbox. There
are no host mounts or worker secrets, and command environments are cleared.
Sandbox backends come from the shared Hummingbird Sandbox library.
Podman disables networking; Kubernetes requires deny-all egress policy in the
dedicated sandbox namespace. Privileged
mock, full builds, and runtime dependency installation are not used. Reusing a container with a reset workspace is not a fresh security boundary: use controlled images and packaging, not mutually hostile tenants. - Deployment isolation. Repository Git operations use private, temporary no-checkout object transport; package files exist only in the offline sandbox. The worker may manage execution pods only in the sandbox namespace once explicit API credentials are available. Those pods run non-root, drop capabilities, disable privilege escalation and token mounts, and use infrastructure-managed pod deadlines and resource limits sufficient for the sweep. Normal context-manager exit deletes sandbox pods; forced termination can leave pods until their active deadline, so operators must remove leftovers. Standalone investigations clean up at run end. Sandbox-produced source trees are never extracted into the worker.
- Secret-free repository subprocesses. Worker Git transport operations use a minimal, allowlisted environment and do not inherit the Jira or model tokens used by the trusted worker and sweep CLI.
- Uncredentialed, hardened git. Repository transport is public and read-only with
no push token. Credential helpers and interactive prompts are disabled,
system/global git config is neutralized (an attacker-controlled
GIT_CONFIG_GLOBALcannot change behaviour), and submodule recursion is off. - Input validation. Package names are restricted to a safe charset (no path
traversal or absolute paths), file reads are confined to the package
directory, saved patch names must be
*.patchand cannot overwrite existing files, and patch URLs must use HTTPS on a recognized upstream forge.
Development
See the main README for development workflows.
Build the local CVE sandbox image for explicit local configuration overrides:
podman build -f hummingbird-sandbox/images/cve/Containerfile \
-t localhost/hb-cve-sandbox:latest hummingbird-sandbox/
make hummingbird-cve-agent/setup
cd hummingbird-cve-agent
PYTHONPATH=.. python3 -m unittest discover -s tests -t .
# Opt into real offline RPM tests using an image with the required macros:
PYTHONPATH=.. HUMMINGBIRD_RPM_TEST_IMAGE=registry.example.com/rpm-prep:dev \
python3 -m unittest tests.test_rpm_integration
The integration suite uses Podman without network access. Only source downloads are mocked; RPM parsing, Go preparation, architecture conditions, source overlays, patch checks, and final-spec replay execute for real. Standard unit discovery skips these tests unless the image variable is set.
The Containerfile pins the RPM repository commit supplying hum in HUM_COMMIT
and records it in the io.hummingbird-project.hum-revision image label. Update
that pin explicitly when refreshing the metadata library. Its fetch layer comes
before Python dependencies, so dependency-only changes reuse the pinned checkout.
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.