This is the multi-page printable view of this section.
Click here to print.
Return to the regular view of this page.
Hummingbird Agent
An event-driven LLM agent that investigates CI/CD failures and posts findings
as GitLab merge request notes. The agent executes markdown-defined workflows
using tool calling, with all data processing running inside an isolated sandbox
container.
Model adapters, provider calls, and shared AI usage metrics come from the
in-process hummingbird-ai library.
For the architectural design rationale, module boundaries, security
invariants, and design decision registry, see
Agent Design. For agent turn flow and tool
handling, see Agent Model Loop. Provider
adapters and wire formats are documented in Hummingbird AI.
flowchart TD
Pipeline["Pipeline Fails"]
MREvent["MR Created / Updated"]
Slash["/hummingbird command"]
Pipeline -->|"event"| Agent
MREvent -->|"event"| Agent
Slash -->|"command"| Agent
subgraph Agent ["Hummingbird Agent"]
APIs["GitLab + Konflux +<br/>Testing Farm"]
Model["LLM<br/>(Gemini / Claude)"]
subgraph Sandbox ["Isolated Sandbox (no network)"]
Tools["jq / python3 / yq<br/>data processing"]
end
APIs <-->|"data"| Model
Model <-->|"commands"| Tools
end
Note["MR Note<br/>(analysis or review)"]
Agent -->|"posts"| Note
Note -->|"reply to continue"| Agent
Features
- Workflow-driven analysis - Investigation logic lives in
.md files,
not in code; easy to iterate without redeployment
- Centralized YAML config - A single config file defines operational
settings, workflows, enabled data sources with token env var names,
project allowlists, and per-project limits
- Sandboxed execution - All untrusted commands (
jq, python3, shell)
run in an isolated container, never on the host
- Five sandbox backends - Podman for local development (network-isolated),
direct K8s pod creation, Deployment-backed pod pool for low-latency
production use (restricted-v2 SCC compliant), KubeVirt VM (direct), or
VMIRS-backed VM pool for heavier workloads requiring full OS isolation
- Data source abstraction - GitLab, Konflux, and Testing Farm are registered
as tool-calling functions the model invokes directly
- Auto-spill for large outputs - Stdout/stderr and data source responses
exceeding 4 KB are automatically saved to sandbox files with a compact preview
returned to the model, keeping context usage bounded
- Prompt caching - Gemini uses implicit server-side caching automatically;
Claude uses explicit sliding-window cache breakpoints that reduce input
token costs by ~80% on multi-turn agent runs
- Token budget management - Per-call context ceiling and iteration-based
soft/hard limits prevent runaway sessions
- Session persistence - Conversation history, transcript, and sandbox
files saved to S3 (production) or local directory (development) for
debugging and future session resumption. Sessions are stored in a
provider-neutral format, enabling model switching between conversations.
Architecture
flowchart LR
subgraph input [Input]
SQS["SQS Queue"]
CLI["CLI --event"]
end
subgraph agentLoop [Agent Loop]
WF["Workflow .md<br/>(system prompt)"]
LLM["LLM<br/>(Gemini / Claude)"]
TR["Tool Registry"]
end
subgraph tools [Tools]
SE["sandbox_exec"]
FTS["fetch_to_sandbox"]
DS["Data Sources"]
end
subgraph sandbox [Sandbox Container]
JQ["jq / python3 / yq"]
Files["Spilled files"]
end
subgraph external [External APIs]
GL["GitLab API"]
KX["Konflux K8s"]
TF["Testing Farm"]
end
subgraph output [Output]
Note["GitLab MR Note"]
Session["Session State"]
end
SQS --> WF
CLI --> WF
WF --> LLM
LLM -->|"tool calls"| TR
TR --> SE --> sandbox
TR --> FTS --> sandbox
TR --> DS
DS --> GL
DS --> KX
DS --> TF
DS -->|"auto-spill"| sandbox
LLM -->|"final text"| Note
LLM --> Session
An event (CLI --event JSON or SQS message) identifies a GitLab project and
MR IID. The config file maps project paths to workflows and provides
action, max_iterations, enabled data sources, and token env var names.
Both run and serve use this config. The markdown body of the prompt file
becomes the LLM system prompt. The agent
loop iterates until the model produces a final text response or hits the
iteration/context limit. Tool calls are dispatched through the ToolRegistry:
sandbox_exec runs shell commands, fetch_to_sandbox pipes data source
output into the sandbox, and direct data source calls return results inline
(or auto-spill large responses to files).
For a detailed walkthrough of the model loop – what gets sent to the model each
iteration, how tool calls flow, the exact wire format, and how user replies
integrate for session resumption – see
Agent Model Loop.
Event-Driven Triggers
In production, the agent consumes events from an SQS queue subscribed to the
central SNS topic. The SNS filter policy delivers three event types:
-
gitlab::pipeline – Fires when a GitLab CI pipeline completes. The
agent triggers on status=failed pipelines from merge_request_event
sources where the triggering user has at least Developer access on the
project. Only workflows with trigger: pipeline are executed. Since the
pipeline stays open until all Konflux external stages resolve, this
naturally waits for all builds and tests to finish before triggering.
-
gitlab::merge_request – Fires on MR open and update events.
Only workflows with trigger: merge_request are executed. The agent
triggers in three cases: a new MR is opened (action=open), code is
pushed to an existing MR (action=update with oldrev), or a draft MR
is marked as ready (action=update with a changes.draft transition
from true to false). All other update events (labels, assignees,
description changes) are skipped – they carry no new code. Draft MRs
are skipped; SHA-based deduplication prevents reviewing the same code
revision twice. Access checks and trigger_rules filtering evaluate the
push author: for open and push events the webhook user is the pusher;
for undraft transitions the agent resolves the actual pusher from MR
system notes so that the person undrafting cannot bypass the access check
on code pushed by an untrusted user.
-
gitlab::note – MR comment events. Two sub-flows:
- Slash command (
/hummingbird <workflow-name>): triggers a specific
workflow. Prefix matching is supported (e.g. /hummingbird analyze
matches analyze-failures). /hummingbird or /hummingbird help lists
available workflows. The note author must have Developer+ access on the
project. Optional runtime overrides can be appended:
/hummingbird code-review model=claude-sonnet-4-6 max_iterations=25.
- Reply to agent note: when a user replies to an existing agent note
(which contains a session marker), the agent loads the previous session
from S3 (conversation history + sandbox files) and continues the
conversation with the user’s reply as input. The system prompt includes
a
CONTINUATION_PROMPT that prevents the model from re-running the
full workflow. If the session is not found (expired/deleted), the agent
falls back to a cold start. Replies may include overrides on a separate
line (e.g. /hummingbird model=claude-opus-4-6); override lines are
stripped from the user message. Overrides persist in the session until
explicitly changed.
Notes generated by the agent itself (containing session markers) are
skipped to prevent infinite loops. Reply threading respects the
internal_notes config: if the project requires internal notes but the
original thread was public, the reply is posted as a new top-level
internal note instead.
Both pipeline and merge_request triggers apply per-workflow trigger
rules filtering. Each workflow may define an ordered trigger_rules
array; the first matching rule decides whether the event is allowed or
denied (implicit deny on fallthrough). Rules can match on pipeline
status, user_regex/user_regexes, branch_regex/branch_regexes,
and title_regex/title_regexes. All conditions within a rule are ANDed;
multiple patterns within a regex field are ORed. The ! prefix negates a
pattern. If no trigger_rules are specified, merge_request triggers
allow all events and pipeline triggers default to allowing only failed
status (preserving backward compatibility).
The legacy ignore_users and ignore_branches fields are still accepted
and automatically desugared into equivalent trigger_rules (deny +
catch-all allow). They cannot be mixed with explicit trigger_rules.
Rate limiting is per-workflow: each workflow’s thread count is tracked
independently via JSON session markers that embed the workflow name and
commit SHA. The max_runs_per_mr limit applies separately to each
workflow on a given MR.
Events flow through a two-stage SQS pipeline. A slim ingress router
forwards webhooks from the standard queue to an SQS FIFO queue, grouped
by discussion_id for note events (ensuring same-discussion ordering)
and by SQS MessageId for pipeline/MR events (no serialization needed).
Phase 1 handlers validate the event and resolve the session, then post a
continuation message back to the FIFO grouped by session_id. Phase 2
picks up the continuation and runs the workflow. This ensures all work
for a given session is serialized even across multiple discussion threads.
The SQS infrastructure is defined in template.yaml (SAM/CloudFormation):
a standard ingress queue (60s visibility timeout for the router hop) and
a FIFO work queue (30-minute visibility timeout for workflow execution).
Model Configuration
The agent supports multiple LLM providers via a unified adapter interface.
Gemini
- API key (local dev) – Set
GOOGLE_API_KEY. Calls
generativelanguage.googleapis.com directly. No region configuration needed.
- Vertex AI (production) – Set
GOOGLE_CLOUD_PROJECT and configure
model_regions in the YAML settings. Uses Application Default Credentials
(ADC) via google-auth. For OpenShift, mount a service account key JSON
file and set GOOGLE_APPLICATION_CREDENTIALS, or use workload identity.
Gemini uses implicit server-side caching – repeated prefixes are
automatically cached by the Vertex AI backend with no opt-in required.
Cached input tokens are billed at 25% of the base input rate. The agent
tracks cached token counts from API responses for cost estimation.
Anthropic Claude
Claude models are accessed via Vertex AI using the same GCP project
(GOOGLE_CLOUD_PROJECT) with the region resolved from model_regions.
Enable the desired model in the GCP Model Garden. Use
model: claude-sonnet-4-20250514 in workflow config.
Prompt caching is enabled automatically for Claude models. The agent places
explicit cache breakpoints on messages so that the full conversation prefix
is served from cache on every turn after the first. Cache writes cost 1.25x
the base input rate; cache reads cost 0.10x (90% discount). For a typical
20-iteration agent run, this reduces input token costs by approximately 80%.
Ephemeral messages (iteration warnings, nudges) are excluded from cache
writes to avoid polluting the cache with transient content.
Region configuration
GCP regions are configured via settings.model_regions in the config file –
a map of model name prefixes to GCP regions. The agent resolves the region by
longest-prefix match on the effective model name (same algorithm as cost
estimation). Example:
settings:
model_regions:
gemini-2.5-pro: us-east5
gemini-3.1-pro: global
claude: global
At least one of GOOGLE_API_KEY or GOOGLE_CLOUD_PROJECT must be set. If both
are set, the API key takes precedence for Gemini models. Claude models always
require Vertex AI mode (GOOGLE_CLOUD_PROJECT + model_regions);
GOOGLE_API_KEY direct mode is for Gemini only.
Prerequisites
- Python 3.11+
- Podman for local sandboxes, or
kubectl and Kubernetes access to a dedicated
sandbox namespace for K8s sandboxes.
- Authentication:
GOOGLE_API_KEY or GOOGLE_CLOUD_PROJECT (see above)
- For Claude models: enable the desired model in GCP Model Garden
- GitLab tokens referenced in the config file (model tool tokens, orchestrator tokens)
- Kubeconfig with access to the Konflux cluster (if using Konflux data sources)
Installation
cd hummingbird-agent
pip install -e .
Usage
Local development (run)
The run command uses the config file for workflow lookup, data source
registration, and token resolution – the same code path as serve. By
default, results are printed to stdout (dry-run). Use --execute to post
the result as a GitLab MR note.
There are two ways to select what to run:
Direct workflow selection (--workflow + --project + --event):
# Run a specific workflow on an MR, Podman sandbox, print to stdout
hummingbird-agent run \
--workflow analyze-failures \
--project org/group/project \
--event '{"iid": 123, "sha": "abc123"}'
# Same but with K8s sandbox
hummingbird-agent run \
--workflow analyze-failures \
--project org/group/project \
--event '{"iid": 123, "sha": "abc123"}' \
--context my-cluster/my-namespace
# Post the result as a GitLab MR note (also saves session to S3 if configured)
hummingbird-agent run \
--workflow analyze-failures \
--project org/group/project \
--event '{"iid": 123, "sha": "abc123"}' \
--execute
# Save session locally for debugging (context.json, transcript.md, sandbox.tar.gz)
hummingbird-agent run \
--workflow analyze-failures \
--project org/group/project \
--event '{"iid": 123, "sha": "abc123"}' \
--save-session /tmp/my-session
# Resume from a saved session with a follow-up question
hummingbird-agent run \
--workflow analyze-failures \
--project org/group/project \
--event '{"iid": 123, "sha": "abc123"}' \
--resume-session /tmp/my-session \
--message "Can you look at the clair-scan timeout more closely?"
# Chain: resume and save the new session for another round
hummingbird-agent run \
--event-file event.json \
--resume-session /tmp/my-session \
--message "What layer hash failed?" \
--save-session /tmp/my-session-2
Event-file replay (--event-file): routes the event through the same
project-index lookup as serve, but skips rate limiting and status filtering:
# Replay a real webhook event, dry-run with Podman
hummingbird-agent run \
--event-file event.json
# Replay with K8s sandbox and post notes
hummingbird-agent run \
--event-file event.json \
--context my-cluster/my-namespace \
--execute
Production (serve)
# Poll SQS queue for events (pool sandbox by default, posts results as MR notes)
CONFIG_PATH=config.yml hummingbird-agent serve
Requires CONFIG_PATH pointing to a config file with settings.sqs_queue_url
and settings.sqs_fifo_queue_url set. The config file defines which workflows
run on which projects, with per-project limits and data source token mappings.
Handles SIGTERM/SIGINT for graceful shutdown. A background router thread
forwards events from the standard queue to the FIFO; the main thread consumes
the FIFO with a semaphore-gated thread pool (controlled by
settings.max_concurrent_agents) so excess messages stay in SQS for other
instances.
Config hot-reload: In serve mode a background thread polls the config file
for changes (every 5 seconds by default). When the file changes, the new
config is validated and atomically swapped in – subsequent event dispatches
use the updated config. If the new config is invalid, the previous config is
kept and a warning is logged. No restart required for config changes.
The serve command also accepts --sandbox, --context, and --namespace
for local development with a different sandbox backend (e.g. Podman).
CLI Options
run subcommand:
| Option |
Description |
Default |
--event |
Inline event JSON string (mutually exclusive with --event-file) |
- |
--event-file |
Read event from a JSON file (mutually exclusive with --event) |
- |
--workflow |
Workflow name from config file (requires --project) |
- |
--project |
GitLab project path (requires --workflow) |
- |
--execute |
Post result as GitLab MR note and save session to S3 |
- |
--save-session |
Save session artifacts to this directory |
- |
--resume-session |
Resume from a saved session directory (requires --message) |
- |
--message |
Follow-up message for session resumption (requires --resume-session) |
- |
--sandbox |
Sandbox backend (podman, k8s, k8spool, kubevirt, or kubevirtpool) |
podman |
--context |
K8s context (implies K8s backend) |
- |
--namespace |
K8s namespace |
- |
-v, --verbose |
Enable debug logging |
- |
serve subcommand:
| Option |
Description |
Default |
--sandbox |
Sandbox backend (podman, k8s, k8spool, kubevirt, or kubevirtpool) |
k8spool |
--context |
K8s context (implies K8s backend) |
- |
--namespace |
K8s namespace |
- |
-v, --verbose |
Enable debug logging |
- |
Configuration
Config file
Both run and serve use a single YAML config file. Set the path via
CONFIG_PATH (default: config.example.yaml). The config has two sections:
settings for operational parameters, and workflows for workflow
definitions. Most settings have defaults; configure the sandbox image and
namespace as needed for the selected backend.
Sandbox options for settings.sandbox are documented by the shared
Hummingbird Sandbox library.
The dedicated agent execution image is not yet published or selected in runtime
configuration. See sandbox images for build and rollout details.
settings:
gitlab_url: https://gitlab.com # GitLab instance URL
sandbox: {}
max_concurrent_agents: 4 # max concurrent workflows (serve)
sqs_queue_url: "" # standard SQS ingress queue URL (serve)
sqs_fifo_queue_url: "" # FIFO work queue URL (serve)
s3_session_bucket: "" # S3 bucket for session persistence
model: gemini-3.1-pro-preview # or claude-sonnet-4-20250514
model_regions: # model prefix -> GCP region
gemini-2.5-pro: us-east5
gemini-3.1-pro: global
claude: global
max_iterations: 30 # default iteration limit
max_runs_per_mr: 5 # default per-MR rate limit
internal_notes: true # default note visibility
docs_url: https://gitlab.com/org/group/project/-/blob/main/docs/agent.md
source_url: https://gitlab.com/org/group/project
slack_url: https://slack.example.com/archives/C0123456789
slack_label: "#my-channel"
workflows:
code-review:
trigger: merge_request # auto-trigger on MR events
description: Performs AI-powered code review
sandbox: k8spool # per-workflow sandbox override (optional)
workflow_url: https://gitlab.com/org/group/project/-/blob/main/workflows/code-review.md
trigger_rules: # ordered rule chain, first match wins
- user_regex: "renovate\\[bot\\]" # deny bot MRs (regex fullmatch)
action: deny
- branch_regex: "chore/.*" # deny maintenance branches
action: deny
- action: allow # allow everything else
prompt: workflows/code-review.md
action: post_gitlab_note
model: gemini-3.1-pro-preview # or claude-sonnet-4-20250514
max_iterations: 15
max_inline_size: 200000 # keep full diffs in context
context_limit: 500000 # Gemini 3.1 Pro / Claude Sonnet 4 have large context windows
data_sources:
gitlab:
token_env: GITLAB_TOKEN_RO
projects:
org/group/project: {}
analyze-failures:
trigger: pipeline # auto-trigger on failed pipelines
description: Investigates CI/CD pipeline failures
sandbox: kubevirtpool # use VM sandbox for heavier workloads
auto_resolve_on_push: true # resolve threads when a new SHA is pushed
auto_resolve_on_success: true # resolve threads when pipeline succeeds
workflow_url: https://gitlab.com/org/group/project/-/blob/main/workflows/analyze-failures.md
prompt: workflows/analyze-failures.md # relative to config file dir
action: post_gitlab_note
model: gemini-3.1-pro-preview # or claude-sonnet-4-20250514
max_iterations: 50 # per-workflow iteration override
data_sources: # model tool tokens (read-only)
gitlab:
token_env: GITLAB_TOKEN_RO # env var name, not the token
konflux:
cluster_url: https://example.com:6443/ns/my-tenant
kubeconfig_env: KUBECONFIG
kubearchive_url: https://kubearchive-api-server-product-kubearchive.apps.example.com
testing_farm: {}
projects:
redhat/hummingbird/containers:
tokens: # per-project model token overrides
gitlab: GITLAB_TOKEN_CONTAINERS_RO
action_tokens: # per-workflow write tokens
gitlab: HUMMINGBIRD_AGENT_ACTION_ANALYZE_FAILURES_GITLAB_TOKEN_CONTAINERS
With --workflow/--project, the workflow and project are looked up directly
in the config. With --event-file, the project is extracted from the event
body and matched against the project index to find applicable workflows.
Pool overrides in this example require the corresponding pool to be provisioned
and selected with --sandbox at startup. Only the CLI-selected pool is
initialized, so these two pool overrides cannot both be used in one service.
Without a matching pool, a workflow can fall back to a direct sandbox instead.
Discussion threads
All workflow results are posted as discussion threads: a placeholder note
starts the discussion and the full result is posted as a reply. The
placeholder is never edited, so email notifications include the actual
result text. For slash commands, the result is posted as a reply in the
triggering discussion.
Auto-resolve
Workflows can opt in to automatic resolution of their discussion threads:
auto_resolve_on_push (default false): When a new commit is pushed
to the MR (i.e. a merge_request event with action: update), all agent
discussion threads for the workflow whose SHA differs from the new HEAD are
resolved. This clears stale failure analyses when the developer pushes a
fix.
auto_resolve_on_success (default false): When the head pipeline
succeeds, all agent discussion threads for the workflow on that MR are
resolved (regardless of SHA). This handles pipeline reruns on the same
SHA where a transient failure is now green.
Both flags are independent and can be combined. Resolution runs before rate
limit checks, so threads are resolved even if the workflow’s per-MR run
limit has been reached.
Web search
Workflows can enable Anthropic’s built-in web search server tool by
listing web_search as a data source:
workflows:
renovate-babysit:
model: claude-sonnet-4-20250514
data_sources:
gitlab: {}
web_search: {}
# ...
When web_search is present in data_sources, the web_search_20250305
server tool is included in API requests to Claude. The model decides
autonomously when to search. Search execution happens server-side (no
client-side tool dispatch), and results appear as server_tool_use /
web_search_tool_result content blocks in the response. These blocks
are preserved through session save/resume.
Unlike other data sources, web_search has no configuration options and
does not register any orchestrator-side tools – it is handled entirely
by the model provider.
If the API returns a pause_turn stop reason (server-side search loop
hit its iteration limit), the adapter automatically re-sends the
conversation to continue, up to 5 continuations per generate() call.
Web search is only supported with Claude models. The setting is ignored
for Gemini.
The first agent-authored note (placeholder) includes a footer with links to
documentation, source code, the Slack channel, the workflow prompt, and a
continuation prompt. These links are configured via global settings:
| Setting |
Description |
settings.docs_url |
Link to agent documentation |
settings.source_url |
Link to agent source repository |
settings.slack_url |
Link to support Slack channel |
settings.slack_label |
Display text for Slack link (default: “Slack”) |
Per-workflow, set workflow_url to link to the workflow’s prompt file.
The footer stays on the placeholder and the result is a separate reply.
Token separation
Tokens are split into three tiers that never mix:
- Model tool tokens (in YAML
data_sources / tokens): read-only tokens
passed to the LLM’s tool calls. Declared in the config file as env var
names. These are user-defined and resolved at runtime from the referenced
env vars. Create as project access tokens with Reporter role and
read_api scope. Reporter is the minimum role required to see internal
(confidential) notes in the discussions tool.
- Workflow action tokens (in YAML
action_tokens, per-project): write
tokens used by the orchestrator for per-workflow GitLab writes (notes,
thread resolution). Each workflow gets a dedicated bot user per project,
providing clear audit trails for which agent produced each note. Declared
in the project config as env var names. Create as project access tokens
with Developer role and api scope. These tokens are never exposed to
the LLM and never enter the ToolRegistry. Naming convention:
HUMMINGBIRD_AGENT_ACTION_<WORKFLOW>_GITLAB_TOKEN_<PROJECT>.
- Orchestrator tokens (
ORCHESTRATOR_* env vars, NOT in YAML):
tokens used by the runner for operational reads (member access checks,
push author lookup, head pipeline queries) and infrastructure notes
(access-denied replies, rate-limit notices). Create as project access
tokens with Developer role and read_api scope (or api for
infrastructure notes). Developer role is required because the discussions
tool trust-filters notes by author access level (>= Developer); if the
orchestrator bot has only Reporter access, its own notes are redacted.
Resolved by convention:
ORCHESTRATOR_GITLAB_TOKEN_<MANGLED_PROJECT> (per-project) or
ORCHESTRATOR_GITLAB_TOKEN (fallback). The ORCHESTRATOR_ prefix makes
these impossible to confuse with model tokens.
Environment Variables
The agent reads only secrets and authentication from environment variables.
All operational settings come from the config file’s settings section.
| Variable |
Required |
Default |
Description |
CONFIG_PATH |
no |
config.example.yaml |
Path to config YAML |
GOOGLE_API_KEY |
yes* |
- |
Gemini API key; Gemini direct mode only (not for Claude) |
GOOGLE_CLOUD_PROJECT |
yes* |
- |
GCP project ID (Vertex AI mode); required for Claude models |
ORCHESTRATOR_GITLAB_TOKEN |
serve |
- |
Orchestrator GitLab token (global fallback) |
ORCHESTRATOR_GITLAB_TOKEN_<PROJECT> |
no |
- |
Per-project orchestrator token |
SENTRY_DSN |
no |
- |
Sentry DSN for error tracking |
*One of GOOGLE_API_KEY or GOOGLE_CLOUD_PROJECT is required. Claude models
require Vertex AI (GOOGLE_CLOUD_PROJECT); GOOGLE_API_KEY is for Gemini
direct mode only.
Model tool tokens (e.g. GITLAB_TOKEN_RO, GITLAB_TOKEN_CONTAINERS_RO) and
data source credentials (e.g. KONFLUX_CLUSTER_URL, KUBECONFIG) are
referenced by name in the config file’s data_sources and tokens sections.
They are not listed in the table above because their names are user-defined.
Sandbox Security
Sandbox isolation, security constraints, permissions, and backend behavior are
documented in the shared Hummingbird Sandbox library. The agent
continues to own workflow selection, CLI options, session persistence, and the
decision to acquire or reuse a sandbox.
Data Sources
Data sources are registered as tool-calling functions. The model invokes them
by name; the orchestrator executes them and returns results (or auto-spills
large responses to the sandbox).
GitLab
| Tool |
Description |
gitlab_get_mr_details |
MR metadata (title, author, state, SHA, labels) |
gitlab_get_mr_unified_diff |
Complete unified diff in patch format |
gitlab_get_mr_diff |
Per-file structured change data |
gitlab_get_mr_commits |
List of commits in a merge request |
gitlab_get_mr_discussions |
Discussion threads with redacted agent transcripts and trust-filtered comments |
gitlab_get_commit_statuses |
CI/CD pipeline statuses for a commit |
gitlab_get_file_at_ref |
Raw file content at a git ref |
gitlab_get_repo_archive |
Repository tar.gz (binary, auto-spilled) |
gitlab_get_job_log |
CI job trace output (ANSI codes stripped) |
Konflux
Fetches Tekton PipelineRuns and TaskRuns from both the live K8s API and
Kubearchive (for completed resources), with deduplication by UID.
| Tool |
Description |
konflux_list_pipelineruns |
PipelineRuns for a commit SHA; optional status allowlist filter |
konflux_list_taskruns |
TaskRuns by commit SHA or parent PipelineRun name; optional status filter |
konflux_list_pods |
Pods for a commit SHA; optional status allowlist filter |
konflux_get_pod |
Full pod resource (spec, status, conditions, container statuses) |
konflux_get_pod_log |
Pod logs; optional container param, fetches all containers when omitted |
Status filtering: All three list tools accept an optional status
parameter — an allowlist of status.conditions[0].status values
("True" = success, "False" = failure, "Unknown" = in-progress).
Only items matching the allowlist are written to the sandbox file. Response
metadata always includes counts (all items seen, by reason), included,
and excluded — the agent knows the full reason breakdown even when fetching
only failures. Items without conditions always pass.
Per-PLR TaskRun fetching: konflux_list_taskruns accepts a pipelinerun
parameter. When set, it uses tekton.dev/pipelineRun=<name> as a server-side
label selector, returning only the ~15-21 TaskRuns for that PipelineRun (~600 KB)
instead of all TaskRuns for the commit (~530 MB for 340 components).
Recommended pattern for analyze-failures: Fetch failing PLRs and TaskRuns
together in one batch with status=["False"]. This captures every failure
reason (Failed, Cancelled, PipelineRunTimeout, StepFailed, etc.)
without enumerating them. Use response metadata counts for the summary.
Use per-PLR fetch only when deeper investigation is needed.
Response metadata includes konflux_ui base URL for building reviewer-facing
links.
Testing Farm
| Tool |
Description |
tf_get_results |
JUnit XML results for a request ID |
tf_get_test_log |
Individual test log by URL (restricted to Testing Farm artifact URLs) |
tf_get_request_status |
Request state, queue/run times |
Response metadata includes artifacts_base URL for building artifact links.
Workflow System
Workflows are .md files whose content becomes the LLM system prompt
verbatim. Workflow metadata (action, model, max_iterations, enabled
data sources, project allowlists) is defined in the config file. The .md
file is pure system prompt text.
Available workflows:
analyze-failures.md - Investigates CI/CD pipeline failures by fetching
MR details, identifying failed pipelines via commit statuses, retrieving
PipelineRuns/TaskRuns from Konflux, analyzing test results from Testing Farm,
and producing a grouped root-cause report with reviewer-facing URLs.
code-review.md - Performs AI-powered code review by fetching MR details,
the unified diff, and prior discussion threads in parallel, then attempting
to load per-project rules from workflows/repo-rules/ (a mandatory step –
rules take precedence over generic standards when present) and producing
structured feedback with severity ratings, code examples, and actionable
suggestions. On follow-up reviews (after SHA updates), the agent sees its
own previous findings, developer responses, and resolved threads – avoiding
duplicate findings and respecting developer explanations. Uses elevated
max_inline_size (200 KB) and context_limit (500K tokens) to keep the
full diff in context.
renovate-babysit.md - Triages a single Renovate-authored MR triggered
by a successful pipeline. Classifies the MR as safe or risky based on diff
scope, upstream dependency changes (from MR description, web search), and
local codebase impact (via gitlab_get_repo_archive + grep). Posts a
structured note with verdict, upstream change summary, and suggested actions
for risky MRs. Requires a Claude model with web_search data source.
Token Budget Management
Prompt caching, cached token accounting, and provider-specific cost estimates
are implemented in hummingbird-ai.
The agent uses a dual-limit approach instead of a cumulative token budget:
- Iteration limit (
settings.max_iterations, default 30,
overridable per-workflow) - Hard cap on tool-calling rounds. A wrap-up
prompt is injected at 80% (SOFT_ITERATION_RATIO).
- Context limit (
CONTEXT_LIMIT, default 60,000 tokens, overridable
per-workflow via context_limit) - Per-call input token ceiling. When
exceeded, a wrap-up prompt forces the model to finalize.
Large outputs are automatically redirected to sandbox files to keep the LLM
context small. The spill threshold defaults to 4 KB (MAX_INLINE_SIZE) but
can be overridden per-workflow via max_inline_size in the config:
sandbox_exec - stdout/stderr exceeding the threshold saved to
/tmp/_out/{N}.txt; model receives a preview (head + tail) with file path
- Data sources - text exceeding the threshold saved to
/tmp/_out/{name}_{N}.txt with preview; binary data saved to .bin
fetch_to_sandbox - always writes to the caller-specified path; returns
metadata only
All tuning constants are centralized in config.py:
| Constant |
Default |
Purpose |
DEFAULT_MAX_ITERATIONS |
30 |
Hard iteration cap |
SOFT_ITERATION_RATIO |
0.8 |
Inject wrap-up at this fraction |
CONTEXT_LIMIT |
60,000 |
Per-call input token ceiling |
OUTPUT_PREVIEW_BYTES |
4,096 |
Preview size for spilled outputs |
OUTPUT_TAIL_BYTES |
512 |
Extra tail appended to previews |
MAX_INLINE_SIZE |
4,096 |
Max inline size for data source responses |
Evaluation Framework
The agent includes an evaluation framework for scientific measurement of
prompt and model quality. See
Hummingbird Agent Evals for the full guide
covering the shared hummingbird_agent.eval library, existing
evaluations, and how to write new ones.
Development
See the main README for development workflows.
make hummingbird-agent/setup # Install dependencies
make check # Lint code (ruff)
make fmt # Format code
make test # Run unit tests
make coverage # Run tests with coverage
License
This project is licensed under the GNU General Public License v3.0 or later -
see the LICENSE file for details.
1 - Hummingbird Agent Model Loop
How the Hummingbird agent runs conversation turns, calls tools, and integrates
user replies.
For the architectural design rationale behind the loop (why full history
replay, why iteration-based budgets, why nudge via system prompt), see
Agent Design – Section 6.
For operational usage documentation, see
Hummingbird Agent.
Provider adapters, wire formats, canonical message conversion, and caching are
documented in Hummingbird AI.
What gets sent to the model
Every call to the model API sends three things:
1. System instruction
A single text string, rebuilt every iteration. The adapter maps it to the
active provider’s request format. It is composed of:
BASE_SYSTEM_PROMPT # agent.py: sandbox rules, tool usage tips
+ tool_notes # per-data-source notes from ToolRegistry
+ workflow_prompt # full content of e.g. workflows/analyze-failures.md
Near the iteration or context limit, a warning suffix is appended to the
system prompt for that call. The suffix escalates through three levels:
ITERATION_WARNING / CONTEXT_WARNING – soft “start wrapping up” at 80%
of the iteration or context limit. Tools remain available.
FINAL_TURN_WARNING – hard stop on the absolute last iteration or after
the context limit is exceeded. Combined with tool-calling disabled for that
request to mechanically prevent further tool calls.
EMPTY_RESPONSE_NUDGE – appended when the model returns an empty response
(no text, no tool calls). Tools remain available.
2. contents
A list of messages representing the full conversation history. It grows
every iteration. Provider-specific content conversion is handled by the
adapter; see Hummingbird AI.
Tool definitions, static across all iterations. Adapters translate the shared
tool definition into the provider’s schema.
On the final turn (last iteration or after context limit exceeded), the
request disables tools to mechanically prevent function calls. The active
adapter applies the provider-specific setting.
This is only sent when FINAL_TURN_WARNING is active. All other iterations
omit this constraint, allowing the model to choose freely between text and tools.
The agent loop, turn by turn
The loop in run_agent_loop runs up to max_iterations times. Each
iteration is one round-trip to the model.
Initialization
system_prompt = BASE_SYSTEM_PROMPT + tool_notes + workflow_prompt
tool_defs = [sandbox_exec, fetch_to_sandbox, fetch_batch_to_sandbox, <data sources...>]
contents = [user: {"project": "org/repo", "iid": 42, "sha": "abc...", "session_id": "uuid"}]
Iteration 1
-> Send: system_instruction + contents (1 item) + tool_defs
<- Response: tool_call(gitlab_get_mr_details, {project: "org/repo", iid: 42})
contents.append(model: response.raw_content) # opaque provider content
execute tool -> result = {"result": "{\"title\": \"Fix auth\",...}"}
contents.append(tool: model.make_tool_responses([(name, result)]))
Iteration 2
-> Send: system_instruction + contents (3 items) + tool_defs
<- Response: tool_call(fetch_batch_to_sandbox, {requests: [...]})
contents.append(model: ...)
execute tool -> result = {"results": [{"saved_to": "/tmp/data/pipelineruns.json", "bytes": 85432}]}
contents.append(tool: ...)
Iteration 3
-> Send: system_instruction + contents (5 items) + tool_defs
<- Response: tool_call(sandbox_exec, {command: "jq '[...]' /tmp/data/pipelineruns.json"})
contents.append(model: ...)
execute tool -> result = {"exit_code": 0, "stdout": "...(preview)...", "stdout_file": "/tmp/_out/0.txt"}
contents.append(tool: ...)
Iteration N (final)
-> Send: system_instruction + contents (2N-1 items) + tool_defs
<- Response: text("## Konflux Failure Analysis\n\n...") // text + no tool calls = DONE
contents.append(model: {text: "## Konflux Failure Analysis..."})
BREAK -- return (text, usage, transcript, contents)
Response handling and termination
Each iteration, the model’s response can contain text, tool calls, both, or
neither. The agent handles each case:
| Response |
Action |
| Text only |
Final response. Save text, break. (Happy path.) |
| Text + tool calls |
Save text as last_text fallback, execute the tool calls, continue loop. The model is thinking aloud while also acting. |
| Tool calls only |
Execute the tool calls, continue loop. |
| Neither |
Empty response – retry up to MAX_EMPTY_RETRIES (2) times. Each retry nudges the model via EMPTY_RESPONSE_NUDGE appended to the system prompt (not injected as a user message, to keep the contents list clean). Tools remain available during nudge retries. |
The text + tool calls case is worth noting: the model’s text is not
returned to the user immediately. It is stored as last_text (a fallback in
case the loop terminates later without a clean final text, e.g. by hitting max
iterations). The provider response is appended to contents unchanged; its
wire format is documented in Hummingbird AI.
The loop terminates on:
- Text only – the model produced a final response.
- Max iterations reached – the final iteration uses
FINAL_TURN_WARNING and disables tools through the adapter to force text output. Falls back
to whatever last_text was seen, or "[Agent did not produce a final response]".
- Context limit exceeded –
input_tokens >= CONTEXT_LIMIT sets
context_exceeded, making the next iteration final (same as #2). A soft
warning (CONTEXT_WARNING) fires earlier at 80% of the limit.
- Fatal model error – a provider error after shared retry handling is
exhausted. Error classification and retries are described in
Hummingbird AI.
- Empty responses exhausted –
MAX_EMPTY_RETRIES (2) nudge retries
failed.
- Unexpected error – any exception not handled by the retry mechanism
(e.g. malformed API response) breaks the loop and returns partial results.
Accumulated contents, transcript, and sandbox are preserved and the
session is saved normally.
Budget escalation
Both iterations and context size use the same two-tier pattern:
|
Soft warning (80%) |
Hard stop (100%) |
| Iterations |
ITERATION_WARNING at 80% of max_iterations |
FINAL_TURN_WARNING on last iteration |
| Context |
CONTEXT_WARNING at 80% of CONTEXT_LIMIT |
FINAL_TURN_WARNING on next iteration after exceeding CONTEXT_LIMIT |
Soft warnings are advisory (“start wrapping up”) and keep tools available.
The hard stop disables tools through the active adapter to mechanically
prevent further tool calls, ensuring the model produces text.
The contents list in detail
The agent treats each entry in contents as opaque adapter content during
execution. Provider representations and canonical conversions are documented
in Hummingbird AI.
model.make_user_content() constructs the initial event turn and
model.make_tool_responses() appends results for dispatched calls. The agent
stores the provider response as raw_content and appends it unchanged.
Typical 8-iteration conversation shape
contents[0] = user: {"project":"org/repo","iid":42,...} # initial event
contents[1] = model: tool_call(gitlab_get_mr_details) # iter 1
contents[2] = tool: result(gitlab_get_mr_details) # iter 1
contents[3] = model: tool_call(fetch_batch_to_sandbox) # iter 2
contents[4] = tool: result(fetch_batch_to_sandbox) # iter 2
contents[5] = model: tool_call(sandbox_exec) # iter 3
contents[6] = tool: result(sandbox_exec) # iter 3
...
contents[13] = model: tool_call(sandbox_exec) # iter 7
contents[14] = tool: result(sandbox_exec) # iter 7
contents[15] = model: text("## Konflux Failure Analysis...") # iter 8 (final)
This entire list is persisted as canonical messages in context.json; the
adapter converts to and from its provider format at the persistence boundary.
See Hummingbird
AI for
details.
When the model’s response contains tool calls, _execute_tool_calls
iterates them sequentially:
tool_registry.execute(ToolCall(name, args))
match name:
"sandbox_exec" -> runs sh -c in container -> {exit_code, stdout, stderr}
"fetch_to_sandbox" -> calls data source func -> writes to sandbox file -> {saved_to, bytes}
"fetch_batch_to_sandbox" -> multiple fetch_to_sandbox in one call
any data source name -> calls func directly -> returns inline or auto-spills large output
Large output handling
When stdout or a data source response exceeds 4KB, it is automatically saved
to a sandbox file (/tmp/_out/N.txt) and only a preview (head + tail) is
returned to the model. The model gets the file path and can use sandbox_exec
with jq/grep/head to process it. This keeps the context window
manageable.
How a user reply integrates (session resumption)
When a user replies to an agent note on GitLab, the orchestrator resumes the
conversation by restoring the previous state and appending the reply.
What gets restored from S3
context.json – the persisted conversation in canonical form; the
active adapter converts it to the in-memory representation before the loop
runs
sandbox.tar.gz – /tmp/_out/ files (PipelineRun JSONs, test logs,
jq output, etc.) restored into the new sandbox
Canonical storage allows sessions to resume with another provider; conversion
details are in Hummingbird AI.
The resumed contents list
// Restored from context.json (previous run)
contents[0] = user: {"project":"org/repo","iid":42,...} # original event
contents[1] = model: tool_call(gitlab_get_mr_details) # iter 1
contents[2] = tool: result(gitlab_get_mr_details) # iter 1
... # all prior turns
contents[15] = model: text("## Konflux Failure Analysis...") # previous final text
// NEW: user reply appended
contents[16] = user: "Can you look at the clair-scan timeout more closely?"
The agent loop continues
ITERATION 1 (resumed):
-> Send: system_instruction (+ CONTINUATION_PROMPT) + contents[0..16] + tool_defs
<- Response: tool_call(sandbox_exec, {command: "grep -i timeout /tmp/data/..."})
^ using restored sandbox files
contents[17] = model: tool_call(sandbox_exec)
contents[18] = tool: result(sandbox_exec)
ITERATION 2 (resumed):
<- Response: text("The clair-scan timeout is caused by...")
contents[19] = model: text("The clair-scan timeout is caused by...")
BREAK
Cold start vs resumed – key differences
| Aspect |
Cold start |
Resumed |
| System prompt |
BASE + tool_notes + workflow |
BASE + tool_notes + workflow + CONTINUATION_PROMPT |
| First user message |
{"project":..., "iid":...} |
{"project":..., "iid":...} (restored) |
| Conversation history |
Empty (just the event) |
Full prior conversation |
| Latest user message |
– |
"Can you look at the clair-scan timeout?" |
| Sandbox files |
Empty |
Restored from archive |
| Tool definitions |
Same |
Same |
| Session format |
Adapter content in memory |
Canonical on disk; adapter content after load |
The CONTINUATION_PROMPT
Without this, the workflow prompt (e.g. analyze-failures.md) tells the model
to follow a rigid workflow: Data.1, Data.2, Data.3, Analysis.1… The model
might try to re-run the entire analysis. The continuation prompt overrides
this:
## Continuation
This is a follow-up to a previous conversation. The conversation history
contains your prior analysis and tool calls. The user is replying with a
question or request about your previous work.
**Do NOT re-run the full workflow from scratch.** Instead:
- Respond directly to the user's question
- Use your tools to investigate further if needed (logs, data are still
available in the sandbox)
- Reference your previous findings where relevant
- Keep your response focused on what the user asked
The model now understands: “I already did the analysis (it’s all in the
conversation history). The user has a specific question. Let me answer it.”
Iteration and token budget for resumed sessions
The resumed session gets a full fresh iteration budget. run_workflow
sets max_iterations from the workflow config (e.g. 50 for
analyze-failures), and the loop counter starts at 0 regardless of how many
iterations the previous session used. The model can make as many tool calls
as it needs to answer the follow-up.
The practical constraint is the context window, not iterations. The
restored contents can be large – a typical 8-iteration cold start uses
~50-100K input tokens. Since the model re-counts the full history every
iteration, the resumed session starts at roughly the token count where the
previous session ended, plus the new user message. Each new tool call adds
more. Token counts logged by the agent are billed tokens (cumulative
across API calls; each call re-sends the full history, so these overlap).
The existing CONTEXT_LIMIT check still applies: at 80% of the limit, a
soft CONTEXT_WARNING tells the model to start wrapping up. If the limit is
exceeded, the next iteration becomes final (FINAL_TURN_WARNING with provider
tools disabled by the adapter).
For the typical case (user asks one focused follow-up, agent does 1-3 tool
calls), this is fine. Multi-turn deep dives will eventually hit the context
limit, at which point the agent wraps up – and the user can start a fresh
conversation if needed.
Provider behavior
Provider wire formats, canonical message conversion, prompt caching, server-side
tools, and adapter continuations are owned by the
hummingbird-ai library.
The Hummingbird agent loop
remains responsible for tool dispatch, iteration/context budgets, and session
resumption.
2 - Hummingbird Agent Design
Architectural design document for the hummingbird-agent. Covers the reasoning
behind every major design choice so that future changes can be made safely,
without accidentally violating invariants that hold the system together.
For operational usage (CLI, config reference, deployment), see
Hummingbird Agent. For agent turn flow and tool
handling, see Agent Model Loop. Provider
adapters and wire formats are documented in Hummingbird AI.
1. Design Philosophy
Five principles shaped the agent’s architecture. Every component traces back
to at least one of these.
Security by default. The agent processes untrusted merge requests. All
LLM-driven commands run inside isolated sandbox containers with no network
and no credentials. Orchestrator write tokens and model read tokens live in
separate code paths that never cross. No cluster-admin is required.
Config over code. Investigation logic lives in markdown workflow files
that become the LLM system prompt verbatim – changing the analysis strategy
requires no code change and no redeployment. Operational settings (data
sources, project allowlists, iteration limits) live in a YAML config file
that is auditable and committable. Secrets are the only thing in environment
variables.
Bounded cost. Every output path has a size cap. Large tool outputs are
auto-spilled to sandbox files with only a preview returned to the model. Each
session has both an iteration limit and a per-call context token limit with
two-tier escalation (soft warning, then hard stop with tool disabling). The
relationship between these constants is documented and centralized.
Partial over nothing. When some data is unavailable (expired pod logs,
unreachable Konflux cluster, Testing Farm 404), the agent continues with
whatever data it has and notes the gap in its output. A partial report is
more valuable than a crash.
Model-agnostic agent loop. The agent loop (agent.py) does not inspect
the internal structure of the contents list. It appends raw_content from
model responses and make_tool_responses() output without looking inside.
Each model backend owns its wire format. This makes it possible to add new
model backends (Claude, GPT) without touching the agent loop.
2. System Architecture
2.1. Component overview
flowchart TB
subgraph input [Event Sources]
SQS["SQS Queue<br/>(gitlab::pipeline, gitlab::note)"]
CLI["CLI<br/>(--event / --event-file)"]
end
subgraph orchestrator [Orchestrator Process]
Events["events.py<br/>SQS consumer"]
Runner["runner.py<br/>Event routing"]
Agent["agent.py<br/>Model loop"]
Tools["tools.py<br/>Tool registry"]
Actions["actions.py<br/>GitLab notes"]
Sessions["sessions.py<br/>Persistence"]
WfConfig["workflow_config.py<br/>YAML config"]
end
subgraph ai [Shared AI Library]
Models["hummingbird-ai<br/>Gemini / Anthropic adapters"]
end
subgraph sandbox [Sandbox Container]
SB["PodmanSandbox / K8sSandbox / KubeVirtSandbox<br/>jq, python3, yq"]
SpillFiles["/tmp/data/_out/ spill files"]
DataFiles["/tmp/data/ working files"]
end
subgraph dataSources [Data Source Modules]
GL["gitlab.py"]
KX["konflux.py"]
TF["testing_farm.py"]
end
subgraph external [External APIs]
GitLabAPI["GitLab API"]
KonfluxAPI["K8s + Kubearchive"]
TFAPI["Testing Farm API"]
end
subgraph storage [Storage]
S3["S3 Sessions"]
MRNote["GitLab MR Notes"]
end
SQS --> Events --> Runner
CLI --> Runner
Runner --> Agent
Agent --> Tools
Tools --> SB
Tools --> dataSources
dataSources --> external
Agent --> Models
Runner --> Actions --> MRNote
Runner --> Sessions --> S3
Runner --> WfConfig
2.2. Request lifecycle
A complete run proceeds through these stages:
-
Event arrival. An SQS message (production) or CLI invocation (dev)
provides a GitLab project path and MR IID.
-
Config lookup. workflow_config.py maps the project to one or more
workflows via the project_index. Each match yields a WorkflowConfig
and ProjectEntry with data source declarations and per-project settings.
-
Rate-limit check and SHA dedup. actions.scan_agent_threads() scans
MR discussions in a single pass, parsing JSON session markers to determine
the per-workflow thread count and whether the current SHA has already been
reviewed. If the SHA was already reviewed, the workflow is skipped. If the
thread count meets or exceeds max_runs_per_mr, the workflow is skipped.
Slash commands and replies bypass this check.
-
Placeholder note. actions.create_placeholder_note() posts a
placeholder so the reviewer knows analysis is in progress. The note
contains a JSON session marker
(<!-- hummingbird-session: {"id":"UUID","wf":"name","sha":"abc"} -->)
for rate limiting, SHA dedup, and session resumption.
-
Sandbox start. sandbox.create_sandbox() starts a Podman container
or K8s pod. /tmp/data/ is pre-created for the model’s use.
-
Data source registration. data_sources.register_selected() resolves
tokens and URLs from the config and registers tool functions on the
ToolRegistry. Only data sources declared in the workflow config are
registered.
-
Agent loop. agent.run_agent_loop() runs the model loop: the
workflow markdown becomes the system prompt, tool definitions are provided,
and the model iterates calling tools and producing text until it emits a
final response or hits a budget limit.
-
Note update. The final text, wrapped with a session marker and reply
prompt, replaces the placeholder note.
-
Session save. Conversation history (contents), transcript, and
sandbox archive are saved to S3 (production) or a local directory (dev).
-
Sandbox cleanup. The container/pod is deleted. K8s pods also have a
configurable activeDeadlineSeconds backstop (default 1800s) in case the
orchestrator dies.
2.3. Architectural boundaries
The codebase is organized around four strict boundaries:
Runner (runner.py) is the orchestration layer. It owns event routing,
the placeholder/update note lifecycle, session save/load, and sandbox
lifecycle. It calls agent.run_agent_loop() but never reaches into the
agent’s internals.
Agent (agent.py) is the model loop. It knows about the model interface,
tool definitions, and the contents list, but nothing about GitLab, SQS,
sessions, or actions. It returns (text, usage, transcript, contents) and
is completely unaware of what happens with those values.
Tools (tools.py) bridge the agent and the sandbox/data-sources. The
agent calls tool_registry.execute(tool_call) and gets a dict back. It
never calls sandbox methods directly. This indirection is what enables
auto-spill: the tool registry can transparently save large outputs to
sandbox files and return previews.
Model providers are supplied by the shared
hummingbird-ai library,
which owns adapter construction,
provider formats, retries, and AI usage accounting. The agent loop uses the
library’s normalized adapter interface and treats conversation content as
opaque.
3. Module Architecture
3.1. Dependency graph
flowchart TD
main["__main__.py"]
config_mod["config.py"]
wf_config["workflow_config.py"]
events_mod["events.py"]
runner_mod["runner.py"]
agent_mod["agent.py"]
tools_mod["tools.py"]
sandbox_mod["sandbox.py"]
actions_mod["actions.py"]
sessions_mod["sessions.py"]
transcript_mod["transcript.py"]
ds_init["data_sources/__init__.py"]
ds_gitlab["data_sources/gitlab.py"]
ds_konflux["data_sources/konflux.py"]
ds_tf["data_sources/testing_farm.py"]
config_watch_mod["config_watch.py"]
ai_pkg["hummingbird-ai library"]
main --> config_mod
main --> config_watch_mod
main --> wf_config
main --> events_mod
main --> runner_mod
main --> sessions_mod
main --> transcript_mod
main --> actions_mod
config_watch_mod --> config_mod
config_watch_mod --> wf_config
runner_mod --> actions_mod
runner_mod --> agent_mod
runner_mod --> config_mod
runner_mod --> ds_init
runner_mod --> sandbox_mod
runner_mod --> sessions_mod
runner_mod --> tools_mod
runner_mod --> ai_pkg
runner_mod --> wf_config
runner_mod --> transcript_mod
agent_mod --> config_mod
agent_mod --> ai_pkg
agent_mod --> tools_mod
tools_mod --> config_mod
tools_mod --> ai_pkg
tools_mod --> sandbox_mod
ds_init --> tools_mod
ds_init --> wf_config
ds_init --> ds_gitlab
ds_init --> ds_konflux
ds_init --> ds_tf
ds_gitlab --> ai_pkg
ds_konflux --> ai_pkg
ds_tf --> ai_pkg
sessions_mod --> sandbox_mod
3.2. Module responsibilities
Each module has a single, well-defined responsibility. The boundary rules
below are invariants – violating them breaks the security model or the
separation of concerns.
config.py – Runtime configuration and constants
- Owns:
Config dataclass and agent tuning constants
(DEFAULT_MAX_ITERATIONS, CONTEXT_LIMIT, OUTPUT_PREVIEW_BYTES, etc.)
- Boundary: Pure data. No I/O except reading env vars in
Config.load().
No imports from other Hummingbird agent modules; model setup and pricing are
owned by hummingbird-ai.
- Invariant: All interdependent budget/spill constants must be defined
here with their relationship documented in the comment block.
workflow_config.py – YAML config loader and token resolution
- Owns:
AgentConfig, WorkflowConfig (with trigger, description,
trigger_rules, ignore_users, ignore_branches fields), TriggerRule,
ProjectEntry, DataSourceConfig dataclasses; load(), validate_env(),
evaluate_trigger_rules(), resolve_tool_token(), resolve_cluster_url(),
get_prompt().
- Boundary: Reads YAML from disk and env vars. Never instantiates
network clients or model objects.
- Invariant:
resolve_tool_token() resolves model tool tokens only.
It must never return orchestrator tokens. The resolution chain is:
project_entry.tokens[ds] -> workflow_cfg.data_sources[ds].token_env -> "" (empty).
events.py – SQS consumer
- Owns:
decode_sns_message() (SNS envelope decoding),
poll_loop() (blocking SQS consumer with concurrency control).
- Boundary: Knows about SQS/SNS wire formats. Calls a generic
EventHandler callback. Does not know about GitLab, workflows, or the
agent.
- Invariant: Failed messages are not deleted from SQS (they go to
the DLQ after visibility timeout expires). Successful messages are
deleted after handler returns.
config_watch.py – Background config hot-reload
- Owns:
ConfigHolder (thread-safe config pair with atomic swap),
start_watcher() (daemon thread that polls config file mtime),
_watch_loop().
- Boundary: Knows about
config.Config.load() and
workflow_config.load(). Does not know about events, the agent, or
any runtime state.
- Invariant: On reload failure (parse error, missing env var), the
previous config is kept and a warning is logged. The watcher never
crashes the serve loop.
runner.py – Event routing and workflow execution
- Owns:
handle_event(), handle_pipeline(), handle_merge_request(),
handle_note(), _execute_workflow(), _handle_reply(), run_workflow(),
_acquire_sandbox(), _resolve_slash_workflow(), _format_help(),
_is_ignored_user(), WorkflowRequest, WorkflowResult, SandboxOpts.
- Boundary: Orchestrates everything: config lookup, rate limiting,
placeholder notes, sandbox lifecycle, agent invocation, session save.
This is the only module that touches both
actions and agent.
- Invariant:
run_workflow() either lingers the sandbox (success)
or cleans it up (exception).
- Owns:
run_agent_loop(), system prompt constants (BASE_SYSTEM_PROMPT,
CONTINUATION_PROMPT, warning/nudge strings), budget logic.
- Boundary: Uses the
ModelAdapter interface from
hummingbird-ai and
tools (for execute()). Does not
know about GitLab, SQS, sessions, or actions, or which backend is in use.
- Invariant: The
contents list is treated as opaque. Items are
appended via response.raw_content and model.make_tool_responses().
The agent never inspects, modifies, or deletes items in the list (except
for empty response retries, which pop() the last appended item before
the model has seen any tool results for it).
- Owns:
ToolRegistry class with sandbox_exec, fetch_to_sandbox,
fetch_batch_to_sandbox built-in tools; data source registration and
dispatch; auto-spill logic.
- Boundary: Owns the sandbox reference and all tool execution. The
agent never touches the sandbox directly.
- Invariant:
_spill_counter is shared across all spill paths
(sandbox exec stdout, sandbox exec stderr, data source auto-spill) to
prevent filename collisions in /tmp/data/_out/.
sandbox.py – Sandbox backends
- Owns:
Sandbox protocol, PodmanSandbox, K8sSandbox,
K8sPoolSandbox, SandboxPool, KubeVirtSandbox, VmiPool,
KubeVirtPoolSandbox, create_sandbox(), ExecResult.
- Boundary: Knows about container/pod/VM lifecycle and command execution.
Does not know about tools, models, or the agent.
- Invariant: All backends must implement the same 8-method protocol
(
start, exec, write_file, stream_to_file, stream_exec,
read_file_iter, cleanup, linger). exec() accepts optional
stdin_data for piping raw bytes. All must pre-create /tmp/data/ in
start(). All must use stdin piping for write_file() (never host
volume mounts).
actions.py – GitLab note lifecycle
- Owns: Orchestrator token resolution (
resolve_orchestrator_token()),
workflow action token client creation (_make_client()), bot user ID
collection (collect_workflow_bot_ids()), note CRUD, JSON marker parsing
(marker_tag, parse_marker), scan_agent_threads (per-workflow rate
limiting + SHA dedup), post_simple_reply, member access checks,
session-for-reply lookup, AgentThreadInfo, SessionRef.
- Boundary: Functions are split by token tier:
- Orchestrator token (operational reads + infrastructure notes, use
_get_client): check_member_access, get_latest_push_author,
get_head_pipeline_id, is_first_note_in_discussion,
reply_access_denied, notify_rate_limit, post_simple_reply.
- Workflow action token (per-workflow-attributable writes, take explicit
token: str): create_placeholder_discussion, post_discussion_note,
resolve_agent_threads, scan_agent_threads.
Never touches model tool tokens, the tool registry, or the agent.
- Invariant: Orchestrator tokens are resolved from
ORCHESTRATOR_* env
vars only. Workflow action tokens are resolved from action_tokens in YAML
and passed as explicit parameters. Neither tier is exposed to the model.
sessions.py – Session persistence
- Owns:
archive_sandbox(), save_local(), save_s3(), load_s3(),
load_local(), restore_sandbox(), SessionData.
- Boundary: Knows about sandbox (for archiving) and S3/filesystem (for
storage). Does not know about the agent, tools, or models. Load/save uses
canonical envelopes with
format_version: 1; unversioned legacy sessions
are not supported.
- Invariant: The sandbox archive is never extracted on the orchestrator.
It is created inside the sandbox (
tar czf), streamed out via
read_file_iter(), and restored inside a new sandbox via
stream_exec("tar xzf -").
transcript.py – Transcript rendering
- Owns:
render_markdown(), per-tool rendering functions, truncation.
- Boundary: Pure transformation from
TranscriptEntry list to markdown.
No I/O, no side effects.
data_sources/__init__.py – Data source registration
- Owns:
register_selected() – resolves tokens/URLs from config and
calls each data source’s register() function.
- Boundary: Bridges
workflow_config (for token/URL resolution) and
tools (for registration). Only registers data sources declared in the
workflow config.
data_sources/gitlab.py, konflux.py, testing_farm.py
- Own:
register() function that creates tool definitions and closures
over credentials, and registers them on the ToolRegistry.
- Boundary: Each module talks to one external API. Credentials are
captured at registration time via closure, never stored globally.
4. Security Model
The agent runs in a shared OpenShift cluster without cluster-admin access,
processing merge requests from repositories where external contributors can
submit code. The security model addresses two threat vectors: (1) the LLM
executing arbitrary commands chosen by an attacker-controlled MR, and (2)
credential leakage between the model, the sandbox, and orchestrator actions.
4.1. Sandbox isolation
All commands generated by the LLM run inside an ephemeral container, never
on the orchestrator host. The sandbox has no credentials, no network, and no
visibility into the orchestrator.
flowchart LR
subgraph orchestrator [Orchestrator]
Agent["Agent Loop"]
Creds["Secrets<br/>(tokens, kubeconfig)"]
end
subgraph sandbox [Sandbox Container]
Shell["sh -c commands"]
Files["/tmp/data/<br/>(incl. _out/ spill dir)"]
end
Agent -->|"stdin pipe<br/>(write_file)"| sandbox
Agent -->|"exec command"| sandbox
sandbox -->|"stdout/stderr"| Agent
sandbox -.-x|"NO network"| Internet["Internet / K8s API"]
sandbox -.-x|"NO access"| Creds
Podman (local development):
--network=none – complete network isolation; curl, wget, pip install all
fail
--user 65532 – fixed non-root UID; no privilege escalation
- No host volume mounts – data enters only via
stdin piping through
write_file()
Kubernetes (production):
Every field in the pod manifest is set explicitly for portability to vanilla
Kubernetes with Pod Security Admission (restricted level), not just reliance
on OpenShift’s restricted-v2 SCC admission:
automountServiceAccountToken: false – no K8s API access from sandbox
runAsNonRoot: true – enforced at pod level
seccompProfile: RuntimeDefault – required by restricted level
allowPrivilegeEscalation: false – container level
capabilities.drop: ["ALL"] – container level
activeDeadlineSeconds: 1800 – pod self-terminates after 30 minutes even
if the orchestrator crashes; prevents orphaned pods
restartPolicy: Never – pod does not restart on failure
NetworkPolicy on the sandbox namespace blocks all egress from all pods
in the namespace (podSelector: {}, egress: [])
The pod does not set runAsUser explicitly. On OpenShift, the SCC assigns a
UID from the namespace range. On vanilla K8s, the image’s USER directive is
used.
4.2. Credential separation (four-tier token model)
Tokens are split into four tiers with strict code-path separation:
flowchart TB
subgraph tier1 [Tier 1: Orchestrator Tokens]
OT["ORCHESTRATOR_GITLAB_TOKEN_*<br/>Operational reads<br/>Env vars only, never in YAML"]
end
subgraph tier2 [Tier 2: Workflow Action Tokens]
AT["action_tokens in YAML<br/>Per-workflow bot identity<br/>Write scope (api)"]
end
subgraph tier3 [Tier 3: Model Tool Tokens]
MT["GITLAB_TOKEN_RO, etc.<br/>Read scope (read_api)<br/>Env var NAMES in YAML"]
end
subgraph tier4 [Tier 4: Sandbox]
SB["Zero credentials<br/>Zero network<br/>Zero SA token"]
end
OT -->|"used by"| OpsReads["actions.py<br/>(member access, push author,<br/>rate-limit notices)"]
AT -->|"used by"| WfWrites["actions.py<br/>(notes, thread resolution)"]
MT -->|"used by"| DataSources["data_sources/<br/>(GitLab, Konflux, TF)"]
SB -->|"used by"| SandboxExec["sandbox_exec<br/>(sh -c commands)"]
OpsReads -.-x|"NEVER"| DataSources
WfWrites -.-x|"NEVER"| DataSources
WfWrites -.-x|"NEVER"| SandboxExec
DataSources -.-x|"NEVER"| OpsReads
DataSources -.-x|"NEVER"| WfWrites
Tier 1 – Orchestrator tokens. Tokens used by actions.py for
operational reads (member access checks, push author lookup, head pipeline
queries) and infrastructure notes (access-denied replies, rate-limit
notices). Resolved by convention from env vars:
ORCHESTRATOR_GITLAB_TOKEN_<MANGLED_PROJECT> (per-project) or
ORCHESTRATOR_GITLAB_TOKEN (global fallback). The ORCHESTRATOR_ prefix is
a structural safeguard – these env var names can never appear in the YAML
config’s data_sources, tokens, or action_tokens sections. These tokens
can be scoped to read_api since all per-workflow writes moved to Tier 2.
Tier 2 – Workflow action tokens. Per-workflow bot identity tokens
(api scope) used by actions.py for all workflow-attributable GitLab
writes: placeholder notes, discussion notes, and thread resolution. Each
workflow gets a dedicated bot user per project, providing clear audit trails
for which agent produced each note. Declared in the YAML config as env var
names at the project level:
projects:
redhat/hummingbird/containers:
action_tokens:
gitlab: HUMMINGBIRD_AGENT_ACTION_CODE_REVIEW_GITLAB_TOKEN_CONTAINERS
Resolved by resolve_action_token(project_entry, ds_name) which reads the
env var name from action_tokens and resolves it from os.environ. These
tokens are passed as explicit token: str parameters to write functions
in actions.py. They NEVER enter the ToolRegistry or model tool calls.
At startup, collect_workflow_bot_ids() authenticates each unique workflow
token to build a set of all workflow bot user IDs. This set is used by
handle_note for self-filtering (skipping notes from any workflow bot)
and by find_session_for_reply for session marker scanning.
The naming convention groups tokens alphabetically:
HUMMINGBIRD_AGENT_ACTION_<WORKFLOW>_GITLAB_TOKEN_<PROJECT>.
Tier 3 – Model tool tokens. Read-only tokens (read_api scope for
GitLab) used by data source modules during tool execution. Declared in the
YAML config as env var names (not values):
data_sources:
gitlab:
token_env: GITLAB_TOKEN_RO # name of the env var
Per-project overrides are possible:
projects:
redhat/hummingbird/containers:
tokens:
gitlab: GITLAB_TOKEN_CONTAINERS_RO # overrides token_env for this project
The resolution chain in resolve_tool_token() is:
project_entry.tokens[ds] -> workflow_cfg.data_sources[ds].token_env -> ""
Storing env var names (not values) in YAML means the config file is safe
to commit and audit. Actual secret values live in environment variables,
injected via K8s Secrets at deployment time.
Tier 4 – Sandbox. The sandbox container has zero credentials, zero
network access, and automountServiceAccountToken: false (no K8s API
access). Data enters the sandbox only via write_file() (stdin piping).
The model cannot instruct the sandbox to reach external APIs – it must use
the orchestrator’s data source tools.
4.3. Namespace separation
Sandbox pods run in a dedicated namespace (hummingbird--agent-sandbox),
separate from the orchestrator namespace (hummingbird--internal). This
limits blast radius: even if a sandbox pod is compromised, it has no
visibility into the orchestrator’s Secrets, Pods, or ServiceAccount tokens.
RBAC setup:
The orchestrator’s ServiceAccount gets a Role in the sandbox namespace
(not its own namespace) granting only:
pods: create, get, list, delete, patch – sandbox pod lifecycle and pool claims
pods/exec: create – command execution via kubectl exec
No CRDs, no custom runtimes, no cluster-scoped resources. The orchestrator
needs only namespace-scoped permissions, so it works with standard
OpenShift RBAC without requesting cluster-admin.
Konflux data is fetched via bearer token from kubeconfig credentials, not
from inside the cluster. The orchestrator’s ServiceAccount does not need
access to Konflux namespaces.
NetworkPolicy:
A deny-all-egress NetworkPolicy in the sandbox namespace uses podSelector: {}
to match all pods and sets egress: []. The sandbox cannot reach the internet,
the K8s API, or other pods in the cluster.
4.4. Security invariants
These must hold for the security model to be effective. Any change that
violates one of these is a security regression:
-
Orchestrator tokens must NEVER flow into ToolRegistry,
data_sources, or model contents. They are resolved in actions.py
only.
-
Workflow action tokens must NEVER flow into ToolRegistry,
data_sources, or model contents. They are passed as explicit
token: str parameters within actions.py and runner.py only.
-
Model tool tokens must NEVER flow into actions.py. They are
resolved in workflow_config.py and consumed in data_sources/.
-
The sandbox must NEVER have network access or credentials. No host
mounts, no SA token, no env vars with secrets.
-
The sandbox archive is NEVER extracted on the orchestrator. It is
created inside one sandbox and restored inside another. The orchestrator
only transports the bytes.
-
No cluster-admin required. Only namespace-scoped resources (Role,
RoleBinding, Pod, NetworkPolicy) are used.
-
Agent-generated notes are skipped on re-processing. Notes containing
SESSION_MARKER_PREFIX are filtered out in handle_note() to prevent
infinite loops. Self-filtering checks against all workflow bot user IDs
(collected at startup) plus the orchestrator bot ID.
-
All auto-triggers require Developer access. handle_pipeline(),
handle_merge_request(), and handle_note() each check
check_member_access() on the event’s user before executing a workflow.
Events from non-developers are silently skipped (or replied to with an
access-denied message for slash commands). This ensures that the target
MR’s work products – its diff, description, CI logs, and commit
messages – originate from a trusted author. Fork MRs from external
contributors are blocked because the MR author lacks Developer+ on the
target project.
Scope limitation: this gate only covers the target MR. Once the
agent is running, the model controls tool arguments and can direct tools
at content beyond the target MR – other MRs, other refs, other job
IDs, even other projects reachable by the read-only token. The current
mitigations are: (a) workflow prompts instruct the model to use the
event’s {project, iid, sha}, (b) the model would need to be
manipulated via prompt injection from already-trusted content to deviate,
(c) the sandbox prevents the model from acting on manipulated reasoning
beyond producing text output, and (d) the token is read-only with
minimal scope. However, if a data source tool is added that returns
user-authored prose from arbitrary resources (e.g. issue bodies, wiki
pages), it should apply per-author trust filtering (invariant #10).
-
Data source tools that accept URLs must validate them against an
allowlist. tf_get_test_log restricts URLs to the Testing Farm
artifacts prefix to prevent the LLM from directing the orchestrator to
fetch arbitrary URLs (SSRF).
-
Third-party commentary entering the model prompt must be
trust-filtered. Data sources that feed text from users other than the
event trigger into contents (discussion comments, issue bodies) must
gate on project membership at Developer+ level per author. Content from
untrusted authors must be redacted to a fixed placeholder, never
sanitized or escaped – there is no reliable way to escape adversarial
text for an LLM prompt. Discussions where all notes are untrusted must
be dropped entirely. See
§9.1 for the reference
implementation in gitlab_get_mr_discussions.
This invariant covers commentary (what other people said about the
MR), not work products (the diff, CI logs, commit messages). Work
products are inherently the input to the agent’s analysis and cannot be
content-filtered without defeating the agent’s purpose. For the target
MR, their trust comes from invariant #8 (the event trigger is
Developer+). For content the model fetches beyond the target MR, trust
depends on the tool: discussion tools must filter per-author (#10), while
code/log/metadata tools rely on the mitigations described in #7.
5. Configuration System
5.1. Two sources, strict separation
Configuration comes from exactly two sources with no overlap:
- YAML config file (
CONFIG_PATH): operational settings, workflow
definitions, project allowlists, data source declarations, and token env
var names. This file is safe to commit, review, and audit.
- Environment variables: secrets only (API keys, GitLab tokens,
kubeconfig paths). These are injected at deployment time via K8s Secrets.
This split exists because YAML provides structure, validation, and audit
trails, while secrets must stay out of version control.
5.2. Config loading
Two config objects are built at startup:
Config (from config.py): loaded by Config.load(settings), where
settings is the settings: section from the YAML file. Auth/bootstrap
fields come from env vars (GOOGLE_API_KEY, GOOGLE_CLOUD_PROJECT, etc.).
Operational fields come from the settings dict with sensible defaults.
AgentConfig (from workflow_config.py): loaded by
workflow_config.load(path). Contains all workflow definitions, project
entries, and a pre-built project_index.
The two are kept separate because Config is needed everywhere (model
construction, sandbox creation, session storage), while AgentConfig is
only needed for event routing and data source registration.
5.3. YAML config structure
settings:
gitlab_url: https://gitlab.com # operational, not a secret
sandbox:
image: quay.io/.../image:tag
namespace: my-namespace # K8s only
model: gemini-3.1-pro-preview
# model: claude-sonnet-4-5@20250929 # Anthropic via Vertex AI (alternative)
max_iterations: 30
max_runs_per_mr: 5
internal_notes: true
max_concurrent_agents: 4
sqs_queue_url: "" # empty = no SQS (local dev)
s3_session_bucket: "" # empty = no S3 (local dev)
trigger_prefix: /hummingbird # slash command prefix
session_marker_prefix: hummingbird-session # HTML comment marker ID
auto_trigger: true # auto-run on failed pipelines
workflows:
analyze-failures:
prompt: workflows/analyze-failures.md # relative to config file dir
action: post_gitlab_note
model: gemini-3.1-pro-preview # per-workflow override
max_iterations: 50 # per-workflow override
data_sources:
gitlab:
token_env: GITLAB_TOKEN_RO # env var name, not the value
konflux:
cluster_url: https://example.com:6443/ns/my-tenant
kubeconfig_env: KUBECONFIG
kubearchive_url: https://kubearchive-api-server-product-kubearchive.apps.example.com
testing_farm: {}
projects:
redhat/hummingbird/containers:
tokens: # per-project token overrides
gitlab: GITLAB_TOKEN_CONTAINERS_RO
Design choices in this structure:
Workflow-first organization. Each workflow owns its project list, not the
other way around. This scopes data source permissions per workflow-project
pair. A future code-review workflow can have different GitLab tokens (with
different scopes) than the analyze-failures workflow, with no ambiguity.
Token env var names in YAML (not values). The YAML file is committed to
the repo. It contains token_env: GITLAB_TOKEN_RO (a name), not the actual
token. The actual secret value is resolved at runtime via
os.environ.get(env_var). This allows the config to be reviewed and
audited without exposing secrets.
Inline cluster_url only. The Konflux cluster URL is operational
configuration (it identifies which cluster to talk to), not a secret.
Putting it inline in YAML makes it visible and auditable. The URL must
include the namespace path (e.g. /ns/my-tenant).
5.4. Project index
At load time, _build_project_index() constructs a reverse lookup:
project_index: dict[str, list[tuple[str, WorkflowConfig, ProjectEntry]]]
# e.g. {"redhat/hummingbird/containers": [("analyze-failures", wf_cfg, proj_entry)]}
This provides O(1) lookup when a pipeline event arrives with a project
path. A single project can appear in multiple workflows (e.g. both
analyze-failures and a future code-review), and each matching workflow
will be triggered independently.
5.5. Token resolution chains
Model tool tokens (for data source API calls):
resolve_tool_token(project_entry, ds_name, workflow_cfg):
1. project_entry.tokens[ds_name] -> per-project override
2. workflow_cfg.data_sources[ds_name].token_env -> workflow default
3. "" (empty string) -> no token
Each step resolves the env var NAME, then reads os.environ[name].
Workflow action tokens (for per-workflow GitLab writes):
resolve_action_token(project_entry, ds_name):
1. project_entry.action_tokens[ds_name] -> env var NAME
2. os.environ[name] -> resolved token value
3. "" (empty) -> not configured
No fallback chain. Missing tokens cause validate_env to fail at startup.
Orchestrator tokens (for operational reads and infrastructure notes):
resolve_orchestrator_token(project_path):
1. ORCHESTRATOR_GITLAB_TOKEN_<MANGLED_PROJECT> -> per-project
2. ORCHESTRATOR_GITLAB_TOKEN -> global fallback
Mangling: "/" -> "_", "-" -> "_", uppercase.
e.g. "redhat/hummingbird/containers" -> ORCHESTRATOR_GITLAB_TOKEN_REDHAT_HUMMINGBIRD_CONTAINERS
Cluster URL (for Konflux):
resolve_cluster_url(ds_cfg):
-> ds_cfg.cluster_url (inline value in YAML)
5.6. Environment validation
validate_env(agent_cfg) checks at startup that all referenced env vars
exist. It walks every workflow’s data sources, project token overrides, and
project action token entries, collecting missing vars into a single error
message. This catches configuration errors early instead of failing mid-run
when a specific data source or workflow action is first used. Missing
action_tokens env vars are treated the same as missing model tool tokens:
startup fails unconditionally.
See Also
3 - Hummingbird Agent Design — Engine
Agent loop, tool system, and sandbox backends for the Hummingbird Agent.
This page covers the execution engine: how the agent iterates, dispatches tools,
and isolates execution. For foundations (design philosophy, system architecture,
modules, security, config), see Hummingbird Agent Design.
6. Agent Loop Design
For the wire-format walkthrough (what bytes go to the model API, what comes back),
see Agent Model Loop. This section
covers the design rationale behind the loop.
6.1. Full history replay
The Gemini and Anthropic APIs are stateless. Every call sends the complete contents list
from the beginning of the conversation. This means every large tool output
sitting in history inflates every subsequent API call.
This property is fundamental to why auto-spill exists (section 7). Without
auto-spill, a single cat of a 32KB file early in the conversation adds
~8K tokens to every remaining API call. Over a 15-iteration run, that is
~120K wasted tokens.
The alternative – conversation compaction (replacing old tool results with
summaries) – was considered and deferred. Each provider has strict
requirements about content structure, and modifying history risks confusing
the model or violating API constraints. Provider-specific format requirements
are documented in Hummingbird AI.
Auto-spill solves 90% of the problem with none of the risk.
6.2. Budget model: iterations + context limit
The agent uses a dual-limit approach rather than a cumulative token budget:
Iteration limit (max_iterations, default 30 per workflow config).
Hard cap on the number of model round-trips. This is the primary cost
control lever. With auto-spill keeping per-call context bounded, iteration
count is roughly proportional to cost.
Per-call context limit (CONTEXT_LIMIT, default 60,000 tokens).
Checked after each API call using response.usage.total_input_tokens. This is
a safety net for cases where auto-spill is not sufficient (e.g., many
small tool results that individually stay under the spill threshold but
cumulatively fill the context).
Why not a cumulative token budget? Because with full history replay, each
API call re-sends everything. “Cumulative billed tokens” double-counts:
call 1 sends 5K, call 2 sends 10K (including the 5K again), so billed
total is 15K but actual new content is only 10K. Iteration count is a
simpler and more predictable proxy for cost.
6.3. Two-tier budget escalation
Both limits use the same escalation pattern:
flowchart LR
Normal["Normal<br/>tools available"]
Soft["Soft Warning (80%)<br/>ITERATION_WARNING or<br/>CONTEXT_WARNING<br/>tools still available"]
Hard["Hard Stop (100%)<br/>FINAL_TURN_WARNING<br/>toolConfig: NONE"]
Normal -->|"80% reached"| Soft
Soft -->|"100% reached"| Hard
Soft warning (80%). An ephemeral user message (ITERATION_WARNING) is
injected into contents for that turn only, then popped before the
response is persisted. Tools remain available so the model can finish
in-progress work.
Hard stop (100%). FINAL_TURN_WARNING is injected as an ephemeral
user message AND tool_defs is set to [] (empty list) to physically
prevent further tool calls. The model must produce text. This is more
reliable than relying on provider-side tool disabling, which models may
ignore.
6.3.1 Ephemeral messages and provider normalization
All per-turn warnings use ephemeral user messages: they are appended to
contents before the API call and popped immediately after. This keeps
the system prompt stable across all iterations and prevents warnings from
polluting the conversation history saved in sessions.
Provider adapters normalize provider-specific conversation requirements while
preserving the agent’s saved history; details are in
Hummingbird AI.
6.4. Empty response handling
Models occasionally return unusable responses (no text and no valid tool calls). The
agent retries up to MAX_EMPTY_RETRIES (2) times:
- Pop the empty response from
contents. It adds nothing and may
confuse the model on the next call.
- Inject
EMPTY_RESPONSE_NUDGE as an ephemeral user message on the
next turn: “Your previous response was empty. Please continue…”
- Continue the loop (consuming an iteration).
The nudge is delivered as an ephemeral user message (injected before the
API call and popped after). This avoids mutating the system prompt and
keeps the conversation history clean for session persistence.
The empty_retries counter resets to 0 after any successful iteration
(one where tool calls were executed). This means the model gets fresh
retries if it produces empty responses at different points in the
conversation.
MALFORMED_FUNCTION_CALL handling. Models sometimes return a
finishReason of MALFORMED_FUNCTION_CALL (Gemini) with no usable tool calls.
This is treated as a special case of empty response: the retry mechanism
kicks in, but the nudge is replaced with MALFORMED_CALL_NUDGE which
tells the model to retry with simpler arguments and avoid large text
payloads in tool call arguments.
6.5. Error recovery
Model API and transport errors. The hummingbird-ai library classifies
provider and transport failures, applies the workflow’s retry budget, and
records AI usage and request metrics. See Hummingbird AI
for provider-specific behavior. The agent loop retains partial results when a
terminal error reaches it.
Unexpected loop errors. run_agent_loop wraps the
_generate_with_retry call in a try/except Exception that logs and
breaks instead of propagating. This ensures the function always returns
partial results (accumulated contents, transcript, sandbox state)
even when an unexpected exception occurs (e.g. JSONDecodeError from a
truncated API response). The caller saves the session normally –
conversation history and sandbox archive are preserved for resumption.
This follows the “partial over nothing” principle: 32 iterations of work
are more valuable than a crash.
No failure-path session save. _execute_workflow does not save a
session when run_workflow raises. With the agent loop catching
unexpected errors, the failure path only fires for infrastructure errors
(sandbox start, config) where there is no useful state. Not saving
avoids overwriting a previous good session when a reply attempt fails.
Tool execution errors. _execute_tool_calls() wraps each
tool_registry.execute() in a try/except. Unhandled exceptions are caught
and returned to the model as {"error": "Tool X failed: ..."}. This
prevents a single broken tool from crashing the entire session – the model
sees the error and can adapt.
Sandbox exec timeout. If a command exceeds EXEC_TIMEOUT (120s), the
TimeoutExpired exception is caught in the tool registry and returned as a
structured error dict. The model can retry with a different command or
proceed without the result.
6.6. Session resumption
When resuming from a previous session:
initial_contents (the full contents list from the previous run) is
prepended, followed by the user’s reply as a new user turn.
CONTINUATION_PROMPT is appended to the system prompt.
- The sandbox is restored from the archived tarball.
- A fresh iteration budget starts from 0.
Saved sessions use a versioned JSON envelope (format_version: 1) whose
contents are in a canonical message format independent of the active model.
Unversioned legacy sessions are not supported. On resume, run_workflow asks
the active adapter to load canonical history; after a successful run, it
converts native contents back to canonical form before persistence. The
library documents the canonical format and provider conversions in
Hummingbird AI.
CONTINUATION_PROMPT is critical. Without it, the workflow prompt (e.g.,
analyze-failures.md) tells the model to follow a rigid multi-phase
workflow: Data.1, Data.2, Analysis.1… The model would try to re-run the
entire analysis. The continuation prompt overrides this: “Do NOT re-run the
full workflow. Respond directly to the user’s question.”
The practical constraint on resumed sessions is the context window, not
iterations. The restored history from an 8-iteration cold start uses
~50-100K input tokens. Each new tool call adds more. The existing
CONTEXT_LIMIT check still applies and will force wrap-up if the context
grows too large.
Storing sessions in canonical form decouples persisted history from any one
provider’s JSON shape. The agent loop treats native contents as opaque; the
library owns provider conversion at load and save boundaries.
6.7. Provider caching and pricing
Provider-specific prompt caching and token-cost estimation are owned by
hummingbird-ai.
ToolRegistry is the single point of dispatch for all tool calls. The
agent calls registry.execute(tool_call) and gets a dict back. It never
calls sandbox methods or data source functions directly.
Three built-in tools are always available:
sandbox_exec – runs sh -c <command> in the sandbox container.
Returns {exit_code, stdout, stderr}, with auto-spill for large outputs.
fetch_to_sandbox – calls a data source function and writes the
result to a specified path in the sandbox. Returns metadata only (path,
byte count, line count). Used when the model wants explicit path control.
fetch_batch_to_sandbox – calls multiple fetch_to_sandbox in one
tool call. Saves iterations vs. sequential calls (e.g., fetching both
pipelineruns and taskruns in one round-trip).
Data source tools are registered dynamically per workflow config. Each data
source module provides a register() function that creates ToolDef
objects and closures over credentials, then calls
registry.register_data_source(tool_def, func, response_metadata).
7.2. Auto-spill architecture
Auto-spill is the key mechanism for keeping the LLM context bounded. Without
it, large outputs accumulate in the contents list and inflate every
subsequent API call (because both APIs replay the full history).
flowchart TD
Output["Tool produces output"]
SizeCheck{"size > OUTPUT_PREVIEW_BYTES<br/>(4 KB)?"}
Inline["Return full output inline"]
Spill["Write full output to<br/>/tmp/data/_out/N.txt"]
Preview["Return preview<br/>(head + tail) +<br/>file path metadata"]
Output --> SizeCheck
SizeCheck -->|"<= 4 KB"| Inline
SizeCheck -->|"> 4 KB"| Spill --> Preview
Three spill paths share the same _spill_counter to prevent filename
collisions:
sandbox_exec spill. When stdout or stderr exceeds
OUTPUT_PREVIEW_BYTES (4096 bytes), the full output is written to
/tmp/data/_out/{counter}.txt via _spill_field(). The model receives:
{
"exit_code": 0,
"stdout": "<first 4KB>",
"stdout_truncated": true,
"stdout_file": "/tmp/data/_out/0.txt",
"stdout_bytes": 32768,
"stdout_lines": 1024,
"stdout_tail": "<last 512 bytes>"
}
The preview (head + tail) gives the model enough context to decide whether
to process the full file with jq/grep/head.
Data source inline spill. When a direct data source call returns text
larger than MAX_INLINE_SIZE (4096 bytes), _spill_data_source() writes
it to /tmp/data/_out/{name}_{counter}.txt and returns:
{
"saved_to": "/tmp/data/_out/konflux_list_pipelineruns_1.txt",
"bytes": 85432,
"lines": 2100,
"preview": "<first 4KB>"
}
Streaming responses (e.g., gitlab_get_repo_archive which returns a
StreamingResponse with suffix .tar.gz) are written via
_spill_streaming(). Text streams (binary=False, the default) capture a
UTF-8 preview from the head while piping chunks to the sandbox. Binary
streams (binary=True) skip preview capture entirely and return only
{saved_to, bytes}, avoiding meaningless decoded output for formats like
tar.gz. The StreamingResponse.suffix field controls the file extension
(.jsonl, .tar.gz, .log).
Non-streaming binary responses are written to .bin files with no preview
via _spill_binary().
Sandbox-optional mode. ToolRegistry accepts sb=None for use cases
that do not require a sandbox (e.g. evaluation harnesses that only use
native model tools like Google Search grounding). When no sandbox is
available:
- Sandbox tools (
sandbox_exec, fetch_to_sandbox, fetch_batch_to_sandbox)
are omitted from get_tool_defs() so the model never sees them.
- Text data sources (inline and streaming) degrade to truncated previews
(
OUTPUT_PREVIEW_BYTES). The model receives {preview, truncated: true}
instead of {saved_to, preview}. Streaming text is consumed into a memory
buffer up to the preview limit and the rest is discarded.
- Binary data (streaming and non-streaming) returns an error with the
byte count (
{error, bytes}). Binary content cannot be meaningfully
previewed as text.
fetch_to_sandbox spill. Always writes to the caller-specified path
(not /tmp/data/_out/). Returns metadata only (no preview). The model uses this
when it wants a specific filename for later processing.
7.3. Why auto-spill instead of rejecting large outputs
The earlier design rejected large data source responses with “use
fetch_to_sandbox instead.” This caused two wasted iterations per rejection:
the model makes the call, gets rejected, then has to repeat with
fetch_to_sandbox. With auto-spill, the data flows to a sandbox file
transparently. Validation showed ~10% fewer iterations with auto-spill.
7.4. Why fetch_to_sandbox still exists alongside auto-spill
Auto-spill handles the common case, but fetch_to_sandbox provides:
- Explicit path control. The model can choose meaningful filenames
(
/tmp/data/pipelineruns.json) instead of getting auto-generated names
(/tmp/data/_out/konflux_list_pipelineruns_1.txt).
- Batch fetching.
fetch_batch_to_sandbox combines multiple fetches
in one tool call, saving iterations.
- No preview overhead.
fetch_to_sandbox returns metadata only, which
is useful when the model knows it will process the file with
sandbox_exec anyway.
ToolDef has an optional notes field for domain knowledge that belongs
in the system prompt but not in the tool’s JSON schema. Examples:
- Konflux notes explain dual K8s/Kubearchive fetch, UID deduplication, and
the two label selectors (BUILD vs. TEST).
- Testing Farm notes explain XML result structure and usage patterns.
ToolRegistry.get_tool_notes() collects all non-None notes into a
## Data Source Notes section that is prepended to the workflow body in
the system prompt:
BASE_SYSTEM_PROMPT + tool_notes + workflow_body
This keeps domain knowledge close to the tool definitions (in the data
source module) rather than duplicated in every workflow .md file.
register_data_source() accepts optional response_metadata – a dict
that is merged into every response from that tool (inline, auto-spill, and
fetch_to_sandbox). Used for:
- Konflux:
{"konflux_ui": "https://..."} so the model can build
reviewer-facing PipelineRun links.
- Testing Farm:
{"artifacts_base": "https://..."} so the model can
build artifact links.
This avoids having the model ask “what is the Konflux UI URL?” – the
information arrives with every tool response.
8. Sandbox Architecture
8.1. The Sandbox protocol
All backends implement an 8-method typing.Protocol:
class Sandbox(Protocol):
def start(self) -> None # create container/pod, pre-create /tmp/data/
def exec(command, *, stdin_data: bytes | None) -> ExecResult # sh -c in sandbox
def write_file(path, data: bytes) -> None # stdin pipe: cat > path
def stream_to_file(path, chunks: Iterable[bytes]) -> tuple[int, int] # Popen stdin pipe, returns (bytes, lines)
def stream_exec(command, chunks: Iterable[bytes]) -> ExecResult # Popen with streaming stdin
def read_file_iter(path) -> Iterator[bytes] # Popen stdout pipe in chunks
def cleanup(self) -> None # rm -f container / delete pod
def linger(session_id) -> None # keep alive for reuse, or fall back to cleanup
ExecResult is (exit_code: int, stdout: str, stderr: str).
stream_to_file() and stream_exec() use subprocess.Popen with a stdin
pipe, writing chunks incrementally. This avoids buffering large payloads in
orchestrator memory (e.g., streaming a tar.gz archive into the sandbox).
read_file_iter() reads from a Popen stdout pipe in 64 KB chunks.
write_file() uses stdin piping (cat > path), never host volume mounts.
This is critical: it means data flows through the orchestrator process, not
through a shared filesystem. The sandbox has no host mounts.
8.2. Backend selection
Five sandbox backends are available:
| Backend |
--sandbox |
When to use |
PodmanSandbox |
podman |
Local development (default for run) |
K8sSandbox |
k8s |
Direct pod creation on a K8s cluster |
K8sPoolSandbox |
k8spool |
Pre-warmed pool via Deployment (default for serve) |
KubeVirtSandbox |
kubevirt |
Direct VMI creation with SSH exec |
KubeVirtPoolSandbox |
kubevirtpool |
Pre-warmed VMI pool via VMIRS |
create_sandbox() factory handles podman, k8s, and kubevirt. Pool
backends (k8spool, kubevirtpool) are handled in __main__.py, which
creates a SandboxPool or VmiPool and passes it to run_workflow().
The sandbox backend can be set per-workflow via the sandbox: field in the
workflow config YAML. Resolution order: workflow.sandbox > CLI --sandbox
default. This allows different workflows to use different backends (e.g.,
lightweight code review uses pod pool, heavier analysis uses VM pool).
K8s is never auto-detected from the environment. It requires explicit CLI
flags. This prevents accidental use of a K8s sandbox when developing
locally.
For K8s namespace resolution (k8s and k8spool):
- If
--namespace is provided, use it.
- If
--context is provided, extract the namespace from the kubeconfig
context. If the context has no default namespace, raise an error.
- If neither is provided (in-cluster), use
sandbox.namespace from the
config file.
8.3. PodmanSandbox
podman run -d --name hb-sandbox-{uuid8} --network=none --user 65532 \
--workdir /tmp {image} sleep infinity
- Unique container name with UUID suffix prevents collisions
sleep infinity keeps the container alive for repeated exec calls
--network=none provides complete network isolation
--user 65532 is a fixed non-root UID (matches nonroot in distroless)
- Cleanup:
podman rm -f (force, in case exec is still running)
8.4. K8sSandbox: the hybrid approach
The K8s sandbox uses a hybrid of two tools:
- kubernetes Python library for pod lifecycle:
create_namespaced_pod,
read_namespaced_pod (poll for Running), delete_namespaced_pod.
kubectl exec subprocess for command execution.
Why not use the kubernetes Python library for exec too? Three problems
discovered during development:
-
No stdin EOF signal in WebSocket v1-v4. The Kubernetes exec protocol
uses WebSocket channels (stdin=0, stdout=1, stderr=2). Protocol versions
1-4 have no mechanism to signal “stdin is done.” Commands like
cat > /file hang forever waiting for more input. Python client v5
support does not exist.
-
BrokenPipeError on large stdin. Sending more than ~1MB through the
WebSocket stream() API causes pipe errors, breaking write_file() for
large data source responses.
-
Unbounded memory from stream(). The stream() function accumulates
all stdout/stderr data in memory with no streaming control. A command
producing megabytes of output would consume unbounded memory.
kubectl exec as a subprocess avoids all three problems and provides the
same interface as podman exec – stdin via subprocess.PIPE, stdout/stderr
captured, exit code from return code. The implementation in exec() is
nearly identical between PodmanSandbox and K8sSandbox.
8.5. Pod manifest design
The K8s pod manifest is built by _build_pod_manifest():
metadata:
labels:
app.kubernetes.io/name: hummingbird-agent-sandbox
spec:
automountServiceAccountToken: false # no K8s API from sandbox
activeDeadlineSeconds: 1800 # 30min hard timeout, backstop
restartPolicy: Never
securityContext:
runAsNonRoot: true
seccompProfile: RuntimeDefault
containers:
- name: sandbox
command: ["sleep", "infinity"]
workingDir: /tmp
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
resources:
requests: { cpu: 100m, memory: 256Mi, ephemeral-storage: 256Mi }
limits: { cpu: "1", memory: 1Gi, ephemeral-storage: 2Gi }
runAsUser is deliberately omitted. On OpenShift, the restricted-v2 SCC
assigns a UID from the namespace UID range. On vanilla K8s, the image’s
USER directive is used. Setting an explicit UID would conflict with
OpenShift’s SCC admission.
8.6. Auth modes
- Local development:
--context flag passes the kubeconfig context to
config.new_client_from_config(context=...), creating a per-instance
ApiClient.
- Production (in-cluster):
context=None triggers
config.load_incluster_config(), using the pod’s ServiceAccount token.
The kubectl exec commands include --context when running locally but
omit it when in-cluster (kubectl uses the default in-cluster config).
8.7. Pre-created directories
Both backends run mkdir -p /tmp/data in start() after the
container/pod is up. This provides a working directory for the model
without consuming an iteration. /tmp/data/_out (the spill directory) is
created on demand by mkdir -p $(dirname ...) in write_file().
8.8. K8sPoolSandbox: Deployment-backed pool
On-demand pod creation adds 5-30 seconds of latency per workflow (image
pull, scheduling, container start). For interactive use (slash commands,
reply-based resumption), this delay is user-facing. The pool eliminates it.
Mechanism. A Kubernetes Deployment maintains a set of pods labelled
hummingbird/role: standby. When a sandbox is needed, SandboxPool.claim()
finds a Running standby pod, patches it to role: active, clears its
ownerReferences (detaching it from the ReplicaSet), and annotates it with
hummingbird/reap-by (an absolute UTC deadline). The Deployment controller
sees the ReplicaSet is under the desired replica count and creates a
replacement.
Pod lifecycle.
- Deployment creates pod ->
role: standby (managed by ReplicaSet)
claim() patches -> role: active, ownerReferences: [], reap-by: now + max_active_seconds
- Agent uses the pod via inherited
K8sSandbox.exec()
- On success:
linger() patches -> reap-by: now + linger_seconds,
session-id: <id> (pod stays alive)
- On reply within linger window:
try_reclaim() patches ->
reap-by: now + max_active_seconds, clears session-id
- On failure or linger expiry: pod is deleted (by
cleanup() or the reaper)
Pod lingering. After a successful workflow, pool sandboxes are kept
alive for linger_seconds (default 300) instead of being deleted
immediately. The pod is annotated with hummingbird/session-id to link it
back to the session. If a user reply arrives within the linger window,
try_reclaim() finds the pod by session-id, clears the annotation (marking
it as in-use), and resets reap-by. This skips pod creation and S3 archive
restoration. If no reply arrives, the reaper deletes the pod when its
reap-by deadline passes.
Concurrency safety. session-id is only present while lingering. Its
absence during active execution prevents concurrent replies from adopting
the same pod. try_reclaim() is serialized by a threading lock; the first
caller wins, others fall back to S3 restore with a fresh pod.
Reaping. reap_expired() is called once per workflow execution (after
sandbox acquisition, before the model loop). It performs a single sorted
pass over all active pods:
- Non-lingering pods (no
session-id) past their reap-by deadline are
deleted immediately.
- Lingering pods (with
session-id) are sorted by reap-by ascending.
The loop deletes pods that are either expired (now > reap-by) or
exceed max_lingering_pods (default 2, configurable), starting with
those closest to their deadline. The loop stops when both conditions are
satisfied (now <= reap-by and remaining count <= limit).
This replaces the previous design where _reap_expired() ran inside
claim() on every poll iteration. Moving it to a once-per-workflow call
reduces API load and centralizes cleanup. The max_lingering_pods cap
prevents unbounded pod accumulation from many short-lived workflows.
Indefinite wait. claim() polls until a pod is available or the
shutdown event fires. It logs at DEBUG level on each poll, escalating to
WARNING after ~2 minutes. This is normal behavior when the pool (Deployment
replicas) is smaller than max_concurrent_agents – the SQS semaphore caps
concurrency, so demand will not permanently exceed supply.
Config simplification. In pool mode, the pod image, resources, metadata,
and security context are defined solely in the Deployment template. The
agent config only needs sandbox.namespace and
sandbox.active_deadline_seconds. This eliminates duplication between the
agent configmap and the Deployment.
Class design. K8sPoolSandbox inherits from K8sSandbox to reuse
exec(), write_file(), read_file_iter(), and cleanup(). It overrides
__init__() (does not call super().__init__() since that resolves
kubeconfig), start() (claims from pool instead of creating a pod), and
linger() (patches annotations instead of deleting). It adds
start_from_existing() for the reclaim path. All sandbox backends
implement linger(session_id): non-pool backends fall back to cleanup().
8.9. KubeVirtSandbox: SSH-based VM execution
KubeVirtSandbox provides full VM isolation using KubeVirt
VirtualMachineInstances. It parallels K8sSandbox but uses
CustomObjectsApi for VMI lifecycle and ssh for command execution
instead of kubectl exec.
Why SSH? VMs have no kubectl exec equivalent. KubeVirt provides
virtctl console (serial) and VNC, but these are not suitable for
programmatic command execution. SSH provides a reliable, well-understood
exec channel with stdin/stdout/stderr piping.
SSH key injection. The VM sandbox image includes a custom
inject-ssh-keys.service that reads SSH public keys from a KubeVirt
Secret volume attached as a virtio disk with serial ssh-pubkeys
(visible at /dev/disk/by-id/virtio-ssh-pubkeys) and installs them to
/root/.ssh/authorized_keys before sshd starts. This avoids the need
for cloudInitNoCloud or qemu-guest-agent.
Single-VM lifecycle:
- Generate an ephemeral ed25519 key pair via
ssh-keygen
- Create a K8s Secret with the public key
- Create the VMI referencing the containerDisk image and the SSH Secret
- Poll VMI status until Running with a pod network IP
- Poll SSH readiness (
ssh sandbox@ip true)
- Execute workflow commands via SSH
- Cleanup: delete VMI, delete Secret, remove temp key files
Polling strategy. Two-phase wait: first poll status.phase via
get_namespaced_custom_object until Running (same interval as pod
polling), then poll SSH readiness with ConnectTimeout=5. Total timeout
matches POD_START_TIMEOUT (300s) for VMI startup plus
_SSH_READY_TIMEOUT (120s) for SSH.
8.10. KubeVirtPoolSandbox: VMIRS-backed pool
KubeVirtPoolSandbox + VmiPool parallel K8sPoolSandbox +
SandboxPool, operating on VMI custom resources instead of Pods.
VirtualMachineInstanceReplicaSet (VMIRS). KubeVirt’s native
controller maintains a desired number of identical VMIs. When a VMI is
claimed (label flipped to active, ownerReferences cleared), the VMIRS
automatically creates a replacement – the same pattern as a Deployment
replenishing claimed pods.
Shared SSH key. Pool mode uses a single key pair for all standby VMIs.
The public key is stored in a K8s Secret referenced by all VMIs in the
VMIRS template. The private key path is provided via
sandbox.vm_ssh_private_key in the config. On pool restart, a new key
pair can be generated; existing VMIs drain naturally with the old key.
Claim/reap/linger semantics. Identical to the pod pool: claim()
polls for Running VMIs with hummingbird/role=standby, patches to
active, clears ownerReferences. reap_expired() deletes VMIs past
their hummingbird/reap-by deadline. try_reclaim() finds lingering
VMIs by hummingbird/session-id. The only difference is the API: all
operations go through CustomObjectsApi with kubevirt.io/v1
virtualmachineinstances.
Class design. KubeVirtPoolSandbox inherits from KubeVirtSandbox
to reuse exec(), write_file(), read_file_iter() (all SSH-based).
It overrides __init__() (takes pool, skips key generation),
start() (claims from pool), cleanup() (only deletes VMI, not
Secret/keys which are pool-owned), and linger() (patches annotations).
4 - Hummingbird Agent Evals
Evaluation framework for scientific measurement of LLM output quality.
For the agent itself, see Hummingbird Agent.
For architecture details, see
Hummingbird Agent Design.
What evals are
Evals measure the quality of LLM outputs under controlled conditions.
They answer questions like “does prompt v5 produce better code reviews
than v4?” or “can an LLM answer this question correctly using only
public documentation?”
Evals are not unit tests. Unit tests verify deterministic behavior;
evals quantify stochastic output quality through structured scoring.
General pattern
Evals generally follow a pattern of:
- Inputs – a set of test cases: golden dataset with scoring
criteria, real MRs to review, or other structured inputs.
- Model execution – run one or more LLM configurations against
the inputs, optionally with tools (web search, sandbox).
- Scoring – evaluate outputs via LLM-as-judge, comparison
against reference answers, or human review.
- Iteration – adjust prompts or config, re-run, check for
improvement.
Not all evals use every step. Baseline measurements may stop at
step 3; comparison evals may skip the golden dataset entirely.
Shared infrastructure
hummingbird_agent.eval
The hummingbird_agent.eval module provides generic evaluation
primitives used across all evals. Key components:
EvalJudge – LLM-as-judge wrapper over Vertex AI with explicit
temperature and token limit control. Supports both strict judging
(low temperature) and open-ended analysis (higher temperature).
- Snapshot management – saves timestamped evaluation results with
metadata, maintains a
latest symlink.
- Variant tracking – persists the best-performing prompt or config
variant across iterations.
- Convergence detection – checks whether scores have plateaued
across consecutive iterations.
- Holdout validation – compares holdout scores against dev scores
using standard error of the mean.
See the module docstrings for the full API.
hummingbird_agent.evals.mr_helpers
MR-experiment-specific helpers built on top of hummingbird_agent.eval.
Provides GitLab auth, MR data I/O, review generation orchestration, and
CLI scaffolding. Used by the code-review and renovate-triage evals.
Running an eval
Prerequisites
-
GCP Application Default Credentials:
gcloud auth application-default login
export GOOGLE_CLOUD_PROJECT=<your-gcp-project>
-
The agent package installed in editable mode:
cd hummingbird-agent
pip install -e .
Execution
Each eval is a Python module under hummingbird_agent/evals/. Run from
hummingbird-agent/:
python -m hummingbird_agent.evals.<name>.<script> [options]
Refer to each eval’s own README for specific options and configuration.
Existing evals
Writing a new eval
Create a new directory under hummingbird_agent/evals/ with:
README.md – what the eval measures and how to run it, linking
back to this document for the general framework.
- Golden dataset – inputs and scoring criteria. Can live outside
the eval directory (e.g. in the documentation repo alongside the
content being measured) and be passed via CLI.
run_*.py – orchestration script using hummingbird_agent.eval
primitives (or hummingbird_agent.evals.mr_helpers for MR-based evals).
The eval’s run script should be importable as a module
(python -m hummingbird_agent.evals.<name>.<script>). Add an __init__.py if needed.
Do not hardcode GCP project IDs; VertexAuth resolves them at
runtime from GOOGLE_CLOUD_PROJECT or ADC credentials.
Authentication
All evals use VertexAuth from hummingbird_ai.http, which
resolves credentials and project in this order:
- Explicit
project= parameter (if passed).
GOOGLE_CLOUD_PROJECT environment variable.
- ADC default project (from
gcloud auth application-default login).
Service account keys work automatically when
GOOGLE_APPLICATION_CREDENTIALS is set.
5 - Documentation Quality Evaluation
Reference for the documentation quality evaluation, which measures whether
public documentation is complete enough for LLMs with web search to answer
questions correctly. For the general evaluation framework, see
Hummingbird Agent Evals.
Config file
A single YAML file combines evaluation settings and the golden dataset.
It is passed to the run script via -c:
python -m hummingbird_agent.evals.docs_eval.run_eval -c path/to/docs_eval.yml
The config file lives in the documentation repo alongside the content
being measured.
Evaluation settings
| Key |
Required |
Default |
Description |
models |
yes |
– |
List of model configurations |
models[].id |
yes |
– |
Vertex AI model identifier |
models[].region |
yes |
– |
Vertex AI region |
models[].google_search |
no |
false |
Enable Google Search grounding (Gemini) |
models[].web_search |
no |
false |
Enable web search tool (Claude) |
judge |
yes |
– |
Judge model configuration |
judge.model |
yes |
– |
Judge model identifier |
judge.region |
no |
global |
Judge model region |
repetitions |
no |
3 |
Runs per (question, model) pair |
max_iterations |
no |
5 |
Agent loop iteration cap per question |
Golden dataset
The questions list defines what the evaluation measures:
| Field |
Required |
Default |
Description |
id |
yes |
– |
Unique identifier (used in output paths) |
question |
yes |
– |
Question sent to each model |
facts |
yes |
– |
List of fact checks |
facts[].question |
yes |
– |
Yes/no question about the response |
facts[].weight |
no |
1 |
Relative importance |
coherence |
no |
– |
Criterion for narrative quality scoring |
coherence_weight |
no |
1.5 |
Weight of coherence in the final score |
threshold |
no |
0.4 |
Minimum weighted score to pass |
CLI options
| Option |
Default |
Description |
-c / --config |
required |
Path to config YAML |
--phase |
all |
ask, judge, or all |
--results-dir |
results/ next to config |
Output directory |
--run-id |
UTC timestamp |
Subdirectory under results |
Scoring
Each response is scored independently:
- Fact scoring – the judge answers each fact question with yes or no.
Passing facts contribute their weight to the numerator.
- Coherence scoring – the judge rates narrative quality from 0.0 to
1.0. The coherence score is multiplied by its weight and added to the
total.
- Weighted score –
sum(passed_fact_weights + coherence_contribution) / total_weight.
A response passes if weighted_score >= threshold.
The summary aggregates per-model averages across all repetitions.
6 - Hummingbird Agent Design — Runtime
Data sources, event pipeline, sessions, workflows, and deployment for the Hummingbird Agent.
This page covers the runtime: how the agent connects to external systems, processes events,
manages sessions, and is deployed. For the execution engine (agent loop, tool system, sandbox),
see Hummingbird Agent Design — Engine.
9. Data Sources
Data sources are external APIs wrapped as tool-calling functions. The model
invokes them by name; the orchestrator executes them and returns results
(inline or auto-spilled). Each data source module follows the same pattern:
- Define
TOOL_DEFS – a list of ToolDef objects with names,
descriptions, parameter schemas, and optional notes.
- Define implementation functions that take a pre-configured client as the
first argument.
- Provide a
register() function that creates the client, wraps
implementations with functools.partial, and calls
registry.register_data_source() for each tool.
Credentials are captured at registration time via functools.partial
closures. They are never stored as global state and never leak into tool
definitions or model contents.
9.1. GitLab
Uses python-gitlab library with retry_transient_errors=True for
automatic retry on transient HTTP errors.
Tools:
gitlab_get_mr_details – MR metadata (title, author, SHA, labels, URLs)
gitlab_get_commit_statuses – all CI/CD statuses for a commit SHA
(paginated automatically). Covers both Konflux external statuses and
native GitLab CI job statuses. Each status includes target_url (job
page link containing the job ID) and allow_failure.
gitlab_get_mr_diff – changed files in the MR
gitlab_get_file_at_ref – raw file content at a git ref
gitlab_get_repo_archive – repository tar.gz via
repository_archive(iterator=True). Returns a StreamingResponse with
chunked binary data (suffix .tar.gz), so the archive streams to the
sandbox without buffering in orchestrator memory.
gitlab_get_job_log – job trace (log output) for a GitLab CI job.
Uses lazy=True on the job object and trace(iterator=True) for
streaming. Each chunk is decoded with errors='replace' and ANSI escape
codes are stripped per-chunk before re-encoding. Returns a
StreamingResponse (suffix .log). Per-chunk ANSI stripping is safe
because escape sequences are <20 bytes and chunks are 1024+ bytes.
No response_metadata is set because GitLab commit statuses already contain
target_url fields that the model uses for linking.
gitlab_get_mr_discussions – trust filtering and redaction.
MR discussions are the first data source that feeds user-generated free text
into the model prompt. Unlike diffs and CI logs (which are code or machine
output), discussion comments can contain arbitrary prose written by anyone
who can post on the MR – including external contributors on public projects.
This creates a prompt injection vector: an attacker posts a comment containing
instructions that manipulate the model’s review output.
The discussions tool addresses this with a three-layer defence:
-
Author trust gate. Each note’s author is checked for Developer+ access
(level >= 30) on the project via members_all.get(author_id). Results are
cached per author_id within a single call to avoid repeated API lookups.
System notes (merge events, label changes) bypass the author check.
Agent-authored notes are trusted via their author’s Developer+ access
(the bot account holds Developer access on configured projects). Note:
agent note detection for redaction purposes (stripping transcripts and
footers) uses session marker presence, but trust is always based on the
author’s access level, not the marker. This prevents marker injection
from granting trust to non-member notes.
-
Redaction, not sanitization. Untrusted notes in a discussion that also
contains trusted notes are replaced with a fixed placeholder
([redacted: non-member comment]). The placeholder preserves the
conversation structure (the model sees that someone replied, but not what
they said). Discussions where all notes are untrusted are dropped
entirely. No attempt is made to sanitize or escape untrusted content –
there is no reliable escaping mechanism for LLM prompts, so the only safe
option is to withhold the content entirely.
-
Agent note stripping. Agent-authored notes contain large
<details><summary>Agent transcript</summary>...</details> blocks
(15K-90K chars of raw tool calls), metadata footers, and session markers.
These are stripped before the note enters the model prompt, leaving only
the review text. This serves dual purposes: token efficiency and avoiding
feeding the model its own raw tool call history (which could cause
degenerate self-referential loops).
Residual risks and scope limitations:
- A compromised Developer+ account can inject adversarial content. This is
accepted as equivalent to the existing risk of a compromised developer
pushing malicious code (which the agent would also process).
- The trust threshold is project-level (Developer role on the project), not
MR-level. A developer on the project can influence any MR’s review.
- Comments posted after the discussions are fetched but before the review
is posted are not seen. This is a TOCTOU gap but has no security impact
(the model simply misses late comments).
9.2. Konflux
Uses raw requests against K8s and Kubearchive APIs (not the kubernetes
Python SDK). This avoids the heavy kubernetes client dependency for what is
essentially bearer-token HTTP with label selectors.
Dual-fetch architecture:
flowchart LR
subgraph fetch [Fetch Phase]
KA["Kubearchive<br/>(historical)"]
K8s["K8s API<br/>(live)"]
end
Combine["Combine items"]
Dedup["Deduplicate<br/>by metadata.uid"]
KA --> Combine
K8s --> Combine
Combine --> Dedup
For each resource type (PipelineRuns, TaskRuns), the client fetches from
both Kubearchive (completed/historical resources) and the live K8s API,
combines the results, and deduplicates by metadata.uid. This ensures no
resources are missed regardless of whether they have been archived yet.
Two label selectors. Konflux uses different labels for BUILD and TEST
PipelineRuns:
- BUILD:
pipelinesascode.tekton.dev/sha=<commit_sha>
- TEST:
pac.test.appstudio.openshift.io/sha=<commit_sha>
Each iter_pipelineruns()/iter_taskruns() call queries both selectors and
combines the results. _iter_resources(endpoint, label_selectors) is the
generic implementation: it creates one session, iterates each selector through
_iter_both_sources(), and deduplicates across selectors by UID.
Per-PLR TaskRun fetching. iter_taskruns() accepts either commit_sha
(two-selector path) or pipelinerun (single selector:
tekton.dev/pipelineRun=<name>). When pipelinerun is set, only the
~15-21 TaskRuns for that PipelineRun are fetched (~600 KB) instead of all
TaskRuns for the commit (~530 MB for large commits). This is server-side
filtering — the K8s API and Kubearchive both support the
tekton.dev/pipelineRun label selector.
Status filtering and streaming counts. _items_to_jsonl(items, status, stats) accepts an optional status: set[str] | None allowlist and a
_StreamStats accumulator. For each item it increments stats.by_reason
(keyed by reason) regardless of the filter (so counts reflect all items
seen), then skips items whose status.conditions[0].status is not in the
allowlist. The allowlist values are "True" (success), "False" (failure),
or "Unknown" (in-progress) — only 3 possible values, covering every Tekton
reason without enumeration. Items without conditions always pass. The
_finalize() generator wrapper exhausts the chunks iterator, then writes
counts, included, and excluded into the StreamingResponse.metadata
dict — after the sandbox file is fully written. The filtering happens O(1)
per item with no buffering.
StreamingResponse.metadata. StreamingResponse (in tools.py) has a
mutable metadata: dict[str, Any] field (default empty dict). Both
_exec_fetch_to_sandbox() and _spill_streaming() merge it into the response
dict after the iterator is exhausted. The Konflux code populates it with
counts, included, and excluded; other data sources can use it for their
own post-streaming summaries without touching tools.py.
Pod log fetching. get_pod_log() accepts an optional container
parameter. When specified, it fetches logs for that single container. When
omitted, it discovers all containers from the pod spec and concatenates
their logs with === container_name === headers. Each individual log
fetch tries Kubearchive first, then falls back to the live K8s API. Logs
may be unavailable from both sources if the pod has expired.
Streaming pagination. K8s list endpoints can return pages of 80+ MB
when a commit touches many components (e.g. 500 PipelineRuns per page).
iter_paginated() uses session.get(stream=True) and ijson.parse() to
stream-parse each page: items are yielded one at a time via
ObjectBuilder, and the metadata.continue pagination token is captured
from the same parse pass. resp.raw.decode_content = True is set so
urllib3 transparently decompresses gzip/deflate Content-Encoding inline
(Kubearchive returns gzip-compressed responses). Only one item is in
memory at a time – O(single_item) regardless of page size. Truncated or
malformed streams raise IncompleteJSONError, which is caught and treated
like a network error (log a warning, stop iterating that source).
Credential resolution. KonfluxClient.__init__() parses the kubeconfig
file to extract the API server URL and bearer token for the cluster. The
cluster domain is extracted from the cluster_url config value. This
approach avoids depending on kubectl or the kubernetes Python library for
API authentication.
Response metadata: Static metadata {"konflux_ui": "...", "konflux_pipelinerun_url_pattern": "..."} is merged into every response so the model can build reviewer-facing links.
Dynamic per-call metadata (counts, included, excluded) is populated
into StreamingResponse.metadata by _finalize() during streaming and
merged by tools.py after the iterator is exhausted.
9.3. Testing Farm
Uses requests.Session with a module-level retry adapter (429, 5xx) for
resilience against transient errors.
Tools:
tf_get_results – JUnit XML results for a request ID
tf_get_test_log – individual test log by URL (from results.xml);
restricted to the ARTIFACTS_BASE prefix to prevent SSRF
tf_get_request_status – request state, queue/run times
ToolDef notes on tf_get_results document the XML structure:
//testsuites/@overall-result, //testcase/@result, //testcase/logs/log
with @name and @href. This domain knowledge goes into the system prompt
so the model knows how to parse the XML with sandbox_exec using
python3 xml.etree.ElementTree.
Response metadata: {"artifacts_base": "https://artifacts.osci.redhat.com/testing-farm"}
is merged into every response so the model can build artifact links like
[{request_id}]({artifacts_base}/{request_id}/).
9.4. Registration flow
data_sources.register_selected() is the entry point called by runner.py.
It iterates the workflow config’s data_sources dict and registers only the
declared sources:
for ds_name, ds_cfg in wf_cfg.data_sources.items():
if ds_name == "gitlab":
token = resolve_tool_token(proj_entry, "gitlab", wf_cfg)
gitlab.register(registry, gitlab_url, token)
elif ds_name == "konflux":
cluster_url = resolve_cluster_url(ds_cfg)
kubeconfig_path = os.environ.get(ds_cfg.kubeconfig_env, "")
konflux.register(registry, cluster_url, kubeconfig_path, ds_cfg.kubearchive_url)
elif ds_name == "testing_farm":
testing_farm.register(registry)
This selective registration means a workflow with data_sources: {gitlab: {...}}
only exposes GitLab tools to the model. Konflux and Testing Farm tools do
not appear in the tool definitions, preventing the model from attempting to
use unconfigured data sources.
9.5. Data flow: streaming vs buffered
Data moves from external APIs through the orchestrator to the model (inline
or auto-spilled to sandbox). The memory profile of each tool depends on
whether the HTTP response is consumed incrementally or buffered entirely.
Streaming (preferred for large/unbounded responses). The HTTP response
is consumed incrementally – via ijson stream parsing, iterator=True
in python-gitlab, or lazy pagination. Only one chunk or item is in memory
at a time. Used by:
iter_paginated() (Konflux) – session.get(stream=True) + ijson
get_commit_statuses() (GitLab) – statuses.list(iterator=True) via
_iter_commit_statuses(), yielding JSONL bytes one status at a time
get_repo_archive() (GitLab) – repository_archive(iterator=True),
returns chunked tar.gz binary via StreamingResponse(suffix=".tar.gz", binary=True) – no text preview is generated
get_job_log() (GitLab) – trace(iterator=True) with per-chunk ANSI
stripping, returns StreamingResponse(suffix=".log")
All produce StreamingResponse objects that flow through
stream_to_file() into the sandbox without accumulating in orchestrator
memory. StreamingResponse.suffix controls the auto-spill filename
extension (.jsonl, .tar.gz, .log). StreamingResponse.binary
controls whether a head preview is captured (False for text, True to
skip for opaque binary formats).
Buffered (acceptable for small/bounded responses). The full response is
loaded into memory as a string or dict. This is fine when the response size
is bounded and small (typically < 1 MB). Used by:
get_mr_details() (GitLab) – single MR object, < 10 KB
get_file_at_ref() (GitLab) – single file, bounded by repo constraints
tf_get_results(), tf_get_test_log(), tf_get_request_status()
(Testing Farm) – XML/text/JSON, typically < 1 MB
Buffered with risk (candidates for future streaming). Same buffered
pattern but the response size is not bounded by design. Auto-spill
mitigates the context growth problem (large outputs are written to sandbox
files), but the orchestrator still spikes RSS during the fetch:
get_pod_log() (Konflux) – resp.text per container, concatenated.
Multi-container pods accumulate all logs.
get_mr_diff() (GitLab) – full diff JSON, scales with MR size.
GitLab truncates server-side but the result can still be large.
Invariant: paginated K8s list endpoints must always use streaming.
Page sizes scale with the number of components in the commit – a single
page can contain hundreds of PipelineRuns or TaskRuns (80+ MB JSON).
Buffering these responses risks OOM under normal production workloads,
especially with concurrent workflows.
10. Event Pipeline and Production Operations
10.1. Two-stage SQS pipeline (ingress router + FIFO worker)
Events flow through a two-stage pipeline to serialize per-session work:
flowchart LR
STD["Standard SQS<br/>(ingress)"]
R["Router"]
FIFO["SQS FIFO<br/>(work queue)"]
W["Worker<br/>(poll_loop)"]
STD -->|ReceiveMessage| R
R -->|"SendMessage<br/>MessageGroupId"| FIFO
W -->|"continuation<br/>group=session_id"| FIFO
FIFO -->|ReceiveMessage| W
Ingress router. events.router_loop() is a single-threaded loop
that long-polls the standard (ingress) queue, peeks into each SNS
envelope to assign a MessageGroupId, forwards the raw message body
to the FIFO queue, then deletes from the standard queue.
- Note events:
MessageGroupId = discussion_id from the webhook
body. This serializes replies to the same agent session – FIFO
delivers at most one in-flight message per group.
- Pipeline / MR events:
MessageGroupId = SQS MessageId (unique
per message). No serialization – each event is its own group.
- MessageDeduplicationId: Always the SQS
MessageId from the
standard queue. Absorbs duplicate deliveries from standard SQS’s
at-least-once semantics (5-minute dedup window).
If SendMessage to FIFO fails, the message is not deleted from the
standard queue and retries via visibility timeout.
FIFO worker. events.poll_loop() long-polls the FIFO queue,
decodes SNS envelopes, and dispatches to handle_event. The
max_concurrent_agents semaphore caps parallel groups being processed.
FIFO guarantees that within a group, only one message is in-flight.
Two-phase processing. When the FIFO worker picks up a webhook
(Phase 1), the handler validates the event, resolves the session_id,
then posts a continuation message back to the same FIFO with
MessageGroupId = session_id. The Phase 1 message is deleted quickly
(~1-5s). Phase 2 processes the continuation: loading session state,
building WorkflowRequest, and calling _execute_workflow. Because
Phase 2 continuations share a MessageGroupId per session, all work
for a given session is serialized – even if it spans multiple GitLab
discussion threads.
Uniform message format. Both webhook messages (from SNS) and
continuation messages use the same SNS-style envelope with gzip+base64
compression. encode_envelope() is the inverse of
decode_sns_message(). The consumer doesn’t need to distinguish
between webhook and continuation messages at the transport layer.
Graceful shutdown. SIGTERM/SIGINT set a shutdown_event. Both
loops exit. Messages in SQS become visible again after the visibility
timeout for other consumers.
Why two queues. The standard queue receives events from SNS (via
subscription). The FIFO queue serializes per-session work. The
gitlab-event-forwarder and SNS subscription are unchanged – the
routing logic lives entirely in the agent codebase.
10.2. SNS envelope decoding
Messages arrive as SNS notifications with:
MessageAttributes.source – event source (e.g., "gitlab")
MessageAttributes.event_type – event type (e.g., "pipeline",
"merge_request", "note")
MessageAttributes.content_encoding – "gzip+base64" for
gitlab-event-forwarder, absent for kubernetes-event-forwarder
Message – the actual event body (JSON string, or gzip+base64 encoded)
decode_sns_message() handles both encoding formats transparently.
10.3. Event routing
runner.handle_event() dispatches on (source, event_type):
flowchart TD
Event["Event arrives"]
Check{"source/type?"}
Cont["handle_continuation()"]
Pipeline["handle_pipeline()"]
MRHandler["handle_merge_request()"]
Note["handle_note()"]
Ignore["Ignore"]
Event --> Check
Check -->|"agent/continuation"| Cont
Check -->|"gitlab/pipeline"| Pipeline
Check -->|"gitlab/merge_request"| MRHandler
Check -->|"gitlab/note"| Note
Check -->|"other"| Ignore
Pipeline --> FilterSource{"source = merge_request_event?"}
FilterSource -->|"yes"| AutoResolve["auto-resolve on success<br/>(unconditional)"]
FilterSource -->|"no"| Skip1["Skip"]
AutoResolve --> PipeRules["evaluate trigger_rules<br/>(status + user + branch + title)"]
PipeRules -->|"allow"| Enqueue1["_enqueue_continuation()"]
PipeRules -->|"deny"| Skip1a["Skip"]
MRHandler --> FilterMRAction{"action in open/update?<br/>draft = false?"}
FilterMRAction -->|"yes"| MRRules["evaluate trigger_rules<br/>(user + branch + title)"]
FilterMRAction -->|"no"| Skip1b["Skip"]
MRRules -->|"allow"| Enqueue2["_enqueue_continuation()"]
MRRules -->|"deny"| Skip1c["Skip"]
Note --> FilterMRNote{"MR note?<br/>action = create?"}
FilterMRNote -->|"yes"| NoteType
FilterMRNote -->|"no"| Skip3["Skip"]
NoteType{"Note type?"}
NoteType -->|"author_id == bot_id"| Skip4["Skip (own note)"]
NoteType -->|"/hummingbird help"| Help["post_simple_reply (help)"]
NoteType -->|"/hummingbird wf-name"| SlashCmd["_resolve_slash_workflow()<br/>→ _enqueue_continuation()"]
NoteType -->|"DiscussionNote reply"| FindSession["find_session_for_reply()"]
NoteType -->|"other"| Skip5["Skip"]
FindSession --> Found{"session found?"}
Found -->|"yes"| Enqueue3["_enqueue_continuation()<br/>(resume_session)"]
Found -->|"no"| Skip6["Skip"]
Cont --> ContType{"work_type?"}
ContType -->|"new_session"| ExecNew["_execute_workflow()"]
ContType -->|"resume_session"| LoadS3["load session from S3<br/>→ _execute_workflow()"]
Phase 1 handlers (handle_pipeline, handle_merge_request,
handle_note) perform validation, auth checks, and session resolution,
then post a WorkOrder continuation message via _enqueue_continuation().
They do not call _execute_workflow directly.
Phase 2 (handle_continuation) receives the WorkOrder, resolves
the workflow config, builds a WorkflowRequest, and calls
_execute_workflow. For resume_session orders, it loads the previous
session from S3. _execute_workflow handles SHA dedup and per-workflow
rate limiting via scan_agent_threads.
Slash commands dispatch to a single named workflow via
_resolve_slash_workflow (exact match, then prefix match). Help and error
replies are posted via post_simple_reply (no session marker, no rate
limit impact).
10.4. Pipeline trigger design
Why gitlab::pipeline instead of kubernetes::PipelineRun: A GitLab
pipeline event fires once when the pipeline completes. Since Konflux
external stages are attached to the pipeline, the event naturally waits for
all builds and tests to finish before triggering. This means the agent sees
the full picture in one event, without needing to deduplicate or wait for
stragglers.
Trigger filters:
status in {failed} – only failed pipelines trigger analysis
source == "merge_request_event" – only MR pipelines, not branch/tag
10.4a. Merge request trigger design
The handle_merge_request handler fires on MR open, reopen, and update
events (action in {"open", "reopen", "update"}). Draft MRs are skipped
(object_attributes.draft == True); marking a draft as ready triggers a
review since the event arrives with draft: false.
Event classification is deliberately simple: the handler does not inspect
oldrev or changes.draft fields. Instead, SHA-based deduplication in
_execute_workflow (via JSON session markers) ensures each code revision
is reviewed at most once. Metadata-only updates (title, label changes) on
an already-reviewed SHA are silently skipped.
10.5. Note trigger design
Two sub-flows:
Slash command (/hummingbird <workflow-name>): Triggers a specific
named workflow, bypassing rate limiting. _resolve_slash_workflow performs
exact-match lookup first, then falls back to unique prefix matching. If
the subcommand is missing or is “help”, _format_help returns a list of
available workflows with descriptions. Ambiguous or unknown subcommands
produce an error message via post_simple_reply. The note author must
have Developer+ access (level >= 30) on the project, checked via
check_member_access(). If denied, the agent replies in the same
discussion thread with a short access-denied message.
Reply to agent note: When a user replies to an existing agent note
(which contains a JSON session marker):
find_session_for_reply() walks the discussion thread, filtering by
bot author ID (get_bot_user_id()), and returns the first matching
session marker. All bot markers in a thread share the same session ID.
handle_note posts a resume_session continuation to the FIFO
(grouped by session_id).
handle_continuation loads the session from S3 (conversation history and
sandbox archive), builds a WorkflowRequest, and calls
_execute_workflow() to resume with the user’s reply.
- If the session is not found (expired/deleted), the agent replies with
a message explaining the session has expired and suggesting to start
a new run.
Reply authors are also subject to the Developer+ access check. If denied,
the agent replies in the discussion thread with the same access-denied
message.
Non-agent-directed notes: Regular comments that are neither slash
commands nor replies to agent threads are silently ignored (debug-level
log). No access check is performed for these.
Self-note filtering: Notes where author_id matches the bot’s own
user ID (resolved via get_bot_user_id()) are skipped immediately. This
prevents infinite loops where the agent’s own output triggers another
agent run. The author-ID check replaces the previous substring check
(marker_prefix_str in note_body), which was vulnerable to
denial-of-service: a user including the marker prefix in their reply
would cause the bot to silently ignore it.
Threading logic for replies: If the project requires internal notes
(internal_notes: true) but the original discussion was public, the reply
is posted as a new top-level internal note instead of replying in the
public thread. This prevents leaking internal analysis into public threads.
10.6. Note lifecycle
1. create_placeholder_note() -> "Running the <workflow> workflow..."
+ JSON session marker (id, wf, sha)
2. run_workflow() -> agent loop
3. update_note() -> replace placeholder with result
+ reply prompt + JSON session marker
Reply notes (from session resumption) omit the reply prompt since the
user is already engaged in the conversation.
On failure, the placeholder is updated to “Hummingbird analysis failed.”
The JSON session marker is always present so the note can be identified as
agent-generated (for rate limiting, SHA dedup, and self-note filtering).
The marker embeds the workflow name and commit SHA, enabling per-workflow
rate limiting and SHA-based deduplication.
10.7. Rate limiting and SHA deduplication
scan_agent_threads() performs a single pass over MR discussions, parsing
JSON session markers (<!-- hummingbird-session: {"id":...,"wf":...,"sha":...} -->)
to determine:
- Per-workflow thread count – how many threads the given workflow has
created on this MR. If the count meets or exceeds
max_runs_per_mr,
the workflow is skipped.
- SHA dedup – whether the current commit SHA has already been reviewed
by this workflow. If so, the workflow is skipped (prevents re-reviewing
the same code on metadata-only MR updates).
Slash commands and replies bypass both checks entirely.
The rate limit is per-workflow per-MR: different workflows maintain
independent thread counts on the same MR. JSON markers are backward
compatible – old plain-UUID markers (<!-- hummingbird-session: UUID -->)
are parsed as {"id": "UUID"} with no workflow or SHA information, so they
are not counted toward any specific workflow’s limit.
11. Session System
Sessions enable conversation continuity: a user can reply to an agent note
and the agent picks up where it left off, with full context and sandbox
files restored.
11.1. What gets saved
Three artifacts are saved after each run:
context.json – the full contents list from the agent loop in the
canonical provider-independent format. It is the minimum state needed to
resume the conversation; format conversion is handled by the AI library.
transcript.md – a human-readable markdown rendering of the run for
debugging and auditing. Not used for resumption.
sandbox.tar.gz – an archive of /tmp/data/ (the sandbox working
directory, which includes the _out/ spill subdirectory). Contains
PipelineRun JSONs, test logs, jq output, and any other files the model
created during the run. Restored into the new sandbox on resumption so
the model can reference its previous work.
11.2. Storage backends
S3 (production):
s3://{bucket}/sessions/{session_id}/context.json
s3://{bucket}/sessions/{session_id}/transcript.md
s3://{bucket}/sessions/{session_id}/sandbox.tar.gz
Local directory (development, --save-session):
{directory}/context.json
{directory}/transcript.md
{directory}/sandbox.tar.gz
Both backends have the same interface. S3 save is best-effort: wrapped in
try/except so a transient S3 error does not prevent the MR note from being
posted. The note always gets delivered first.
11.3. Sandbox archive transport
The sandbox archive requires special handling because the orchestrator
cannot directly access the sandbox filesystem (no volume mounts):
Archive (inside sandbox):
sb.exec("tar czf /tmp/_archive.tar.gz -C /tmp data")
Stream out (sandbox -> host temp file):
for chunk in sb.read_file_iter("/tmp/_archive.tar.gz"):
tmp.write(chunk) # 64 KB chunks, no base64
Restore (host temp file -> new sandbox):
sb.stream_exec("tar xzf - -C /tmp", file_chunks())
Both directions use streaming binary I/O via Popen stdin/stdout pipes.
No base64 encoding is needed: read_file_iter() yields raw bytes from
the sandbox via a Popen stdout pipe, and stream_exec() feeds raw bytes
into a Popen stdin pipe. The archive is never extracted on the
orchestrator – it exists only as opaque bytes being transported between
sandboxes. This is a security invariant: the orchestrator never parses or
inspects the archive contents.
11.4. Resumption flow
flowchart TD
Reply["User replies to agent note"]
FindSession["find_session_for_reply()<br/>walks discussion thread"]
LoadS3["load_s3(session_id)"]
NotFound{"session found?"}
Expired["Post session-expired reply"]
NewSandbox["Start new sandbox"]
Restore["restore_sandbox(archive)"]
BuildContents["contents = old_contents + user_reply"]
RunLoop["run_agent_loop(<br/>initial_contents, CONTINUATION_PROMPT)"]
Reply --> FindSession --> LoadS3
LoadS3 --> NotFound
NotFound -->|"no"| Expired
NotFound -->|"yes"| NewSandbox
NewSandbox --> Restore --> BuildContents --> RunLoop
Key aspects:
- Fresh iteration budget. The resumed session starts iteration 0 with
the full
max_iterations budget, regardless of how many iterations the
previous session used.
- Context window is the real limit. A typical 8-iteration cold start
uses ~50-100K input tokens. The restored history is sent in full on every
API call. The
CONTEXT_LIMIT check applies and will force wrap-up if
needed.
- Session ID reuse. The resumed session keeps the original session
ID. S3 state is overwritten in place with the updated conversation
history. Since GitLab discussions are linear (not branching), there is
no need for a tree of sessions – the conversation is a single
sequential thread.
- Graceful degradation. If the S3 session is not found (expired,
deleted), the agent posts a reply explaining the session has expired and
suggesting to start a new run. It does not fall back to a cold start.
11.5. Session markers
Every agent note contains a hidden HTML comment with a JSON payload:
<!-- hummingbird-session: {"id":"UUID","wf":"code-review","sha":"abc123"} -->
The JSON payload contains:
id – session UUID (always present)
wf – workflow name (present for new-format markers)
sha – commit SHA at the time of review (present for auto-triggered runs)
This marker serves four purposes:
- Resumption:
find_session_for_reply() searches discussion threads
for this marker to find the session ID. Only bot-authored notes are
considered; the first matching marker wins.
- Per-workflow rate limiting:
scan_agent_threads() counts discussion
threads for a specific workflow using the wf field. Only bot-authored
notes are scanned.
- SHA deduplication:
scan_agent_threads() checks whether the current
SHA has already been reviewed by the workflow using the sha field.
- Self-filtering:
handle_note() compares the webhook event’s
author_id against the bot’s own user ID (via get_bot_user_id())
to prevent infinite loops.
All marker-scanning functions resolve the bot user ID via gl.auth()
(GET /user) on the orchestrator token. This ensures markers in
non-bot notes (user replies, external contributors) are never parsed,
preventing session hijacking via injected markers.
Old plain-UUID markers (<!-- hummingbird-session: UUID -->) are parsed
as {"id": "UUID"} for backward compatibility. They are counted for
resumption but not for per-workflow rate limiting or SHA dedup (no
wf/sha information).
12. Workflow System
A workflow has two parts that live in different places:
- Prompt (
.md file): the LLM system prompt, verbatim. This is the
investigation strategy, tool usage guidance, data patterns, and output
format. Pure text, no code.
- Metadata (YAML config): operational settings –
action,
model, max_iterations, data_sources, projects. This controls
what the orchestrator does with the workflow, not what the model does.
This separation means changing the analysis strategy (e.g., adding a new
investigation step) requires editing a markdown file. Changing operational
parameters (e.g., which projects use this workflow, which model to use)
requires editing the YAML config. Neither requires a code change.
12.2. System prompt layering
The system prompt sent to the model is built from three layers:
BASE_SYSTEM_PROMPT # agent.py: sandbox rules, tool usage tips
+ tool_notes # from ToolRegistry: per-data-source domain knowledge
+ workflow_body # full content of e.g. workflows/analyze-failures.md
On session resumption, a fourth layer is appended:
+ CONTINUATION_PROMPT # agent.py: "do not re-run the full workflow"
The system prompt is rebuilt every iteration (to allow warning suffixes to
be appended), but the base content is stable. Warning suffixes are appended
at the end so they override earlier instructions.
12.3. Design choice: strategy, not procedure
The workflow .md file describes strategy and guidance, not a rigid script.
The model decides when and how to use each tool based on what it sees.
This matters because merge request failures are diverse. A fixed procedure
would either miss edge cases (e.g., build failures before tests ran) or
waste iterations on steps that don’t apply. By giving the model a strategy
(“identify failing tests, fetch details, analyze root causes, group by
similarity”), it can adapt to whatever it encounters.
The workflow does structure the investigation into phases (Data Collection,
Analysis, Output) for clarity, but these are guidelines, not enforced
checkpoints.
12.4. Workflow file anatomy (analyze-failures.md)
The primary workflow is structured as:
- Scope and approach – what this workflow analyzes and what it ignores
- Input – event JSON format (
project, iid)
- Data Collection Phase:
- Data.1: Fetch MR details (direct call, small response)
- Data.2: Fetch commit statuses, identify failures
- Data.3: Batch fetch PipelineRuns + TaskRuns (fetch_batch_to_sandbox)
- Data.4: Process with jq, extract Testing Farm data in bulk
- Analysis Phase:
- Analysis.1a: Investigate each failed PipelineRun individually
- Analysis.1b: Summarize and group by root cause
- Output – markdown template with root causes, collapsible details,
clickable links (PipelineRun, Testing Farm, test logs)
- Error Handling – partial report philosophy
Design considerations embedded in the workflow:
- Efficient bulk fetching: PipelineRuns and TaskRuns are fetched in one
fetch_batch_to_sandbox call, not individually.
- Testing Farm data extracted in Data.4, analyzed in Analysis.1: All TF
results.xml are fetched in bulk before analysis starts, enabling
cross-failure pattern detection.
- Log fetching is selective: Only 1-2 representative logs per failure
pattern, not all logs. This keeps iteration count bounded.
- Reviewer-facing URLs: The output template instructs the model to
include clickable links using
konflux_ui and artifacts_base from
response metadata.
12.5. Prompt file resolution
get_prompt(workflow_cfg, config_dir) resolves the prompt: field relative
to the config file’s directory:
prompt_path = config_dir / workflow_cfg.prompt
# e.g. /app/config.yaml with prompt: workflows/analyze-failures.md
# -> /app/workflows/analyze-failures.md
In the container image, workflows are baked in at /app/workflows/. A
ConfigMap can override them at deploy time by mounting at the same path.
13. Deployment and Container
13.1. Container build strategy
The Containerfile uses an all-RPM builder+installroot+scratch pattern (no
pip, no venv):
- Builder stage: Uses a Fedora-based builder image with dnf. Installs
all dependencies as RPMs into a clean
--installroot.
- Application code: Copies
hummingbird_agent/ and workflows/ into
the installroot at /app/.
- Final stage:
FROM scratch, copies the entire installroot. No
package manager, no shell beyond what RPMs provide.
RPM dependencies: python3-boto3, python3-google-auth+requests,
python3-gitlab, python3-kubernetes, python3-pyyaml, python3-requests,
python3-sentry-sdk, kubernetes1.35-client (for kubectl).
This approach was chosen over pip because:
- All deps come from Fedora’s package repository – no PyPI supply chain risk
- Smaller image (~22 MB saved vs. google-genai SDK alone)
- Reproducible builds from known RPM versions
- No compilation step (no gcc/python3-devel in the image)
13.2. Container runtime properties
CMD ["python3", "-m", "hummingbird_agent", "serve"]
WORKDIR /app
USER 65532
The default CMD runs serve mode for production. Local development uses
run mode via explicit command override. USER 65532 matches the standard
nonroot UID used by distroless images and the Podman sandbox.
13.3. K8s deployment manifests
Located in hummingbird-agent/kubernetes/:
deployment.yaml:
- Single replica with rolling update (
maxSurge: 1, maxUnavailable: 0)
terminationGracePeriodSeconds: 900 (15 minutes) to allow in-flight
agent runs to complete on shutdown
- ServiceAccount:
hummingbird-agent
- Secrets mounted from K8s Secret
hummingbird-agent
- Commented-out mounts for workflow ConfigMap and custom CA trust
rbac.yaml (applied in the sandbox namespace):
- ServiceAccount
hummingbird-agent
- Role with
pods: create, get, list, delete, patch and pods/exec: create
- RoleBinding linking the SA to the Role
patch is required for pool mode (relabeling pods during claim)
sandbox-pool.yaml (applied in the sandbox namespace):
- Deployment with
replicas: 3 (tuned to balance latency vs cost)
- Pods labelled
hummingbird/role: standby for pool discovery
- Same security context and resource limits as direct-creation pods
revisionHistoryLimit: 2, rolling update with maxSurge: 1,
maxUnavailable: 0
networkpolicy.yaml:
podSelector: {} – applies to all pods in the namespace
egress: [] – deny all outbound traffic
secret.yaml:
- Template with all required env vars (API keys, tokens, URLs)
- Values must be populated per deployment
13.4. Workflow mounting
Workflows are baked into the image at /app/workflows/. To update workflows
without rebuilding the image:
- Create a ConfigMap from the workflows directory
- Uncomment the volume mount in the deployment YAML
- The ConfigMap mount replaces the baked-in directory (all-or-nothing)
This is useful for rapid iteration in staging without waiting for a new
image build.
13.5. Custom CA trust
For clusters with internal CA certificates (common in enterprise
environments), the deployment supports OpenShift’s CA injection:
- Create a ConfigMap with the
config.openshift.io/inject-trusted-cabundle
label
- Mount it at
/etc/pki/custom
- Set
REQUESTS_CA_BUNDLE=/etc/pki/custom/ca-bundle.crt
OpenShift automatically injects the cluster’s CA bundle into the ConfigMap.
7 - Hummingbird Agent Design — Decisions
Design Decision Registry (DDR) for the Hummingbird Agent — every major
architectural choice, the alternatives considered, why the chosen approach won,
and what would break if reversed. For the full design context, see
Hummingbird Agent Design. Provider adapter
decisions are in Hummingbird AI Design.
Design Decisions
kubectl exec over kubernetes Python exec API
- Chosen:
kubectl exec as subprocess for sandbox command execution.
- Alternative: kubernetes Python client
stream() API.
- Why: Three showstopper bugs in the Python client: (1) WebSocket
v1-v4 has no stdin EOF signal, so
cat > /file hangs forever; (2)
BrokenPipeError on stdin larger than ~1MB; (3) stream() accumulates
all stdout in memory with no control. kubectl avoids all three and
provides the same subprocess interface as Podman.
- If reversed:
write_file() would hang or fail on large data. Pod
log retrieval with large outputs would OOM. Sandbox reliability would
drop significantly.
YAML config over env vars for operational settings
- Chosen: Single YAML file for workflows, projects, limits, data
source declarations.
- Alternative: Everything in env vars (original design).
- Why: Env vars cannot express structured data (workflow-project
mappings, per-project token overrides, data source config with multiple
fields). YAML provides structure, validation, and audit trails.
- If reversed: Token scoping would be lost (no per-workflow-project
token overrides). Project allowlists would be impossible. The config
would be unauditable.
Token env var names in YAML, not values
- Chosen: YAML contains
token_env: GITLAB_TOKEN_RO (the env var
name), not the actual token value.
- Alternative: Inline secrets in YAML, or env vars for everything.
- Why: The YAML file can be committed, reviewed, and audited. Actual
secrets stay in env vars (injected via K8s Secrets). Inline secrets
would make the config file a secret itself.
- If reversed: The config file would become a secret, breaking audit
trails and code review workflows.
Orchestrator token prefix convention (ORCHESTRATOR_*)
- Chosen: Orchestrator tokens use
ORCHESTRATOR_GITLAB_TOKEN_* env
var names by convention.
- Alternative: Same env var namespace as model tokens, distinguished
by context.
- Why: Structural separation makes it impossible to accidentally pass
an orchestrator token to a model tool (or vice versa). The
ORCHESTRATOR_ prefix is never used in YAML token_env fields.
- If reversed: A misconfiguration could leak write-capable tokens to
the model, which could then expose them via tool calls.
Auto-spill over conversation compaction
- Chosen: Large outputs are saved to sandbox files with previews
returned to the model.
- Alternative: Replace old tool results in
contents with compact
summaries (conversation compaction).
- Why: Compaction requires modifying the
contents list, which risks
violating provider conversation constraints. Auto-spill achieves ~90% of
the token reduction without modifying model history. See
Hummingbird AI for adapter
format requirements.
- If reversed: Token usage would increase ~15-30%. Per-call context
would grow unbounded. Sessions would hit the context limit much sooner.
Conversation compaction could be added on top of auto-spill in the
future, but is not needed with current workloads.
Iteration count over cumulative token budget
- Chosen:
max_iterations as the primary cost control lever.
- Alternative: Cumulative token budget (stop when total billed tokens
exceed a threshold).
- Why: With full history replay, each API call re-sends everything.
Cumulative billed tokens double-count: call 1 = 5K, call 2 = 10K
(including 5K again), total = 15K billed but only 10K new content. With
auto-spill keeping per-call size bounded, iteration count is a much
simpler and more predictable proxy for actual cost.
- If reversed: The budget model would be confusing and inaccurate.
Cost estimates would be wrong. The cumulative metric is still logged
for observability, but it does not drive termination.
- Chosen: Domain knowledge (Konflux dual-fetch, TF XML structure) goes
in
ToolDef.notes, injected into the system prompt.
- Alternative: Put everything in the tool schema
description.
- Why: Tool schemas have character limits and are sent in the
tools
field of every API call. Long descriptions waste tokens on tool defs.
System prompt notes are sent once and can be arbitrarily detailed.
- If reversed: Tool descriptions would be bloated. Domain knowledge
would need to be duplicated in every workflow
.md file.
Full history replay (no compaction)
- Chosen: The
contents list grows monotonically. Items are never
removed or modified (except empty response retries).
- Alternative: Compact old turns to reduce context size.
- Why: Provider APIs impose strict conversation structure. Modifying or
removing items risks invalid history. Auto-spill handles the growth problem
at the source (preventing large items from entering history).
- If reversed: Risk of provider API errors from malformed conversation
structure and of confusing the model with summarized results.
fetch_to_sandbox kept alongside auto-spill
- Chosen: Both
fetch_to_sandbox and auto-spill coexist.
- Alternative: Remove
fetch_to_sandbox since auto-spill handles
large outputs automatically.
- Why:
fetch_to_sandbox provides explicit path control (model can
choose meaningful filenames), batch fetching (one tool call for multiple
fetches), and no-preview responses (useful when the model will process
with jq anyway).
- If reversed: The model would lose path control and batch fetching
would require multiple auto-spilled calls. The workflow would need more
iterations to achieve the same result.
K8s library for lifecycle + kubectl for exec (hybrid)
- Chosen: Use the
kubernetes Python library for pod
create/read/delete, kubectl subprocess for exec.
- Alternative: All-kubectl (subprocess for everything) or all-library.
- Why: The library provides typed pod status polling and clean error
handling for lifecycle.
kubectl provides reliable exec (see “kubectl
exec over kubernetes Python exec API” above). Using the library for exec
would require solving the three WebSocket bugs.
- If reversed: Either lifecycle management would be fragile (parsing
kubectl JSON output for pod status) or exec would be unreliable.
Single-file YAML config over multiple config files
- Chosen: Everything in one
config.yaml with settings and
workflows sections.
- Alternative: Separate files per workflow, or settings.yaml +
workflows.yaml.
- Why: Single source of truth. All project-workflow mappings visible in
one place. Settings-level defaults flow down to all projects.
No file discovery logic needed.
- If reversed: Token resolution would need cross-file lookups. Project
index would need multi-file aggregation. Config validation would be more
complex.
- Chosen: The workflow
.md file is pure system prompt text. Metadata
(action, model, max_iterations, data_sources, projects) lives
in the YAML config.
- Alternative: YAML frontmatter in the
.md file (original design).
- Why: Separation of concerns. The
.md file is the model’s
instructions – it should be readable and editable by anyone writing
prompts. The YAML config is the orchestrator’s instructions – it
controls routing, limits, and credentials. Mixing them in one file
conflates two audiences.
- If reversed: Prompt authors would need to understand YAML config
structure. Token/project config would be scattered across
.md files
instead of centralized.
Nudge via system prompt suffix, not user message
- Chosen: Empty response nudge and budget warnings are appended to the
system prompt as suffixes.
- Alternative: Inject synthetic user messages into
contents.
- Why: User messages in
contents must come from the actual user (the
initial event, or a reply). Injecting synthetic messages pollutes the
conversation history that is saved in sessions. System prompt suffixes
are transient – they affect one API call without permanently modifying
the conversation state.
- If reversed: Saved sessions would contain synthetic user messages.
Resumed conversations would be confusing. The model might respond to the
synthetic messages instead of the user’s actual input.
Session ID reuse over new-ID-per-resume
- Chosen: Resumed sessions keep the original session ID. S3 state is
overwritten in place.
- Alternative: Each resume generates a new UUID, saving alongside the
original (append-only tree of sessions).
- Why: GitLab discussions are linear, not branching. There is no
scenario where two different sessions from the same thread are both
valid. A new UUID per resume creates orphaned S3 snapshots that are
never referenced again, since
find_session_for_reply() always picks
the latest marker. Reusing the ID is simpler, uses less storage, and
matches the linear conversation model.
- If reversed: S3 would accumulate orphaned session snapshots. Each
resume would need a new note marker, but the old markers would still be
in the thread, creating confusion about which session is current.
Reply to unauthorized agent-directed notes instead of silent skip
- Chosen: When an unauthorized user sends a slash command or replies to
an agent thread, the agent replies in the same discussion with a short
access-denied message. Non-agent-directed notes are silently ignored
(debug log, no access check).
- Alternative: Log a warning and skip silently for all unauthorized
notes (the original behavior).
- Why: Silent skip gives no feedback to someone who intentionally tried
to engage the agent, which is confusing. On the other hand, checking
access and logging a warning for every random comment on a public project
is noisy and pointless. Moving the access check after intent detection
cleanly separates the two cases. The reply uses the discussions API so it
automatically inherits confidentiality from the parent note/thread.
- If reversed: Users without sufficient access who try
/hummingbird
would get no feedback. The agent log would be noisy with warnings for
every comment on public MRs.
Deployment-backed pod pool over Python-thread pool
- Chosen: A Kubernetes Deployment maintains pre-warmed standby pods.
The agent claims a pod by relabeling it; the Deployment replaces it.
- Alternative: A Python-side thread pool that pre-creates pods and
queues them for use.
- Why: The Deployment provides self-healing (restart crashed pods),
native scaling (
kubectl scale), rolling updates (image changes), and
monitoring via standard K8s tooling. A Python pool would need to
reimplement all of these.
- If reversed: Pod replenishment, crash recovery, and image updates
would all need custom code. Scaling would require agent redeployment.
Pod isolation: never reuse sandbox pods
- Chosen: Each workflow run gets its own pod. After cleanup, the pod is
deleted. Claimed pods are detached from the ReplicaSet.
- Alternative: Return used pods to the pool and reset them.
- Why: Residual state from a previous run (files, environment, running
processes) could leak between MR investigations, creating security and
correctness risks. Deletion is simple and foolproof.
- If reversed: Would need a reliable pod-reset mechanism and auditing
that no state survives between runs.
Absolute reap-by deadline over relative claimed-at age
- Chosen: Active pods are annotated with
hummingbird/reap-by (an
absolute UTC timestamp). The reaper deletes pods where now > reap-by.
- Alternative: Annotate with
claimed-at and compute age relative to
max_active_seconds; or use activeDeadlineSeconds on the pod spec.
- Why: An absolute deadline simplifies the reaper to a single
comparison. It also supports varying deadlines: claim sets
reap-by = now + max_active_seconds, while linger sets
reap-by = now + linger_seconds. With a relative timestamp, the reaper
would need to know which mode the pod is in.
activeDeadlineSeconds applies from pod creation, not from claim –
standby pods would expire before being used.
- If reversed: The reaper would need mode-aware age calculations.
Lingering pods would require a separate annotation or reaper path.
Indefinite claim wait over timeout
- Chosen:
SandboxPool.claim() polls indefinitely (governed by
shutdown event), logging at DEBUG then WARNING.
- Alternative: Timeout after N seconds and raise an error.
- Why: The pool is typically smaller than
max_concurrent_agents for
cost reasons. Waiting for a replacement pod is normal operational
behavior, not an error. A timeout would cause spurious failures during
burst traffic. The SQS semaphore already bounds concurrency.
- If reversed: Burst traffic would cause avoidable failures instead of
brief delays.
Pool config in Deployment only, not in agent configmap
- Chosen: In pool mode (
k8spool), the pod image, resources, labels,
and security context are defined solely in the Deployment template. The
agent config only needs namespace and active_deadline_seconds.
- Alternative: Keep image/resources/metadata in the agent configmap too
(as in
k8s mode).
- Why: Eliminates config duplication. The Deployment template is the
single source of truth. Changes to pod resources or image only require
updating one place and re-rolling the Deployment.
- If reversed: Config drift between the Deployment and the agent
configmap would be a constant risk.
- Chosen: After a successful workflow, pool sandbox pods linger for
linger_seconds (default 300) with a session-id annotation. Replies
within the window reclaim the pod via try_reclaim(), skipping pod
creation and S3 archive restoration.
- Alternative: Always delete the pod immediately and restore from S3 on
every reply.
- Why: Reply latency drops from seconds (pod claim + S3 restore) to
near-zero. The S3 archive is still saved as a fallback if the pod is
gone. The
reap-by annotation ensures lingering pods are cleaned up if
no reply arrives. The session-id annotation doubles as a concurrency
guard: it is only present while lingering, preventing active pods from
being reclaimed.
- If reversed: Every reply would pay full pod + restore latency, even
for immediate follow-ups. User experience for conversational interactions
would degrade noticeably.
linger() on Sandbox protocol over isinstance checks in runner
- Chosen: All sandbox backends implement
linger(session_id). Non-pool
backends fall back to cleanup(). The runner calls sb.linger() without
type checks.
- Alternative:
isinstance(sb, K8sPoolSandbox) in run_workflow().
- Why: Keeps the runner backend-agnostic. Adding a new backend requires
only implementing the protocol, not touching the runner. The fallback
behavior is co-located with each backend.
- If reversed: The runner would need to know about every backend type
and their linger capabilities.
GCP SA key over Workload Identity Federation
- Chosen: GCP service account key stored in Vault, rotated via
cki-tools credential manager (prepare/switch/clean cycle).
- Alternative: Workload Identity Federation (WIF) with projected SA
token. Two sub-options: (a) automatic OIDC discovery if the cluster
issuer is public, (b) manual JWKS upload for internal issuers.
- Why: mpp-prod’s OIDC issuer is
https://kubernetes.default.svc
(internal, not publicly reachable), so GCP STS cannot discover it
automatically. Manual JWKS upload works but requires re-upload after
SRE-triggered signing key rotations. SA key integrates with the
existing credential manager rotation infrastructure (same pattern as
AWS keys and GitLab tokens), needs no OIDC reachability, and enables
automated validate/update via google-auth in CI. The application code
(hummingbird_ai.http.VertexAuth / google.auth.default()) works identically
with both approaches – switching to WIF later requires only infrastructure
changes.
- If reversed: Replace SA key with WIF projected token + credential
config ConfigMap. The
gcp_service_account_key token type in
cki-tools would no longer be needed for this use case.
- Chosen:
gitlab_get_mr_discussions checks each note author’s project
access level (Developer+ / >= 30). Untrusted notes are replaced with a fixed
placeholder in mixed-trust discussions, or the entire discussion is dropped
if all notes are untrusted. Agent notes are always trusted (detected by
session marker) but have transcripts and metadata stripped.
- Alternatives considered:
- (a) Include all comments unfiltered. Simplest, but any external
contributor can craft comments that manipulate the model’s review output
(prompt injection).
- (b) Sanitize/escape untrusted content (strip markdown, quote as code
blocks, prefix with “[external]”). There is no reliable escaping mechanism
for LLM prompts – the model interprets natural language regardless of
formatting. Escaping gives a false sense of security.
- (c) Only include agent-authored notes (skip all human comments). Safe, but
defeats the purpose: the model would never see developer responses to its
own findings.
- (d) Include only discussions where the agent participated. Better, but
still misses developer-initiated review threads that provide relevant
context.
- Why: Developer+ is the same threshold used for pipeline trigger
authorization (invariant #8) and slash command access. It matches the trust
boundary already established: people who can push code and approve MRs are
trusted to provide review context. The placeholder approach preserves
discussion structure (the model sees that someone replied) without
exposing the content. Full-drop for all-untrusted discussions avoids
noise from discussions that contain zero useful context.
- If reversed: (a) opens a prompt injection vector on any project that
accepts external MRs. (b) provides no actual protection. (c) makes
follow-up reviews unable to see developer explanations, causing repeated
false positives. (d) misses developer-initiated context.
Bot-author filtering for session marker parsing over unfiltered note scanning
- Chosen:
find_session_for_reply(), scan_agent_threads(), and the
handle_note() self-filter all resolve the bot user ID via
get_bot_user_id() (which calls gl.auth() on the orchestrator token)
and only consider notes where author.id matches. In
find_session_for_reply(), the first matching marker wins.
- Alternative: Parse markers from any note and keep the last match;
substring self-filter in
handle_note (previous behavior).
- Why: The previous unfiltered approach allowed session hijacking: a
user reply or agent review prose containing an example marker
(
<!-- hummingbird-session: ... -->) was parsed as a real session,
causing “session expired” errors in production. For handle_note, the
substring check caused the reverse problem: a user embedding the marker
prefix in a reply silently suppressed the event (denial of service). The
gl.auth() call is one GET /user per invocation – negligible cost.
_get_client_with_bot_id() returns both the authenticated client and
the bot user ID, avoiding redundant client construction.
- If reversed: Any MR participant can inject a marker to hijack the
session ID or inflate rate-limit counts. The agent’s own review prose
containing example markers causes spurious “session expired” messages.
ijson streaming over buffered JSON for K8s list responses
- Chosen:
session.get(stream=True) + ijson.parse(resp.raw) for
iter_paginated(). Items are yielded one at a time via ObjectBuilder;
the metadata.continue pagination token is captured in the same parse
pass. resp.raw.decode_content = True is required because Kubearchive
returns gzip Content-Encoding; without it ijson sees compressed bytes.
- Alternative:
session.get().json() loads the full page into memory
(the original implementation).
- Why: A single K8s list page can contain 500 PipelineRuns (80+ MB
JSON). Parsing this with
.json() creates a +247 MB RSS spike (raw
bytes + decoded string + parsed dict coexist). With 4 concurrent
workflows, this exceeds any reasonable pod memory limit. ijson
stream-parsing yields items one at a time, reducing the peak to +12 MB
for the same data – a 95% reduction. The continue token appears in
the metadata object (before or after items depending on the server);
the event-driven parser captures it regardless of order.
- If reversed: Large MRs (100+ components) would OOM the agent pod.
Concurrent workflows would multiply the problem. The pod memory limit
would need to scale with the largest possible page size, which is
unbounded.
Action token tier for per-workflow identity
- Chosen: A dedicated Tier 2 of workflow action tokens, separate
from both orchestrator tokens (Tier 1) and model tool tokens (Tier 3). Each workflow gets
a dedicated GitLab bot user per project for all write operations (notes,
thread resolution).
- Alternative: Extend orchestrator tokens with a per-workflow fallback
chain (
ORCHESTRATOR_<WORKFLOW>_GITLAB_TOKEN_<PROJECT> →
ORCHESTRATOR_GITLAB_TOKEN_<PROJECT> → ORCHESTRATOR_GITLAB_TOKEN).
- Why: Orchestrator tokens serve a fundamentally different purpose
(operational reads + infrastructure notes) than workflow output (analysis
results, review comments). Mixing them in one fallback chain risks a
single bot identity posting notes on behalf of multiple workflows,
eliminating the audit trail. A separate tier with explicit
token: str
parameters makes the code path unambiguous: callers must choose which
tier they use. The naming convention
(HUMMINGBIRD_AGENT_ACTION_<WORKFLOW>_GITLAB_TOKEN_<PROJECT>) groups
tokens alphabetically by workflow, making env var auditing
straightforward.
- If reversed: All workflow notes would appear from the same bot user,
making it impossible to distinguish which agent (code-review vs
analyze-failures vs renovate-babysit) produced a given note in the MR
timeline. GitLab’s audit log would attribute all actions to one identity.
Per-project-only action token config (no workflow-level default)
- Chosen:
action_tokens is defined at the project level only in YAML
config. No workflow-level action_tokens default is supported.
- Alternative: Allow workflow-level
action_tokens that projects
inherit by default, with per-project overrides.
- Why: GitLab project access tokens are project-scoped. A
workflow-level default would be misleading – a single token cannot
write to multiple projects. Forcing per-project declaration makes the
mapping explicit and prevents configuration errors where a cross-project
default silently fails.
- If reversed: Operators could set a workflow-level token that only
works for one project, causing silent 403 errors for other projects in
the same workflow.
SSH over kubectl exec for KubeVirt VMs
- Chosen: SSH (
ssh -T sandbox@ip command) for VM command execution.
- Alternative:
virtctl console (serial console), KubeVirt websocket
API, or qemu-guest-agent exec.
- Why: VMs have no
kubectl exec equivalent. virtctl console provides
a serial terminal, not a programmable exec channel. The websocket API
requires a VNC or serial connection. qemu-guest-agent is not installed
in the VM image. SSH provides clean stdin/stdout/stderr separation,
timeout control, and is already well-tested in subprocess pipelines.
- If reversed: Would need
qemu-guest-agent in the image and the
exec endpoint, which has poorer error reporting and no streaming stdin.
Secret volume for SSH key injection
- Chosen: Kubernetes Secret mounted as a virtio disk (serial
ssh-pubkeys) read by the VM’s inject-ssh-keys.service at boot.
- Alternative:
cloudInitNoCloud or accessCredentials with
qemu-guest-agent.
- Why: The VM image already has
inject-ssh-keys.service which reads
from the virtio disk. cloudInitNoCloud requires cloud-init in the
image. accessCredentials requires qemu-guest-agent. The Secret
volume approach requires no additional software in the VM image.
- If reversed: Would need cloud-init or guest agent in the bootc
image, adding complexity and attack surface.
VMIRS for VM pool replenishment
- Chosen:
VirtualMachineInstanceReplicaSet (VMIRS) as the
replenishment controller for the VM pool.
- Alternative: Custom controller, or VirtualMachine with RunStrategy.
- Why: VMIRS is KubeVirt’s native equivalent of a Deployment’s
ReplicaSet – it maintains a desired count of identical VMIs. When a
VMI is claimed (ownerReferences cleared), the VMIRS creates a
replacement automatically. Same pattern as the pod pool with Deployment.
- If reversed: Would need a custom controller to maintain the VMI pool,
adding operational complexity.
Per-workflow sandbox backend
- Chosen:
sandbox: field in workflow config, resolved per-workflow
with fallback to CLI --sandbox flag.
- Alternative: Global-only sandbox selection via CLI.
- Why: Different workflows have different isolation requirements.
Code review needs only a lightweight container, while failure analysis
may need a full VM with root access. Per-workflow selection avoids
running all workflows in VMs (wasteful) or all in containers
(insufficient isolation).
- If reversed: All workflows would share the same sandbox backend,
requiring either over-provisioning (VMs for everything) or accepting
weaker isolation for some workflows.