# Hummingbird Agent Design
LLMS index: [llms.txt](https://hummingbird-project.io/llms.txt) | Full content: [llms-full.txt](https://hummingbird-project.io/llms-full.txt)
---
Architectural design document for the hummingbird-agent. Covers the reasoning
behind every major design choice so that future changes can be made safely,
without accidentally violating invariants that hold the system together.
For operational usage (CLI, config reference, deployment), see
[Hummingbird Agent](https://hummingbird-project.io/docs/background/tools/hummingbird-agent/). For the model loop wire format,
see [Agent Model Loop](https://hummingbird-project.io/docs/background/tools/hummingbird-agent-model-loop/).
## 1. Design Philosophy
Five principles shaped the agent's architecture. Every component traces back
to at least one of these.
**Security by default.** The agent processes untrusted merge requests. All
LLM-driven commands run inside isolated sandbox containers with no network
and no credentials. Orchestrator write tokens and model read tokens live in
separate code paths that never cross. No cluster-admin is required.
**Config over code.** Investigation logic lives in markdown workflow files
that become the LLM system prompt verbatim -- changing the analysis strategy
requires no code change and no redeployment. Operational settings (data
sources, project allowlists, iteration limits) live in a YAML config file
that is auditable and committable. Secrets are the only thing in environment
variables.
**Bounded cost.** Every output path has a size cap. Large tool outputs are
auto-spilled to sandbox files with only a preview returned to the model. Each
session has both an iteration limit and a per-call context token limit with
two-tier escalation (soft warning, then hard stop with tool disabling). The
relationship between these constants is documented and centralized.
**Partial over nothing.** When some data is unavailable (expired pod logs,
unreachable Konflux cluster, Testing Farm 404), the agent continues with
whatever data it has and notes the gap in its output. A partial report is
more valuable than a crash.
**Model-agnostic agent loop.** The agent loop (`agent.py`) does not inspect
the internal structure of the `contents` list. It appends `raw_content` from
model responses and `make_tool_responses()` output without looking inside.
Each model backend owns its wire format. This makes it possible to add new
model backends (Claude, GPT) without touching the agent loop.
## 2. System Architecture
### 2.1. Component overview
```mermaid
flowchart TB
subgraph input [Event Sources]
SQS["SQS Queue
(gitlab::pipeline, gitlab::note)"]
CLI["CLI
(--event / --event-file)"]
end
subgraph orchestrator [Orchestrator Process]
Events["events.py
SQS consumer"]
Runner["runner.py
Event routing"]
Agent["agent.py
Model loop"]
Tools["tools.py
Tool registry"]
Actions["actions.py
GitLab notes"]
Sessions["sessions.py
Persistence"]
WfConfig["workflow_config.py
YAML config"]
end
subgraph models [Model Backends]
Gemini["models/gemini.py
Gemini API / Vertex AI"]
Anthropic["models/anthropic.py
Anthropic via Vertex AI"]
end
subgraph sandbox [Sandbox Container]
SB["PodmanSandbox / K8sSandbox / KubeVirtSandbox
jq, python3, yq"]
SpillFiles["/tmp/data/_out/ spill files"]
DataFiles["/tmp/data/ working files"]
end
subgraph dataSources [Data Source Modules]
GL["gitlab.py"]
KX["konflux.py"]
TF["testing_farm.py"]
end
subgraph external [External APIs]
GitLabAPI["GitLab API"]
KonfluxAPI["K8s + Kubearchive"]
TFAPI["Testing Farm API"]
end
subgraph storage [Storage]
S3["S3 Sessions"]
MRNote["GitLab MR Notes"]
end
SQS --> Events --> Runner
CLI --> Runner
Runner --> Agent
Agent --> Tools
Tools --> SB
Tools --> dataSources
dataSources --> external
Agent --> Gemini
Agent --> Anthropic
Runner --> Actions --> MRNote
Runner --> Sessions --> S3
Runner --> WfConfig
```
### 2.2. Request lifecycle
A complete run proceeds through these stages:
1. **Event arrival.** An SQS message (production) or CLI invocation (dev)
provides a GitLab project path and MR IID.
2. **Config lookup.** `workflow_config.py` maps the project to one or more
workflows via the `project_index`. Each match yields a `WorkflowConfig`
and `ProjectEntry` with data source declarations and per-project settings.
3. **Rate-limit check and SHA dedup.** `actions.scan_agent_threads()` scans
MR discussions in a single pass, parsing JSON session markers to determine
the per-workflow thread count and whether the current SHA has already been
reviewed. If the SHA was already reviewed, the workflow is skipped. If the
thread count meets or exceeds `max_runs_per_mr`, the workflow is skipped.
Slash commands and replies bypass this check.
4. **Placeholder note.** `actions.create_placeholder_note()` posts a
placeholder so the reviewer knows analysis is in progress. The note
contains a JSON session marker
(``)
for rate limiting, SHA dedup, and session resumption.
5. **Sandbox start.** `sandbox.create_sandbox()` starts a Podman container
or K8s pod. `/tmp/data/` is pre-created for the model's use.
6. **Data source registration.** `data_sources.register_selected()` resolves
tokens and URLs from the config and registers tool functions on the
`ToolRegistry`. Only data sources declared in the workflow config are
registered.
7. **Agent loop.** `agent.run_agent_loop()` runs the model loop: the
workflow markdown becomes the system prompt, tool definitions are provided,
and the model iterates calling tools and producing text until it emits a
final response or hits a budget limit.
8. **Note update.** The final text, wrapped with a session marker and reply
prompt, replaces the placeholder note.
9. **Session save.** Conversation history (`contents`), transcript, and
sandbox archive are saved to S3 (production) or a local directory (dev).
10. **Sandbox cleanup.** The container/pod is deleted. K8s pods also have a
configurable `activeDeadlineSeconds` backstop (default 1800s) in case the
orchestrator dies.
### 2.3. Architectural boundaries
The codebase is organized around four strict boundaries:
**Runner** (`runner.py`) is the orchestration layer. It owns event routing,
the placeholder/update note lifecycle, session save/load, and sandbox
lifecycle. It calls `agent.run_agent_loop()` but never reaches into the
agent's internals.
**Agent** (`agent.py`) is the model loop. It knows about the model interface,
tool definitions, and the `contents` list, but nothing about GitLab, SQS,
sessions, or actions. It returns `(text, usage, transcript, contents)` and
is completely unaware of what happens with those values.
**Tools** (`tools.py`) bridge the agent and the sandbox/data-sources. The
agent calls `tool_registry.execute(tool_call)` and gets a dict back. It
never calls sandbox methods directly. This indirection is what enables
auto-spill: the tool registry can transparently save large outputs to
sandbox files and return previews.
**Models** (`models/`) own model construction, region selection, pricing, and
wire format conversion. Each model adapter's
`generate()` accepts internal types (`contents`, `ToolDef`) and returns a
`ModelResponse`. `make_user_content()` and `make_tool_responses()` produce
the model-specific dicts that go into `contents` (Gemini and Anthropic
backends each implement the `ModelAdapter` protocol). The agent treats these
as opaque values -- it appends them but never inspects their internal structure.
## 3. Module Architecture
### 3.1. Dependency graph
```mermaid
flowchart TD
main["__main__.py"]
config_mod["config.py"]
wf_config["workflow_config.py"]
events_mod["events.py"]
runner_mod["runner.py"]
agent_mod["agent.py"]
tools_mod["tools.py"]
sandbox_mod["sandbox.py"]
actions_mod["actions.py"]
sessions_mod["sessions.py"]
transcript_mod["transcript.py"]
ds_init["data_sources/__init__.py"]
ds_gitlab["data_sources/gitlab.py"]
ds_konflux["data_sources/konflux.py"]
ds_tf["data_sources/testing_farm.py"]
http_mod["_http.py"]
config_watch_mod["config_watch.py"]
models_init["models/__init__.py"]
models_types["models/_types.py"]
models_gemini["models/gemini.py"]
models_anthropic["models/anthropic.py"]
main --> config_mod
main --> config_watch_mod
main --> wf_config
main --> events_mod
main --> runner_mod
main --> sessions_mod
main --> transcript_mod
main --> actions_mod
config_watch_mod --> config_mod
config_watch_mod --> wf_config
runner_mod --> actions_mod
runner_mod --> agent_mod
runner_mod --> config_mod
runner_mod --> ds_init
runner_mod --> sandbox_mod
runner_mod --> sessions_mod
runner_mod --> tools_mod
runner_mod --> models_init
runner_mod --> wf_config
runner_mod --> transcript_mod
agent_mod --> config_mod
agent_mod --> models_init
agent_mod --> tools_mod
tools_mod --> config_mod
tools_mod --> models_init
tools_mod --> sandbox_mod
ds_init --> tools_mod
ds_init --> wf_config
ds_init --> ds_gitlab
ds_init --> ds_konflux
ds_init --> ds_tf
ds_konflux --> http_mod
sessions_mod --> sandbox_mod
models_init --> models_types
models_init --> models_gemini
models_init --> models_anthropic
models_gemini --> http_mod
models_gemini --> models_types
models_anthropic --> http_mod
models_anthropic --> models_types
```
### 3.2. Module responsibilities
Each module has a single, well-defined responsibility. The boundary rules
below are invariants -- violating them breaks the security model or the
separation of concerns.
#### `config.py` -- Runtime configuration and constants
- **Owns:** `Config` dataclass and agent tuning constants
(`DEFAULT_MAX_ITERATIONS`, `CONTEXT_LIMIT`, `OUTPUT_PREVIEW_BYTES`, etc.)
- **Boundary:** Pure data. No I/O except reading env vars in `Config.load()`.
No imports from other hummingbird modules; model setup and pricing are
owned by `models/`.
- **Invariant:** All interdependent budget/spill constants must be defined
here with their relationship documented in the comment block.
#### `workflow_config.py` -- YAML config loader and token resolution
- **Owns:** `AgentConfig`, `WorkflowConfig` (with `trigger`, `description`,
`trigger_rules`, `ignore_users`, `ignore_branches` fields), `TriggerRule`,
`ProjectEntry`, `DataSourceConfig` dataclasses; `load()`, `validate_env()`,
`evaluate_trigger_rules()`, `resolve_tool_token()`, `resolve_cluster_url()`,
`get_prompt()`.
- **Boundary:** Reads YAML from disk and env vars. Never instantiates
network clients or model objects.
- **Invariant:** `resolve_tool_token()` resolves model tool tokens only.
It must never return orchestrator tokens. The resolution chain is:
`project_entry.tokens[ds] -> workflow_cfg.data_sources[ds].token_env ->
"" (empty)`.
#### `events.py` -- SQS consumer
- **Owns:** `decode_sns_message()` (SNS envelope decoding),
`poll_loop()` (blocking SQS consumer with concurrency control).
- **Boundary:** Knows about SQS/SNS wire formats. Calls a generic
`EventHandler` callback. Does not know about GitLab, workflows, or the
agent.
- **Invariant:** Failed messages are not deleted from SQS (they go to
the DLQ after visibility timeout expires). Successful messages are
deleted after handler returns.
#### `config_watch.py` -- Background config hot-reload
- **Owns:** `ConfigHolder` (thread-safe config pair with atomic swap),
`start_watcher()` (daemon thread that polls config file mtime),
`_watch_loop()`.
- **Boundary:** Knows about `config.Config.load()` and
`workflow_config.load()`. Does not know about events, the agent, or
any runtime state.
- **Invariant:** On reload failure (parse error, missing env var), the
previous config is kept and a warning is logged. The watcher never
crashes the serve loop.
#### `runner.py` -- Event routing and workflow execution
- **Owns:** `handle_event()`, `handle_pipeline()`, `handle_merge_request()`,
`handle_note()`, `_execute_workflow()`, `_handle_reply()`, `run_workflow()`,
`_acquire_sandbox()`, `_resolve_slash_workflow()`, `_format_help()`,
`_is_ignored_user()`, `WorkflowRequest`, `WorkflowResult`, `SandboxOpts`.
- **Boundary:** Orchestrates everything: config lookup, rate limiting,
placeholder notes, sandbox lifecycle, agent invocation, session save.
This is the only module that touches both `actions` and `agent`.
- **Invariant:** `run_workflow()` either lingers the sandbox (success)
or cleans it up (exception).
#### `agent.py` -- Model-agnostic tool-calling loop
- **Owns:** `run_agent_loop()`, system prompt constants (`BASE_SYSTEM_PROMPT`,
`CONTINUATION_PROMPT`, warning/nudge strings), budget logic.
- **Boundary:** Knows about `models` (the `ModelAdapter` protocol: `generate()`
and content construction) and `tools` (for `execute()`). Does not know about GitLab,
SQS, sessions, or actions. Does not know which model backend is in use.
- **Invariant:** The `contents` list is treated as opaque. Items are
appended via `response.raw_content` and `model.make_tool_responses()`.
The agent never inspects, modifies, or deletes items in the list (except
for empty response retries, which `pop()` the last appended item before
the model has seen any tool results for it).
#### `tools.py` -- Tool registry
- **Owns:** `ToolRegistry` class with `sandbox_exec`, `fetch_to_sandbox`,
`fetch_batch_to_sandbox` built-in tools; data source registration and
dispatch; auto-spill logic.
- **Boundary:** Owns the sandbox reference and all tool execution. The
agent never touches the sandbox directly.
- **Invariant:** `_spill_counter` is shared across all spill paths
(sandbox exec stdout, sandbox exec stderr, data source auto-spill) to
prevent filename collisions in `/tmp/data/_out/`.
#### `sandbox.py` -- Sandbox backends
- **Owns:** `Sandbox` protocol, `PodmanSandbox`, `K8sSandbox`,
`K8sPoolSandbox`, `SandboxPool`, `KubeVirtSandbox`, `VmiPool`,
`KubeVirtPoolSandbox`, `create_sandbox()`, `ExecResult`.
- **Boundary:** Knows about container/pod/VM lifecycle and command execution.
Does not know about tools, models, or the agent.
- **Invariant:** All backends must implement the same 8-method protocol
(`start`, `exec`, `write_file`, `stream_to_file`, `stream_exec`,
`read_file_iter`, `cleanup`, `linger`). `exec()` accepts optional
`stdin_data` for piping raw bytes. All must pre-create `/tmp/data/` in
`start()`. All must use `stdin` piping for `write_file()` (never host
volume mounts).
#### `actions.py` -- GitLab note lifecycle
- **Owns:** Orchestrator token resolution (`resolve_orchestrator_token()`),
workflow action token client creation (`_make_client()`), bot user ID
collection (`collect_workflow_bot_ids()`), note CRUD, JSON marker parsing
(`marker_tag`, `parse_marker`), `scan_agent_threads` (per-workflow rate
limiting + SHA dedup), `post_simple_reply`, member access checks,
session-for-reply lookup, `AgentThreadInfo`, `SessionRef`.
- **Boundary:** Functions are split by token tier:
- *Orchestrator token* (operational reads + infrastructure notes, use
`_get_client`): `check_member_access`, `get_latest_push_author`,
`get_head_pipeline_id`, `is_first_note_in_discussion`,
`reply_access_denied`, `notify_rate_limit`, `post_simple_reply`.
- *Workflow action token* (per-workflow-attributable writes, take explicit
`token: str`): `create_placeholder_discussion`, `post_discussion_note`,
`resolve_agent_threads`, `scan_agent_threads`.
Never touches model tool tokens, the tool registry, or the agent.
- **Invariant:** Orchestrator tokens are resolved from `ORCHESTRATOR_*` env
vars only. Workflow action tokens are resolved from `action_tokens` in YAML
and passed as explicit parameters. Neither tier is exposed to the model.
#### `sessions.py` -- Session persistence
- **Owns:** `archive_sandbox()`, `save_local()`, `save_s3()`, `load_s3()`,
`load_local()`, `restore_sandbox()`, `SessionData` (including
`format_version` for canonical vs legacy session JSON).
- **Boundary:** Knows about sandbox (for archiving) and S3/filesystem (for
storage). Does not know about the agent, tools, or models. Load/save detects
canonical envelope (`format_version: 1`) vs legacy raw `contents` lists.
- **Invariant:** The sandbox archive is never extracted on the orchestrator.
It is created inside the sandbox (`tar czf`), streamed out via
`read_file_iter()`, and restored inside a new sandbox via
`stream_exec("tar xzf -")`.
#### `transcript.py` -- Transcript rendering
- **Owns:** `render_markdown()`, per-tool rendering functions, truncation.
- **Boundary:** Pure transformation from `TranscriptEntry` list to markdown.
No I/O, no side effects.
#### `data_sources/__init__.py` -- Data source registration
- **Owns:** `register_selected()` -- resolves tokens/URLs from config and
calls each data source's `register()` function.
- **Boundary:** Bridges `workflow_config` (for token/URL resolution) and
`tools` (for registration). Only registers data sources declared in the
workflow config.
#### `data_sources/gitlab.py`, `konflux.py`, `testing_farm.py`
- **Own:** `register()` function that creates tool definitions and closures
over credentials, and registers them on the `ToolRegistry`.
- **Boundary:** Each module talks to one external API. Credentials are
captured at registration time via closure, never stored globally.
#### `models/_types.py` -- Shared data classes
- **Owns:** `ModelError`, `ToolDef`, `ToolCall`, `Usage`, `ModelResponse`,
`TranscriptEntry`; `ModelAdapter` protocol (implemented by model backends).
- **Boundary:** Pure data. No imports from other hummingbird modules.
#### `_http.py` -- Shared HTTP infrastructure
- **Owns:** `new_session()` (requests session with retry adapter and
configurable 429 handling), `VertexAuth` (Google ADC credentials).
- **Boundary:** Used by model backends (`models/gemini.py`) and data
sources (`data_sources/konflux.py`). No model-specific or data-source-
specific logic.
#### `models/gemini.py` -- Gemini model adapter
- **Owns:** `GeminiModel` with `generate()`, `make_user_content()`,
`make_tool_responses()`, `to_canonical()`, `from_canonical()`.
- **Boundary:** Translates between internal types and the Gemini REST API.
Supports API key (direct) and Vertex AI (ADC) authentication modes.
- **Note:** Parses `cachedContentTokenCount` from Gemini's implicit
server-side caching into `Usage.cache_read_tokens`.
#### `models/anthropic.py` -- Anthropic model adapter (Vertex AI)
- **Owns:** `AnthropicVertexModel` with `generate()`, `make_user_content()`,
`make_tool_responses()`, `to_canonical()`, `from_canonical()`.
- **Boundary:** Translates between internal types and the Anthropic Messages
API via Vertex AI `rawPredict`.
- **Note:** `make_tool_responses()` uses `_last_tool_ids` instance state to
pair tool results with tool-use IDs.
- **Note:** `generate()` merges consecutive user messages before sending
(Anthropic requires strict user/assistant alternation).
- **Note:** `generate()` annotates messages with sliding-window cache
breakpoints (`cache_control`) on shallow copies to avoid mutating `contents`.
The `ephemeral` parameter skips cache writes for transient warning messages.
## 4. Security Model
The agent runs in a shared OpenShift cluster without cluster-admin access,
processing merge requests from repositories where external contributors can
submit code. The security model addresses two threat vectors: (1) the LLM
executing arbitrary commands chosen by an attacker-controlled MR, and (2)
credential leakage between the model, the sandbox, and orchestrator actions.
### 4.1. Sandbox isolation
All commands generated by the LLM run inside an ephemeral container, never
on the orchestrator host. The sandbox has no credentials, no network, and no
visibility into the orchestrator.
```mermaid
flowchart LR
subgraph orchestrator [Orchestrator]
Agent["Agent Loop"]
Creds["Secrets
(tokens, kubeconfig)"]
end
subgraph sandbox [Sandbox Container]
Shell["sh -c commands"]
Files["/tmp/data/
(incl. _out/ spill dir)"]
end
Agent -->|"stdin pipe
(write_file)"| sandbox
Agent -->|"exec command"| sandbox
sandbox -->|"stdout/stderr"| Agent
sandbox -.-x|"NO network"| Internet["Internet / K8s API"]
sandbox -.-x|"NO access"| Creds
```
**Podman (local development):**
- `--network=none` -- complete network isolation; curl, wget, pip install all
fail
- `--user 65532` -- fixed non-root UID; no privilege escalation
- No host volume mounts -- data enters only via `stdin` piping through
`write_file()`
**Kubernetes (production):**
Every field in the pod manifest is set explicitly for portability to vanilla
Kubernetes with Pod Security Admission (restricted level), not just reliance
on OpenShift's `restricted-v2` SCC admission:
- `automountServiceAccountToken: false` -- no K8s API access from sandbox
- `runAsNonRoot: true` -- enforced at pod level
- `seccompProfile: RuntimeDefault` -- required by restricted level
- `allowPrivilegeEscalation: false` -- container level
- `capabilities.drop: ["ALL"]` -- container level
- `activeDeadlineSeconds: 1800` -- pod self-terminates after 30 minutes even
if the orchestrator crashes; prevents orphaned pods
- `restartPolicy: Never` -- pod does not restart on failure
- `NetworkPolicy` on the sandbox namespace blocks all egress from all pods
in the namespace (`podSelector: {}`, `egress: []`)
The pod does not set `runAsUser` explicitly. On OpenShift, the SCC assigns a
UID from the namespace range. On vanilla K8s, the image's USER directive is
used.
### 4.2. Credential separation (four-tier token model)
Tokens are split into four tiers with strict code-path separation:
```mermaid
flowchart TB
subgraph tier1 [Tier 1: Orchestrator Tokens]
OT["ORCHESTRATOR_GITLAB_TOKEN_*
Operational reads
Env vars only, never in YAML"]
end
subgraph tier2 [Tier 2: Workflow Action Tokens]
AT["action_tokens in YAML
Per-workflow bot identity
Write scope (api)"]
end
subgraph tier3 [Tier 3: Model Tool Tokens]
MT["GITLAB_TOKEN_RO, etc.
Read scope (read_api)
Env var NAMES in YAML"]
end
subgraph tier4 [Tier 4: Sandbox]
SB["Zero credentials
Zero network
Zero SA token"]
end
OT -->|"used by"| OpsReads["actions.py
(member access, push author,
rate-limit notices)"]
AT -->|"used by"| WfWrites["actions.py
(notes, thread resolution)"]
MT -->|"used by"| DataSources["data_sources/
(GitLab, Konflux, TF)"]
SB -->|"used by"| SandboxExec["sandbox_exec
(sh -c commands)"]
OpsReads -.-x|"NEVER"| DataSources
WfWrites -.-x|"NEVER"| DataSources
WfWrites -.-x|"NEVER"| SandboxExec
DataSources -.-x|"NEVER"| OpsReads
DataSources -.-x|"NEVER"| WfWrites
```
**Tier 1 -- Orchestrator tokens.** Tokens used by `actions.py` for
operational reads (member access checks, push author lookup, head pipeline
queries) and infrastructure notes (access-denied replies, rate-limit
notices). Resolved by convention from env vars:
`ORCHESTRATOR_GITLAB_TOKEN_` (per-project) or
`ORCHESTRATOR_GITLAB_TOKEN` (global fallback). The `ORCHESTRATOR_` prefix is
a structural safeguard -- these env var names can never appear in the YAML
config's `data_sources`, `tokens`, or `action_tokens` sections. These tokens
can be scoped to `read_api` since all per-workflow writes moved to Tier 2.
**Tier 2 -- Workflow action tokens.** Per-workflow bot identity tokens
(`api` scope) used by `actions.py` for all workflow-attributable GitLab
writes: placeholder notes, discussion notes, and thread resolution. Each
workflow gets a dedicated bot user per project, providing clear audit trails
for which agent produced each note. Declared in the YAML config as env var
*names* at the project level:
```yaml
projects:
redhat/hummingbird/containers:
action_tokens:
gitlab: HUMMINGBIRD_AGENT_ACTION_CODE_REVIEW_GITLAB_TOKEN_CONTAINERS
```
Resolved by `resolve_action_token(project_entry, ds_name)` which reads the
env var name from `action_tokens` and resolves it from `os.environ`. These
tokens are passed as explicit `token: str` parameters to write functions
in `actions.py`. They NEVER enter the `ToolRegistry` or model tool calls.
At startup, `collect_workflow_bot_ids()` authenticates each unique workflow
token to build a set of all workflow bot user IDs. This set is used by
`handle_note` for self-filtering (skipping notes from any workflow bot)
and by `find_session_for_reply` for session marker scanning.
The naming convention groups tokens alphabetically:
`HUMMINGBIRD_AGENT_ACTION__GITLAB_TOKEN_`.
**Tier 3 -- Model tool tokens.** Read-only tokens (`read_api` scope for
GitLab) used by data source modules during tool execution. Declared in the
YAML config as env var *names* (not values):
```yaml
data_sources:
gitlab:
token_env: GITLAB_TOKEN_RO # name of the env var
```
Per-project overrides are possible:
```yaml
projects:
redhat/hummingbird/containers:
tokens:
gitlab: GITLAB_TOKEN_CONTAINERS_RO # overrides token_env for this project
```
The resolution chain in `resolve_tool_token()` is:
`project_entry.tokens[ds] -> workflow_cfg.data_sources[ds].token_env -> ""`
Storing env var *names* (not values) in YAML means the config file is safe
to commit and audit. Actual secret values live in environment variables,
injected via K8s Secrets at deployment time.
**Tier 4 -- Sandbox.** The sandbox container has zero credentials, zero
network access, and `automountServiceAccountToken: false` (no K8s API
access). Data enters the sandbox only via `write_file()` (stdin piping).
The model cannot instruct the sandbox to reach external APIs -- it must use
the orchestrator's data source tools.
### 4.3. Namespace separation
Sandbox pods run in a dedicated namespace (`hummingbird--agent-sandbox`),
separate from the orchestrator namespace (`hummingbird--internal`). This
limits blast radius: even if a sandbox pod is compromised, it has no
visibility into the orchestrator's Secrets, Pods, or ServiceAccount tokens.
**RBAC setup:**
The orchestrator's ServiceAccount gets a Role in the *sandbox* namespace
(not its own namespace) granting only:
- `pods: create, get, list, delete, patch` -- sandbox pod lifecycle and pool claims
- `pods/exec: create` -- command execution via `kubectl exec`
No CRDs, no custom runtimes, no cluster-scoped resources. The orchestrator
needs only namespace-scoped permissions, so it works with standard
OpenShift RBAC without requesting cluster-admin.
Konflux data is fetched via bearer token from kubeconfig credentials, not
from inside the cluster. The orchestrator's ServiceAccount does not need
access to Konflux namespaces.
**NetworkPolicy:**
A deny-all-egress NetworkPolicy in the sandbox namespace uses `podSelector: {}`
to match all pods and sets `egress: []`. The sandbox cannot reach the internet,
the K8s API, or other pods in the cluster.
### 4.4. Security invariants
These must hold for the security model to be effective. Any change that
violates one of these is a security regression:
1. **Orchestrator tokens must NEVER flow into `ToolRegistry`,
`data_sources`, or model `contents`.** They are resolved in `actions.py`
only.
2. **Workflow action tokens must NEVER flow into `ToolRegistry`,
`data_sources`, or model `contents`.** They are passed as explicit
`token: str` parameters within `actions.py` and `runner.py` only.
3. **Model tool tokens must NEVER flow into `actions.py`.** They are
resolved in `workflow_config.py` and consumed in `data_sources/`.
4. **The sandbox must NEVER have network access or credentials.** No host
mounts, no SA token, no env vars with secrets.
5. **The sandbox archive is NEVER extracted on the orchestrator.** It is
created inside one sandbox and restored inside another. The orchestrator
only transports the bytes.
6. **No cluster-admin required.** Only namespace-scoped resources (Role,
RoleBinding, Pod, NetworkPolicy) are used.
7. **Agent-generated notes are skipped on re-processing.** Notes containing
`SESSION_MARKER_PREFIX` are filtered out in `handle_note()` to prevent
infinite loops. Self-filtering checks against all workflow bot user IDs
(collected at startup) plus the orchestrator bot ID.
8. **All auto-triggers require Developer access.** `handle_pipeline()`,
`handle_merge_request()`, and `handle_note()` each check
`check_member_access()` on the event's user before executing a workflow.
Events from non-developers are silently skipped (or replied to with an
access-denied message for slash commands). This ensures that the *target
MR's* work products -- its diff, description, CI logs, and commit
messages -- originate from a trusted author. Fork MRs from external
contributors are blocked because the MR author lacks Developer+ on the
target project.
**Scope limitation:** this gate only covers the target MR. Once the
agent is running, the model controls tool arguments and can direct tools
at content beyond the target MR -- other MRs, other refs, other job
IDs, even other projects reachable by the read-only token. The current
mitigations are: (a) workflow prompts instruct the model to use the
event's `{project, iid, sha}`, (b) the model would need to be
manipulated via prompt injection from already-trusted content to deviate,
(c) the sandbox prevents the model from acting on manipulated reasoning
beyond producing text output, and (d) the token is read-only with
minimal scope. However, if a data source tool is added that returns
user-authored prose from arbitrary resources (e.g. issue bodies, wiki
pages), it should apply per-author trust filtering (invariant #10).
9. **Data source tools that accept URLs must validate them against an
allowlist.** `tf_get_test_log` restricts URLs to the Testing Farm
artifacts prefix to prevent the LLM from directing the orchestrator to
fetch arbitrary URLs (SSRF).
10. **Third-party commentary entering the model prompt must be
trust-filtered.** Data sources that feed text from users *other than the
event trigger* into `contents` (discussion comments, issue bodies) must
gate on project membership at Developer+ level per author. Content from
untrusted authors must be redacted to a fixed placeholder, never
sanitized or escaped -- there is no reliable way to escape adversarial
text for an LLM prompt. Discussions where *all* notes are untrusted must
be dropped entirely. See §9.1 for the reference implementation in
`gitlab_get_mr_discussions`.
This invariant covers *commentary* (what other people said about the
MR), not *work products* (the diff, CI logs, commit messages). Work
products are inherently the input to the agent's analysis and cannot be
content-filtered without defeating the agent's purpose. For the target
MR, their trust comes from invariant #8 (the event trigger is
Developer+). For content the model fetches beyond the target MR, trust
depends on the tool: discussion tools must filter per-author (#10), while
code/log/metadata tools rely on the mitigations described in #7.
## 5. Configuration System
### 5.1. Two sources, strict separation
Configuration comes from exactly two sources with no overlap:
- **YAML config file** (`CONFIG_PATH`): operational settings, workflow
definitions, project allowlists, data source declarations, and token env
var *names*. This file is safe to commit, review, and audit.
- **Environment variables**: secrets only (API keys, GitLab tokens,
kubeconfig paths). These are injected at deployment time via K8s Secrets.
This split exists because YAML provides structure, validation, and audit
trails, while secrets must stay out of version control.
### 5.2. Config loading
Two config objects are built at startup:
**`Config`** (from `config.py`): loaded by `Config.load(settings)`, where
`settings` is the `settings:` section from the YAML file. Auth/bootstrap
fields come from env vars (`GOOGLE_API_KEY`, `GOOGLE_CLOUD_PROJECT`, etc.).
Operational fields come from the settings dict with sensible defaults.
**`AgentConfig`** (from `workflow_config.py`): loaded by
`workflow_config.load(path)`. Contains all workflow definitions, project
entries, and a pre-built `project_index`.
The two are kept separate because `Config` is needed everywhere (model
construction, sandbox creation, session storage), while `AgentConfig` is
only needed for event routing and data source registration.
### 5.3. YAML config structure
```yaml
settings:
gitlab_url: https://gitlab.com # operational, not a secret
sandbox:
image: quay.io/.../image:tag
namespace: my-namespace # K8s only
model: gemini-3.1-pro-preview
# model: claude-sonnet-4-5@20250929 # Anthropic via Vertex AI (alternative)
max_iterations: 30
max_runs_per_mr: 5
internal_notes: true
max_concurrent_agents: 4
sqs_queue_url: "" # empty = no SQS (local dev)
s3_session_bucket: "" # empty = no S3 (local dev)
trigger_prefix: /hummingbird # slash command prefix
session_marker_prefix: hummingbird-session # HTML comment marker ID
auto_trigger: true # auto-run on failed pipelines
workflows:
analyze-failures:
prompt: workflows/analyze-failures.md # relative to config file dir
action: post_gitlab_note
model: gemini-3.1-pro-preview # per-workflow override
max_iterations: 50 # per-workflow override
data_sources:
gitlab:
token_env: GITLAB_TOKEN_RO # env var name, not the value
konflux:
cluster_url: https://example.com:6443/ns/my-tenant
kubeconfig_env: KUBECONFIG
kubearchive_url: https://kubearchive-api-server-product-kubearchive.apps.example.com
testing_farm: {}
projects:
redhat/hummingbird/containers:
tokens: # per-project token overrides
gitlab: GITLAB_TOKEN_CONTAINERS_RO
```
Design choices in this structure:
**Workflow-first organization.** Each workflow owns its project list, not the
other way around. This scopes data source permissions per workflow-project
pair. A future code-review workflow can have different GitLab tokens (with
different scopes) than the analyze-failures workflow, with no ambiguity.
**Token env var names in YAML (not values).** The YAML file is committed to
the repo. It contains `token_env: GITLAB_TOKEN_RO` (a name), not the actual
token. The actual secret value is resolved at runtime via
`os.environ.get(env_var)`. This allows the config to be reviewed and
audited without exposing secrets.
**Inline `cluster_url` only.** The Konflux cluster URL is operational
configuration (it identifies which cluster to talk to), not a secret.
Putting it inline in YAML makes it visible and auditable. The URL must
include the namespace path (e.g. `/ns/my-tenant`).
### 5.4. Project index
At load time, `_build_project_index()` constructs a reverse lookup:
```python
project_index: dict[str, list[tuple[str, WorkflowConfig, ProjectEntry]]]
# e.g. {"redhat/hummingbird/containers": [("analyze-failures", wf_cfg, proj_entry)]}
```
This provides O(1) lookup when a pipeline event arrives with a project
path. A single project can appear in multiple workflows (e.g. both
`analyze-failures` and a future `code-review`), and each matching workflow
will be triggered independently.
### 5.5. Token resolution chains
**Model tool tokens** (for data source API calls):
```text
resolve_tool_token(project_entry, ds_name, workflow_cfg):
1. project_entry.tokens[ds_name] -> per-project override
2. workflow_cfg.data_sources[ds_name].token_env -> workflow default
3. "" (empty string) -> no token
Each step resolves the env var NAME, then reads os.environ[name].
```
**Workflow action tokens** (for per-workflow GitLab writes):
```text
resolve_action_token(project_entry, ds_name):
1. project_entry.action_tokens[ds_name] -> env var NAME
2. os.environ[name] -> resolved token value
3. "" (empty) -> not configured
No fallback chain. Missing tokens cause validate_env to fail at startup.
```
**Orchestrator tokens** (for operational reads and infrastructure notes):
```text
resolve_orchestrator_token(project_path):
1. ORCHESTRATOR_GITLAB_TOKEN_ -> per-project
2. ORCHESTRATOR_GITLAB_TOKEN -> global fallback
Mangling: "/" -> "_", "-" -> "_", uppercase.
e.g. "redhat/hummingbird/containers" -> ORCHESTRATOR_GITLAB_TOKEN_REDHAT_HUMMINGBIRD_CONTAINERS
```
**Cluster URL** (for Konflux):
```text
resolve_cluster_url(ds_cfg):
-> ds_cfg.cluster_url (inline value in YAML)
```
### 5.6. Environment validation
`validate_env(agent_cfg)` checks at startup that all referenced env vars
exist. It walks every workflow's data sources, project token overrides, and
project action token entries, collecting missing vars into a single error
message. This catches configuration errors early instead of failing mid-run
when a specific data source or workflow action is first used. Missing
`action_tokens` env vars are treated the same as missing model tool tokens:
startup fails unconditionally.
## 6. Agent Loop Design
For the wire-format walkthrough (what bytes go to the model API, what comes back),
see [Agent Model Loop](https://hummingbird-project.io/docs/background/tools/hummingbird-agent-model-loop/). This section
covers the *design rationale* behind the loop.
### 6.1. Full history replay
The Gemini and Anthropic APIs are stateless. Every call sends the complete `contents` list
from the beginning of the conversation. This means every large tool output
sitting in history inflates every subsequent API call.
This property is fundamental to why auto-spill exists (section 7). Without
auto-spill, a single `cat` of a 32KB file early in the conversation adds
~8K tokens to every remaining API call. Over a 15-iteration run, that is
~120K wasted tokens.
The alternative -- conversation compaction (replacing old tool results with
summaries) -- was considered and deferred. Each provider has strict
requirements about content structure (e.g. Gemini model turns must match the
preceding tool turns; Anthropic enforces user/assistant alternation), and
modifying history risks confusing the model or violating API constraints.
Auto-spill solves 90% of the problem with none of the risk.
### 6.2. Budget model: iterations + context limit
The agent uses a dual-limit approach rather than a cumulative token budget:
**Iteration limit** (`max_iterations`, default 30 per workflow config).
Hard cap on the number of model round-trips. This is the primary cost
control lever. With auto-spill keeping per-call context bounded, iteration
count is roughly proportional to cost.
**Per-call context limit** (`CONTEXT_LIMIT`, default 60,000 tokens).
Checked after each API call using `response.usage.input_tokens`. This is
a safety net for cases where auto-spill is not sufficient (e.g., many
small tool results that individually stay under the spill threshold but
cumulatively fill the context).
Why not a cumulative token budget? Because with full history replay, each
API call re-sends everything. "Cumulative billed tokens" double-counts:
call 1 sends 5K, call 2 sends 10K (including the 5K again), so billed
total is 15K but actual new content is only 10K. Iteration count is a
simpler and more predictable proxy for cost.
### 6.3. Two-tier budget escalation
Both limits use the same escalation pattern:
```mermaid
flowchart LR
Normal["Normal
tools available"]
Soft["Soft Warning (80%)
ITERATION_WARNING or
CONTEXT_WARNING
tools still available"]
Hard["Hard Stop (100%)
FINAL_TURN_WARNING
toolConfig: NONE"]
Normal -->|"80% reached"| Soft
Soft -->|"100% reached"| Hard
```
**Soft warning (80%).** An ephemeral user message (`ITERATION_WARNING`) is
injected into `contents` for that turn only, then popped before the
response is persisted. Tools remain available so the model can finish
in-progress work.
**Hard stop (100%).** `FINAL_TURN_WARNING` is injected as an ephemeral
user message AND `tool_defs` is set to `[]` (empty list) to physically
prevent further tool calls. The model must produce text. This is more
reliable than disabling tools at the API layer (`toolConfig.functionCallingConfig.mode: NONE`
on Gemini, `tool_choice: {"type": "none"}` on Anthropic), which models
sometimes ignore (Gemini may return `UNEXPECTED_TOOL_CALL`).
### 6.3.1 Ephemeral messages and Anthropic alternation
All per-turn warnings use ephemeral user messages: they are appended to
`contents` before the API call and popped immediately after. This keeps
the system prompt stable across all iterations and prevents warnings from
polluting the conversation history saved in sessions.
Anthropic requires strict user/assistant alternation. The Anthropic adapter's
`generate()` merges consecutive user messages on a copy of `contents` before
sending the request, so ephemeral warnings (and other adjacent user turns) do
not break the API contract.
### 6.4. Empty response handling
Models occasionally return empty responses (no text, no tool calls). The
agent retries up to `MAX_EMPTY_RETRIES` (2) times:
1. **Pop the empty response** from `contents`. It adds nothing and may
confuse the model on the next call.
2. **Inject `EMPTY_RESPONSE_NUDGE`** as an ephemeral user message on the
next turn: "Your previous response was empty. Please continue..."
3. **Continue the loop** (consuming an iteration).
The nudge is delivered as an ephemeral user message (injected before the
API call and popped after). This avoids mutating the system prompt and
keeps the conversation history clean for session persistence.
The `empty_retries` counter resets to 0 after any successful iteration
(one where tool calls were executed). This means the model gets fresh
retries if it produces empty responses at different points in the
conversation.
**`MALFORMED_FUNCTION_CALL` handling.** Models sometimes return a
`finishReason` of `MALFORMED_FUNCTION_CALL` (Gemini) with no usable tool calls.
This is treated as a special case of empty response: the retry mechanism
kicks in, but the nudge is replaced with `MALFORMED_CALL_NUDGE` which
tells the model to retry with simpler arguments and avoid large text
payloads in tool call arguments.
### 6.5. Error recovery
**Model API errors.** `_generate_with_retry()` catches `ModelError` and
retries up to `MODEL_RETRY_COUNT` (4) times if `retryable` is True (HTTP
5xx and 429). Non-retryable errors (4xx, auth failures) fail immediately.
Retries use exponential backoff: `MODEL_RETRY_BASE_DELAY * 2^attempt`,
capped at `MODEL_RETRY_MAX_DELAY` (60s), giving delays of 5s, 10s, 20s,
40s. Transport-level 429 retry is disabled on the model's HTTP session
(`retry_429=False`) so that rate-limit responses are handled at the model
retry layer with proper backoff instead of being silently retried by
urllib3.
**Transport errors.** Each model adapter's `generate()` catches
`requests.Timeout` and `requests.ConnectionError` from the HTTP call and
wraps them as retryable `ModelError`. This makes timeouts (read and
connect) and connection failures subject to the same retry logic as HTTP
5xx/429. `Timeout` is caught before `ConnectionError` because
`ConnectTimeout` inherits from both. Note that the urllib3 retry adapter
does not retry POST requests (not idempotent), so transport errors from
model calls always propagate to our code.
**Unexpected loop errors.** `run_agent_loop` wraps the
`_generate_with_retry` call in a `try/except Exception` that logs and
breaks instead of propagating. This ensures the function always returns
partial results (accumulated `contents`, `transcript`, sandbox state)
even when an unexpected exception occurs (e.g. `JSONDecodeError` from a
truncated API response). The caller saves the session normally --
conversation history and sandbox archive are preserved for resumption.
This follows the "partial over nothing" principle: 32 iterations of work
are more valuable than a crash.
**No failure-path session save.** `_execute_workflow` does not save a
session when `run_workflow` raises. With the agent loop catching
unexpected errors, the failure path only fires for infrastructure errors
(sandbox start, config) where there is no useful state. Not saving
avoids overwriting a previous good session when a reply attempt fails.
**Tool execution errors.** `_execute_tool_calls()` wraps each
`tool_registry.execute()` in a try/except. Unhandled exceptions are caught
and returned to the model as `{"error": "Tool X failed: ..."}`. This
prevents a single broken tool from crashing the entire session -- the model
sees the error and can adapt.
**Sandbox exec timeout.** If a command exceeds `EXEC_TIMEOUT` (120s), the
`TimeoutExpired` exception is caught in the tool registry and returned as a
structured error dict. The model can retry with a different command or
proceed without the result.
### 6.6. Session resumption
When resuming from a previous session:
1. `initial_contents` (the full `contents` list from the previous run) is
prepended, followed by the user's reply as a new user turn.
2. `CONTINUATION_PROMPT` is appended to the system prompt.
3. The sandbox is restored from the archived tarball.
4. A fresh iteration budget starts from 0.
Saved sessions use a versioned JSON envelope (`format_version: 1`) whose
`contents` are in a canonical (OpenAI-style) message format, independent of
whether the run used Gemini or Anthropic. On resume, `run_workflow` converts
`initial_contents` into the active model's native wire format: for
`format_version >= 1`, via `model.from_canonical()`; for legacy sessions
without a version, `GeminiModel.to_canonical()` migrates Gemini-native history
to canonical form first, then `model.from_canonical()` loads it into the
current backend. After a successful run, `model.to_canonical()` converts the
native `contents` back to canonical form before persistence.
`CONTINUATION_PROMPT` is critical. Without it, the workflow prompt (e.g.,
`analyze-failures.md`) tells the model to follow a rigid multi-phase
workflow: Data.1, Data.2, Analysis.1... The model would try to re-run the
entire analysis. The continuation prompt overrides this: "Do NOT re-run the
full workflow. Respond directly to the user's question."
The practical constraint on resumed sessions is the context window, not
iterations. The restored history from an 8-iteration cold start uses
~50-100K input tokens. Each new tool call adds more. The existing
`CONTEXT_LIMIT` check still applies and will force wrap-up if the context
grows too large.
#### Canonical session format (rationale)
Storing sessions in canonical form decouples persisted history from any one
provider's JSON shape. The same saved thread can be resumed on a different
model adapter (including cross-provider migration) because conversion happens
at load and save boundaries only; the agent loop continues to treat native
`contents` as opaque between those steps.
### 6.7. Prompt caching
Since the agent replays the full conversation history on every API call
(see 6.1), prompt caching reduces the cost of re-processing unchanged
prefixes. The two model backends handle caching differently.
**Gemini.** Vertex AI caches prefixes implicitly on the server side. The
adapter parses `cachedContentTokenCount` from `usageMetadata` and reports it
as `cache_read_tokens`. No request-side changes are needed.
**Claude.** Vertex AI does not support Anthropic's automatic caching
(which requires opt-in at the API level and is not yet available on Vertex).
The adapter uses explicit `cache_control: {"type": "ephemeral"}` annotations
on message content blocks. Up to 4 breakpoint slots are available per
request; the agent uses 2.
**Why 2 breakpoints on messages, not 4 on system/tools/messages.** The
Anthropic prefix hash is cumulative: it covers everything from the start of
the request (tools, system prompt, messages) up to the annotated block. A
single breakpoint on a message already caches the entire prefix including
tools and system. Separate breakpoints on earlier components would be
redundant and waste slots. Using only 2 of the 4 slots leaves room for
future use.
**Why a sliding window.** The conversation grows by 2 messages per turn
(assistant response + user tool result). Two breakpoints slide forward in
lockstep:
- B2 on the last message writes the full prefix to cache.
- B1 on the previous B2 position reads the prior prefix from cache.
Each call after the first gets a cache hit for the entire prefix minus the
newest turn. `_prev_cache_index` on the model instance tracks where B2 was
placed so that B1 can be positioned on the next call.
**Why shallow copies.** The `contents` list is owned by the agent loop and
persisted in sessions. `_annotate_cache_breakpoints()` creates a shallow
copy of the messages list and replaces only the B1/B2 entries with copies
that have `cache_control` injected. The originals are never mutated, so no
cleanup is needed after the API call and `cache_control` never leaks into
saved sessions.
**Ephemeral message interaction.** When the agent loop injects a transient
message (see 6.3.1), `generate()` receives `ephemeral=True`. B2 is not
placed (no cache write) so the transient content never enters the cache.
`_prev_cache_index` is not updated, so the next non-ephemeral call's B1
still points to the last valid write and produces a cache hit. B1 is still
placed to provide a cache read for the stable prefix on the ephemeral call
itself.
**Cost estimation.** `MODEL_PRICING` in `models/__init__.py` stores a 4-tuple per
model prefix: `(input, output, cache_read, cache_write)` per million tokens.
`models.estimate_cost()` computes: `uncached * input + cache_read_tokens *
cache_read_rate + cache_creation_tokens * cache_write_rate + output *
output_rate`. Gemini has `cache_write = 0` (implicit caching has no write
surcharge); Claude has non-zero `cache_write` (1.25x the base input rate for
explicit breakpoints).
## 7. Tool System and Auto-Spill
### 7.1. Tool registry design
`ToolRegistry` is the single point of dispatch for all tool calls. The
agent calls `registry.execute(tool_call)` and gets a `dict` back. It never
calls sandbox methods or data source functions directly.
Three built-in tools are always available:
- **`sandbox_exec`** -- runs `sh -c ` in the sandbox container.
Returns `{exit_code, stdout, stderr}`, with auto-spill for large outputs.
- **`fetch_to_sandbox`** -- calls a data source function and writes the
result to a specified path in the sandbox. Returns metadata only (path,
byte count, line count). Used when the model wants explicit path control.
- **`fetch_batch_to_sandbox`** -- calls multiple `fetch_to_sandbox` in one
tool call. Saves iterations vs. sequential calls (e.g., fetching both
pipelineruns and taskruns in one round-trip).
Data source tools are registered dynamically per workflow config. Each data
source module provides a `register()` function that creates `ToolDef`
objects and closures over credentials, then calls
`registry.register_data_source(tool_def, func, response_metadata)`.
### 7.2. Auto-spill architecture
Auto-spill is the key mechanism for keeping the LLM context bounded. Without
it, large outputs accumulate in the `contents` list and inflate every
subsequent API call (because both APIs replay the full history).
```mermaid
flowchart TD
Output["Tool produces output"]
SizeCheck{"size > OUTPUT_PREVIEW_BYTES
(4 KB)?"}
Inline["Return full output inline"]
Spill["Write full output to
/tmp/data/_out/N.txt"]
Preview["Return preview
(head + tail) +
file path metadata"]
Output --> SizeCheck
SizeCheck -->|"<= 4 KB"| Inline
SizeCheck -->|"> 4 KB"| Spill --> Preview
```
Three spill paths share the same `_spill_counter` to prevent filename
collisions:
**`sandbox_exec` spill.** When stdout or stderr exceeds
`OUTPUT_PREVIEW_BYTES` (4096 bytes), the full output is written to
`/tmp/data/_out/{counter}.txt` via `_spill_field()`. The model receives:
```json
{
"exit_code": 0,
"stdout": "",
"stdout_truncated": true,
"stdout_file": "/tmp/data/_out/0.txt",
"stdout_bytes": 32768,
"stdout_lines": 1024,
"stdout_tail": ""
}
```
The preview (head + tail) gives the model enough context to decide whether
to process the full file with `jq`/`grep`/`head`.
**Data source inline spill.** When a direct data source call returns text
larger than `MAX_INLINE_SIZE` (4096 bytes), `_spill_data_source()` writes
it to `/tmp/data/_out/{name}_{counter}.txt` and returns:
```json
{
"saved_to": "/tmp/data/_out/konflux_list_pipelineruns_1.txt",
"bytes": 85432,
"lines": 2100,
"preview": ""
}
```
**Streaming responses** (e.g., `gitlab_get_repo_archive` which returns a
`StreamingResponse` with suffix `.tar.gz`) are written via
`_spill_streaming()`. Text streams (`binary=False`, the default) capture a
UTF-8 preview from the head while piping chunks to the sandbox. Binary
streams (`binary=True`) skip preview capture entirely and return only
`{saved_to, bytes}`, avoiding meaningless decoded output for formats like
tar.gz. The `StreamingResponse.suffix` field controls the file extension
(`.jsonl`, `.tar.gz`, `.log`).
Non-streaming binary responses are written to `.bin` files with no preview
via `_spill_binary()`.
**Sandbox-optional mode.** `ToolRegistry` accepts `sb=None` for use cases
that do not require a sandbox (e.g. evaluation harnesses that only use
native model tools like Google Search grounding). When no sandbox is
available:
- Sandbox tools (`sandbox_exec`, `fetch_to_sandbox`, `fetch_batch_to_sandbox`)
are omitted from `get_tool_defs()` so the model never sees them.
- **Text data sources** (inline and streaming) degrade to truncated previews
(`OUTPUT_PREVIEW_BYTES`). The model receives `{preview, truncated: true}`
instead of `{saved_to, preview}`. Streaming text is consumed into a memory
buffer up to the preview limit and the rest is discarded.
- **Binary data** (streaming and non-streaming) returns an error with the
byte count (`{error, bytes}`). Binary content cannot be meaningfully
previewed as text.
**`fetch_to_sandbox` spill.** Always writes to the caller-specified path
(not `/tmp/data/_out/`). Returns metadata only (no preview). The model uses this
when it wants a specific filename for later processing.
### 7.3. Why auto-spill instead of rejecting large outputs
The earlier design rejected large data source responses with "use
`fetch_to_sandbox` instead." This caused two wasted iterations per rejection:
the model makes the call, gets rejected, then has to repeat with
`fetch_to_sandbox`. With auto-spill, the data flows to a sandbox file
transparently. Validation showed ~10% fewer iterations with auto-spill.
### 7.4. Why fetch_to_sandbox still exists alongside auto-spill
Auto-spill handles the common case, but `fetch_to_sandbox` provides:
- **Explicit path control.** The model can choose meaningful filenames
(`/tmp/data/pipelineruns.json`) instead of getting auto-generated names
(`/tmp/data/_out/konflux_list_pipelineruns_1.txt`).
- **Batch fetching.** `fetch_batch_to_sandbox` combines multiple fetches
in one tool call, saving iterations.
- **No preview overhead.** `fetch_to_sandbox` returns metadata only, which
is useful when the model knows it will process the file with
`sandbox_exec` anyway.
### 7.5. ToolDef notes
`ToolDef` has an optional `notes` field for domain knowledge that belongs
in the system prompt but not in the tool's JSON schema. Examples:
- Konflux notes explain dual K8s/Kubearchive fetch, UID deduplication, and
the two label selectors (BUILD vs. TEST).
- Testing Farm notes explain XML result structure and usage patterns.
`ToolRegistry.get_tool_notes()` collects all non-None notes into a
`## Data Source Notes` section that is prepended to the workflow body in
the system prompt:
```text
BASE_SYSTEM_PROMPT + tool_notes + workflow_body
```
This keeps domain knowledge close to the tool definitions (in the data
source module) rather than duplicated in every workflow `.md` file.
### 7.6. Response metadata
`register_data_source()` accepts optional `response_metadata` -- a dict
that is merged into every response from that tool (inline, auto-spill, and
`fetch_to_sandbox`). Used for:
- **Konflux:** `{"konflux_ui": "https://..."}` so the model can build
reviewer-facing PipelineRun links.
- **Testing Farm:** `{"artifacts_base": "https://..."}` so the model can
build artifact links.
This avoids having the model ask "what is the Konflux UI URL?" -- the
information arrives with every tool response.
## 8. Sandbox Architecture
### 8.1. The Sandbox protocol
All backends implement an 8-method `typing.Protocol`:
```python
class Sandbox(Protocol):
def start(self) -> None # create container/pod, pre-create /tmp/data/
def exec(command, *, stdin_data: bytes | None) -> ExecResult # sh -c in sandbox
def write_file(path, data: bytes) -> None # stdin pipe: cat > path
def stream_to_file(path, chunks: Iterable[bytes]) -> tuple[int, int] # Popen stdin pipe, returns (bytes, lines)
def stream_exec(command, chunks: Iterable[bytes]) -> ExecResult # Popen with streaming stdin
def read_file_iter(path) -> Iterator[bytes] # Popen stdout pipe in chunks
def cleanup(self) -> None # rm -f container / delete pod
def linger(session_id) -> None # keep alive for reuse, or fall back to cleanup
```
`ExecResult` is `(exit_code: int, stdout: str, stderr: str)`.
`stream_to_file()` and `stream_exec()` use `subprocess.Popen` with a stdin
pipe, writing chunks incrementally. This avoids buffering large payloads in
orchestrator memory (e.g., streaming a tar.gz archive into the sandbox).
`read_file_iter()` reads from a Popen stdout pipe in 64 KB chunks.
`write_file()` uses stdin piping (`cat > path`), never host volume mounts.
This is critical: it means data flows through the orchestrator process, not
through a shared filesystem. The sandbox has no host mounts.
### 8.2. Backend selection
Five sandbox backends are available:
| Backend | `--sandbox` | When to use |
| --------------------- | --------------- | ---------------------------------------------------- |
| `PodmanSandbox` | `podman` | Local development (default for `run`) |
| `K8sSandbox` | `k8s` | Direct pod creation on a K8s cluster |
| `K8sPoolSandbox` | `k8spool` | Pre-warmed pool via Deployment (default for `serve`) |
| `KubeVirtSandbox` | `kubevirt` | Direct VMI creation with SSH exec |
| `KubeVirtPoolSandbox` | `kubevirtpool` | Pre-warmed VMI pool via VMIRS |
`create_sandbox()` factory handles `podman`, `k8s`, and `kubevirt`. Pool
backends (`k8spool`, `kubevirtpool`) are handled in `__main__.py`, which
creates a `SandboxPool` or `VmiPool` and passes it to `run_workflow()`.
The sandbox backend can be set per-workflow via the `sandbox:` field in the
workflow config YAML. Resolution order: `workflow.sandbox` > CLI `--sandbox`
> default. This allows different workflows to use different backends (e.g.,
lightweight code review uses pod pool, heavier analysis uses VM pool).
K8s is never auto-detected from the environment. It requires explicit CLI
flags. This prevents accidental use of a K8s sandbox when developing
locally.
For K8s namespace resolution (`k8s` and `k8spool`):
1. If `--namespace` is provided, use it.
2. If `--context` is provided, extract the namespace from the kubeconfig
context. If the context has no default namespace, raise an error.
3. If neither is provided (in-cluster), use `sandbox.namespace` from the
config file.
### 8.3. PodmanSandbox
```bash
podman run -d --name hb-sandbox-{uuid8} --network=none --user 65532 \
--workdir /tmp {image} sleep infinity
```
- Unique container name with UUID suffix prevents collisions
- `sleep infinity` keeps the container alive for repeated `exec` calls
- `--network=none` provides complete network isolation
- `--user 65532` is a fixed non-root UID (matches `nonroot` in distroless)
- Cleanup: `podman rm -f` (force, in case exec is still running)
### 8.4. K8sSandbox: the hybrid approach
The K8s sandbox uses a hybrid of two tools:
- **kubernetes Python library** for pod lifecycle: `create_namespaced_pod`,
`read_namespaced_pod` (poll for Running), `delete_namespaced_pod`.
- **`kubectl exec` subprocess** for command execution.
Why not use the kubernetes Python library for exec too? Three problems
discovered during development:
1. **No stdin EOF signal in WebSocket v1-v4.** The Kubernetes exec protocol
uses WebSocket channels (stdin=0, stdout=1, stderr=2). Protocol versions
1-4 have no mechanism to signal "stdin is done." Commands like
`cat > /file` hang forever waiting for more input. Python client v5
support does not exist.
2. **`BrokenPipeError` on large stdin.** Sending more than ~1MB through the
WebSocket `stream()` API causes pipe errors, breaking `write_file()` for
large data source responses.
3. **Unbounded memory from `stream()`.** The `stream()` function accumulates
all stdout/stderr data in memory with no streaming control. A command
producing megabytes of output would consume unbounded memory.
`kubectl exec` as a subprocess avoids all three problems and provides the
same interface as `podman exec` -- stdin via `subprocess.PIPE`, stdout/stderr
captured, exit code from return code. The implementation in `exec()` is
nearly identical between `PodmanSandbox` and `K8sSandbox`.
### 8.5. Pod manifest design
The K8s pod manifest is built by `_build_pod_manifest()`:
```yaml
metadata:
labels:
app.kubernetes.io/name: hummingbird-agent-sandbox
spec:
automountServiceAccountToken: false # no K8s API from sandbox
activeDeadlineSeconds: 1800 # 30min hard timeout, backstop
restartPolicy: Never
securityContext:
runAsNonRoot: true
seccompProfile: RuntimeDefault
containers:
- name: sandbox
command: ["sleep", "infinity"]
workingDir: /tmp
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
resources:
requests: {cpu: 100m, memory: 256Mi, ephemeral-storage: 256Mi}
limits: {cpu: "1", memory: 1Gi, ephemeral-storage: 2Gi}
```
`runAsUser` is deliberately omitted. On OpenShift, the `restricted-v2` SCC
assigns a UID from the namespace UID range. On vanilla K8s, the image's
`USER` directive is used. Setting an explicit UID would conflict with
OpenShift's SCC admission.
### 8.6. Auth modes
- **Local development:** `--context` flag passes the kubeconfig context to
`config.new_client_from_config(context=...)`, creating a per-instance
`ApiClient`.
- **Production (in-cluster):** `context=None` triggers
`config.load_incluster_config()`, using the pod's ServiceAccount token.
The `kubectl exec` commands include `--context` when running locally but
omit it when in-cluster (kubectl uses the default in-cluster config).
### 8.7. Pre-created directories
Both backends run `mkdir -p /tmp/data` in `start()` after the
container/pod is up. This provides a working directory for the model
without consuming an iteration. `/tmp/data/_out` (the spill directory) is
created on demand by `mkdir -p $(dirname ...)` in `write_file()`.
### 8.8. K8sPoolSandbox: Deployment-backed pool
On-demand pod creation adds 5-30 seconds of latency per workflow (image
pull, scheduling, container start). For interactive use (slash commands,
reply-based resumption), this delay is user-facing. The pool eliminates it.
**Mechanism.** A Kubernetes Deployment maintains a set of pods labelled
`hummingbird/role: standby`. When a sandbox is needed, `SandboxPool.claim()`
finds a Running standby pod, patches it to `role: active`, clears its
`ownerReferences` (detaching it from the ReplicaSet), and annotates it with
`hummingbird/reap-by` (an absolute UTC deadline). The Deployment controller
sees the ReplicaSet is under the desired replica count and creates a
replacement.
**Pod lifecycle.**
1. Deployment creates pod -> `role: standby` (managed by ReplicaSet)
2. `claim()` patches -> `role: active`, `ownerReferences: []`, `reap-by: now + max_active_seconds`
3. Agent uses the pod via inherited `K8sSandbox.exec()`
4. On success: `linger()` patches -> `reap-by: now + linger_seconds`,
`session-id: ` (pod stays alive)
5. On reply within linger window: `try_reclaim()` patches ->
`reap-by: now + max_active_seconds`, clears `session-id`
6. On failure or linger expiry: pod is deleted (by `cleanup()` or the reaper)
**Pod lingering.** After a successful workflow, pool sandboxes are kept
alive for `linger_seconds` (default 300) instead of being deleted
immediately. The pod is annotated with `hummingbird/session-id` to link it
back to the session. If a user reply arrives within the linger window,
`try_reclaim()` finds the pod by session-id, clears the annotation (marking
it as in-use), and resets `reap-by`. This skips pod creation and S3 archive
restoration. If no reply arrives, the reaper deletes the pod when its
`reap-by` deadline passes.
**Concurrency safety.** `session-id` is only present while lingering. Its
absence during active execution prevents concurrent replies from adopting
the same pod. `try_reclaim()` is serialized by a threading lock; the first
caller wins, others fall back to S3 restore with a fresh pod.
**Reaping.** `reap_expired()` is called once per workflow execution (after
sandbox acquisition, before the model loop). It performs a single sorted
pass over all active pods:
1. Non-lingering pods (no `session-id`) past their `reap-by` deadline are
deleted immediately.
2. Lingering pods (with `session-id`) are sorted by `reap-by` ascending.
The loop deletes pods that are either expired (`now > reap-by`) or
exceed `max_lingering_pods` (default 2, configurable), starting with
those closest to their deadline. The loop stops when both conditions are
satisfied (`now <= reap-by` and remaining count <= limit).
This replaces the previous design where `_reap_expired()` ran inside
`claim()` on every poll iteration. Moving it to a once-per-workflow call
reduces API load and centralizes cleanup. The `max_lingering_pods` cap
prevents unbounded pod accumulation from many short-lived workflows.
**Indefinite wait.** `claim()` polls until a pod is available or the
shutdown event fires. It logs at DEBUG level on each poll, escalating to
WARNING after ~2 minutes. This is normal behavior when the pool (Deployment
replicas) is smaller than `max_concurrent_agents` -- the SQS semaphore caps
concurrency, so demand will not permanently exceed supply.
**Config simplification.** In pool mode, the pod image, resources, metadata,
and security context are defined solely in the Deployment template. The
agent config only needs `sandbox.namespace` and
`sandbox.active_deadline_seconds`. This eliminates duplication between the
agent configmap and the Deployment.
**Class design.** `K8sPoolSandbox` inherits from `K8sSandbox` to reuse
`exec()`, `write_file()`, `read_file_iter()`, and `cleanup()`. It overrides
`__init__()` (does not call `super().__init__()` since that resolves
kubeconfig), `start()` (claims from pool instead of creating a pod), and
`linger()` (patches annotations instead of deleting). It adds
`start_from_existing()` for the reclaim path. All sandbox backends
implement `linger(session_id)`: non-pool backends fall back to `cleanup()`.
### 8.9. KubeVirtSandbox: SSH-based VM execution
`KubeVirtSandbox` provides full VM isolation using KubeVirt
VirtualMachineInstances. It parallels `K8sSandbox` but uses
`CustomObjectsApi` for VMI lifecycle and `ssh` for command execution
instead of `kubectl exec`.
**Why SSH?** VMs have no `kubectl exec` equivalent. KubeVirt provides
`virtctl console` (serial) and VNC, but these are not suitable for
programmatic command execution. SSH provides a reliable, well-understood
exec channel with stdin/stdout/stderr piping.
**SSH key injection.** The VM sandbox image includes a custom
`inject-ssh-keys.service` that reads SSH public keys from a KubeVirt
Secret volume attached as a virtio disk with serial `ssh-pubkeys`
(visible at `/dev/disk/by-id/virtio-ssh-pubkeys`) and installs them to
`/root/.ssh/authorized_keys` before sshd starts. This avoids the need
for `cloudInitNoCloud` or `qemu-guest-agent`.
**Single-VM lifecycle:**
1. Generate an ephemeral ed25519 key pair via `ssh-keygen`
2. Create a K8s Secret with the public key
3. Create the VMI referencing the containerDisk image and the SSH Secret
4. Poll VMI status until Running with a pod network IP
5. Poll SSH readiness (`ssh sandbox@ip true`)
6. Execute workflow commands via SSH
7. Cleanup: delete VMI, delete Secret, remove temp key files
**Polling strategy.** Two-phase wait: first poll `status.phase` via
`get_namespaced_custom_object` until Running (same interval as pod
polling), then poll SSH readiness with `ConnectTimeout=5`. Total timeout
matches `POD_START_TIMEOUT` (300s) for VMI startup plus
`_SSH_READY_TIMEOUT` (120s) for SSH.
### 8.10. KubeVirtPoolSandbox: VMIRS-backed pool
`KubeVirtPoolSandbox` + `VmiPool` parallel `K8sPoolSandbox` +
`SandboxPool`, operating on VMI custom resources instead of Pods.
**VirtualMachineInstanceReplicaSet (VMIRS).** KubeVirt's native
controller maintains a desired number of identical VMIs. When a VMI is
claimed (label flipped to `active`, ownerReferences cleared), the VMIRS
automatically creates a replacement -- the same pattern as a Deployment
replenishing claimed pods.
**Shared SSH key.** Pool mode uses a single key pair for all standby VMIs.
The public key is stored in a K8s Secret referenced by all VMIs in the
VMIRS template. The private key path is provided via
`sandbox.vm_ssh_private_key` in the config. On pool restart, a new key
pair can be generated; existing VMIs drain naturally with the old key.
**Claim/reap/linger semantics.** Identical to the pod pool: `claim()`
polls for `Running` VMIs with `hummingbird/role=standby`, patches to
`active`, clears ownerReferences. `reap_expired()` deletes VMIs past
their `hummingbird/reap-by` deadline. `try_reclaim()` finds lingering
VMIs by `hummingbird/session-id`. The only difference is the API: all
operations go through `CustomObjectsApi` with `kubevirt.io/v1`
`virtualmachineinstances`.
**Class design.** `KubeVirtPoolSandbox` inherits from `KubeVirtSandbox`
to reuse `exec()`, `write_file()`, `read_file_iter()` (all SSH-based).
It overrides `__init__()` (takes pool, skips key generation),
`start()` (claims from pool), `cleanup()` (only deletes VMI, not
Secret/keys which are pool-owned), and `linger()` (patches annotations).
## 9. Data Sources
Data sources are external APIs wrapped as tool-calling functions. The model
invokes them by name; the orchestrator executes them and returns results
(inline or auto-spilled). Each data source module follows the same pattern:
1. Define `TOOL_DEFS` -- a list of `ToolDef` objects with names,
descriptions, parameter schemas, and optional notes.
2. Define implementation functions that take a pre-configured client as the
first argument.
3. Provide a `register()` function that creates the client, wraps
implementations with `functools.partial`, and calls
`registry.register_data_source()` for each tool.
Credentials are captured at registration time via `functools.partial`
closures. They are never stored as global state and never leak into tool
definitions or model contents.
### 9.1. GitLab
Uses `python-gitlab` library with `retry_transient_errors=True` for
automatic retry on transient HTTP errors.
**Tools:**
- `gitlab_get_mr_details` -- MR metadata (title, author, SHA, labels, URLs)
- `gitlab_get_commit_statuses` -- all CI/CD statuses for a commit SHA
(paginated automatically). Covers both Konflux external statuses and
native GitLab CI job statuses. Each status includes `target_url` (job
page link containing the job ID) and `allow_failure`.
- `gitlab_get_mr_diff` -- changed files in the MR
- `gitlab_get_file_at_ref` -- raw file content at a git ref
- `gitlab_get_repo_archive` -- repository tar.gz via
`repository_archive(iterator=True)`. Returns a `StreamingResponse` with
chunked binary data (suffix `.tar.gz`), so the archive streams to the
sandbox without buffering in orchestrator memory.
- `gitlab_get_job_log` -- job trace (log output) for a GitLab CI job.
Uses `lazy=True` on the job object and `trace(iterator=True)` for
streaming. Each chunk is decoded with `errors='replace'` and ANSI escape
codes are stripped per-chunk before re-encoding. Returns a
`StreamingResponse` (suffix `.log`). Per-chunk ANSI stripping is safe
because escape sequences are <20 bytes and chunks are 1024+ bytes.
No `response_metadata` is set because GitLab commit statuses already contain
`target_url` fields that the model uses for linking.
**`gitlab_get_mr_discussions` -- trust filtering and redaction.**
MR discussions are the first data source that feeds *user-generated free text*
into the model prompt. Unlike diffs and CI logs (which are code or machine
output), discussion comments can contain arbitrary prose written by anyone
who can post on the MR -- including external contributors on public projects.
This creates a prompt injection vector: an attacker posts a comment containing
instructions that manipulate the model's review output.
The discussions tool addresses this with a three-layer defence:
1. **Author trust gate.** Each note's author is checked for Developer+ access
(level >= 30) on the project via `members_all.get(author_id)`. Results are
cached per `author_id` within a single call to avoid repeated API lookups.
System notes (merge events, label changes) bypass the author check.
Agent-authored notes are trusted via their author's Developer+ access
(the bot account holds Developer access on configured projects). Note:
agent note detection for *redaction* purposes (stripping transcripts and
footers) uses session marker presence, but trust is always based on the
author's access level, not the marker. This prevents marker injection
from granting trust to non-member notes.
2. **Redaction, not sanitization.** Untrusted notes in a discussion that also
contains trusted notes are replaced with a fixed placeholder
(`[redacted: non-member comment]`). The placeholder preserves the
conversation structure (the model sees that someone replied, but not what
they said). Discussions where *all* notes are untrusted are dropped
entirely. No attempt is made to sanitize or escape untrusted content --
there is no reliable escaping mechanism for LLM prompts, so the only safe
option is to withhold the content entirely.
3. **Agent note stripping.** Agent-authored notes contain large
`Agent transcript
...
(historical)"]
K8s["K8s API
(live)"]
end
Combine["Combine items"]
Dedup["Deduplicate
by metadata.uid"]
KA --> Combine
K8s --> Combine
Combine --> Dedup
```
For each resource type (PipelineRuns, TaskRuns), the client fetches from
both Kubearchive (completed/historical resources) and the live K8s API,
combines the results, and deduplicates by `metadata.uid`. This ensures no
resources are missed regardless of whether they have been archived yet.
**Two label selectors.** Konflux uses different labels for BUILD and TEST
PipelineRuns:
- BUILD: `pipelinesascode.tekton.dev/sha=`
- TEST: `pac.test.appstudio.openshift.io/sha=`
Each `get_pipelineruns()`/`get_taskruns()` call queries both selectors and
combines the results.
**Pod log fetching.** `get_pod_log()` accepts an optional `container`
parameter. When specified, it fetches logs for that single container. When
omitted, it discovers all containers from the pod spec and concatenates
their logs with `=== container_name ===` headers. Each individual log
fetch tries Kubearchive first, then falls back to the live K8s API. Logs
may be unavailable from both sources if the pod has expired.
**Streaming pagination.** K8s list endpoints can return pages of 80+ MB
when a commit touches many components (e.g. 500 PipelineRuns per page).
`iter_paginated()` uses `session.get(stream=True)` and `ijson.parse()` to
stream-parse each page: items are yielded one at a time via
`ObjectBuilder`, and the `metadata.continue` pagination token is captured
from the same parse pass. `resp.raw.decode_content = True` is set so
urllib3 transparently decompresses gzip/deflate Content-Encoding inline
(Kubearchive returns gzip-compressed responses). Only one item is in
memory at a time -- O(single\_item) regardless of page size. Truncated or
malformed streams raise `IncompleteJSONError`, which is caught and treated
like a network error (log a warning, stop iterating that source).
**Credential resolution.** `KonfluxClient.__init__()` parses the kubeconfig
file to extract the API server URL and bearer token for the cluster. The
cluster domain is extracted from the `cluster_url` config value. This
approach avoids depending on `kubectl` or the kubernetes Python library for
API authentication.
**Response metadata:** `{"konflux_ui": "https://konflux-ui.apps./ns/"}` is
merged into every response so the model can build reviewer-facing links
like `[{name}]({konflux_ui}/pipelinerun/{name})`.
### 9.3. Testing Farm
Uses `requests.Session` with a module-level retry adapter (429, 5xx) for
resilience against transient errors.
**Tools:**
- `tf_get_results` -- JUnit XML results for a request ID
- `tf_get_test_log` -- individual test log by URL (from results.xml);
restricted to the `ARTIFACTS_BASE` prefix to prevent SSRF
- `tf_get_request_status` -- request state, queue/run times
**ToolDef notes** on `tf_get_results` document the XML structure:
`//testsuites/@overall-result`, `//testcase/@result`, `//testcase/logs/log`
with `@name` and `@href`. This domain knowledge goes into the system prompt
so the model knows how to parse the XML with `sandbox_exec` using
`python3 xml.etree.ElementTree`.
**Response metadata:** `{"artifacts_base": "https://artifacts.osci.redhat.com/testing-farm"}`
is merged into every response so the model can build artifact links like
`[{request_id}]({artifacts_base}/{request_id}/)`.
### 9.4. Registration flow
`data_sources.register_selected()` is the entry point called by `runner.py`.
It iterates the workflow config's `data_sources` dict and registers only the
declared sources:
```python
for ds_name, ds_cfg in wf_cfg.data_sources.items():
if ds_name == "gitlab":
token = resolve_tool_token(proj_entry, "gitlab", wf_cfg)
gitlab.register(registry, gitlab_url, token)
elif ds_name == "konflux":
cluster_url = resolve_cluster_url(ds_cfg)
kubeconfig_path = os.environ.get(ds_cfg.kubeconfig_env, "")
konflux.register(registry, cluster_url, kubeconfig_path, ds_cfg.kubearchive_url)
elif ds_name == "testing_farm":
testing_farm.register(registry)
```
This selective registration means a workflow with `data_sources: {gitlab: {...}}`
only exposes GitLab tools to the model. Konflux and Testing Farm tools do
not appear in the tool definitions, preventing the model from attempting to
use unconfigured data sources.
### 9.5. Data flow: streaming vs buffered
Data moves from external APIs through the orchestrator to the model (inline
or auto-spilled to sandbox). The memory profile of each tool depends on
whether the HTTP response is consumed incrementally or buffered entirely.
**Streaming (preferred for large/unbounded responses).** The HTTP response
is consumed incrementally -- via `ijson` stream parsing, `iterator=True`
in python-gitlab, or lazy pagination. Only one chunk or item is in memory
at a time. Used by:
- `iter_paginated()` (Konflux) -- `session.get(stream=True)` + `ijson`
- `get_commit_statuses()` (GitLab) -- `statuses.list(iterator=True)` via
`_iter_commit_statuses()`, yielding JSONL bytes one status at a time
- `get_repo_archive()` (GitLab) -- `repository_archive(iterator=True)`,
returns chunked tar.gz binary via `StreamingResponse(suffix=".tar.gz",
binary=True)` -- no text preview is generated
- `get_job_log()` (GitLab) -- `trace(iterator=True)` with per-chunk ANSI
stripping, returns `StreamingResponse(suffix=".log")`
All produce `StreamingResponse` objects that flow through
`stream_to_file()` into the sandbox without accumulating in orchestrator
memory. `StreamingResponse.suffix` controls the auto-spill filename
extension (`.jsonl`, `.tar.gz`, `.log`). `StreamingResponse.binary`
controls whether a head preview is captured (`False` for text, `True` to
skip for opaque binary formats).
**Buffered (acceptable for small/bounded responses).** The full response is
loaded into memory as a string or dict. This is fine when the response size
is bounded and small (typically < 1 MB). Used by:
- `get_mr_details()` (GitLab) -- single MR object, < 10 KB
- `get_file_at_ref()` (GitLab) -- single file, bounded by repo constraints
- `tf_get_results()`, `tf_get_test_log()`, `tf_get_request_status()`
(Testing Farm) -- XML/text/JSON, typically < 1 MB
**Buffered with risk (candidates for future streaming).** Same buffered
pattern but the response size is not bounded by design. Auto-spill
mitigates the context growth problem (large outputs are written to sandbox
files), but the orchestrator still spikes RSS during the fetch:
- `get_pod_log()` (Konflux) -- `resp.text` per container, concatenated.
Multi-container pods accumulate all logs.
- `get_mr_diff()` (GitLab) -- full diff JSON, scales with MR size.
GitLab truncates server-side but the result can still be large.
**Invariant: paginated K8s list endpoints must always use streaming.**
Page sizes scale with the number of components in the commit -- a single
page can contain hundreds of PipelineRuns or TaskRuns (80+ MB JSON).
Buffering these responses risks OOM under normal production workloads,
especially with concurrent workflows.
## 10. Event Pipeline and Production Operations
### 10.1. Two-stage SQS pipeline (ingress router + FIFO worker)
Events flow through a two-stage pipeline to serialize per-session work:
```mermaid
flowchart LR
STD["Standard SQS
(ingress)"]
R["Router"]
FIFO["SQS FIFO
(work queue)"]
W["Worker
(poll_loop)"]
STD -->|ReceiveMessage| R
R -->|"SendMessage
MessageGroupId"| FIFO
W -->|"continuation
group=session_id"| FIFO
FIFO -->|ReceiveMessage| W
```
**Ingress router.** `events.router_loop()` is a single-threaded loop
that long-polls the standard (ingress) queue, peeks into each SNS
envelope to assign a `MessageGroupId`, forwards the raw message body
to the FIFO queue, then deletes from the standard queue.
- **Note events:** `MessageGroupId = discussion_id` from the webhook
body. This serializes replies to the same agent session -- FIFO
delivers at most one in-flight message per group.
- **Pipeline / MR events:** `MessageGroupId = SQS MessageId` (unique
per message). No serialization -- each event is its own group.
- **MessageDeduplicationId:** Always the SQS `MessageId` from the
standard queue. Absorbs duplicate deliveries from standard SQS's
at-least-once semantics (5-minute dedup window).
If `SendMessage` to FIFO fails, the message is not deleted from the
standard queue and retries via visibility timeout.
**FIFO worker.** `events.poll_loop()` long-polls the FIFO queue,
decodes SNS envelopes, and dispatches to `handle_event`. The
`max_concurrent_agents` semaphore caps parallel groups being processed.
FIFO guarantees that within a group, only one message is in-flight.
**Two-phase processing.** When the FIFO worker picks up a webhook
(Phase 1), the handler validates the event, resolves the `session_id`,
then posts a **continuation message** back to the same FIFO with
`MessageGroupId = session_id`. The Phase 1 message is deleted quickly
(~1-5s). Phase 2 processes the continuation: loading session state,
building `WorkflowRequest`, and calling `_execute_workflow`. Because
Phase 2 continuations share a `MessageGroupId` per session, all work
for a given session is serialized -- even if it spans multiple GitLab
discussion threads.
**Uniform message format.** Both webhook messages (from SNS) and
continuation messages use the same SNS-style envelope with gzip+base64
compression. `encode_envelope()` is the inverse of
`decode_sns_message()`. The consumer doesn't need to distinguish
between webhook and continuation messages at the transport layer.
**Graceful shutdown.** SIGTERM/SIGINT set a `shutdown_event`. Both
loops exit. Messages in SQS become visible again after the visibility
timeout for other consumers.
**Why two queues.** The standard queue receives events from SNS (via
subscription). The FIFO queue serializes per-session work. The
gitlab-event-forwarder and SNS subscription are unchanged -- the
routing logic lives entirely in the agent codebase.
### 10.2. SNS envelope decoding
Messages arrive as SNS notifications with:
- `MessageAttributes.source` -- event source (e.g., `"gitlab"`)
- `MessageAttributes.event_type` -- event type (e.g., `"pipeline"`,
`"merge_request"`, `"note"`)
- `MessageAttributes.content_encoding` -- `"gzip+base64"` for
gitlab-event-forwarder, absent for kubernetes-event-forwarder
- `Message` -- the actual event body (JSON string, or gzip+base64 encoded)
`decode_sns_message()` handles both encoding formats transparently.
### 10.3. Event routing
`runner.handle_event()` dispatches on `(source, event_type)`:
```mermaid
flowchart TD
Event["Event arrives"]
Check{"source/type?"}
Cont["handle_continuation()"]
Pipeline["handle_pipeline()"]
MRHandler["handle_merge_request()"]
Note["handle_note()"]
Ignore["Ignore"]
Event --> Check
Check -->|"agent/continuation"| Cont
Check -->|"gitlab/pipeline"| Pipeline
Check -->|"gitlab/merge_request"| MRHandler
Check -->|"gitlab/note"| Note
Check -->|"other"| Ignore
Pipeline --> FilterSource{"source = merge_request_event?"}
FilterSource -->|"yes"| AutoResolve["auto-resolve on success
(unconditional)"]
FilterSource -->|"no"| Skip1["Skip"]
AutoResolve --> PipeRules["evaluate trigger_rules
(status + user + branch + title)"]
PipeRules -->|"allow"| Enqueue1["_enqueue_continuation()"]
PipeRules -->|"deny"| Skip1a["Skip"]
MRHandler --> FilterMRAction{"action in open/update?
draft = false?"}
FilterMRAction -->|"yes"| MRRules["evaluate trigger_rules
(user + branch + title)"]
FilterMRAction -->|"no"| Skip1b["Skip"]
MRRules -->|"allow"| Enqueue2["_enqueue_continuation()"]
MRRules -->|"deny"| Skip1c["Skip"]
Note --> FilterMRNote{"MR note?
action = create?"}
FilterMRNote -->|"yes"| NoteType
FilterMRNote -->|"no"| Skip3["Skip"]
NoteType{"Note type?"}
NoteType -->|"author_id == bot_id"| Skip4["Skip (own note)"]
NoteType -->|"/hummingbird help"| Help["post_simple_reply (help)"]
NoteType -->|"/hummingbird wf-name"| SlashCmd["_resolve_slash_workflow()
→ _enqueue_continuation()"]
NoteType -->|"DiscussionNote reply"| FindSession["find_session_for_reply()"]
NoteType -->|"other"| Skip5["Skip"]
FindSession --> Found{"session found?"}
Found -->|"yes"| Enqueue3["_enqueue_continuation()
(resume_session)"]
Found -->|"no"| Skip6["Skip"]
Cont --> ContType{"work_type?"}
ContType -->|"new_session"| ExecNew["_execute_workflow()"]
ContType -->|"resume_session"| LoadS3["load session from S3
→ _execute_workflow()"]
```
**Phase 1 handlers** (`handle_pipeline`, `handle_merge_request`,
`handle_note`) perform validation, auth checks, and session resolution,
then post a `WorkOrder` continuation message via `_enqueue_continuation()`.
They do not call `_execute_workflow` directly.
**Phase 2** (`handle_continuation`) receives the `WorkOrder`, resolves
the workflow config, builds a `WorkflowRequest`, and calls
`_execute_workflow`. For `resume_session` orders, it loads the previous
session from S3. `_execute_workflow` handles SHA dedup and per-workflow
rate limiting via `scan_agent_threads`.
Slash commands dispatch to a single named workflow via
`_resolve_slash_workflow` (exact match, then prefix match). Help and error
replies are posted via `post_simple_reply` (no session marker, no rate
limit impact).
### 10.4. Pipeline trigger design
**Why `gitlab::pipeline` instead of `kubernetes::PipelineRun`:** A GitLab
pipeline event fires once when the pipeline completes. Since Konflux
external stages are attached to the pipeline, the event naturally waits for
all builds and tests to finish before triggering. This means the agent sees
the full picture in one event, without needing to deduplicate or wait for
stragglers.
**Trigger filters:**
- `status` in `{failed}` -- only failed pipelines trigger analysis
- `source == "merge_request_event"` -- only MR pipelines, not branch/tag
### 10.4a. Merge request trigger design
The `handle_merge_request` handler fires on MR open, reopen, and update
events (`action in {"open", "reopen", "update"}`). Draft MRs are skipped
(`object_attributes.draft == True`); marking a draft as ready triggers a
review since the event arrives with `draft: false`.
Event classification is deliberately simple: the handler does not inspect
`oldrev` or `changes.draft` fields. Instead, SHA-based deduplication in
`_execute_workflow` (via JSON session markers) ensures each code revision
is reviewed at most once. Metadata-only updates (title, label changes) on
an already-reviewed SHA are silently skipped.
### 10.5. Note trigger design
Two sub-flows:
**Slash command** (`/hummingbird `): Triggers a specific
named workflow, bypassing rate limiting. `_resolve_slash_workflow` performs
exact-match lookup first, then falls back to unique prefix matching. If
the subcommand is missing or is "help", `_format_help` returns a list of
available workflows with descriptions. Ambiguous or unknown subcommands
produce an error message via `post_simple_reply`. The note author must
have Developer+ access (level >= 30) on the project, checked via
`check_member_access()`. If denied, the agent replies in the same
discussion thread with a short access-denied message.
**Reply to agent note:** When a user replies to an existing agent note
(which contains a JSON session marker):
1. `find_session_for_reply()` walks the discussion thread, filtering by
bot author ID (`get_bot_user_id()`), and returns the first matching
session marker. All bot markers in a thread share the same session ID.
2. `handle_note` posts a `resume_session` continuation to the FIFO
(grouped by `session_id`).
3. `handle_continuation` loads the session from S3 (conversation history and
sandbox archive), builds a `WorkflowRequest`, and calls
`_execute_workflow()` to resume with the user's reply.
4. If the session is not found (expired/deleted), the agent replies with
a message explaining the session has expired and suggesting to start
a new run.
Reply authors are also subject to the Developer+ access check. If denied,
the agent replies in the discussion thread with the same access-denied
message.
**Non-agent-directed notes:** Regular comments that are neither slash
commands nor replies to agent threads are silently ignored (debug-level
log). No access check is performed for these.
**Self-note filtering:** Notes where `author_id` matches the bot's own
user ID (resolved via `get_bot_user_id()`) are skipped immediately. This
prevents infinite loops where the agent's own output triggers another
agent run. The author-ID check replaces the previous substring check
(`marker_prefix_str in note_body`), which was vulnerable to
denial-of-service: a user including the marker prefix in their reply
would cause the bot to silently ignore it.
**Threading logic for replies:** If the project requires internal notes
(`internal_notes: true`) but the original discussion was public, the reply
is posted as a new top-level internal note instead of replying in the
public thread. This prevents leaking internal analysis into public threads.
### 10.6. Note lifecycle
```text
1. create_placeholder_note() -> "Running the workflow..."
+ JSON session marker (id, wf, sha)
2. run_workflow() -> agent loop
3. update_note() -> replace placeholder with result
+ reply prompt + JSON session marker
Reply notes (from session resumption) omit the reply prompt since the
user is already engaged in the conversation.
```
On failure, the placeholder is updated to "Hummingbird analysis failed."
The JSON session marker is always present so the note can be identified as
agent-generated (for rate limiting, SHA dedup, and self-note filtering).
The marker embeds the workflow name and commit SHA, enabling per-workflow
rate limiting and SHA-based deduplication.
### 10.7. Rate limiting and SHA deduplication
`scan_agent_threads()` performs a single pass over MR discussions, parsing
JSON session markers (``)
to determine:
1. **Per-workflow thread count** -- how many threads the given workflow has
created on this MR. If the count meets or exceeds `max_runs_per_mr`,
the workflow is skipped.
2. **SHA dedup** -- whether the current commit SHA has already been reviewed
by this workflow. If so, the workflow is skipped (prevents re-reviewing
the same code on metadata-only MR updates).
Slash commands and replies bypass both checks entirely.
The rate limit is per-workflow per-MR: different workflows maintain
independent thread counts on the same MR. JSON markers are backward
compatible -- old plain-UUID markers (``)
are parsed as `{"id": "UUID"}` with no workflow or SHA information, so they
are not counted toward any specific workflow's limit.
## 11. Session System
Sessions enable conversation continuity: a user can reply to an agent note
and the agent picks up where it left off, with full context and sandbox
files restored.
### 11.1. What gets saved
Three artifacts are saved after each run:
- **`context.json`** -- the full `contents` list from the agent loop. This
is the complete conversation history in Gemini wire format (user turns,
model turns with tool calls, tool response turns). It is the minimum
state needed to resume the conversation.
- **`transcript.md`** -- a human-readable markdown rendering of the run for
debugging and auditing. Not used for resumption.
- **`sandbox.tar.gz`** -- an archive of `/tmp/data/` (the sandbox working
directory, which includes the `_out/` spill subdirectory). Contains
PipelineRun JSONs, test logs, jq output, and any other files the model
created during the run. Restored into the new sandbox on resumption so
the model can reference its previous work.
### 11.2. Storage backends
**S3 (production):**
```text
s3://{bucket}/sessions/{session_id}/context.json
s3://{bucket}/sessions/{session_id}/transcript.md
s3://{bucket}/sessions/{session_id}/sandbox.tar.gz
```
**Local directory (development, `--save-session`):**
```text
{directory}/context.json
{directory}/transcript.md
{directory}/sandbox.tar.gz
```
Both backends have the same interface. S3 save is best-effort: wrapped in
try/except so a transient S3 error does not prevent the MR note from being
posted. The note always gets delivered first.
### 11.3. Sandbox archive transport
The sandbox archive requires special handling because the orchestrator
cannot directly access the sandbox filesystem (no volume mounts):
```text
Archive (inside sandbox):
sb.exec("tar czf /tmp/_archive.tar.gz -C /tmp data")
Stream out (sandbox -> host temp file):
for chunk in sb.read_file_iter("/tmp/_archive.tar.gz"):
tmp.write(chunk) # 64 KB chunks, no base64
Restore (host temp file -> new sandbox):
sb.stream_exec("tar xzf - -C /tmp", file_chunks())
```
Both directions use streaming binary I/O via Popen stdin/stdout pipes.
No base64 encoding is needed: `read_file_iter()` yields raw bytes from
the sandbox via a Popen stdout pipe, and `stream_exec()` feeds raw bytes
into a Popen stdin pipe. The archive is never extracted on the
orchestrator -- it exists only as opaque bytes being transported between
sandboxes. This is a security invariant: the orchestrator never parses or
inspects the archive contents.
### 11.4. Resumption flow
```mermaid
flowchart TD
Reply["User replies to agent note"]
FindSession["find_session_for_reply()
walks discussion thread"]
LoadS3["load_s3(session_id)"]
NotFound{"session found?"}
Expired["Post session-expired reply"]
NewSandbox["Start new sandbox"]
Restore["restore_sandbox(archive)"]
BuildContents["contents = old_contents + user_reply"]
RunLoop["run_agent_loop(
initial_contents, CONTINUATION_PROMPT)"]
Reply --> FindSession --> LoadS3
LoadS3 --> NotFound
NotFound -->|"no"| Expired
NotFound -->|"yes"| NewSandbox
NewSandbox --> Restore --> BuildContents --> RunLoop
```
Key aspects:
- **Fresh iteration budget.** The resumed session starts iteration 0 with
the full `max_iterations` budget, regardless of how many iterations the
previous session used.
- **Context window is the real limit.** A typical 8-iteration cold start
uses ~50-100K input tokens. The restored history is sent in full on every
API call. The `CONTEXT_LIMIT` check applies and will force wrap-up if
needed.
- **Session ID reuse.** The resumed session keeps the original session
ID. S3 state is overwritten in place with the updated conversation
history. Since GitLab discussions are linear (not branching), there is
no need for a tree of sessions -- the conversation is a single
sequential thread.
- **Graceful degradation.** If the S3 session is not found (expired,
deleted), the agent posts a reply explaining the session has expired and
suggesting to start a new run. It does not fall back to a cold start.
### 11.5. Session markers
Every agent note contains a hidden HTML comment with a JSON payload:
```html
```
The JSON payload contains:
- `id` -- session UUID (always present)
- `wf` -- workflow name (present for new-format markers)
- `sha` -- commit SHA at the time of review (present for auto-triggered runs)
This marker serves four purposes:
1. **Resumption:** `find_session_for_reply()` searches discussion threads
for this marker to find the session ID. Only bot-authored notes are
considered; the first matching marker wins.
2. **Per-workflow rate limiting:** `scan_agent_threads()` counts discussion
threads for a specific workflow using the `wf` field. Only bot-authored
notes are scanned.
3. **SHA deduplication:** `scan_agent_threads()` checks whether the current
SHA has already been reviewed by the workflow using the `sha` field.
4. **Self-filtering:** `handle_note()` compares the webhook event's
`author_id` against the bot's own user ID (via `get_bot_user_id()`)
to prevent infinite loops.
All marker-scanning functions resolve the bot user ID via `gl.auth()`
(`GET /user`) on the orchestrator token. This ensures markers in
non-bot notes (user replies, external contributors) are never parsed,
preventing session hijacking via injected markers.
Old plain-UUID markers (``) are parsed
as `{"id": "UUID"}` for backward compatibility. They are counted for
resumption but not for per-workflow rate limiting or SHA dedup (no
`wf`/`sha` information).
## 12. Workflow System
### 12.1. Separation of prompt and metadata
A workflow has two parts that live in different places:
- **Prompt** (`.md` file): the LLM system prompt, verbatim. This is the
investigation strategy, tool usage guidance, data patterns, and output
format. Pure text, no code.
- **Metadata** (YAML config): operational settings -- `action`,
`model`, `max_iterations`, `data_sources`, `projects`. This controls
what the orchestrator does with the workflow, not what the model does.
This separation means changing the analysis strategy (e.g., adding a new
investigation step) requires editing a markdown file. Changing operational
parameters (e.g., which projects use this workflow, which model to use)
requires editing the YAML config. Neither requires a code change.
### 12.2. System prompt layering
The system prompt sent to the model is built from three layers:
```text
BASE_SYSTEM_PROMPT # agent.py: sandbox rules, tool usage tips
+ tool_notes # from ToolRegistry: per-data-source domain knowledge
+ workflow_body # full content of e.g. workflows/analyze-failures.md
```
On session resumption, a fourth layer is appended:
```text
+ CONTINUATION_PROMPT # agent.py: "do not re-run the full workflow"
```
The system prompt is rebuilt every iteration (to allow warning suffixes to
be appended), but the base content is stable. Warning suffixes are appended
at the end so they override earlier instructions.
### 12.3. Design choice: strategy, not procedure
The workflow `.md` file describes strategy and guidance, not a rigid script.
The model decides when and how to use each tool based on what it sees.
This matters because merge request failures are diverse. A fixed procedure
would either miss edge cases (e.g., build failures before tests ran) or
waste iterations on steps that don't apply. By giving the model a strategy
("identify failing tests, fetch details, analyze root causes, group by
similarity"), it can adapt to whatever it encounters.
The workflow does structure the investigation into phases (Data Collection,
Analysis, Output) for clarity, but these are guidelines, not enforced
checkpoints.
### 12.4. Workflow file anatomy (analyze-failures.md)
The primary workflow is structured as:
1. **Scope and approach** -- what this workflow analyzes and what it ignores
2. **Input** -- event JSON format (`project`, `iid`)
3. **Data Collection Phase:**
- Data.1: Fetch MR details (direct call, small response)
- Data.2: Fetch commit statuses, identify failures
- Data.3: Batch fetch PipelineRuns + TaskRuns (fetch_batch_to_sandbox)
- Data.4: Process with jq, extract Testing Farm data in bulk
4. **Analysis Phase:**
- Analysis.1a: Investigate each failed PipelineRun individually
- Analysis.1b: Summarize and group by root cause
5. **Output** -- markdown template with root causes, collapsible details,
clickable links (PipelineRun, Testing Farm, test logs)
6. **Error Handling** -- partial report philosophy
Design considerations embedded in the workflow:
- **Efficient bulk fetching:** PipelineRuns and TaskRuns are fetched in one
`fetch_batch_to_sandbox` call, not individually.
- **Testing Farm data extracted in Data.4, analyzed in Analysis.1:** All TF
results.xml are fetched in bulk before analysis starts, enabling
cross-failure pattern detection.
- **Log fetching is selective:** Only 1-2 representative logs per failure
pattern, not all logs. This keeps iteration count bounded.
- **Reviewer-facing URLs:** The output template instructs the model to
include clickable links using `konflux_ui` and `artifacts_base` from
response metadata.
### 12.5. Prompt file resolution
`get_prompt(workflow_cfg, config_dir)` resolves the `prompt:` field relative
to the config file's directory:
```python
prompt_path = config_dir / workflow_cfg.prompt
# e.g. /app/config.yaml with prompt: workflows/analyze-failures.md
# -> /app/workflows/analyze-failures.md
```
In the container image, workflows are baked in at `/app/workflows/`. A
ConfigMap can override them at deploy time by mounting at the same path.
## 13. Deployment and Container
### 13.1. Container build strategy
The Containerfile uses an all-RPM builder+installroot+scratch pattern (no
pip, no venv):
1. **Builder stage:** Uses a Fedora-based builder image with dnf. Installs
all dependencies as RPMs into a clean `--installroot`.
2. **Application code:** Copies `hummingbird_agent/` and `workflows/` into
the installroot at `/app/`.
3. **Final stage:** `FROM scratch`, copies the entire installroot. No
package manager, no shell beyond what RPMs provide.
RPM dependencies: `python3-boto3`, `python3-google-auth+requests`,
`python3-gitlab`, `python3-kubernetes`, `python3-pyyaml`, `python3-requests`,
`python3-sentry-sdk`, `kubernetes1.35-client` (for `kubectl`).
This approach was chosen over pip because:
- All deps come from Fedora's package repository -- no PyPI supply chain risk
- Smaller image (~22 MB saved vs. google-genai SDK alone)
- Reproducible builds from known RPM versions
- No compilation step (no gcc/python3-devel in the image)
### 13.2. Container runtime properties
```dockerfile
CMD ["python3", "-m", "hummingbird_agent", "serve"]
WORKDIR /app
USER 65532
```
The default CMD runs `serve` mode for production. Local development uses
`run` mode via explicit command override. `USER 65532` matches the standard
nonroot UID used by distroless images and the Podman sandbox.
### 13.3. K8s deployment manifests
Located in `hummingbird-agent/kubernetes/`:
**`deployment.yaml`:**
- Single replica with rolling update (`maxSurge: 1`, `maxUnavailable: 0`)
- `terminationGracePeriodSeconds: 900` (15 minutes) to allow in-flight
agent runs to complete on shutdown
- ServiceAccount: `hummingbird-agent`
- Secrets mounted from K8s Secret `hummingbird-agent`
- Commented-out mounts for workflow ConfigMap and custom CA trust
**`rbac.yaml`** (applied in the *sandbox* namespace):
- ServiceAccount `hummingbird-agent`
- Role with `pods: create, get, list, delete, patch` and `pods/exec: create`
- RoleBinding linking the SA to the Role
- `patch` is required for pool mode (relabeling pods during claim)
**`sandbox-pool.yaml`** (applied in the *sandbox* namespace):
- Deployment with `replicas: 3` (tuned to balance latency vs cost)
- Pods labelled `hummingbird/role: standby` for pool discovery
- Same security context and resource limits as direct-creation pods
- `revisionHistoryLimit: 2`, rolling update with `maxSurge: 1`,
`maxUnavailable: 0`
**`networkpolicy.yaml`:**
- `podSelector: {}` -- applies to all pods in the namespace
- `egress: []` -- deny all outbound traffic
**`secret.yaml`:**
- Template with all required env vars (API keys, tokens, URLs)
- Values must be populated per deployment
### 13.4. Workflow mounting
Workflows are baked into the image at `/app/workflows/`. To update workflows
without rebuilding the image:
1. Create a ConfigMap from the workflows directory
2. Uncomment the volume mount in the deployment YAML
3. The ConfigMap mount replaces the baked-in directory (all-or-nothing)
This is useful for rapid iteration in staging without waiting for a new
image build.
### 13.5. Custom CA trust
For clusters with internal CA certificates (common in enterprise
environments), the deployment supports OpenShift's CA injection:
1. Create a ConfigMap with the `config.openshift.io/inject-trusted-cabundle`
label
2. Mount it at `/etc/pki/custom`
3. Set `REQUESTS_CA_BUNDLE=/etc/pki/custom/ca-bundle.crt`
OpenShift automatically injects the cluster's CA bundle into the ConfigMap.
## 14. Design Decision Registry
Each entry records a decision, the alternatives considered, why the chosen
approach won, and what would break if the decision were reversed. This is
the most important section for avoiding regressions.
### kubectl exec over kubernetes Python exec API
- **Chosen:** `kubectl exec` as subprocess for sandbox command execution.
- **Alternative:** kubernetes Python client `stream()` API.
- **Why:** Three showstopper bugs in the Python client: (1) WebSocket
v1-v4 has no stdin EOF signal, so `cat > /file` hangs forever; (2)
`BrokenPipeError` on stdin larger than ~1MB; (3) `stream()` accumulates
all stdout in memory with no control. `kubectl` avoids all three and
provides the same subprocess interface as Podman.
- **If reversed:** `write_file()` would hang or fail on large data. Pod
log retrieval with large outputs would OOM. Sandbox reliability would
drop significantly.
### YAML config over env vars for operational settings
- **Chosen:** Single YAML file for workflows, projects, limits, data
source declarations.
- **Alternative:** Everything in env vars (original design).
- **Why:** Env vars cannot express structured data (workflow-project
mappings, per-project token overrides, data source config with multiple
fields). YAML provides structure, validation, and audit trails.
- **If reversed:** Token scoping would be lost (no per-workflow-project
token overrides). Project allowlists would be impossible. The config
would be unauditable.
### Token env var names in YAML, not values
- **Chosen:** YAML contains `token_env: GITLAB_TOKEN_RO` (the env var
name), not the actual token value.
- **Alternative:** Inline secrets in YAML, or env vars for everything.
- **Why:** The YAML file can be committed, reviewed, and audited. Actual
secrets stay in env vars (injected via K8s Secrets). Inline secrets
would make the config file a secret itself.
- **If reversed:** The config file would become a secret, breaking audit
trails and code review workflows.
### Orchestrator token prefix convention (ORCHESTRATOR_*)
- **Chosen:** Orchestrator tokens use `ORCHESTRATOR_GITLAB_TOKEN_*` env
var names by convention.
- **Alternative:** Same env var namespace as model tokens, distinguished
by context.
- **Why:** Structural separation makes it impossible to accidentally pass
an orchestrator token to a model tool (or vice versa). The
`ORCHESTRATOR_` prefix is never used in YAML `token_env` fields.
- **If reversed:** A misconfiguration could leak write-capable tokens to
the model, which could then expose them via tool calls.
### Auto-spill over conversation compaction
- **Chosen:** Large outputs are saved to sandbox files with previews
returned to the model.
- **Alternative:** Replace old tool results in `contents` with compact
summaries (conversation compaction).
- **Why:** Compaction requires modifying the `contents` list, which risks
violating Gemini API constraints (model turns must match preceding tool
turns). Auto-spill achieves ~90% of the token reduction with zero risk
of breaking the conversation structure.
- **If reversed:** Token usage would increase ~15-30%. Per-call context
would grow unbounded. Sessions would hit the context limit much sooner.
Conversation compaction could be added *on top of* auto-spill in the
future, but is not needed with current workloads.
### Iteration count over cumulative token budget
- **Chosen:** `max_iterations` as the primary cost control lever.
- **Alternative:** Cumulative token budget (stop when total billed tokens
exceed a threshold).
- **Why:** With full history replay, each API call re-sends everything.
Cumulative billed tokens double-count: call 1 = 5K, call 2 = 10K
(including 5K again), total = 15K billed but only 10K new content. With
auto-spill keeping per-call size bounded, iteration count is a much
simpler and more predictable proxy for actual cost.
- **If reversed:** The budget model would be confusing and inaccurate.
Cost estimates would be wrong. The cumulative metric is still logged
for observability, but it does not drive termination.
### ToolDef notes in system prompt, not tool schema
- **Chosen:** Domain knowledge (Konflux dual-fetch, TF XML structure) goes
in `ToolDef.notes`, injected into the system prompt.
- **Alternative:** Put everything in the tool schema `description`.
- **Why:** Tool schemas have character limits and are sent in the `tools`
field of every API call. Long descriptions waste tokens on tool defs.
System prompt notes are sent once and can be arbitrarily detailed.
- **If reversed:** Tool descriptions would be bloated. Domain knowledge
would need to be duplicated in every workflow `.md` file.
### Full history replay (no compaction)
- **Chosen:** The `contents` list grows monotonically. Items are never
removed or modified (except empty response retries).
- **Alternative:** Compact old turns to reduce context size.
- **Why:** The Gemini API requires strict turn-by-turn structure. Modifying
or removing items risks invalid conversation structure. Auto-spill handles
the growth problem at the source (preventing large items from entering
history).
- **If reversed:** Risk of Gemini API errors from malformed conversation
structure. Risk of confusing the model (it references previous results
that have been summarized away).
### fetch_to_sandbox kept alongside auto-spill
- **Chosen:** Both `fetch_to_sandbox` and auto-spill coexist.
- **Alternative:** Remove `fetch_to_sandbox` since auto-spill handles
large outputs automatically.
- **Why:** `fetch_to_sandbox` provides explicit path control (model can
choose meaningful filenames), batch fetching (one tool call for multiple
fetches), and no-preview responses (useful when the model will process
with jq anyway).
- **If reversed:** The model would lose path control and batch fetching
would require multiple auto-spilled calls. The workflow would need more
iterations to achieve the same result.
### K8s library for lifecycle + kubectl for exec (hybrid)
- **Chosen:** Use the `kubernetes` Python library for pod
create/read/delete, `kubectl` subprocess for exec.
- **Alternative:** All-kubectl (subprocess for everything) or all-library.
- **Why:** The library provides typed pod status polling and clean error
handling for lifecycle. `kubectl` provides reliable exec (see "kubectl
exec over kubernetes Python exec API" above). Using the library for exec
would require solving the three WebSocket bugs.
- **If reversed:** Either lifecycle management would be fragile (parsing
kubectl JSON output for pod status) or exec would be unreliable.
### Single-file YAML config over multiple config files
- **Chosen:** Everything in one `config.yaml` with `settings` and
`workflows` sections.
- **Alternative:** Separate files per workflow, or settings.yaml +
workflows.yaml.
- **Why:** Single source of truth. All project-workflow mappings visible in
one place. Settings-level defaults flow down to all projects.
No file discovery logic needed.
- **If reversed:** Token resolution would need cross-file lookups. Project
index would need multi-file aggregation. Config validation would be more
complex.
### Workflow .md as pure prompt, metadata in YAML
- **Chosen:** The workflow `.md` file is pure system prompt text. Metadata
(`action`, `model`, `max_iterations`, `data_sources`, `projects`) lives
in the YAML config.
- **Alternative:** YAML frontmatter in the `.md` file (original design).
- **Why:** Separation of concerns. The `.md` file is the model's
instructions -- it should be readable and editable by anyone writing
prompts. The YAML config is the orchestrator's instructions -- it
controls routing, limits, and credentials. Mixing them in one file
conflates two audiences.
- **If reversed:** Prompt authors would need to understand YAML config
structure. Token/project config would be scattered across `.md` files
instead of centralized.
### Raw HTTP for Gemini instead of google-genai SDK
- **Chosen:** `requests` + `google-auth` for Gemini API calls.
- **Alternative:** `google-genai` Python SDK.
- **Why:** The SDK brings ~22 MB of transitive deps (pydantic, httpx,
websockets). The Gemini REST API is simple camelCase JSON over HTTPS --
one endpoint, one request format. Raw HTTP enables: smaller container,
simpler tests (mock `requests.post`), plain dict conversation history
(easy session serialization), no SDK breakage risk, and a `models/`
package structure that supports multiple backends.
- **If reversed:** Container would be ~22 MB larger. `contents` would use
SDK objects instead of plain dicts, complicating session serialization.
Adding Claude support would require a separate approach.
### Nudge via system prompt suffix, not user message
- **Chosen:** Empty response nudge and budget warnings are appended to the
system prompt as suffixes.
- **Alternative:** Inject synthetic user messages into `contents`.
- **Why:** User messages in `contents` must come from the actual user (the
initial event, or a reply). Injecting synthetic messages pollutes the
conversation history that is saved in sessions. System prompt suffixes
are transient -- they affect one API call without permanently modifying
the conversation state.
- **If reversed:** Saved sessions would contain synthetic user messages.
Resumed conversations would be confusing. The model might respond to the
synthetic messages instead of the user's actual input.
### Session ID reuse over new-ID-per-resume
- **Chosen:** Resumed sessions keep the original session ID. S3 state is
overwritten in place.
- **Alternative:** Each resume generates a new UUID, saving alongside the
original (append-only tree of sessions).
- **Why:** GitLab discussions are linear, not branching. There is no
scenario where two different sessions from the same thread are both
valid. A new UUID per resume creates orphaned S3 snapshots that are
never referenced again, since `find_session_for_reply()` always picks
the latest marker. Reusing the ID is simpler, uses less storage, and
matches the linear conversation model.
- **If reversed:** S3 would accumulate orphaned session snapshots. Each
resume would need a new note marker, but the old markers would still be
in the thread, creating confusion about which session is current.
### Reply to unauthorized agent-directed notes instead of silent skip
- **Chosen:** When an unauthorized user sends a slash command or replies to
an agent thread, the agent replies in the same discussion with a short
access-denied message. Non-agent-directed notes are silently ignored
(debug log, no access check).
- **Alternative:** Log a warning and skip silently for all unauthorized
notes (the original behavior).
- **Why:** Silent skip gives no feedback to someone who intentionally tried
to engage the agent, which is confusing. On the other hand, checking
access and logging a warning for every random comment on a public project
is noisy and pointless. Moving the access check after intent detection
cleanly separates the two cases. The reply uses the discussions API so it
automatically inherits confidentiality from the parent note/thread.
- **If reversed:** Users without sufficient access who try `/hummingbird`
would get no feedback. The agent log would be noisy with warnings for
every comment on public MRs.
### Deployment-backed pod pool over Python-thread pool
- **Chosen:** A Kubernetes Deployment maintains pre-warmed standby pods.
The agent claims a pod by relabeling it; the Deployment replaces it.
- **Alternative:** A Python-side thread pool that pre-creates pods and
queues them for use.
- **Why:** The Deployment provides self-healing (restart crashed pods),
native scaling (`kubectl scale`), rolling updates (image changes), and
monitoring via standard K8s tooling. A Python pool would need to
reimplement all of these.
- **If reversed:** Pod replenishment, crash recovery, and image updates
would all need custom code. Scaling would require agent redeployment.
### Pod isolation: never reuse sandbox pods
- **Chosen:** Each workflow run gets its own pod. After cleanup, the pod is
deleted. Claimed pods are detached from the ReplicaSet.
- **Alternative:** Return used pods to the pool and reset them.
- **Why:** Residual state from a previous run (files, environment, running
processes) could leak between MR investigations, creating security and
correctness risks. Deletion is simple and foolproof.
- **If reversed:** Would need a reliable pod-reset mechanism and auditing
that no state survives between runs.
### Absolute reap-by deadline over relative claimed-at age
- **Chosen:** Active pods are annotated with `hummingbird/reap-by` (an
absolute UTC timestamp). The reaper deletes pods where `now > reap-by`.
- **Alternative:** Annotate with `claimed-at` and compute age relative to
`max_active_seconds`; or use `activeDeadlineSeconds` on the pod spec.
- **Why:** An absolute deadline simplifies the reaper to a single
comparison. It also supports varying deadlines: claim sets
`reap-by = now + max_active_seconds`, while linger sets
`reap-by = now + linger_seconds`. With a relative timestamp, the reaper
would need to know which mode the pod is in.
`activeDeadlineSeconds` applies from pod creation, not from claim --
standby pods would expire before being used.
- **If reversed:** The reaper would need mode-aware age calculations.
Lingering pods would require a separate annotation or reaper path.
### Indefinite claim wait over timeout
- **Chosen:** `SandboxPool.claim()` polls indefinitely (governed by
shutdown event), logging at DEBUG then WARNING.
- **Alternative:** Timeout after N seconds and raise an error.
- **Why:** The pool is typically smaller than `max_concurrent_agents` for
cost reasons. Waiting for a replacement pod is normal operational
behavior, not an error. A timeout would cause spurious failures during
burst traffic. The SQS semaphore already bounds concurrency.
- **If reversed:** Burst traffic would cause avoidable failures instead of
brief delays.
### Pool config in Deployment only, not in agent configmap
- **Chosen:** In pool mode (`k8spool`), the pod image, resources, labels,
and security context are defined solely in the Deployment template. The
agent config only needs `namespace` and `active_deadline_seconds`.
- **Alternative:** Keep image/resources/metadata in the agent configmap too
(as in `k8s` mode).
- **Why:** Eliminates config duplication. The Deployment template is the
single source of truth. Changes to pod resources or image only require
updating one place and re-rolling the Deployment.
- **If reversed:** Config drift between the Deployment and the agent
configmap would be a constant risk.
### Pod lingering over immediate cleanup for reply latency
- **Chosen:** After a successful workflow, pool sandbox pods linger for
`linger_seconds` (default 300) with a `session-id` annotation. Replies
within the window reclaim the pod via `try_reclaim()`, skipping pod
creation and S3 archive restoration.
- **Alternative:** Always delete the pod immediately and restore from S3 on
every reply.
- **Why:** Reply latency drops from seconds (pod claim + S3 restore) to
near-zero. The S3 archive is still saved as a fallback if the pod is
gone. The `reap-by` annotation ensures lingering pods are cleaned up if
no reply arrives. The `session-id` annotation doubles as a concurrency
guard: it is only present while lingering, preventing active pods from
being reclaimed.
- **If reversed:** Every reply would pay full pod + restore latency, even
for immediate follow-ups. User experience for conversational interactions
would degrade noticeably.
### linger() on Sandbox protocol over isinstance checks in runner
- **Chosen:** All sandbox backends implement `linger(session_id)`. Non-pool
backends fall back to `cleanup()`. The runner calls `sb.linger()` without
type checks.
- **Alternative:** `isinstance(sb, K8sPoolSandbox)` in `run_workflow()`.
- **Why:** Keeps the runner backend-agnostic. Adding a new backend requires
only implementing the protocol, not touching the runner. The fallback
behavior is co-located with each backend.
- **If reversed:** The runner would need to know about every backend type
and their linger capabilities.
### GCP SA key over Workload Identity Federation
- **Chosen:** GCP service account key stored in Vault, rotated via
cki-tools credential manager (prepare/switch/clean cycle).
- **Alternative:** Workload Identity Federation (WIF) with projected SA
token. Two sub-options: (a) automatic OIDC discovery if the cluster
issuer is public, (b) manual JWKS upload for internal issuers.
- **Why:** mpp-prod's OIDC issuer is `https://kubernetes.default.svc`
(internal, not publicly reachable), so GCP STS cannot discover it
automatically. Manual JWKS upload works but requires re-upload after
SRE-triggered signing key rotations. SA key integrates with the
existing credential manager rotation infrastructure (same pattern as
AWS keys and GitLab tokens), needs no OIDC reachability, and enables
automated validate/update via `google-auth` in CI. The application
code (`VertexAuth` / `google.auth.default()`) works identically with
both approaches -- switching to WIF later requires only infrastructure
changes.
- **If reversed:** Replace SA key with WIF projected token + credential
config ConfigMap. The `gcp_service_account_key` token type in
cki-tools would no longer be needed for this use case.
### Sliding-window breakpoints over per-component breakpoints (Claude)
- **Chosen:** Two breakpoints on messages (B1 at the previous write position,
B2 at the latest message) that slide forward each turn.
- **Alternative:** Separate breakpoints on system prompt, tools, and messages
(using 3-4 of the 4 available slots).
- **Why:** The Anthropic prefix hash is cumulative -- it covers everything from
the start of the request (tools, system, messages) up to the breakpoint. A
single breakpoint on a message already caches the entire prefix. Separate
breakpoints on earlier components would be redundant and waste slots. Two
sliding breakpoints cover the full conversation history with cache reads on
every turn after the first.
- **If reversed:** Three breakpoint slots wasted on content already covered by
the message breakpoint. Only one slot left for the sliding window, making it
impossible to have both a read (B1) and write (B2) breakpoint on messages.
### Developer+ trust filtering over unfiltered or sanitized discussion comments
- **Chosen:** `gitlab_get_mr_discussions` checks each note author's project
access level (Developer+ / >= 30). Untrusted notes are replaced with a fixed
placeholder in mixed-trust discussions, or the entire discussion is dropped
if all notes are untrusted. Agent notes are always trusted (detected by
session marker) but have transcripts and metadata stripped.
- **Alternatives considered:**
- (a) Include all comments unfiltered. Simplest, but any external
contributor can craft comments that manipulate the model's review output
(prompt injection).
- (b) Sanitize/escape untrusted content (strip markdown, quote as code
blocks, prefix with "[external]"). There is no reliable escaping mechanism
for LLM prompts -- the model interprets natural language regardless of
formatting. Escaping gives a false sense of security.
- (c) Only include agent-authored notes (skip all human comments). Safe, but
defeats the purpose: the model would never see developer responses to its
own findings.
- (d) Include only discussions where the agent participated. Better, but
still misses developer-initiated review threads that provide relevant
context.
- **Why:** Developer+ is the same threshold used for pipeline trigger
authorization (invariant #8) and slash command access. It matches the trust
boundary already established: people who can push code and approve MRs are
trusted to provide review context. The placeholder approach preserves
discussion structure (the model sees that *someone* replied) without
exposing the content. Full-drop for all-untrusted discussions avoids
noise from discussions that contain zero useful context.
- **If reversed:** (a) opens a prompt injection vector on any project that
accepts external MRs. (b) provides no actual protection. (c) makes
follow-up reviews unable to see developer explanations, causing repeated
false positives. (d) misses developer-initiated context.
### Bot-author filtering for session marker parsing over unfiltered note scanning
- **Chosen:** `find_session_for_reply()`, `scan_agent_threads()`, and the
`handle_note()` self-filter all resolve the bot user ID via
`get_bot_user_id()` (which calls `gl.auth()` on the orchestrator token)
and only consider notes where `author.id` matches. In
`find_session_for_reply()`, the first matching marker wins.
- **Alternative:** Parse markers from any note and keep the last match;
substring self-filter in `handle_note` (previous behavior).
- **Why:** The previous unfiltered approach allowed session hijacking: a
user reply or agent review prose containing an example marker
(``) was parsed as a real session,
causing "session expired" errors in production. For `handle_note`, the
substring check caused the reverse problem: a user embedding the marker
prefix in a reply silently suppressed the event (denial of service). The
`gl.auth()` call is one `GET /user` per invocation -- negligible cost.
`_get_client_with_bot_id()` returns both the authenticated client and
the bot user ID, avoiding redundant client construction.
- **If reversed:** Any MR participant can inject a marker to hijack the
session ID or inflate rate-limit counts. The agent's own review prose
containing example markers causes spurious "session expired" messages.
### ijson streaming over buffered JSON for K8s list responses
- **Chosen:** `session.get(stream=True)` + `ijson.parse(resp.raw)` for
`iter_paginated()`. Items are yielded one at a time via `ObjectBuilder`;
the `metadata.continue` pagination token is captured in the same parse
pass. `resp.raw.decode_content = True` is required because Kubearchive
returns gzip Content-Encoding; without it `ijson` sees compressed bytes.
- **Alternative:** `session.get().json()` loads the full page into memory
(the original implementation).
- **Why:** A single K8s list page can contain 500 PipelineRuns (80+ MB
JSON). Parsing this with `.json()` creates a +247 MB RSS spike (raw
bytes + decoded string + parsed dict coexist). With 4 concurrent
workflows, this exceeds any reasonable pod memory limit. `ijson`
stream-parsing yields items one at a time, reducing the peak to +12 MB
for the same data -- a 95% reduction. The continue token appears in
the `metadata` object (before or after `items` depending on the server);
the event-driven parser captures it regardless of order.
- **If reversed:** Large MRs (100+ components) would OOM the agent pod.
Concurrent workflows would multiply the problem. The pod memory limit
would need to scale with the largest possible page size, which is
unbounded.
### Action token tier for per-workflow identity
- **Chosen:** A dedicated Tier 2 of workflow action tokens, separate
from both orchestrator tokens (Tier 1) and model tool tokens (Tier 3). Each workflow gets
a dedicated GitLab bot user per project for all write operations (notes,
thread resolution).
- **Alternative:** Extend orchestrator tokens with a per-workflow fallback
chain (`ORCHESTRATOR__GITLAB_TOKEN_` →
`ORCHESTRATOR_GITLAB_TOKEN_` → `ORCHESTRATOR_GITLAB_TOKEN`).
- **Why:** Orchestrator tokens serve a fundamentally different purpose
(operational reads + infrastructure notes) than workflow output (analysis
results, review comments). Mixing them in one fallback chain risks a
single bot identity posting notes on behalf of multiple workflows,
eliminating the audit trail. A separate tier with explicit `token: str`
parameters makes the code path unambiguous: callers must choose which
tier they use. The naming convention
(`HUMMINGBIRD_AGENT_ACTION__GITLAB_TOKEN_`) groups
tokens alphabetically by workflow, making env var auditing
straightforward.
- **If reversed:** All workflow notes would appear from the same bot user,
making it impossible to distinguish which agent (code-review vs
analyze-failures vs renovate-babysit) produced a given note in the MR
timeline. GitLab's audit log would attribute all actions to one identity.
### Per-project-only action token config (no workflow-level default)
- **Chosen:** `action_tokens` is defined at the project level only in YAML
config. No workflow-level `action_tokens` default is supported.
- **Alternative:** Allow workflow-level `action_tokens` that projects
inherit by default, with per-project overrides.
- **Why:** GitLab project access tokens are project-scoped. A
workflow-level default would be misleading -- a single token cannot
write to multiple projects. Forcing per-project declaration makes the
mapping explicit and prevents configuration errors where a cross-project
default silently fails.
- **If reversed:** Operators could set a workflow-level token that only
works for one project, causing silent 403 errors for other projects in
the same workflow.
### Migration: workflow action tokens (first deployment)
On the first deployment with workflow action tokens, `scan_agent_threads`
and `resolve_agent_threads` authenticate as the workflow bot and attribute
notes exclusively by that bot's user ID. Threads previously authored by
the orchestrator bot will not be visible to these functions. Expect:
- One extra workflow run per open MR on the first trigger after deployment
(`sha_reviewed` returns `False` for already-reviewed SHAs).
- Pre-migration orchestrator-authored threads will not be auto-resolved on
push or pipeline success.
Pre-migration threads can be manually closed, or they will age out
naturally as new workflow-bot-authored threads are created for subsequent
MR activity.
### SSH over kubectl exec for KubeVirt VMs
- **Chosen:** SSH (`ssh -T sandbox@ip command`) for VM command execution.
- **Alternative:** `virtctl console` (serial console), KubeVirt websocket
API, or `qemu-guest-agent exec`.
- **Why:** VMs have no `kubectl exec` equivalent. `virtctl console` provides
a serial terminal, not a programmable exec channel. The websocket API
requires a VNC or serial connection. `qemu-guest-agent` is not installed
in the VM image. SSH provides clean stdin/stdout/stderr separation,
timeout control, and is already well-tested in subprocess pipelines.
- **If reversed:** Would need `qemu-guest-agent` in the image and the
`exec` endpoint, which has poorer error reporting and no streaming stdin.
### Secret volume for SSH key injection
- **Chosen:** Kubernetes Secret mounted as a virtio disk (serial
`ssh-pubkeys`) read by the VM's `inject-ssh-keys.service` at boot.
- **Alternative:** `cloudInitNoCloud` or `accessCredentials` with
`qemu-guest-agent`.
- **Why:** The VM image already has `inject-ssh-keys.service` which reads
from the virtio disk. `cloudInitNoCloud` requires cloud-init in the
image. `accessCredentials` requires `qemu-guest-agent`. The Secret
volume approach requires no additional software in the VM image.
- **If reversed:** Would need cloud-init or guest agent in the bootc
image, adding complexity and attack surface.
### VMIRS for VM pool replenishment
- **Chosen:** `VirtualMachineInstanceReplicaSet` (VMIRS) as the
replenishment controller for the VM pool.
- **Alternative:** Custom controller, or VirtualMachine with RunStrategy.
- **Why:** VMIRS is KubeVirt's native equivalent of a Deployment's
ReplicaSet -- it maintains a desired count of identical VMIs. When a
VMI is claimed (ownerReferences cleared), the VMIRS creates a
replacement automatically. Same pattern as the pod pool with Deployment.
- **If reversed:** Would need a custom controller to maintain the VMI pool,
adding operational complexity.
### Per-workflow sandbox backend
- **Chosen:** `sandbox:` field in workflow config, resolved per-workflow
with fallback to CLI `--sandbox` flag.
- **Alternative:** Global-only sandbox selection via CLI.
- **Why:** Different workflows have different isolation requirements.
Code review needs only a lightweight container, while failure analysis
may need a full VM with root access. Per-workflow selection avoids
running all workflows in VMs (wasteful) or all in containers
(insufficient isolation).
- **If reversed:** All workflows would share the same sandbox backend,
requiring either over-provisioning (VMs for everything) or accepting
weaker isolation for some workflows.