# Hummingbird Agent Design — Engine

LLMS index: [llms.txt](https://hummingbird-project.io/llms.txt) | Full content: [llms-full.txt](https://hummingbird-project.io/llms-full.txt)

---

Agent loop, tool system, and sandbox backends for the Hummingbird Agent.
This page covers the execution engine: how the agent iterates, dispatches tools,
and isolates execution. For foundations (design philosophy, system architecture,
modules, security, config), see [Hummingbird Agent Design](https://hummingbird-project.io/docs/background/tools/hummingbird-agent-design/).

## 6. Agent Loop Design

For the wire-format walkthrough (what bytes go to the model API, what comes back),
see [Agent Model Loop](https://hummingbird-project.io/docs/background/tools/hummingbird-agent-model-loop/). This section
covers the _design rationale_ behind the loop.

### 6.1. Full history replay

The Gemini and Anthropic APIs are stateless. Every call sends the complete `contents` list
from the beginning of the conversation. This means every large tool output
sitting in history inflates every subsequent API call.

This property is fundamental to why auto-spill exists (section 7). Without
auto-spill, a single `cat` of a 32KB file early in the conversation adds
~8K tokens to every remaining API call. Over a 15-iteration run, that is
~120K wasted tokens.

The alternative -- conversation compaction (replacing old tool results with
summaries) -- was considered and deferred. Each provider has strict
requirements about content structure, and modifying history risks confusing
the model or violating API constraints. Provider-specific format requirements
are documented in [Hummingbird AI](https://hummingbird-project.io/docs/background/tools/hummingbird-ai/).
Auto-spill solves 90% of the problem with none of the risk.

### 6.2. Budget model: iterations + context limit

The agent uses a dual-limit approach rather than a cumulative token budget:

**Iteration limit** (`max_iterations`, default 30 per workflow config).
Hard cap on the number of model round-trips. This is the primary cost
control lever. With auto-spill keeping per-call context bounded, iteration
count is roughly proportional to cost.

**Per-call context limit** (`CONTEXT_LIMIT`, default 60,000 tokens).
Checked after each API call using `response.usage.total_input_tokens`. This is
a safety net for cases where auto-spill is not sufficient (e.g., many
small tool results that individually stay under the spill threshold but
cumulatively fill the context).

Why not a cumulative token budget? Because with full history replay, each
API call re-sends everything. "Cumulative billed tokens" double-counts:
call 1 sends 5K, call 2 sends 10K (including the 5K again), so billed
total is 15K but actual new content is only 10K. Iteration count is a
simpler and more predictable proxy for cost.

### 6.3. Two-tier budget escalation

Both limits use the same escalation pattern:

```mermaid
flowchart LR
    Normal["Normal<br/>tools available"]
    Soft["Soft Warning (80%)<br/>ITERATION_WARNING or<br/>CONTEXT_WARNING<br/>tools still available"]
    Hard["Hard Stop (100%)<br/>FINAL_TURN_WARNING<br/>toolConfig: NONE"]

    Normal -->|"80% reached"| Soft
    Soft -->|"100% reached"| Hard
```

**Soft warning (80%).** An ephemeral user message (`ITERATION_WARNING`) is
injected into `contents` for that turn only, then popped before the
response is persisted. Tools remain available so the model can finish
in-progress work.

**Hard stop (100%).** `FINAL_TURN_WARNING` is injected as an ephemeral
user message AND `tool_defs` is set to `[]` (empty list) to physically
prevent further tool calls. The model must produce text. This is more
reliable than relying on provider-side tool disabling, which models may
ignore.

### 6.3.1 Ephemeral messages and provider normalization

All per-turn warnings use ephemeral user messages: they are appended to
`contents` before the API call and popped immediately after. This keeps
the system prompt stable across all iterations and prevents warnings from
polluting the conversation history saved in sessions.

Provider adapters normalize provider-specific conversation requirements while
preserving the agent's saved history; details are in
[Hummingbird AI](https://hummingbird-project.io/docs/background/tools/hummingbird-ai/).

### 6.4. Empty response handling

Models occasionally return unusable responses (no text and no valid tool calls). The
agent retries up to `MAX_EMPTY_RETRIES` (2) times:

1. **Pop the empty response** from `contents`. It adds nothing and may
   confuse the model on the next call.
2. **Inject `EMPTY_RESPONSE_NUDGE`** as an ephemeral user message on the
   next turn: "Your previous response was empty. Please continue..."
3. **Continue the loop** (consuming an iteration).

The nudge is delivered as an ephemeral user message (injected before the
API call and popped after). This avoids mutating the system prompt and
keeps the conversation history clean for session persistence.

The `empty_retries` counter resets to 0 after any successful iteration
(one where tool calls were executed). This means the model gets fresh
retries if it produces empty responses at different points in the
conversation.

**`MALFORMED_FUNCTION_CALL` handling.** Models sometimes return a
`finishReason` of `MALFORMED_FUNCTION_CALL` (Gemini) with no usable tool calls.
This is treated as a special case of empty response: the retry mechanism
kicks in, but the nudge is replaced with `MALFORMED_CALL_NUDGE` which
tells the model to retry with simpler arguments and avoid large text
payloads in tool call arguments.

### 6.5. Error recovery

**Model API and transport errors.** The `hummingbird-ai` library classifies
provider and transport failures, applies the workflow's retry budget, and
records AI usage and request metrics. See [Hummingbird AI](https://hummingbird-project.io/docs/background/tools/hummingbird-ai/)
for provider-specific behavior. The agent loop retains partial results when a
terminal error reaches it.

**Unexpected loop errors.** `run_agent_loop` wraps the
`_generate_with_retry` call in a `try/except Exception` that logs and
breaks instead of propagating. This ensures the function always returns
partial results (accumulated `contents`, `transcript`, sandbox state)
even when an unexpected exception occurs (e.g. `JSONDecodeError` from a
truncated API response). The caller saves the session normally --
conversation history and sandbox archive are preserved for resumption.
This follows the "partial over nothing" principle: 32 iterations of work
are more valuable than a crash.

**No failure-path session save.** `_execute_workflow` does not save a
session when `run_workflow` raises. With the agent loop catching
unexpected errors, the failure path only fires for infrastructure errors
(sandbox start, config) where there is no useful state. Not saving
avoids overwriting a previous good session when a reply attempt fails.

**Tool execution errors.** `_execute_tool_calls()` wraps each
`tool_registry.execute()` in a try/except. Unhandled exceptions are caught
and returned to the model as `{"error": "Tool X failed: ..."}`. This
prevents a single broken tool from crashing the entire session -- the model
sees the error and can adapt.

**Sandbox exec timeout.** If a command exceeds `EXEC_TIMEOUT` (120s), the
`TimeoutExpired` exception is caught in the tool registry and returned as a
structured error dict. The model can retry with a different command or
proceed without the result.

### 6.6. Session resumption

When resuming from a previous session:

1. `initial_contents` (the full `contents` list from the previous run) is
   prepended, followed by the user's reply as a new user turn.
2. `CONTINUATION_PROMPT` is appended to the system prompt.
3. The sandbox is restored from the archived tarball.
4. A fresh iteration budget starts from 0.

Saved sessions use a versioned JSON envelope (`format_version: 1`) whose
`contents` are in a canonical message format independent of the active model.
Unversioned legacy sessions are not supported. On resume, `run_workflow` asks
the active adapter to load canonical history; after a successful run, it
converts native contents back to canonical form before persistence. The
library documents the canonical format and provider conversions in
[Hummingbird AI](https://hummingbird-project.io/docs/background/tools/hummingbird-ai/).

`CONTINUATION_PROMPT` is critical. Without it, the workflow prompt (e.g.,
`analyze-failures.md`) tells the model to follow a rigid multi-phase
workflow: Data.1, Data.2, Analysis.1... The model would try to re-run the
entire analysis. The continuation prompt overrides this: "Do NOT re-run the
full workflow. Respond directly to the user's question."

The practical constraint on resumed sessions is the context window, not
iterations. The restored history from an 8-iteration cold start uses
~50-100K input tokens. Each new tool call adds more. The existing
`CONTEXT_LIMIT` check still applies and will force wrap-up if the context
grows too large.

#### Canonical session format (rationale)

Storing sessions in canonical form decouples persisted history from any one
provider's JSON shape. The agent loop treats native `contents` as opaque; the
library owns provider conversion at load and save boundaries.

### 6.7. Provider caching and pricing

Provider-specific prompt caching and token-cost estimation are owned by
[`hummingbird-ai`](https://hummingbird-project.io/docs/background/tools/hummingbird-ai/).

## 7. Tool System and Auto-Spill

### 7.1. Tool registry design

`ToolRegistry` is the single point of dispatch for all tool calls. The
agent calls `registry.execute(tool_call)` and gets a `dict` back. It never
calls sandbox methods or data source functions directly.

Three built-in tools are always available:

- **`sandbox_exec`** -- runs `sh -c <command>` in the sandbox container.
  Returns `{exit_code, stdout, stderr}`, with auto-spill for large outputs.
- **`fetch_to_sandbox`** -- calls a data source function and writes the
  result to a specified path in the sandbox. Returns metadata only (path,
  byte count, line count). Used when the model wants explicit path control.
- **`fetch_batch_to_sandbox`** -- calls multiple `fetch_to_sandbox` in one
  tool call. Saves iterations vs. sequential calls (e.g., fetching both
  pipelineruns and taskruns in one round-trip).

Data source tools are registered dynamically per workflow config. Each data
source module provides a `register()` function that creates `ToolDef`
objects and closures over credentials, then calls
`registry.register_data_source(tool_def, func, response_metadata)`.

### 7.2. Auto-spill architecture

Auto-spill is the key mechanism for keeping the LLM context bounded. Without
it, large outputs accumulate in the `contents` list and inflate every
subsequent API call (because both APIs replay the full history).

```mermaid
flowchart TD
    Output["Tool produces output"]
    SizeCheck{"size > OUTPUT_PREVIEW_BYTES<br/>(4 KB)?"}
    Inline["Return full output inline"]
    Spill["Write full output to<br/>/tmp/data/_out/N.txt"]
    Preview["Return preview<br/>(head + tail) +<br/>file path metadata"]

    Output --> SizeCheck
    SizeCheck -->|"<= 4 KB"| Inline
    SizeCheck -->|"> 4 KB"| Spill --> Preview
```

Three spill paths share the same `_spill_counter` to prevent filename
collisions:

**`sandbox_exec` spill.** When stdout or stderr exceeds
`OUTPUT_PREVIEW_BYTES` (4096 bytes), the full output is written to
`/tmp/data/_out/{counter}.txt` via `_spill_field()`. The model receives:

```json
{
  "exit_code": 0,
  "stdout": "<first 4KB>",
  "stdout_truncated": true,
  "stdout_file": "/tmp/data/_out/0.txt",
  "stdout_bytes": 32768,
  "stdout_lines": 1024,
  "stdout_tail": "<last 512 bytes>"
}
```

The preview (head + tail) gives the model enough context to decide whether
to process the full file with `jq`/`grep`/`head`.

**Data source inline spill.** When a direct data source call returns text
larger than `MAX_INLINE_SIZE` (4096 bytes), `_spill_data_source()` writes
it to `/tmp/data/_out/{name}_{counter}.txt` and returns:

```json
{
  "saved_to": "/tmp/data/_out/konflux_list_pipelineruns_1.txt",
  "bytes": 85432,
  "lines": 2100,
  "preview": "<first 4KB>"
}
```

**Streaming responses** (e.g., `gitlab_get_repo_archive` which returns a
`StreamingResponse` with suffix `.tar.gz`) are written via
`_spill_streaming()`. Text streams (`binary=False`, the default) capture a
UTF-8 preview from the head while piping chunks to the sandbox. Binary
streams (`binary=True`) skip preview capture entirely and return only
`{saved_to, bytes}`, avoiding meaningless decoded output for formats like
tar.gz. The `StreamingResponse.suffix` field controls the file extension
(`.jsonl`, `.tar.gz`, `.log`).

Non-streaming binary responses are written to `.bin` files with no preview
via `_spill_binary()`.

**Sandbox-optional mode.** `ToolRegistry` accepts `sb=None` for use cases
that do not require a sandbox (e.g. evaluation harnesses that only use
native model tools like Google Search grounding). When no sandbox is
available:

- Sandbox tools (`sandbox_exec`, `fetch_to_sandbox`, `fetch_batch_to_sandbox`)
  are omitted from `get_tool_defs()` so the model never sees them.
- **Text data sources** (inline and streaming) degrade to truncated previews
  (`OUTPUT_PREVIEW_BYTES`). The model receives `{preview, truncated: true}`
  instead of `{saved_to, preview}`. Streaming text is consumed into a memory
  buffer up to the preview limit and the rest is discarded.
- **Binary data** (streaming and non-streaming) returns an error with the
  byte count (`{error, bytes}`). Binary content cannot be meaningfully
  previewed as text.

**`fetch_to_sandbox` spill.** Always writes to the caller-specified path
(not `/tmp/data/_out/`). Returns metadata only (no preview). The model uses this
when it wants a specific filename for later processing.

### 7.3. Why auto-spill instead of rejecting large outputs

The earlier design rejected large data source responses with "use
`fetch_to_sandbox` instead." This caused two wasted iterations per rejection:
the model makes the call, gets rejected, then has to repeat with
`fetch_to_sandbox`. With auto-spill, the data flows to a sandbox file
transparently. Validation showed ~10% fewer iterations with auto-spill.

### 7.4. Why fetch_to_sandbox still exists alongside auto-spill

Auto-spill handles the common case, but `fetch_to_sandbox` provides:

- **Explicit path control.** The model can choose meaningful filenames
  (`/tmp/data/pipelineruns.json`) instead of getting auto-generated names
  (`/tmp/data/_out/konflux_list_pipelineruns_1.txt`).
- **Batch fetching.** `fetch_batch_to_sandbox` combines multiple fetches
  in one tool call, saving iterations.
- **No preview overhead.** `fetch_to_sandbox` returns metadata only, which
  is useful when the model knows it will process the file with
  `sandbox_exec` anyway.

### 7.5. ToolDef notes

`ToolDef` has an optional `notes` field for domain knowledge that belongs
in the system prompt but not in the tool's JSON schema. Examples:

- Konflux notes explain dual K8s/Kubearchive fetch, UID deduplication, and
  the two label selectors (BUILD vs. TEST).
- Testing Farm notes explain XML result structure and usage patterns.

`ToolRegistry.get_tool_notes()` collects all non-None notes into a
`## Data Source Notes` section that is prepended to the workflow body in
the system prompt:

```text
BASE_SYSTEM_PROMPT + tool_notes + workflow_body
```

This keeps domain knowledge close to the tool definitions (in the data
source module) rather than duplicated in every workflow `.md` file.

### 7.6. Response metadata

`register_data_source()` accepts optional `response_metadata` -- a dict
that is merged into every response from that tool (inline, auto-spill, and
`fetch_to_sandbox`). Used for:

- **Konflux:** `{"konflux_ui": "https://..."}` so the model can build
  reviewer-facing PipelineRun links.
- **Testing Farm:** `{"artifacts_base": "https://..."}` so the model can
  build artifact links.

This avoids having the model ask "what is the Konflux UI URL?" -- the
information arrives with every tool response.

## 8. Sandbox Architecture

### 8.1. The Sandbox protocol

All backends implement an 8-method `typing.Protocol`:

```python
class Sandbox(Protocol):
    def start(self) -> None                                                  # create container/pod, pre-create /tmp/data/
    def exec(command, *, stdin_data: bytes | None) -> ExecResult             # sh -c in sandbox
    def write_file(path, data: bytes) -> None                               # stdin pipe: cat > path
    def stream_to_file(path, chunks: Iterable[bytes]) -> tuple[int, int]    # Popen stdin pipe, returns (bytes, lines)
    def stream_exec(command, chunks: Iterable[bytes]) -> ExecResult         # Popen with streaming stdin
    def read_file_iter(path) -> Iterator[bytes]                             # Popen stdout pipe in chunks
    def cleanup(self) -> None                                               # rm -f container / delete pod
    def linger(session_id) -> None                                          # keep alive for reuse, or fall back to cleanup
```

`ExecResult` is `(exit_code: int, stdout: str, stderr: str)`.

`stream_to_file()` and `stream_exec()` use `subprocess.Popen` with a stdin
pipe, writing chunks incrementally. This avoids buffering large payloads in
orchestrator memory (e.g., streaming a tar.gz archive into the sandbox).
`read_file_iter()` reads from a Popen stdout pipe in 64 KB chunks.

`write_file()` uses stdin piping (`cat > path`), never host volume mounts.
This is critical: it means data flows through the orchestrator process, not
through a shared filesystem. The sandbox has no host mounts.

### 8.2. Backend selection

Five sandbox backends are available:

| Backend               | `--sandbox`    | When to use                                          |
| --------------------- | -------------- | ---------------------------------------------------- |
| `PodmanSandbox`       | `podman`       | Local development (default for `run`)                |
| `K8sSandbox`          | `k8s`          | Direct pod creation on a K8s cluster                 |
| `K8sPoolSandbox`      | `k8spool`      | Pre-warmed pool via Deployment (default for `serve`) |
| `KubeVirtSandbox`     | `kubevirt`     | Direct VMI creation with SSH exec                    |
| `KubeVirtPoolSandbox` | `kubevirtpool` | Pre-warmed VMI pool via VMIRS                        |

`create_sandbox()` factory handles `podman`, `k8s`, and `kubevirt`. Pool
backends (`k8spool`, `kubevirtpool`) are handled in `__main__.py`, which
creates a `SandboxPool` or `VmiPool` and passes it to `run_workflow()`.

The sandbox backend can be set per-workflow via the `sandbox:` field in the
workflow config YAML. Resolution order: `workflow.sandbox` > CLI `--sandbox`

> default. This allows different workflows to use different backends (e.g.,
> lightweight code review uses pod pool, heavier analysis uses VM pool).

K8s is never auto-detected from the environment. It requires explicit CLI
flags. This prevents accidental use of a K8s sandbox when developing
locally.

For K8s namespace resolution (`k8s` and `k8spool`):

1. If `--namespace` is provided, use it.
2. If `--context` is provided, extract the namespace from the kubeconfig
   context. If the context has no default namespace, raise an error.
3. If neither is provided (in-cluster), use `sandbox.namespace` from the
   config file.

### 8.3. PodmanSandbox

```bash
podman run -d --name hb-sandbox-{uuid8} --network=none --user 65532 \
    --workdir /tmp {image} sleep infinity
```

- Unique container name with UUID suffix prevents collisions
- `sleep infinity` keeps the container alive for repeated `exec` calls
- `--network=none` provides complete network isolation
- `--user 65532` is a fixed non-root UID (matches `nonroot` in distroless)
- Cleanup: `podman rm -f` (force, in case exec is still running)

### 8.4. K8sSandbox: the hybrid approach

The K8s sandbox uses a hybrid of two tools:

- **kubernetes Python library** for pod lifecycle: `create_namespaced_pod`,
  `read_namespaced_pod` (poll for Running), `delete_namespaced_pod`.
- **`kubectl exec` subprocess** for command execution.

Why not use the kubernetes Python library for exec too? Three problems
discovered during development:

1. **No stdin EOF signal in WebSocket v1-v4.** The Kubernetes exec protocol
   uses WebSocket channels (stdin=0, stdout=1, stderr=2). Protocol versions
   1-4 have no mechanism to signal "stdin is done." Commands like
   `cat > /file` hang forever waiting for more input. Python client v5
   support does not exist.

2. **`BrokenPipeError` on large stdin.** Sending more than ~1MB through the
   WebSocket `stream()` API causes pipe errors, breaking `write_file()` for
   large data source responses.

3. **Unbounded memory from `stream()`.** The `stream()` function accumulates
   all stdout/stderr data in memory with no streaming control. A command
   producing megabytes of output would consume unbounded memory.

`kubectl exec` as a subprocess avoids all three problems and provides the
same interface as `podman exec` -- stdin via `subprocess.PIPE`, stdout/stderr
captured, exit code from return code. The implementation in `exec()` is
nearly identical between `PodmanSandbox` and `K8sSandbox`.

### 8.5. Pod manifest design

The K8s pod manifest is built by `_build_pod_manifest()`:

```yaml
metadata:
  labels:
    app.kubernetes.io/name: hummingbird-agent-sandbox
spec:
  automountServiceAccountToken: false # no K8s API from sandbox
  activeDeadlineSeconds: 1800 # 30min hard timeout, backstop
  restartPolicy: Never
  securityContext:
    runAsNonRoot: true
    seccompProfile: RuntimeDefault
  containers:
    - name: sandbox
      command: ["sleep", "infinity"]
      workingDir: /tmp
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
      resources:
        requests: { cpu: 100m, memory: 256Mi, ephemeral-storage: 256Mi }
        limits: { cpu: "1", memory: 1Gi, ephemeral-storage: 2Gi }
```

`runAsUser` is deliberately omitted. On OpenShift, the `restricted-v2` SCC
assigns a UID from the namespace UID range. On vanilla K8s, the image's
`USER` directive is used. Setting an explicit UID would conflict with
OpenShift's SCC admission.

### 8.6. Auth modes

- **Local development:** `--context` flag passes the kubeconfig context to
  `config.new_client_from_config(context=...)`, creating a per-instance
  `ApiClient`.
- **Production (in-cluster):** `context=None` triggers
  `config.load_incluster_config()`, using the pod's ServiceAccount token.

The `kubectl exec` commands include `--context` when running locally but
omit it when in-cluster (kubectl uses the default in-cluster config).

### 8.7. Pre-created directories

Both backends run `mkdir -p /tmp/data` in `start()` after the
container/pod is up. This provides a working directory for the model
without consuming an iteration. `/tmp/data/_out` (the spill directory) is
created on demand by `mkdir -p $(dirname ...)` in `write_file()`.

### 8.8. K8sPoolSandbox: Deployment-backed pool

On-demand pod creation adds 5-30 seconds of latency per workflow (image
pull, scheduling, container start). For interactive use (slash commands,
reply-based resumption), this delay is user-facing. The pool eliminates it.

**Mechanism.** A Kubernetes Deployment maintains a set of pods labelled
`hummingbird/role: standby`. When a sandbox is needed, `SandboxPool.claim()`
finds a Running standby pod, patches it to `role: active`, clears its
`ownerReferences` (detaching it from the ReplicaSet), and annotates it with
`hummingbird/reap-by` (an absolute UTC deadline). The Deployment controller
sees the ReplicaSet is under the desired replica count and creates a
replacement.

**Pod lifecycle.**

1. Deployment creates pod -> `role: standby` (managed by ReplicaSet)
2. `claim()` patches -> `role: active`, `ownerReferences: []`, `reap-by: now + max_active_seconds`
3. Agent uses the pod via inherited `K8sSandbox.exec()`
4. On success: `linger()` patches -> `reap-by: now + linger_seconds`,
   `session-id: <id>` (pod stays alive)
5. On reply within linger window: `try_reclaim()` patches ->
   `reap-by: now + max_active_seconds`, clears `session-id`
6. On failure or linger expiry: pod is deleted (by `cleanup()` or the reaper)

**Pod lingering.** After a successful workflow, pool sandboxes are kept
alive for `linger_seconds` (default 300) instead of being deleted
immediately. The pod is annotated with `hummingbird/session-id` to link it
back to the session. If a user reply arrives within the linger window,
`try_reclaim()` finds the pod by session-id, clears the annotation (marking
it as in-use), and resets `reap-by`. This skips pod creation and S3 archive
restoration. If no reply arrives, the reaper deletes the pod when its
`reap-by` deadline passes.

**Concurrency safety.** `session-id` is only present while lingering. Its
absence during active execution prevents concurrent replies from adopting
the same pod. `try_reclaim()` is serialized by a threading lock; the first
caller wins, others fall back to S3 restore with a fresh pod.

**Reaping.** `reap_expired()` is called once per workflow execution (after
sandbox acquisition, before the model loop). It performs a single sorted
pass over all active pods:

1. Non-lingering pods (no `session-id`) past their `reap-by` deadline are
   deleted immediately.
2. Lingering pods (with `session-id`) are sorted by `reap-by` ascending.
   The loop deletes pods that are either expired (`now > reap-by`) or
   exceed `max_lingering_pods` (default 2, configurable), starting with
   those closest to their deadline. The loop stops when both conditions are
   satisfied (`now <= reap-by` and remaining count <= limit).

This replaces the previous design where `_reap_expired()` ran inside
`claim()` on every poll iteration. Moving it to a once-per-workflow call
reduces API load and centralizes cleanup. The `max_lingering_pods` cap
prevents unbounded pod accumulation from many short-lived workflows.

**Indefinite wait.** `claim()` polls until a pod is available or the
shutdown event fires. It logs at DEBUG level on each poll, escalating to
WARNING after ~2 minutes. This is normal behavior when the pool (Deployment
replicas) is smaller than `max_concurrent_agents` -- the SQS semaphore caps
concurrency, so demand will not permanently exceed supply.

**Config simplification.** In pool mode, the pod image, resources, metadata,
and security context are defined solely in the Deployment template. The
agent config only needs `sandbox.namespace` and
`sandbox.active_deadline_seconds`. This eliminates duplication between the
agent configmap and the Deployment.

**Class design.** `K8sPoolSandbox` inherits from `K8sSandbox` to reuse
`exec()`, `write_file()`, `read_file_iter()`, and `cleanup()`. It overrides
`__init__()` (does not call `super().__init__()` since that resolves
kubeconfig), `start()` (claims from pool instead of creating a pod), and
`linger()` (patches annotations instead of deleting). It adds
`start_from_existing()` for the reclaim path. All sandbox backends
implement `linger(session_id)`: non-pool backends fall back to `cleanup()`.

### 8.9. KubeVirtSandbox: SSH-based VM execution

`KubeVirtSandbox` provides full VM isolation using KubeVirt
VirtualMachineInstances. It parallels `K8sSandbox` but uses
`CustomObjectsApi` for VMI lifecycle and `ssh` for command execution
instead of `kubectl exec`.

**Why SSH?** VMs have no `kubectl exec` equivalent. KubeVirt provides
`virtctl console` (serial) and VNC, but these are not suitable for
programmatic command execution. SSH provides a reliable, well-understood
exec channel with stdin/stdout/stderr piping.

**SSH key injection.** The VM sandbox image includes a custom
`inject-ssh-keys.service` that reads SSH public keys from a KubeVirt
Secret volume attached as a virtio disk with serial `ssh-pubkeys`
(visible at `/dev/disk/by-id/virtio-ssh-pubkeys`) and installs them to
`/root/.ssh/authorized_keys` before sshd starts. This avoids the need
for `cloudInitNoCloud` or `qemu-guest-agent`.

**Single-VM lifecycle:**

1. Generate an ephemeral ed25519 key pair via `ssh-keygen`
2. Create a K8s Secret with the public key
3. Create the VMI referencing the containerDisk image and the SSH Secret
4. Poll VMI status until Running with a pod network IP
5. Poll SSH readiness (`ssh sandbox@ip true`)
6. Execute workflow commands via SSH
7. Cleanup: delete VMI, delete Secret, remove temp key files

**Polling strategy.** Two-phase wait: first poll `status.phase` via
`get_namespaced_custom_object` until Running (same interval as pod
polling), then poll SSH readiness with `ConnectTimeout=5`. Total timeout
matches `POD_START_TIMEOUT` (300s) for VMI startup plus
`_SSH_READY_TIMEOUT` (120s) for SSH.

### 8.10. KubeVirtPoolSandbox: VMIRS-backed pool

`KubeVirtPoolSandbox` + `VmiPool` parallel `K8sPoolSandbox` +
`SandboxPool`, operating on VMI custom resources instead of Pods.

**VirtualMachineInstanceReplicaSet (VMIRS).** KubeVirt's native
controller maintains a desired number of identical VMIs. When a VMI is
claimed (label flipped to `active`, ownerReferences cleared), the VMIRS
automatically creates a replacement -- the same pattern as a Deployment
replenishing claimed pods.

**Shared SSH key.** Pool mode uses a single key pair for all standby VMIs.
The public key is stored in a K8s Secret referenced by all VMIs in the
VMIRS template. The private key path is provided via
`sandbox.vm_ssh_private_key` in the config. On pool restart, a new key
pair can be generated; existing VMIs drain naturally with the old key.

**Claim/reap/linger semantics.** Identical to the pod pool: `claim()`
polls for `Running` VMIs with `hummingbird/role=standby`, patches to
`active`, clears ownerReferences. `reap_expired()` deletes VMIs past
their `hummingbird/reap-by` deadline. `try_reclaim()` finds lingering
VMIs by `hummingbird/session-id`. The only difference is the API: all
operations go through `CustomObjectsApi` with `kubevirt.io/v1`
`virtualmachineinstances`.

**Class design.** `KubeVirtPoolSandbox` inherits from `KubeVirtSandbox`
to reuse `exec()`, `write_file()`, `read_file_iter()` (all SSH-based).
It overrides `__init__()` (takes pool, skips key generation),
`start()` (claims from pool), `cleanup()` (only deletes VMI, not
Secret/keys which are pool-owned), and `linger()` (patches annotations).