Hummingbird Agent Design — Engine

Agent loop, tool system, and sandbox backends for the Hummingbird Agent. This page covers the execution engine: how the agent iterates, dispatches tools, and isolates execution. For foundations (design philosophy, system architecture, modules, security, config), see Hummingbird Agent Design.

6. Agent Loop Design

For the wire-format walkthrough (what bytes go to the model API, what comes back), see Agent Model Loop. This section covers the design rationale behind the loop.

6.1. Full history replay

The Gemini and Anthropic APIs are stateless. Every call sends the complete contents list from the beginning of the conversation. This means every large tool output sitting in history inflates every subsequent API call.

This property is fundamental to why auto-spill exists (section 7). Without auto-spill, a single cat of a 32KB file early in the conversation adds ~8K tokens to every remaining API call. Over a 15-iteration run, that is ~120K wasted tokens.

The alternative – conversation compaction (replacing old tool results with summaries) – was considered and deferred. Each provider has strict requirements about content structure, and modifying history risks confusing the model or violating API constraints. Provider-specific format requirements are documented in Hummingbird AI. Auto-spill solves 90% of the problem with none of the risk.

6.2. Budget model: iterations + context limit

The agent uses a dual-limit approach rather than a cumulative token budget:

Iteration limit (max_iterations, default 30 per workflow config). Hard cap on the number of model round-trips. This is the primary cost control lever. With auto-spill keeping per-call context bounded, iteration count is roughly proportional to cost.

Per-call context limit (CONTEXT_LIMIT, default 60,000 tokens). Checked after each API call using response.usage.total_input_tokens. This is a safety net for cases where auto-spill is not sufficient (e.g., many small tool results that individually stay under the spill threshold but cumulatively fill the context).

Why not a cumulative token budget? Because with full history replay, each API call re-sends everything. “Cumulative billed tokens” double-counts: call 1 sends 5K, call 2 sends 10K (including the 5K again), so billed total is 15K but actual new content is only 10K. Iteration count is a simpler and more predictable proxy for cost.

6.3. Two-tier budget escalation

Both limits use the same escalation pattern:

flowchart LR
    Normal["Normal<br/>tools available"]
    Soft["Soft Warning (80%)<br/>ITERATION_WARNING or<br/>CONTEXT_WARNING<br/>tools still available"]
    Hard["Hard Stop (100%)<br/>FINAL_TURN_WARNING<br/>toolConfig: NONE"]

    Normal -->|"80% reached"| Soft
    Soft -->|"100% reached"| Hard

Soft warning (80%). An ephemeral user message (ITERATION_WARNING) is injected into contents for that turn only, then popped before the response is persisted. Tools remain available so the model can finish in-progress work.

Hard stop (100%). FINAL_TURN_WARNING is injected as an ephemeral user message AND tool_defs is set to [] (empty list) to physically prevent further tool calls. The model must produce text. This is more reliable than relying on provider-side tool disabling, which models may ignore.

6.3.1 Ephemeral messages and provider normalization

All per-turn warnings use ephemeral user messages: they are appended to contents before the API call and popped immediately after. This keeps the system prompt stable across all iterations and prevents warnings from polluting the conversation history saved in sessions.

Provider adapters normalize provider-specific conversation requirements while preserving the agent’s saved history; details are in Hummingbird AI.

6.4. Empty response handling

Models occasionally return unusable responses (no text and no valid tool calls). The agent retries up to MAX_EMPTY_RETRIES (2) times:

  1. Pop the empty response from contents. It adds nothing and may confuse the model on the next call.
  2. Inject EMPTY_RESPONSE_NUDGE as an ephemeral user message on the next turn: “Your previous response was empty. Please continue…”
  3. Continue the loop (consuming an iteration).

The nudge is delivered as an ephemeral user message (injected before the API call and popped after). This avoids mutating the system prompt and keeps the conversation history clean for session persistence.

The empty_retries counter resets to 0 after any successful iteration (one where tool calls were executed). This means the model gets fresh retries if it produces empty responses at different points in the conversation.

MALFORMED_FUNCTION_CALL handling. Models sometimes return a finishReason of MALFORMED_FUNCTION_CALL (Gemini) with no usable tool calls. This is treated as a special case of empty response: the retry mechanism kicks in, but the nudge is replaced with MALFORMED_CALL_NUDGE which tells the model to retry with simpler arguments and avoid large text payloads in tool call arguments.

6.5. Error recovery

Model API and transport errors. The hummingbird-ai library classifies provider and transport failures, applies the workflow’s retry budget, and records AI usage and request metrics. See Hummingbird AI for provider-specific behavior. The agent loop retains partial results when a terminal error reaches it.

Unexpected loop errors. run_agent_loop wraps the _generate_with_retry call in a try/except Exception that logs and breaks instead of propagating. This ensures the function always returns partial results (accumulated contents, transcript, sandbox state) even when an unexpected exception occurs (e.g. JSONDecodeError from a truncated API response). The caller saves the session normally – conversation history and sandbox archive are preserved for resumption. This follows the “partial over nothing” principle: 32 iterations of work are more valuable than a crash.

No failure-path session save. _execute_workflow does not save a session when run_workflow raises. With the agent loop catching unexpected errors, the failure path only fires for infrastructure errors (sandbox start, config) where there is no useful state. Not saving avoids overwriting a previous good session when a reply attempt fails.

Tool execution errors. _execute_tool_calls() wraps each tool_registry.execute() in a try/except. Unhandled exceptions are caught and returned to the model as {"error": "Tool X failed: ..."}. This prevents a single broken tool from crashing the entire session – the model sees the error and can adapt.

Sandbox exec timeout. If a command exceeds EXEC_TIMEOUT (120s), the TimeoutExpired exception is caught in the tool registry and returned as a structured error dict. The model can retry with a different command or proceed without the result.

6.6. Session resumption

When resuming from a previous session:

  1. initial_contents (the full contents list from the previous run) is prepended, followed by the user’s reply as a new user turn.
  2. CONTINUATION_PROMPT is appended to the system prompt.
  3. The sandbox is restored from the archived tarball.
  4. A fresh iteration budget starts from 0.

Saved sessions use a versioned JSON envelope (format_version: 1) whose contents are in a canonical message format independent of the active model. Unversioned legacy sessions are not supported. On resume, run_workflow asks the active adapter to load canonical history; after a successful run, it converts native contents back to canonical form before persistence. The library documents the canonical format and provider conversions in Hummingbird AI.

CONTINUATION_PROMPT is critical. Without it, the workflow prompt (e.g., analyze-failures.md) tells the model to follow a rigid multi-phase workflow: Data.1, Data.2, Analysis.1… The model would try to re-run the entire analysis. The continuation prompt overrides this: “Do NOT re-run the full workflow. Respond directly to the user’s question.”

The practical constraint on resumed sessions is the context window, not iterations. The restored history from an 8-iteration cold start uses ~50-100K input tokens. Each new tool call adds more. The existing CONTEXT_LIMIT check still applies and will force wrap-up if the context grows too large.

Canonical session format (rationale)

Storing sessions in canonical form decouples persisted history from any one provider’s JSON shape. The agent loop treats native contents as opaque; the library owns provider conversion at load and save boundaries.

6.7. Provider caching and pricing

Provider-specific prompt caching and token-cost estimation are owned by hummingbird-ai.

7. Tool System and Auto-Spill

7.1. Tool registry design

ToolRegistry is the single point of dispatch for all tool calls. The agent calls registry.execute(tool_call) and gets a dict back. It never calls sandbox methods or data source functions directly.

Three built-in tools are always available:

  • sandbox_exec – runs sh -c <command> in the sandbox container. Returns {exit_code, stdout, stderr}, with auto-spill for large outputs.
  • fetch_to_sandbox – calls a data source function and writes the result to a specified path in the sandbox. Returns metadata only (path, byte count, line count). Used when the model wants explicit path control.
  • fetch_batch_to_sandbox – calls multiple fetch_to_sandbox in one tool call. Saves iterations vs. sequential calls (e.g., fetching both pipelineruns and taskruns in one round-trip).

Data source tools are registered dynamically per workflow config. Each data source module provides a register() function that creates ToolDef objects and closures over credentials, then calls registry.register_data_source(tool_def, func, response_metadata).

7.2. Auto-spill architecture

Auto-spill is the key mechanism for keeping the LLM context bounded. Without it, large outputs accumulate in the contents list and inflate every subsequent API call (because both APIs replay the full history).

flowchart TD
    Output["Tool produces output"]
    SizeCheck{"size > OUTPUT_PREVIEW_BYTES<br/>(4 KB)?"}
    Inline["Return full output inline"]
    Spill["Write full output to<br/>/tmp/data/_out/N.txt"]
    Preview["Return preview<br/>(head + tail) +<br/>file path metadata"]

    Output --> SizeCheck
    SizeCheck -->|"<= 4 KB"| Inline
    SizeCheck -->|"> 4 KB"| Spill --> Preview

Three spill paths share the same _spill_counter to prevent filename collisions:

sandbox_exec spill. When stdout or stderr exceeds OUTPUT_PREVIEW_BYTES (4096 bytes), the full output is written to /tmp/data/_out/{counter}.txt via _spill_field(). The model receives:

{
  "exit_code": 0,
  "stdout": "<first 4KB>",
  "stdout_truncated": true,
  "stdout_file": "/tmp/data/_out/0.txt",
  "stdout_bytes": 32768,
  "stdout_lines": 1024,
  "stdout_tail": "<last 512 bytes>"
}

The preview (head + tail) gives the model enough context to decide whether to process the full file with jq/grep/head.

Data source inline spill. When a direct data source call returns text larger than MAX_INLINE_SIZE (4096 bytes), _spill_data_source() writes it to /tmp/data/_out/{name}_{counter}.txt and returns:

{
  "saved_to": "/tmp/data/_out/konflux_list_pipelineruns_1.txt",
  "bytes": 85432,
  "lines": 2100,
  "preview": "<first 4KB>"
}

Streaming responses (e.g., gitlab_get_repo_archive which returns a StreamingResponse with suffix .tar.gz) are written via _spill_streaming(). Text streams (binary=False, the default) capture a UTF-8 preview from the head while piping chunks to the sandbox. Binary streams (binary=True) skip preview capture entirely and return only {saved_to, bytes}, avoiding meaningless decoded output for formats like tar.gz. The StreamingResponse.suffix field controls the file extension (.jsonl, .tar.gz, .log).

Non-streaming binary responses are written to .bin files with no preview via _spill_binary().

Sandbox-optional mode. ToolRegistry accepts sb=None for use cases that do not require a sandbox (e.g. evaluation harnesses that only use native model tools like Google Search grounding). When no sandbox is available:

  • Sandbox tools (sandbox_exec, fetch_to_sandbox, fetch_batch_to_sandbox) are omitted from get_tool_defs() so the model never sees them.
  • Text data sources (inline and streaming) degrade to truncated previews (OUTPUT_PREVIEW_BYTES). The model receives {preview, truncated: true} instead of {saved_to, preview}. Streaming text is consumed into a memory buffer up to the preview limit and the rest is discarded.
  • Binary data (streaming and non-streaming) returns an error with the byte count ({error, bytes}). Binary content cannot be meaningfully previewed as text.

fetch_to_sandbox spill. Always writes to the caller-specified path (not /tmp/data/_out/). Returns metadata only (no preview). The model uses this when it wants a specific filename for later processing.

7.3. Why auto-spill instead of rejecting large outputs

The earlier design rejected large data source responses with “use fetch_to_sandbox instead.” This caused two wasted iterations per rejection: the model makes the call, gets rejected, then has to repeat with fetch_to_sandbox. With auto-spill, the data flows to a sandbox file transparently. Validation showed ~10% fewer iterations with auto-spill.

7.4. Why fetch_to_sandbox still exists alongside auto-spill

Auto-spill handles the common case, but fetch_to_sandbox provides:

  • Explicit path control. The model can choose meaningful filenames (/tmp/data/pipelineruns.json) instead of getting auto-generated names (/tmp/data/_out/konflux_list_pipelineruns_1.txt).
  • Batch fetching. fetch_batch_to_sandbox combines multiple fetches in one tool call, saving iterations.
  • No preview overhead. fetch_to_sandbox returns metadata only, which is useful when the model knows it will process the file with sandbox_exec anyway.

7.5. ToolDef notes

ToolDef has an optional notes field for domain knowledge that belongs in the system prompt but not in the tool’s JSON schema. Examples:

  • Konflux notes explain dual K8s/Kubearchive fetch, UID deduplication, and the two label selectors (BUILD vs. TEST).
  • Testing Farm notes explain XML result structure and usage patterns.

ToolRegistry.get_tool_notes() collects all non-None notes into a ## Data Source Notes section that is prepended to the workflow body in the system prompt:

BASE_SYSTEM_PROMPT + tool_notes + workflow_body

This keeps domain knowledge close to the tool definitions (in the data source module) rather than duplicated in every workflow .md file.

7.6. Response metadata

register_data_source() accepts optional response_metadata – a dict that is merged into every response from that tool (inline, auto-spill, and fetch_to_sandbox). Used for:

  • Konflux: {"konflux_ui": "https://..."} so the model can build reviewer-facing PipelineRun links.
  • Testing Farm: {"artifacts_base": "https://..."} so the model can build artifact links.

This avoids having the model ask “what is the Konflux UI URL?” – the information arrives with every tool response.

8. Sandbox Architecture

8.1. The Sandbox protocol

All backends implement an 8-method typing.Protocol:

class Sandbox(Protocol):
    def start(self) -> None                                                  # create container/pod, pre-create /tmp/data/
    def exec(command, *, stdin_data: bytes | None) -> ExecResult             # sh -c in sandbox
    def write_file(path, data: bytes) -> None                               # stdin pipe: cat > path
    def stream_to_file(path, chunks: Iterable[bytes]) -> tuple[int, int]    # Popen stdin pipe, returns (bytes, lines)
    def stream_exec(command, chunks: Iterable[bytes]) -> ExecResult         # Popen with streaming stdin
    def read_file_iter(path) -> Iterator[bytes]                             # Popen stdout pipe in chunks
    def cleanup(self) -> None                                               # rm -f container / delete pod
    def linger(session_id) -> None                                          # keep alive for reuse, or fall back to cleanup

ExecResult is (exit_code: int, stdout: str, stderr: str).

stream_to_file() and stream_exec() use subprocess.Popen with a stdin pipe, writing chunks incrementally. This avoids buffering large payloads in orchestrator memory (e.g., streaming a tar.gz archive into the sandbox). read_file_iter() reads from a Popen stdout pipe in 64 KB chunks.

write_file() uses stdin piping (cat > path), never host volume mounts. This is critical: it means data flows through the orchestrator process, not through a shared filesystem. The sandbox has no host mounts.

8.2. Backend selection

Five sandbox backends are available:

Backend --sandbox When to use
PodmanSandbox podman Local development (default for run)
K8sSandbox k8s Direct pod creation on a K8s cluster
K8sPoolSandbox k8spool Pre-warmed pool via Deployment (default for serve)
KubeVirtSandbox kubevirt Direct VMI creation with SSH exec
KubeVirtPoolSandbox kubevirtpool Pre-warmed VMI pool via VMIRS

create_sandbox() factory handles podman, k8s, and kubevirt. Pool backends (k8spool, kubevirtpool) are handled in __main__.py, which creates a SandboxPool or VmiPool and passes it to run_workflow().

The sandbox backend can be set per-workflow via the sandbox: field in the workflow config YAML. Resolution order: workflow.sandbox > CLI --sandbox

default. This allows different workflows to use different backends (e.g., lightweight code review uses pod pool, heavier analysis uses VM pool).

K8s is never auto-detected from the environment. It requires explicit CLI flags. This prevents accidental use of a K8s sandbox when developing locally.

For K8s namespace resolution (k8s and k8spool):

  1. If --namespace is provided, use it.
  2. If --context is provided, extract the namespace from the kubeconfig context. If the context has no default namespace, raise an error.
  3. If neither is provided (in-cluster), use sandbox.namespace from the config file.

8.3. PodmanSandbox

podman run -d --name hb-sandbox-{uuid8} --network=none --user 65532 \
    --workdir /tmp {image} sleep infinity
  • Unique container name with UUID suffix prevents collisions
  • sleep infinity keeps the container alive for repeated exec calls
  • --network=none provides complete network isolation
  • --user 65532 is a fixed non-root UID (matches nonroot in distroless)
  • Cleanup: podman rm -f (force, in case exec is still running)

8.4. K8sSandbox: the hybrid approach

The K8s sandbox uses a hybrid of two tools:

  • kubernetes Python library for pod lifecycle: create_namespaced_pod, read_namespaced_pod (poll for Running), delete_namespaced_pod.
  • kubectl exec subprocess for command execution.

Why not use the kubernetes Python library for exec too? Three problems discovered during development:

  1. No stdin EOF signal in WebSocket v1-v4. The Kubernetes exec protocol uses WebSocket channels (stdin=0, stdout=1, stderr=2). Protocol versions 1-4 have no mechanism to signal “stdin is done.” Commands like cat > /file hang forever waiting for more input. Python client v5 support does not exist.

  2. BrokenPipeError on large stdin. Sending more than ~1MB through the WebSocket stream() API causes pipe errors, breaking write_file() for large data source responses.

  3. Unbounded memory from stream(). The stream() function accumulates all stdout/stderr data in memory with no streaming control. A command producing megabytes of output would consume unbounded memory.

kubectl exec as a subprocess avoids all three problems and provides the same interface as podman exec – stdin via subprocess.PIPE, stdout/stderr captured, exit code from return code. The implementation in exec() is nearly identical between PodmanSandbox and K8sSandbox.

8.5. Pod manifest design

The K8s pod manifest is built by _build_pod_manifest():

metadata:
  labels:
    app.kubernetes.io/name: hummingbird-agent-sandbox
spec:
  automountServiceAccountToken: false # no K8s API from sandbox
  activeDeadlineSeconds: 1800 # 30min hard timeout, backstop
  restartPolicy: Never
  securityContext:
    runAsNonRoot: true
    seccompProfile: RuntimeDefault
  containers:
    - name: sandbox
      command: ["sleep", "infinity"]
      workingDir: /tmp
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
      resources:
        requests: { cpu: 100m, memory: 256Mi, ephemeral-storage: 256Mi }
        limits: { cpu: "1", memory: 1Gi, ephemeral-storage: 2Gi }

runAsUser is deliberately omitted. On OpenShift, the restricted-v2 SCC assigns a UID from the namespace UID range. On vanilla K8s, the image’s USER directive is used. Setting an explicit UID would conflict with OpenShift’s SCC admission.

8.6. Auth modes

  • Local development: --context flag passes the kubeconfig context to config.new_client_from_config(context=...), creating a per-instance ApiClient.
  • Production (in-cluster): context=None triggers config.load_incluster_config(), using the pod’s ServiceAccount token.

The kubectl exec commands include --context when running locally but omit it when in-cluster (kubectl uses the default in-cluster config).

8.7. Pre-created directories

Both backends run mkdir -p /tmp/data in start() after the container/pod is up. This provides a working directory for the model without consuming an iteration. /tmp/data/_out (the spill directory) is created on demand by mkdir -p $(dirname ...) in write_file().

8.8. K8sPoolSandbox: Deployment-backed pool

On-demand pod creation adds 5-30 seconds of latency per workflow (image pull, scheduling, container start). For interactive use (slash commands, reply-based resumption), this delay is user-facing. The pool eliminates it.

Mechanism. A Kubernetes Deployment maintains a set of pods labelled hummingbird/role: standby. When a sandbox is needed, SandboxPool.claim() finds a Running standby pod, patches it to role: active, clears its ownerReferences (detaching it from the ReplicaSet), and annotates it with hummingbird/reap-by (an absolute UTC deadline). The Deployment controller sees the ReplicaSet is under the desired replica count and creates a replacement.

Pod lifecycle.

  1. Deployment creates pod -> role: standby (managed by ReplicaSet)
  2. claim() patches -> role: active, ownerReferences: [], reap-by: now + max_active_seconds
  3. Agent uses the pod via inherited K8sSandbox.exec()
  4. On success: linger() patches -> reap-by: now + linger_seconds, session-id: <id> (pod stays alive)
  5. On reply within linger window: try_reclaim() patches -> reap-by: now + max_active_seconds, clears session-id
  6. On failure or linger expiry: pod is deleted (by cleanup() or the reaper)

Pod lingering. After a successful workflow, pool sandboxes are kept alive for linger_seconds (default 300) instead of being deleted immediately. The pod is annotated with hummingbird/session-id to link it back to the session. If a user reply arrives within the linger window, try_reclaim() finds the pod by session-id, clears the annotation (marking it as in-use), and resets reap-by. This skips pod creation and S3 archive restoration. If no reply arrives, the reaper deletes the pod when its reap-by deadline passes.

Concurrency safety. session-id is only present while lingering. Its absence during active execution prevents concurrent replies from adopting the same pod. try_reclaim() is serialized by a threading lock; the first caller wins, others fall back to S3 restore with a fresh pod.

Reaping. reap_expired() is called once per workflow execution (after sandbox acquisition, before the model loop). It performs a single sorted pass over all active pods:

  1. Non-lingering pods (no session-id) past their reap-by deadline are deleted immediately.
  2. Lingering pods (with session-id) are sorted by reap-by ascending. The loop deletes pods that are either expired (now > reap-by) or exceed max_lingering_pods (default 2, configurable), starting with those closest to their deadline. The loop stops when both conditions are satisfied (now <= reap-by and remaining count <= limit).

This replaces the previous design where _reap_expired() ran inside claim() on every poll iteration. Moving it to a once-per-workflow call reduces API load and centralizes cleanup. The max_lingering_pods cap prevents unbounded pod accumulation from many short-lived workflows.

Indefinite wait. claim() polls until a pod is available or the shutdown event fires. It logs at DEBUG level on each poll, escalating to WARNING after ~2 minutes. This is normal behavior when the pool (Deployment replicas) is smaller than max_concurrent_agents – the SQS semaphore caps concurrency, so demand will not permanently exceed supply.

Config simplification. In pool mode, the pod image, resources, metadata, and security context are defined solely in the Deployment template. The agent config only needs sandbox.namespace and sandbox.active_deadline_seconds. This eliminates duplication between the agent configmap and the Deployment.

Class design. K8sPoolSandbox inherits from K8sSandbox to reuse exec(), write_file(), read_file_iter(), and cleanup(). It overrides __init__() (does not call super().__init__() since that resolves kubeconfig), start() (claims from pool instead of creating a pod), and linger() (patches annotations instead of deleting). It adds start_from_existing() for the reclaim path. All sandbox backends implement linger(session_id): non-pool backends fall back to cleanup().

8.9. KubeVirtSandbox: SSH-based VM execution

KubeVirtSandbox provides full VM isolation using KubeVirt VirtualMachineInstances. It parallels K8sSandbox but uses CustomObjectsApi for VMI lifecycle and ssh for command execution instead of kubectl exec.

Why SSH? VMs have no kubectl exec equivalent. KubeVirt provides virtctl console (serial) and VNC, but these are not suitable for programmatic command execution. SSH provides a reliable, well-understood exec channel with stdin/stdout/stderr piping.

SSH key injection. The VM sandbox image includes a custom inject-ssh-keys.service that reads SSH public keys from a KubeVirt Secret volume attached as a virtio disk with serial ssh-pubkeys (visible at /dev/disk/by-id/virtio-ssh-pubkeys) and installs them to /root/.ssh/authorized_keys before sshd starts. This avoids the need for cloudInitNoCloud or qemu-guest-agent.

Single-VM lifecycle:

  1. Generate an ephemeral ed25519 key pair via ssh-keygen
  2. Create a K8s Secret with the public key
  3. Create the VMI referencing the containerDisk image and the SSH Secret
  4. Poll VMI status until Running with a pod network IP
  5. Poll SSH readiness (ssh sandbox@ip true)
  6. Execute workflow commands via SSH
  7. Cleanup: delete VMI, delete Secret, remove temp key files

Polling strategy. Two-phase wait: first poll status.phase via get_namespaced_custom_object until Running (same interval as pod polling), then poll SSH readiness with ConnectTimeout=5. Total timeout matches POD_START_TIMEOUT (300s) for VMI startup plus _SSH_READY_TIMEOUT (120s) for SSH.

8.10. KubeVirtPoolSandbox: VMIRS-backed pool

KubeVirtPoolSandbox + VmiPool parallel K8sPoolSandbox + SandboxPool, operating on VMI custom resources instead of Pods.

VirtualMachineInstanceReplicaSet (VMIRS). KubeVirt’s native controller maintains a desired number of identical VMIs. When a VMI is claimed (label flipped to active, ownerReferences cleared), the VMIRS automatically creates a replacement – the same pattern as a Deployment replenishing claimed pods.

Shared SSH key. Pool mode uses a single key pair for all standby VMIs. The public key is stored in a K8s Secret referenced by all VMIs in the VMIRS template. The private key path is provided via sandbox.vm_ssh_private_key in the config. On pool restart, a new key pair can be generated; existing VMIs drain naturally with the old key.

Claim/reap/linger semantics. Identical to the pod pool: claim() polls for Running VMIs with hummingbird/role=standby, patches to active, clears ownerReferences. reap_expired() deletes VMIs past their hummingbird/reap-by deadline. try_reclaim() finds lingering VMIs by hummingbird/session-id. The only difference is the API: all operations go through CustomObjectsApi with kubevirt.io/v1 virtualmachineinstances.

Class design. KubeVirtPoolSandbox inherits from KubeVirtSandbox to reuse exec(), write_file(), read_file_iter() (all SSH-based). It overrides __init__() (takes pool, skips key generation), start() (claims from pool), cleanup() (only deletes VMI, not Secret/keys which are pool-owned), and linger() (patches annotations).