This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Hummingbird Agent

An event-driven LLM agent that investigates CI/CD failures and posts findings as GitLab merge request notes. The agent executes markdown-defined workflows using tool calling, with all data processing running inside an isolated sandbox container.

Model adapters, provider calls, and shared AI usage metrics come from the in-process hummingbird-ai library.

For the architectural design rationale, module boundaries, security invariants, and design decision registry, see Agent Design. For agent turn flow and tool handling, see Agent Model Loop. Provider adapters and wire formats are documented in Hummingbird AI.

flowchart TD
    Pipeline["Pipeline Fails"]
    MREvent["MR Created / Updated"]
    Slash["/hummingbird command"]

    Pipeline -->|"event"| Agent
    MREvent -->|"event"| Agent
    Slash -->|"command"| Agent

    subgraph Agent ["Hummingbird Agent"]
        APIs["GitLab + Konflux +<br/>Testing Farm"]
        Model["LLM<br/>(Gemini / Claude)"]
        subgraph Sandbox ["Isolated Sandbox (no network)"]
            Tools["jq / python3 / yq<br/>data processing"]
        end
        APIs <-->|"data"| Model
        Model <-->|"commands"| Tools
    end

    Note["MR Note<br/>(analysis or review)"]

    Agent -->|"posts"| Note
    Note -->|"reply to continue"| Agent

Features

  • Workflow-driven analysis - Investigation logic lives in .md files, not in code; easy to iterate without redeployment
  • Centralized YAML config - A single config file defines operational settings, workflows, enabled data sources with token env var names, project allowlists, and per-project limits
  • Sandboxed execution - All untrusted commands (jq, python3, shell) run in an isolated container, never on the host
  • Five sandbox backends - Podman for local development (network-isolated), direct K8s pod creation, Deployment-backed pod pool for low-latency production use (restricted-v2 SCC compliant), KubeVirt VM (direct), or VMIRS-backed VM pool for heavier workloads requiring full OS isolation
  • Data source abstraction - GitLab, Konflux, and Testing Farm are registered as tool-calling functions the model invokes directly
  • Auto-spill for large outputs - Stdout/stderr and data source responses exceeding 4 KB are automatically saved to sandbox files with a compact preview returned to the model, keeping context usage bounded
  • Prompt caching - Gemini uses implicit server-side caching automatically; Claude uses explicit sliding-window cache breakpoints that reduce input token costs by ~80% on multi-turn agent runs
  • Token budget management - Per-call context ceiling and iteration-based soft/hard limits prevent runaway sessions
  • Session persistence - Conversation history, transcript, and sandbox files saved to S3 (production) or local directory (development) for debugging and future session resumption. Sessions are stored in a provider-neutral format, enabling model switching between conversations.

Architecture

flowchart LR
    subgraph input [Input]
        SQS["SQS Queue"]
        CLI["CLI --event"]
    end

    subgraph agentLoop [Agent Loop]
        WF["Workflow .md<br/>(system prompt)"]
        LLM["LLM&lt;br/&gt;(Gemini / Claude)"]
        TR["Tool Registry"]
    end

    subgraph tools [Tools]
        SE["sandbox_exec"]
        FTS["fetch_to_sandbox"]
        DS["Data Sources"]
    end

    subgraph sandbox [Sandbox Container]
        JQ["jq / python3 / yq"]
        Files["Spilled files"]
    end

    subgraph external [External APIs]
        GL["GitLab API"]
        KX["Konflux K8s"]
        TF["Testing Farm"]
    end

    subgraph output [Output]
        Note["GitLab MR Note"]
        Session["Session State"]
    end

    SQS --> WF
    CLI --> WF
    WF --> LLM
    LLM -->|"tool calls"| TR
    TR --> SE --> sandbox
    TR --> FTS --> sandbox
    TR --> DS
    DS --> GL
    DS --> KX
    DS --> TF
    DS -->|"auto-spill"| sandbox
    LLM -->|"final text"| Note
    LLM --> Session

An event (CLI --event JSON or SQS message) identifies a GitLab project and MR IID. The config file maps project paths to workflows and provides action, max_iterations, enabled data sources, and token env var names. Both run and serve use this config. The markdown body of the prompt file becomes the LLM system prompt. The agent loop iterates until the model produces a final text response or hits the iteration/context limit. Tool calls are dispatched through the ToolRegistry: sandbox_exec runs shell commands, fetch_to_sandbox pipes data source output into the sandbox, and direct data source calls return results inline (or auto-spill large responses to files).

For a detailed walkthrough of the model loop – what gets sent to the model each iteration, how tool calls flow, the exact wire format, and how user replies integrate for session resumption – see Agent Model Loop.

Event-Driven Triggers

In production, the agent consumes events from an SQS queue subscribed to the central SNS topic. The SNS filter policy delivers three event types:

  • gitlab::pipeline – Fires when a GitLab CI pipeline completes. The agent triggers on status=failed pipelines from merge_request_event sources where the triggering user has at least Developer access on the project. Only workflows with trigger: pipeline are executed. Since the pipeline stays open until all Konflux external stages resolve, this naturally waits for all builds and tests to finish before triggering.

  • gitlab::merge_request – Fires on MR open and update events. Only workflows with trigger: merge_request are executed. The agent triggers in three cases: a new MR is opened (action=open), code is pushed to an existing MR (action=update with oldrev), or a draft MR is marked as ready (action=update with a changes.draft transition from true to false). All other update events (labels, assignees, description changes) are skipped – they carry no new code. Draft MRs are skipped; SHA-based deduplication prevents reviewing the same code revision twice. Access checks and trigger_rules filtering evaluate the push author: for open and push events the webhook user is the pusher; for undraft transitions the agent resolves the actual pusher from MR system notes so that the person undrafting cannot bypass the access check on code pushed by an untrusted user.

  • gitlab::note – MR comment events. Two sub-flows:

    • Slash command (/hummingbird <workflow-name>): triggers a specific workflow. Prefix matching is supported (e.g. /hummingbird analyze matches analyze-failures). /hummingbird or /hummingbird help lists available workflows. The note author must have Developer+ access on the project. Optional runtime overrides can be appended: /hummingbird code-review model=claude-sonnet-4-6 max_iterations=25.
    • Reply to agent note: when a user replies to an existing agent note (which contains a session marker), the agent loads the previous session from S3 (conversation history + sandbox files) and continues the conversation with the user’s reply as input. The system prompt includes a CONTINUATION_PROMPT that prevents the model from re-running the full workflow. If the session is not found (expired/deleted), the agent falls back to a cold start. Replies may include overrides on a separate line (e.g. /hummingbird model=claude-opus-4-6); override lines are stripped from the user message. Overrides persist in the session until explicitly changed.

    Notes generated by the agent itself (containing session markers) are skipped to prevent infinite loops. Reply threading respects the internal_notes config: if the project requires internal notes but the original thread was public, the reply is posted as a new top-level internal note instead.

Both pipeline and merge_request triggers apply per-workflow trigger rules filtering. Each workflow may define an ordered trigger_rules array; the first matching rule decides whether the event is allowed or denied (implicit deny on fallthrough). Rules can match on pipeline status, user_regex/user_regexes, branch_regex/branch_regexes, and title_regex/title_regexes. All conditions within a rule are ANDed; multiple patterns within a regex field are ORed. The ! prefix negates a pattern. If no trigger_rules are specified, merge_request triggers allow all events and pipeline triggers default to allowing only failed status (preserving backward compatibility).

The legacy ignore_users and ignore_branches fields are still accepted and automatically desugared into equivalent trigger_rules (deny + catch-all allow). They cannot be mixed with explicit trigger_rules.

Rate limiting is per-workflow: each workflow’s thread count is tracked independently via JSON session markers that embed the workflow name and commit SHA. The max_runs_per_mr limit applies separately to each workflow on a given MR.

Events flow through a two-stage SQS pipeline. A slim ingress router forwards webhooks from the standard queue to an SQS FIFO queue, grouped by discussion_id for note events (ensuring same-discussion ordering) and by SQS MessageId for pipeline/MR events (no serialization needed). Phase 1 handlers validate the event and resolve the session, then post a continuation message back to the FIFO grouped by session_id. Phase 2 picks up the continuation and runs the workflow. This ensures all work for a given session is serialized even across multiple discussion threads.

The SQS infrastructure is defined in template.yaml (SAM/CloudFormation): a standard ingress queue (60s visibility timeout for the router hop) and a FIFO work queue (30-minute visibility timeout for workflow execution).

Model Configuration

The agent supports multiple LLM providers via a unified adapter interface.

Gemini

  • API key (local dev) – Set GOOGLE_API_KEY. Calls generativelanguage.googleapis.com directly. No region configuration needed.
  • Vertex AI (production) – Set GOOGLE_CLOUD_PROJECT and configure model_regions in the YAML settings. Uses Application Default Credentials (ADC) via google-auth. For OpenShift, mount a service account key JSON file and set GOOGLE_APPLICATION_CREDENTIALS, or use workload identity.

Gemini uses implicit server-side caching – repeated prefixes are automatically cached by the Vertex AI backend with no opt-in required. Cached input tokens are billed at 25% of the base input rate. The agent tracks cached token counts from API responses for cost estimation.

Anthropic Claude

Claude models are accessed via Vertex AI using the same GCP project (GOOGLE_CLOUD_PROJECT) with the region resolved from model_regions. Enable the desired model in the GCP Model Garden. Use model: claude-sonnet-4-20250514 in workflow config.

Prompt caching is enabled automatically for Claude models. The agent places explicit cache breakpoints on messages so that the full conversation prefix is served from cache on every turn after the first. Cache writes cost 1.25x the base input rate; cache reads cost 0.10x (90% discount). For a typical 20-iteration agent run, this reduces input token costs by approximately 80%. Ephemeral messages (iteration warnings, nudges) are excluded from cache writes to avoid polluting the cache with transient content.

Region configuration

GCP regions are configured via settings.model_regions in the config file – a map of model name prefixes to GCP regions. The agent resolves the region by longest-prefix match on the effective model name (same algorithm as cost estimation). Example:

settings:
  model_regions:
    gemini-2.5-pro: us-east5
    gemini-3.1-pro: global
    claude: global

At least one of GOOGLE_API_KEY or GOOGLE_CLOUD_PROJECT must be set. If both are set, the API key takes precedence for Gemini models. Claude models always require Vertex AI mode (GOOGLE_CLOUD_PROJECT + model_regions); GOOGLE_API_KEY direct mode is for Gemini only.

Prerequisites

  • Python 3.11+
  • Podman for local sandboxes, or kubectl and Kubernetes access to a dedicated sandbox namespace for K8s sandboxes.
  • Authentication: GOOGLE_API_KEY or GOOGLE_CLOUD_PROJECT (see above)
  • For Claude models: enable the desired model in GCP Model Garden
  • GitLab tokens referenced in the config file (model tool tokens, orchestrator tokens)
  • Kubeconfig with access to the Konflux cluster (if using Konflux data sources)

Installation

cd hummingbird-agent
pip install -e .

Usage

Local development (run)

The run command uses the config file for workflow lookup, data source registration, and token resolution – the same code path as serve. By default, results are printed to stdout (dry-run). Use --execute to post the result as a GitLab MR note.

There are two ways to select what to run:

Direct workflow selection (--workflow + --project + --event):

# Run a specific workflow on an MR, Podman sandbox, print to stdout
hummingbird-agent run \
    --workflow analyze-failures \
    --project org/group/project \
    --event '{"iid": 123, "sha": "abc123"}'

# Same but with K8s sandbox
hummingbird-agent run \
    --workflow analyze-failures \
    --project org/group/project \
    --event '{"iid": 123, "sha": "abc123"}' \
    --context my-cluster/my-namespace

# Post the result as a GitLab MR note (also saves session to S3 if configured)
hummingbird-agent run \
    --workflow analyze-failures \
    --project org/group/project \
    --event '{"iid": 123, "sha": "abc123"}' \
    --execute

# Save session locally for debugging (context.json, transcript.md, sandbox.tar.gz)
hummingbird-agent run \
    --workflow analyze-failures \
    --project org/group/project \
    --event '{"iid": 123, "sha": "abc123"}' \
    --save-session /tmp/my-session

# Resume from a saved session with a follow-up question
hummingbird-agent run \
    --workflow analyze-failures \
    --project org/group/project \
    --event '{"iid": 123, "sha": "abc123"}' \
    --resume-session /tmp/my-session \
    --message "Can you look at the clair-scan timeout more closely?"

# Chain: resume and save the new session for another round
hummingbird-agent run \
    --event-file event.json \
    --resume-session /tmp/my-session \
    --message "What layer hash failed?" \
    --save-session /tmp/my-session-2

Event-file replay (--event-file): routes the event through the same project-index lookup as serve, but skips rate limiting and status filtering:

# Replay a real webhook event, dry-run with Podman
hummingbird-agent run \
    --event-file event.json

# Replay with K8s sandbox and post notes
hummingbird-agent run \
    --event-file event.json \
    --context my-cluster/my-namespace \
    --execute

Production (serve)

# Poll SQS queue for events (pool sandbox by default, posts results as MR notes)
CONFIG_PATH=config.yml hummingbird-agent serve

Requires CONFIG_PATH pointing to a config file with settings.sqs_queue_url and settings.sqs_fifo_queue_url set. The config file defines which workflows run on which projects, with per-project limits and data source token mappings. Handles SIGTERM/SIGINT for graceful shutdown. A background router thread forwards events from the standard queue to the FIFO; the main thread consumes the FIFO with a semaphore-gated thread pool (controlled by settings.max_concurrent_agents) so excess messages stay in SQS for other instances.

Config hot-reload: In serve mode a background thread polls the config file for changes (every 5 seconds by default). When the file changes, the new config is validated and atomically swapped in – subsequent event dispatches use the updated config. If the new config is invalid, the previous config is kept and a warning is logged. No restart required for config changes.

The serve command also accepts --sandbox, --context, and --namespace for local development with a different sandbox backend (e.g. Podman).

CLI Options

run subcommand:

Option Description Default
--event Inline event JSON string (mutually exclusive with --event-file) -
--event-file Read event from a JSON file (mutually exclusive with --event) -
--workflow Workflow name from config file (requires --project) -
--project GitLab project path (requires --workflow) -
--execute Post result as GitLab MR note and save session to S3 -
--save-session Save session artifacts to this directory -
--resume-session Resume from a saved session directory (requires --message) -
--message Follow-up message for session resumption (requires --resume-session) -
--sandbox Sandbox backend (podman, k8s, k8spool, kubevirt, or kubevirtpool) podman
--context K8s context (implies K8s backend) -
--namespace K8s namespace -
-v, --verbose Enable debug logging -

serve subcommand:

Option Description Default
--sandbox Sandbox backend (podman, k8s, k8spool, kubevirt, or kubevirtpool) k8spool
--context K8s context (implies K8s backend) -
--namespace K8s namespace -
-v, --verbose Enable debug logging -

Configuration

Config file

Both run and serve use a single YAML config file. Set the path via CONFIG_PATH (default: config.example.yaml). The config has two sections: settings for operational parameters, and workflows for workflow definitions. Most settings have defaults; configure the sandbox image and namespace as needed for the selected backend. Sandbox options for settings.sandbox are documented by the shared Hummingbird Sandbox library.

The dedicated agent execution image is not yet published or selected in runtime configuration. See sandbox images for build and rollout details.

settings:
  gitlab_url: https://gitlab.com # GitLab instance URL
  sandbox: {}
  max_concurrent_agents: 4 # max concurrent workflows (serve)
  sqs_queue_url: "" # standard SQS ingress queue URL (serve)
  sqs_fifo_queue_url: "" # FIFO work queue URL (serve)
  s3_session_bucket: "" # S3 bucket for session persistence
  model: gemini-3.1-pro-preview # or claude-sonnet-4-20250514
  model_regions: # model prefix -> GCP region
    gemini-2.5-pro: us-east5
    gemini-3.1-pro: global
    claude: global
  max_iterations: 30 # default iteration limit
  max_runs_per_mr: 5 # default per-MR rate limit
  internal_notes: true # default note visibility
  docs_url: https://gitlab.com/org/group/project/-/blob/main/docs/agent.md
  source_url: https://gitlab.com/org/group/project
  slack_url: https://slack.example.com/archives/C0123456789
  slack_label: "#my-channel"

workflows:
  code-review:
    trigger: merge_request # auto-trigger on MR events
    description: Performs AI-powered code review
    sandbox: k8spool # per-workflow sandbox override (optional)
    workflow_url: https://gitlab.com/org/group/project/-/blob/main/workflows/code-review.md
    trigger_rules: # ordered rule chain, first match wins
      - user_regex: "renovate\\[bot\\]" # deny bot MRs (regex fullmatch)
        action: deny
      - branch_regex: "chore/.*" # deny maintenance branches
        action: deny
      - action: allow # allow everything else
    prompt: workflows/code-review.md
    action: post_gitlab_note
    model: gemini-3.1-pro-preview # or claude-sonnet-4-20250514
    max_iterations: 15
    max_inline_size: 200000 # keep full diffs in context
    context_limit: 500000 # Gemini 3.1 Pro / Claude Sonnet 4 have large context windows
    data_sources:
      gitlab:
        token_env: GITLAB_TOKEN_RO
    projects:
      org/group/project: {}

  analyze-failures:
    trigger: pipeline # auto-trigger on failed pipelines
    description: Investigates CI/CD pipeline failures
    sandbox: kubevirtpool # use VM sandbox for heavier workloads
    auto_resolve_on_push: true # resolve threads when a new SHA is pushed
    auto_resolve_on_success: true # resolve threads when pipeline succeeds
    workflow_url: https://gitlab.com/org/group/project/-/blob/main/workflows/analyze-failures.md
    prompt: workflows/analyze-failures.md # relative to config file dir
    action: post_gitlab_note
    model: gemini-3.1-pro-preview # or claude-sonnet-4-20250514
    max_iterations: 50 # per-workflow iteration override

    data_sources: # model tool tokens (read-only)
      gitlab:
        token_env: GITLAB_TOKEN_RO # env var name, not the token
      konflux:
        cluster_url: https://example.com:6443/ns/my-tenant
        kubeconfig_env: KUBECONFIG
        kubearchive_url: https://kubearchive-api-server-product-kubearchive.apps.example.com
      testing_farm: {}

    projects:
      redhat/hummingbird/containers:
        tokens: # per-project model token overrides
          gitlab: GITLAB_TOKEN_CONTAINERS_RO
        action_tokens: # per-workflow write tokens
          gitlab: HUMMINGBIRD_AGENT_ACTION_ANALYZE_FAILURES_GITLAB_TOKEN_CONTAINERS

With --workflow/--project, the workflow and project are looked up directly in the config. With --event-file, the project is extracted from the event body and matched against the project index to find applicable workflows. Pool overrides in this example require the corresponding pool to be provisioned and selected with --sandbox at startup. Only the CLI-selected pool is initialized, so these two pool overrides cannot both be used in one service. Without a matching pool, a workflow can fall back to a direct sandbox instead.

Discussion threads

All workflow results are posted as discussion threads: a placeholder note starts the discussion and the full result is posted as a reply. The placeholder is never edited, so email notifications include the actual result text. For slash commands, the result is posted as a reply in the triggering discussion.

Auto-resolve

Workflows can opt in to automatic resolution of their discussion threads:

  • auto_resolve_on_push (default false): When a new commit is pushed to the MR (i.e. a merge_request event with action: update), all agent discussion threads for the workflow whose SHA differs from the new HEAD are resolved. This clears stale failure analyses when the developer pushes a fix.
  • auto_resolve_on_success (default false): When the head pipeline succeeds, all agent discussion threads for the workflow on that MR are resolved (regardless of SHA). This handles pipeline reruns on the same SHA where a transient failure is now green.

Both flags are independent and can be combined. Resolution runs before rate limit checks, so threads are resolved even if the workflow’s per-MR run limit has been reached.

Workflows can enable Anthropic’s built-in web search server tool by listing web_search as a data source:

workflows:
  renovate-babysit:
    model: claude-sonnet-4-20250514
    data_sources:
      gitlab: {}
      web_search: {}
    # ...

When web_search is present in data_sources, the web_search_20250305 server tool is included in API requests to Claude. The model decides autonomously when to search. Search execution happens server-side (no client-side tool dispatch), and results appear as server_tool_use / web_search_tool_result content blocks in the response. These blocks are preserved through session save/resume.

Unlike other data sources, web_search has no configuration options and does not register any orchestrator-side tools – it is handled entirely by the model provider.

If the API returns a pause_turn stop reason (server-side search loop hit its iteration limit), the adapter automatically re-sends the conversation to continue, up to 5 continuations per generate() call.

Web search is only supported with Claude models. The setting is ignored for Gemini.

The first agent-authored note (placeholder) includes a footer with links to documentation, source code, the Slack channel, the workflow prompt, and a continuation prompt. These links are configured via global settings:

Setting Description
settings.docs_url Link to agent documentation
settings.source_url Link to agent source repository
settings.slack_url Link to support Slack channel
settings.slack_label Display text for Slack link (default: “Slack”)

Per-workflow, set workflow_url to link to the workflow’s prompt file. The footer stays on the placeholder and the result is a separate reply.

Token separation

Tokens are split into three tiers that never mix:

  • Model tool tokens (in YAML data_sources / tokens): read-only tokens passed to the LLM’s tool calls. Declared in the config file as env var names. These are user-defined and resolved at runtime from the referenced env vars. Create as project access tokens with Reporter role and read_api scope. Reporter is the minimum role required to see internal (confidential) notes in the discussions tool.
  • Workflow action tokens (in YAML action_tokens, per-project): write tokens used by the orchestrator for per-workflow GitLab writes (notes, thread resolution). Each workflow gets a dedicated bot user per project, providing clear audit trails for which agent produced each note. Declared in the project config as env var names. Create as project access tokens with Developer role and api scope. These tokens are never exposed to the LLM and never enter the ToolRegistry. Naming convention: HUMMINGBIRD_AGENT_ACTION_<WORKFLOW>_GITLAB_TOKEN_<PROJECT>.
  • Orchestrator tokens (ORCHESTRATOR_* env vars, NOT in YAML): tokens used by the runner for operational reads (member access checks, push author lookup, head pipeline queries) and infrastructure notes (access-denied replies, rate-limit notices). Create as project access tokens with Developer role and read_api scope (or api for infrastructure notes). Developer role is required because the discussions tool trust-filters notes by author access level (>= Developer); if the orchestrator bot has only Reporter access, its own notes are redacted. Resolved by convention: ORCHESTRATOR_GITLAB_TOKEN_<MANGLED_PROJECT> (per-project) or ORCHESTRATOR_GITLAB_TOKEN (fallback). The ORCHESTRATOR_ prefix makes these impossible to confuse with model tokens.

Environment Variables

The agent reads only secrets and authentication from environment variables. All operational settings come from the config file’s settings section.

Variable Required Default Description
CONFIG_PATH no config.example.yaml Path to config YAML
GOOGLE_API_KEY yes* - Gemini API key; Gemini direct mode only (not for Claude)
GOOGLE_CLOUD_PROJECT yes* - GCP project ID (Vertex AI mode); required for Claude models
ORCHESTRATOR_GITLAB_TOKEN serve - Orchestrator GitLab token (global fallback)
ORCHESTRATOR_GITLAB_TOKEN_<PROJECT> no - Per-project orchestrator token
SENTRY_DSN no - Sentry DSN for error tracking

*One of GOOGLE_API_KEY or GOOGLE_CLOUD_PROJECT is required. Claude models require Vertex AI (GOOGLE_CLOUD_PROJECT); GOOGLE_API_KEY is for Gemini direct mode only.

Model tool tokens (e.g. GITLAB_TOKEN_RO, GITLAB_TOKEN_CONTAINERS_RO) and data source credentials (e.g. KONFLUX_CLUSTER_URL, KUBECONFIG) are referenced by name in the config file’s data_sources and tokens sections. They are not listed in the table above because their names are user-defined.

Sandbox Security

Sandbox isolation, security constraints, permissions, and backend behavior are documented in the shared Hummingbird Sandbox library. The agent continues to own workflow selection, CLI options, session persistence, and the decision to acquire or reuse a sandbox.

Data Sources

Data sources are registered as tool-calling functions. The model invokes them by name; the orchestrator executes them and returns results (or auto-spills large responses to the sandbox).

GitLab

Tool Description
gitlab_get_mr_details MR metadata (title, author, state, SHA, labels)
gitlab_get_mr_unified_diff Complete unified diff in patch format
gitlab_get_mr_diff Per-file structured change data
gitlab_get_mr_commits List of commits in a merge request
gitlab_get_mr_discussions Discussion threads with redacted agent transcripts and trust-filtered comments
gitlab_get_commit_statuses CI/CD pipeline statuses for a commit
gitlab_get_file_at_ref Raw file content at a git ref
gitlab_get_repo_archive Repository tar.gz (binary, auto-spilled)
gitlab_get_job_log CI job trace output (ANSI codes stripped)

Konflux

Fetches Tekton PipelineRuns and TaskRuns from both the live K8s API and Kubearchive (for completed resources), with deduplication by UID.

Tool Description
konflux_list_pipelineruns PipelineRuns for a commit SHA; optional status allowlist filter
konflux_list_taskruns TaskRuns by commit SHA or parent PipelineRun name; optional status filter
konflux_list_pods Pods for a commit SHA; optional status allowlist filter
konflux_get_pod Full pod resource (spec, status, conditions, container statuses)
konflux_get_pod_log Pod logs; optional container param, fetches all containers when omitted

Status filtering: All three list tools accept an optional status parameter — an allowlist of status.conditions[0].status values ("True" = success, "False" = failure, "Unknown" = in-progress). Only items matching the allowlist are written to the sandbox file. Response metadata always includes counts (all items seen, by reason), included, and excluded — the agent knows the full reason breakdown even when fetching only failures. Items without conditions always pass.

Per-PLR TaskRun fetching: konflux_list_taskruns accepts a pipelinerun parameter. When set, it uses tekton.dev/pipelineRun=<name> as a server-side label selector, returning only the ~15-21 TaskRuns for that PipelineRun (~600 KB) instead of all TaskRuns for the commit (~530 MB for 340 components).

Recommended pattern for analyze-failures: Fetch failing PLRs and TaskRuns together in one batch with status=["False"]. This captures every failure reason (Failed, Cancelled, PipelineRunTimeout, StepFailed, etc.) without enumerating them. Use response metadata counts for the summary. Use per-PLR fetch only when deeper investigation is needed.

Response metadata includes konflux_ui base URL for building reviewer-facing links.

Testing Farm

Tool Description
tf_get_results JUnit XML results for a request ID
tf_get_test_log Individual test log by URL (restricted to Testing Farm artifact URLs)
tf_get_request_status Request state, queue/run times

Response metadata includes artifacts_base URL for building artifact links.

Workflow System

Workflows are .md files whose content becomes the LLM system prompt verbatim. Workflow metadata (action, model, max_iterations, enabled data sources, project allowlists) is defined in the config file. The .md file is pure system prompt text.

Available workflows:

  • analyze-failures.md - Investigates CI/CD pipeline failures by fetching MR details, identifying failed pipelines via commit statuses, retrieving PipelineRuns/TaskRuns from Konflux, analyzing test results from Testing Farm, and producing a grouped root-cause report with reviewer-facing URLs.
  • code-review.md - Performs AI-powered code review by fetching MR details, the unified diff, and prior discussion threads in parallel, then attempting to load per-project rules from workflows/repo-rules/ (a mandatory step – rules take precedence over generic standards when present) and producing structured feedback with severity ratings, code examples, and actionable suggestions. On follow-up reviews (after SHA updates), the agent sees its own previous findings, developer responses, and resolved threads – avoiding duplicate findings and respecting developer explanations. Uses elevated max_inline_size (200 KB) and context_limit (500K tokens) to keep the full diff in context.
  • renovate-babysit.md - Triages a single Renovate-authored MR triggered by a successful pipeline. Classifies the MR as safe or risky based on diff scope, upstream dependency changes (from MR description, web search), and local codebase impact (via gitlab_get_repo_archive + grep). Posts a structured note with verdict, upstream change summary, and suggested actions for risky MRs. Requires a Claude model with web_search data source.

Token Budget Management

Prompt caching, cached token accounting, and provider-specific cost estimates are implemented in hummingbird-ai.

The agent uses a dual-limit approach instead of a cumulative token budget:

  • Iteration limit (settings.max_iterations, default 30, overridable per-workflow) - Hard cap on tool-calling rounds. A wrap-up prompt is injected at 80% (SOFT_ITERATION_RATIO).
  • Context limit (CONTEXT_LIMIT, default 60,000 tokens, overridable per-workflow via context_limit) - Per-call input token ceiling. When exceeded, a wrap-up prompt forces the model to finalize.

Large outputs are automatically redirected to sandbox files to keep the LLM context small. The spill threshold defaults to 4 KB (MAX_INLINE_SIZE) but can be overridden per-workflow via max_inline_size in the config:

  • sandbox_exec - stdout/stderr exceeding the threshold saved to /tmp/_out/{N}.txt; model receives a preview (head + tail) with file path
  • Data sources - text exceeding the threshold saved to /tmp/_out/{name}_{N}.txt with preview; binary data saved to .bin
  • fetch_to_sandbox - always writes to the caller-specified path; returns metadata only

All tuning constants are centralized in config.py:

Constant Default Purpose
DEFAULT_MAX_ITERATIONS 30 Hard iteration cap
SOFT_ITERATION_RATIO 0.8 Inject wrap-up at this fraction
CONTEXT_LIMIT 60,000 Per-call input token ceiling
OUTPUT_PREVIEW_BYTES 4,096 Preview size for spilled outputs
OUTPUT_TAIL_BYTES 512 Extra tail appended to previews
MAX_INLINE_SIZE 4,096 Max inline size for data source responses

Evaluation Framework

The agent includes an evaluation framework for scientific measurement of prompt and model quality. See Hummingbird Agent Evals for the full guide covering the shared hummingbird_agent.eval library, existing evaluations, and how to write new ones.

Development

See the main README for development workflows.

make hummingbird-agent/setup  # Install dependencies
make check                    # Lint code (ruff)
make fmt                      # Format code
make test                     # Run unit tests
make coverage                 # Run tests with coverage

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

1 - Hummingbird Agent Model Loop

How the Hummingbird agent runs conversation turns, calls tools, and integrates user replies.

For the architectural design rationale behind the loop (why full history replay, why iteration-based budgets, why nudge via system prompt), see Agent Design – Section 6. For operational usage documentation, see Hummingbird Agent. Provider adapters, wire formats, canonical message conversion, and caching are documented in Hummingbird AI.

What gets sent to the model

Every call to the model API sends three things:

1. System instruction

A single text string, rebuilt every iteration. The adapter maps it to the active provider’s request format. It is composed of:

BASE_SYSTEM_PROMPT            # agent.py: sandbox rules, tool usage tips
+ tool_notes                  # per-data-source notes from ToolRegistry
+ workflow_prompt             # full content of e.g. workflows/analyze-failures.md

Near the iteration or context limit, a warning suffix is appended to the system prompt for that call. The suffix escalates through three levels:

  • ITERATION_WARNING / CONTEXT_WARNING – soft “start wrapping up” at 80% of the iteration or context limit. Tools remain available.
  • FINAL_TURN_WARNING – hard stop on the absolute last iteration or after the context limit is exceeded. Combined with tool-calling disabled for that request to mechanically prevent further tool calls.
  • EMPTY_RESPONSE_NUDGE – appended when the model returns an empty response (no text, no tool calls). Tools remain available.

2. contents

A list of messages representing the full conversation history. It grows every iteration. Provider-specific content conversion is handled by the adapter; see Hummingbird AI.

3. tools

Tool definitions, static across all iterations. Adapters translate the shared tool definition into the provider’s schema.

4. Tool availability (conditional)

On the final turn (last iteration or after context limit exceeded), the request disables tools to mechanically prevent function calls. The active adapter applies the provider-specific setting.

This is only sent when FINAL_TURN_WARNING is active. All other iterations omit this constraint, allowing the model to choose freely between text and tools.

The agent loop, turn by turn

The loop in run_agent_loop runs up to max_iterations times. Each iteration is one round-trip to the model.

Initialization

system_prompt = BASE_SYSTEM_PROMPT + tool_notes + workflow_prompt
tool_defs     = [sandbox_exec, fetch_to_sandbox, fetch_batch_to_sandbox, <data sources...>]
contents      = [user: {"project": "org/repo", "iid": 42, "sha": "abc...", "session_id": "uuid"}]

Iteration 1

-> Send: system_instruction + contents (1 item) + tool_defs
<- Response: tool_call(gitlab_get_mr_details, {project: "org/repo", iid: 42})

contents.append(model: response.raw_content)  # opaque provider content
  execute tool -> result = {"result": "{\"title\": \"Fix auth\",...}"}
contents.append(tool: model.make_tool_responses([(name, result)]))

Iteration 2

-> Send: system_instruction + contents (3 items) + tool_defs
<- Response: tool_call(fetch_batch_to_sandbox, {requests: [...]})

contents.append(model: ...)
  execute tool -> result = {"results": [{"saved_to": "/tmp/data/pipelineruns.json", "bytes": 85432}]}
contents.append(tool: ...)

Iteration 3

-> Send: system_instruction + contents (5 items) + tool_defs
<- Response: tool_call(sandbox_exec, {command: "jq '[...]' /tmp/data/pipelineruns.json"})

contents.append(model: ...)
  execute tool -> result = {"exit_code": 0, "stdout": "...(preview)...", "stdout_file": "/tmp/_out/0.txt"}
contents.append(tool: ...)

Iteration N (final)

-> Send: system_instruction + contents (2N-1 items) + tool_defs
<- Response: text("## Konflux Failure Analysis\n\n...")    // text + no tool calls = DONE

contents.append(model: {text: "## Konflux Failure Analysis..."})
BREAK -- return (text, usage, transcript, contents)

Response handling and termination

Each iteration, the model’s response can contain text, tool calls, both, or neither. The agent handles each case:

Response Action
Text only Final response. Save text, break. (Happy path.)
Text + tool calls Save text as last_text fallback, execute the tool calls, continue loop. The model is thinking aloud while also acting.
Tool calls only Execute the tool calls, continue loop.
Neither Empty response – retry up to MAX_EMPTY_RETRIES (2) times. Each retry nudges the model via EMPTY_RESPONSE_NUDGE appended to the system prompt (not injected as a user message, to keep the contents list clean). Tools remain available during nudge retries.

The text + tool calls case is worth noting: the model’s text is not returned to the user immediately. It is stored as last_text (a fallback in case the loop terminates later without a clean final text, e.g. by hitting max iterations). The provider response is appended to contents unchanged; its wire format is documented in Hummingbird AI.

The loop terminates on:

  1. Text only – the model produced a final response.
  2. Max iterations reached – the final iteration uses FINAL_TURN_WARNING and disables tools through the adapter to force text output. Falls back to whatever last_text was seen, or "[Agent did not produce a final response]".
  3. Context limit exceeded – input_tokens >= CONTEXT_LIMIT sets context_exceeded, making the next iteration final (same as #2). A soft warning (CONTEXT_WARNING) fires earlier at 80% of the limit.
  4. Fatal model error – a provider error after shared retry handling is exhausted. Error classification and retries are described in Hummingbird AI.
  5. Empty responses exhausted – MAX_EMPTY_RETRIES (2) nudge retries failed.
  6. Unexpected error – any exception not handled by the retry mechanism (e.g. malformed API response) breaks the loop and returns partial results. Accumulated contents, transcript, and sandbox are preserved and the session is saved normally.

Budget escalation

Both iterations and context size use the same two-tier pattern:

Soft warning (80%) Hard stop (100%)
Iterations ITERATION_WARNING at 80% of max_iterations FINAL_TURN_WARNING on last iteration
Context CONTEXT_WARNING at 80% of CONTEXT_LIMIT FINAL_TURN_WARNING on next iteration after exceeding CONTEXT_LIMIT

Soft warnings are advisory (“start wrapping up”) and keep tools available. The hard stop disables tools through the active adapter to mechanically prevent further tool calls, ensuring the model produces text.

The contents list in detail

The agent treats each entry in contents as opaque adapter content during execution. Provider representations and canonical conversions are documented in Hummingbird AI.

model.make_user_content() constructs the initial event turn and model.make_tool_responses() appends results for dispatched calls. The agent stores the provider response as raw_content and appends it unchanged.

Typical 8-iteration conversation shape

contents[0]  = user:  {"project":"org/repo","iid":42,...}          # initial event
contents[1]  = model: tool_call(gitlab_get_mr_details)             # iter 1
contents[2]  = tool:  result(gitlab_get_mr_details)                # iter 1
contents[3]  = model: tool_call(fetch_batch_to_sandbox)            # iter 2
contents[4]  = tool:  result(fetch_batch_to_sandbox)               # iter 2
contents[5]  = model: tool_call(sandbox_exec)                      # iter 3
contents[6]  = tool:  result(sandbox_exec)                         # iter 3
...
contents[13] = model: tool_call(sandbox_exec)                      # iter 7
contents[14] = tool:  result(sandbox_exec)                         # iter 7
contents[15] = model: text("## Konflux Failure Analysis...")       # iter 8 (final)

This entire list is persisted as canonical messages in context.json; the adapter converts to and from its provider format at the persistence boundary. See Hummingbird AI for details.

How tools are called

When the model’s response contains tool calls, _execute_tool_calls iterates them sequentially:

tool_registry.execute(ToolCall(name, args))
  match name:
     "sandbox_exec"           -> runs sh -c in container -> {exit_code, stdout, stderr}
     "fetch_to_sandbox"       -> calls data source func -> writes to sandbox file -> {saved_to, bytes}
     "fetch_batch_to_sandbox" -> multiple fetch_to_sandbox in one call
     any data source name     -> calls func directly -> returns inline or auto-spills large output

Large output handling

When stdout or a data source response exceeds 4KB, it is automatically saved to a sandbox file (/tmp/_out/N.txt) and only a preview (head + tail) is returned to the model. The model gets the file path and can use sandbox_exec with jq/grep/head to process it. This keeps the context window manageable.

How a user reply integrates (session resumption)

When a user replies to an agent note on GitLab, the orchestrator resumes the conversation by restoring the previous state and appending the reply.

What gets restored from S3

  • context.json – the persisted conversation in canonical form; the active adapter converts it to the in-memory representation before the loop runs
  • sandbox.tar.gz – /tmp/_out/ files (PipelineRun JSONs, test logs, jq output, etc.) restored into the new sandbox

Canonical storage allows sessions to resume with another provider; conversion details are in Hummingbird AI.

The resumed contents list

// Restored from context.json (previous run)
contents[0]  = user:  {"project":"org/repo","iid":42,...}           # original event
contents[1]  = model: tool_call(gitlab_get_mr_details)              # iter 1
contents[2]  = tool:  result(gitlab_get_mr_details)                 # iter 1
...                                                                  # all prior turns
contents[15] = model: text("## Konflux Failure Analysis...")        # previous final text

// NEW: user reply appended
contents[16] = user:  "Can you look at the clair-scan timeout more closely?"

The agent loop continues

ITERATION 1 (resumed):
  -> Send: system_instruction (+ CONTINUATION_PROMPT) + contents[0..16] + tool_defs
  <- Response: tool_call(sandbox_exec, {command: "grep -i timeout /tmp/data/..."})
                                                       ^ using restored sandbox files
  contents[17] = model: tool_call(sandbox_exec)
  contents[18] = tool:  result(sandbox_exec)

ITERATION 2 (resumed):
  <- Response: text("The clair-scan timeout is caused by...")
  contents[19] = model: text("The clair-scan timeout is caused by...")
  BREAK

Cold start vs resumed – key differences

Aspect Cold start Resumed
System prompt BASE + tool_notes + workflow BASE + tool_notes + workflow + CONTINUATION_PROMPT
First user message {"project":..., "iid":...} {"project":..., "iid":...} (restored)
Conversation history Empty (just the event) Full prior conversation
Latest user message – "Can you look at the clair-scan timeout?"
Sandbox files Empty Restored from archive
Tool definitions Same Same
Session format Adapter content in memory Canonical on disk; adapter content after load

The CONTINUATION_PROMPT

Without this, the workflow prompt (e.g. analyze-failures.md) tells the model to follow a rigid workflow: Data.1, Data.2, Data.3, Analysis.1… The model might try to re-run the entire analysis. The continuation prompt overrides this:

## Continuation

This is a follow-up to a previous conversation. The conversation history
contains your prior analysis and tool calls. The user is replying with a
question or request about your previous work.

**Do NOT re-run the full workflow from scratch.** Instead:
- Respond directly to the user's question
- Use your tools to investigate further if needed (logs, data are still
  available in the sandbox)
- Reference your previous findings where relevant
- Keep your response focused on what the user asked

The model now understands: “I already did the analysis (it’s all in the conversation history). The user has a specific question. Let me answer it.”

Iteration and token budget for resumed sessions

The resumed session gets a full fresh iteration budget. run_workflow sets max_iterations from the workflow config (e.g. 50 for analyze-failures), and the loop counter starts at 0 regardless of how many iterations the previous session used. The model can make as many tool calls as it needs to answer the follow-up.

The practical constraint is the context window, not iterations. The restored contents can be large – a typical 8-iteration cold start uses ~50-100K input tokens. Since the model re-counts the full history every iteration, the resumed session starts at roughly the token count where the previous session ended, plus the new user message. Each new tool call adds more. Token counts logged by the agent are billed tokens (cumulative across API calls; each call re-sends the full history, so these overlap).

The existing CONTEXT_LIMIT check still applies: at 80% of the limit, a soft CONTEXT_WARNING tells the model to start wrapping up. If the limit is exceeded, the next iteration becomes final (FINAL_TURN_WARNING with provider tools disabled by the adapter).

For the typical case (user asks one focused follow-up, agent does 1-3 tool calls), this is fine. Multi-turn deep dives will eventually hit the context limit, at which point the agent wraps up – and the user can start a fresh conversation if needed.

Provider behavior

Provider wire formats, canonical message conversion, prompt caching, server-side tools, and adapter continuations are owned by the hummingbird-ai library. The Hummingbird agent loop remains responsible for tool dispatch, iteration/context budgets, and session resumption.

2 - Hummingbird Agent Design

Architectural design document for the hummingbird-agent. Covers the reasoning behind every major design choice so that future changes can be made safely, without accidentally violating invariants that hold the system together.

For operational usage (CLI, config reference, deployment), see Hummingbird Agent. For agent turn flow and tool handling, see Agent Model Loop. Provider adapters and wire formats are documented in Hummingbird AI.

1. Design Philosophy

Five principles shaped the agent’s architecture. Every component traces back to at least one of these.

Security by default. The agent processes untrusted merge requests. All LLM-driven commands run inside isolated sandbox containers with no network and no credentials. Orchestrator write tokens and model read tokens live in separate code paths that never cross. No cluster-admin is required.

Config over code. Investigation logic lives in markdown workflow files that become the LLM system prompt verbatim – changing the analysis strategy requires no code change and no redeployment. Operational settings (data sources, project allowlists, iteration limits) live in a YAML config file that is auditable and committable. Secrets are the only thing in environment variables.

Bounded cost. Every output path has a size cap. Large tool outputs are auto-spilled to sandbox files with only a preview returned to the model. Each session has both an iteration limit and a per-call context token limit with two-tier escalation (soft warning, then hard stop with tool disabling). The relationship between these constants is documented and centralized.

Partial over nothing. When some data is unavailable (expired pod logs, unreachable Konflux cluster, Testing Farm 404), the agent continues with whatever data it has and notes the gap in its output. A partial report is more valuable than a crash.

Model-agnostic agent loop. The agent loop (agent.py) does not inspect the internal structure of the contents list. It appends raw_content from model responses and make_tool_responses() output without looking inside. Each model backend owns its wire format. This makes it possible to add new model backends (Claude, GPT) without touching the agent loop.

2. System Architecture

2.1. Component overview

flowchart TB
    subgraph input [Event Sources]
        SQS["SQS Queue<br/>(gitlab::pipeline, gitlab::note)"]
        CLI["CLI<br/>(--event / --event-file)"]
    end

    subgraph orchestrator [Orchestrator Process]
        Events["events.py<br/>SQS consumer"]
        Runner["runner.py<br/>Event routing"]
        Agent["agent.py<br/>Model loop"]
        Tools["tools.py<br/>Tool registry"]
        Actions["actions.py<br/>GitLab notes"]
        Sessions["sessions.py<br/>Persistence"]
        WfConfig["workflow_config.py<br/>YAML config"]
    end

    subgraph ai [Shared AI Library]
        Models["hummingbird-ai<br/>Gemini / Anthropic adapters"]
    end

    subgraph sandbox [Sandbox Container]
        SB["PodmanSandbox / K8sSandbox / KubeVirtSandbox<br/>jq, python3, yq"]
        SpillFiles["/tmp/data/_out/ spill files"]
        DataFiles["/tmp/data/ working files"]
    end

    subgraph dataSources [Data Source Modules]
        GL["gitlab.py"]
        KX["konflux.py"]
        TF["testing_farm.py"]
    end

    subgraph external [External APIs]
        GitLabAPI["GitLab API"]
        KonfluxAPI["K8s + Kubearchive"]
        TFAPI["Testing Farm API"]
    end

    subgraph storage [Storage]
        S3["S3 Sessions"]
        MRNote["GitLab MR Notes"]
    end

    SQS --> Events --> Runner
    CLI --> Runner
    Runner --> Agent
    Agent --> Tools
    Tools --> SB
    Tools --> dataSources
    dataSources --> external
    Agent --> Models
    Runner --> Actions --> MRNote
    Runner --> Sessions --> S3
    Runner --> WfConfig

2.2. Request lifecycle

A complete run proceeds through these stages:

  1. Event arrival. An SQS message (production) or CLI invocation (dev) provides a GitLab project path and MR IID.

  2. Config lookup. workflow_config.py maps the project to one or more workflows via the project_index. Each match yields a WorkflowConfig and ProjectEntry with data source declarations and per-project settings.

  3. Rate-limit check and SHA dedup. actions.scan_agent_threads() scans MR discussions in a single pass, parsing JSON session markers to determine the per-workflow thread count and whether the current SHA has already been reviewed. If the SHA was already reviewed, the workflow is skipped. If the thread count meets or exceeds max_runs_per_mr, the workflow is skipped. Slash commands and replies bypass this check.

  4. Placeholder note. actions.create_placeholder_note() posts a placeholder so the reviewer knows analysis is in progress. The note contains a JSON session marker (<!-- hummingbird-session: {"id":"UUID","wf":"name","sha":"abc"} -->) for rate limiting, SHA dedup, and session resumption.

  5. Sandbox start. sandbox.create_sandbox() starts a Podman container or K8s pod. /tmp/data/ is pre-created for the model’s use.

  6. Data source registration. data_sources.register_selected() resolves tokens and URLs from the config and registers tool functions on the ToolRegistry. Only data sources declared in the workflow config are registered.

  7. Agent loop. agent.run_agent_loop() runs the model loop: the workflow markdown becomes the system prompt, tool definitions are provided, and the model iterates calling tools and producing text until it emits a final response or hits a budget limit.

  8. Note update. The final text, wrapped with a session marker and reply prompt, replaces the placeholder note.

  9. Session save. Conversation history (contents), transcript, and sandbox archive are saved to S3 (production) or a local directory (dev).

  10. Sandbox cleanup. The container/pod is deleted. K8s pods also have a configurable activeDeadlineSeconds backstop (default 1800s) in case the orchestrator dies.

2.3. Architectural boundaries

The codebase is organized around four strict boundaries:

Runner (runner.py) is the orchestration layer. It owns event routing, the placeholder/update note lifecycle, session save/load, and sandbox lifecycle. It calls agent.run_agent_loop() but never reaches into the agent’s internals.

Agent (agent.py) is the model loop. It knows about the model interface, tool definitions, and the contents list, but nothing about GitLab, SQS, sessions, or actions. It returns (text, usage, transcript, contents) and is completely unaware of what happens with those values.

Tools (tools.py) bridge the agent and the sandbox/data-sources. The agent calls tool_registry.execute(tool_call) and gets a dict back. It never calls sandbox methods directly. This indirection is what enables auto-spill: the tool registry can transparently save large outputs to sandbox files and return previews.

Model providers are supplied by the shared hummingbird-ai library, which owns adapter construction, provider formats, retries, and AI usage accounting. The agent loop uses the library’s normalized adapter interface and treats conversation content as opaque.

3. Module Architecture

3.1. Dependency graph

flowchart TD
    main["__main__.py"]
    config_mod["config.py"]
    wf_config["workflow_config.py"]
    events_mod["events.py"]
    runner_mod["runner.py"]
    agent_mod["agent.py"]
    tools_mod["tools.py"]
    sandbox_mod["sandbox.py"]
    actions_mod["actions.py"]
    sessions_mod["sessions.py"]
    transcript_mod["transcript.py"]
    ds_init["data_sources/__init__.py"]
    ds_gitlab["data_sources/gitlab.py"]
    ds_konflux["data_sources/konflux.py"]
    ds_tf["data_sources/testing_farm.py"]
    config_watch_mod["config_watch.py"]
    ai_pkg["hummingbird-ai library"]

    main --> config_mod
    main --> config_watch_mod
    main --> wf_config
    main --> events_mod
    main --> runner_mod
    main --> sessions_mod
    main --> transcript_mod
    main --> actions_mod

    config_watch_mod --> config_mod
    config_watch_mod --> wf_config

    runner_mod --> actions_mod
    runner_mod --> agent_mod
    runner_mod --> config_mod
    runner_mod --> ds_init
    runner_mod --> sandbox_mod
    runner_mod --> sessions_mod
    runner_mod --> tools_mod
    runner_mod --> ai_pkg
    runner_mod --> wf_config
    runner_mod --> transcript_mod

    agent_mod --> config_mod
    agent_mod --> ai_pkg
    agent_mod --> tools_mod

    tools_mod --> config_mod
    tools_mod --> ai_pkg
    tools_mod --> sandbox_mod

    ds_init --> tools_mod
    ds_init --> wf_config
    ds_init --> ds_gitlab
    ds_init --> ds_konflux
    ds_init --> ds_tf

    ds_gitlab --> ai_pkg
    ds_konflux --> ai_pkg
    ds_tf --> ai_pkg

    sessions_mod --> sandbox_mod

3.2. Module responsibilities

Each module has a single, well-defined responsibility. The boundary rules below are invariants – violating them breaks the security model or the separation of concerns.

config.py – Runtime configuration and constants

  • Owns: Config dataclass and agent tuning constants (DEFAULT_MAX_ITERATIONS, CONTEXT_LIMIT, OUTPUT_PREVIEW_BYTES, etc.)
  • Boundary: Pure data. No I/O except reading env vars in Config.load(). No imports from other Hummingbird agent modules; model setup and pricing are owned by hummingbird-ai.
  • Invariant: All interdependent budget/spill constants must be defined here with their relationship documented in the comment block.

workflow_config.py – YAML config loader and token resolution

  • Owns: AgentConfig, WorkflowConfig (with trigger, description, trigger_rules, ignore_users, ignore_branches fields), TriggerRule, ProjectEntry, DataSourceConfig dataclasses; load(), validate_env(), evaluate_trigger_rules(), resolve_tool_token(), resolve_cluster_url(), get_prompt().
  • Boundary: Reads YAML from disk and env vars. Never instantiates network clients or model objects.
  • Invariant: resolve_tool_token() resolves model tool tokens only. It must never return orchestrator tokens. The resolution chain is: project_entry.tokens[ds] -> workflow_cfg.data_sources[ds].token_env -> "" (empty).

events.py – SQS consumer

  • Owns: decode_sns_message() (SNS envelope decoding), poll_loop() (blocking SQS consumer with concurrency control).
  • Boundary: Knows about SQS/SNS wire formats. Calls a generic EventHandler callback. Does not know about GitLab, workflows, or the agent.
  • Invariant: Failed messages are not deleted from SQS (they go to the DLQ after visibility timeout expires). Successful messages are deleted after handler returns.

config_watch.py – Background config hot-reload

  • Owns: ConfigHolder (thread-safe config pair with atomic swap), start_watcher() (daemon thread that polls config file mtime), _watch_loop().
  • Boundary: Knows about config.Config.load() and workflow_config.load(). Does not know about events, the agent, or any runtime state.
  • Invariant: On reload failure (parse error, missing env var), the previous config is kept and a warning is logged. The watcher never crashes the serve loop.

runner.py – Event routing and workflow execution

  • Owns: handle_event(), handle_pipeline(), handle_merge_request(), handle_note(), _execute_workflow(), _handle_reply(), run_workflow(), _acquire_sandbox(), _resolve_slash_workflow(), _format_help(), _is_ignored_user(), WorkflowRequest, WorkflowResult, SandboxOpts.
  • Boundary: Orchestrates everything: config lookup, rate limiting, placeholder notes, sandbox lifecycle, agent invocation, session save. This is the only module that touches both actions and agent.
  • Invariant: run_workflow() either lingers the sandbox (success) or cleans it up (exception).

agent.py – Model-agnostic tool-calling loop

  • Owns: run_agent_loop(), system prompt constants (BASE_SYSTEM_PROMPT, CONTINUATION_PROMPT, warning/nudge strings), budget logic.
  • Boundary: Uses the ModelAdapter interface from hummingbird-ai and tools (for execute()). Does not know about GitLab, SQS, sessions, or actions, or which backend is in use.
  • Invariant: The contents list is treated as opaque. Items are appended via response.raw_content and model.make_tool_responses(). The agent never inspects, modifies, or deletes items in the list (except for empty response retries, which pop() the last appended item before the model has seen any tool results for it).

tools.py – Tool registry

  • Owns: ToolRegistry class with sandbox_exec, fetch_to_sandbox, fetch_batch_to_sandbox built-in tools; data source registration and dispatch; auto-spill logic.
  • Boundary: Owns the sandbox reference and all tool execution. The agent never touches the sandbox directly.
  • Invariant: _spill_counter is shared across all spill paths (sandbox exec stdout, sandbox exec stderr, data source auto-spill) to prevent filename collisions in /tmp/data/_out/.

sandbox.py – Sandbox backends

  • Owns: Sandbox protocol, PodmanSandbox, K8sSandbox, K8sPoolSandbox, SandboxPool, KubeVirtSandbox, VmiPool, KubeVirtPoolSandbox, create_sandbox(), ExecResult.
  • Boundary: Knows about container/pod/VM lifecycle and command execution. Does not know about tools, models, or the agent.
  • Invariant: All backends must implement the same 8-method protocol (start, exec, write_file, stream_to_file, stream_exec, read_file_iter, cleanup, linger). exec() accepts optional stdin_data for piping raw bytes. All must pre-create /tmp/data/ in start(). All must use stdin piping for write_file() (never host volume mounts).

actions.py – GitLab note lifecycle

  • Owns: Orchestrator token resolution (resolve_orchestrator_token()), workflow action token client creation (_make_client()), bot user ID collection (collect_workflow_bot_ids()), note CRUD, JSON marker parsing (marker_tag, parse_marker), scan_agent_threads (per-workflow rate limiting + SHA dedup), post_simple_reply, member access checks, session-for-reply lookup, AgentThreadInfo, SessionRef.
  • Boundary: Functions are split by token tier:
    • Orchestrator token (operational reads + infrastructure notes, use _get_client): check_member_access, get_latest_push_author, get_head_pipeline_id, is_first_note_in_discussion, reply_access_denied, notify_rate_limit, post_simple_reply.
    • Workflow action token (per-workflow-attributable writes, take explicit token: str): create_placeholder_discussion, post_discussion_note, resolve_agent_threads, scan_agent_threads. Never touches model tool tokens, the tool registry, or the agent.
  • Invariant: Orchestrator tokens are resolved from ORCHESTRATOR_* env vars only. Workflow action tokens are resolved from action_tokens in YAML and passed as explicit parameters. Neither tier is exposed to the model.

sessions.py – Session persistence

  • Owns: archive_sandbox(), save_local(), save_s3(), load_s3(), load_local(), restore_sandbox(), SessionData.
  • Boundary: Knows about sandbox (for archiving) and S3/filesystem (for storage). Does not know about the agent, tools, or models. Load/save uses canonical envelopes with format_version: 1; unversioned legacy sessions are not supported.
  • Invariant: The sandbox archive is never extracted on the orchestrator. It is created inside the sandbox (tar czf), streamed out via read_file_iter(), and restored inside a new sandbox via stream_exec("tar xzf -").

transcript.py – Transcript rendering

  • Owns: render_markdown(), per-tool rendering functions, truncation.
  • Boundary: Pure transformation from TranscriptEntry list to markdown. No I/O, no side effects.

data_sources/__init__.py – Data source registration

  • Owns: register_selected() – resolves tokens/URLs from config and calls each data source’s register() function.
  • Boundary: Bridges workflow_config (for token/URL resolution) and tools (for registration). Only registers data sources declared in the workflow config.

data_sources/gitlab.py, konflux.py, testing_farm.py

  • Own: register() function that creates tool definitions and closures over credentials, and registers them on the ToolRegistry.
  • Boundary: Each module talks to one external API. Credentials are captured at registration time via closure, never stored globally.

4. Security Model

The agent runs in a shared OpenShift cluster without cluster-admin access, processing merge requests from repositories where external contributors can submit code. The security model addresses two threat vectors: (1) the LLM executing arbitrary commands chosen by an attacker-controlled MR, and (2) credential leakage between the model, the sandbox, and orchestrator actions.

4.1. Sandbox isolation

All commands generated by the LLM run inside an ephemeral container, never on the orchestrator host. The sandbox has no credentials, no network, and no visibility into the orchestrator.

flowchart LR
    subgraph orchestrator [Orchestrator]
        Agent["Agent Loop"]
        Creds["Secrets<br/>(tokens, kubeconfig)"]
    end
    subgraph sandbox [Sandbox Container]
        Shell["sh -c commands"]
        Files["/tmp/data/<br/>(incl. _out/ spill dir)"]
    end
    Agent -->|"stdin pipe<br/>(write_file)"| sandbox
    Agent -->|"exec command"| sandbox
    sandbox -->|"stdout/stderr"| Agent
    sandbox -.-x|"NO network"| Internet["Internet / K8s API"]
    sandbox -.-x|"NO access"| Creds

Podman (local development):

  • --network=none – complete network isolation; curl, wget, pip install all fail
  • --user 65532 – fixed non-root UID; no privilege escalation
  • No host volume mounts – data enters only via stdin piping through write_file()

Kubernetes (production):

Every field in the pod manifest is set explicitly for portability to vanilla Kubernetes with Pod Security Admission (restricted level), not just reliance on OpenShift’s restricted-v2 SCC admission:

  • automountServiceAccountToken: false – no K8s API access from sandbox
  • runAsNonRoot: true – enforced at pod level
  • seccompProfile: RuntimeDefault – required by restricted level
  • allowPrivilegeEscalation: false – container level
  • capabilities.drop: ["ALL"] – container level
  • activeDeadlineSeconds: 1800 – pod self-terminates after 30 minutes even if the orchestrator crashes; prevents orphaned pods
  • restartPolicy: Never – pod does not restart on failure
  • NetworkPolicy on the sandbox namespace blocks all egress from all pods in the namespace (podSelector: {}, egress: [])

The pod does not set runAsUser explicitly. On OpenShift, the SCC assigns a UID from the namespace range. On vanilla K8s, the image’s USER directive is used.

4.2. Credential separation (four-tier token model)

Tokens are split into four tiers with strict code-path separation:

flowchart TB
    subgraph tier1 [Tier 1: Orchestrator Tokens]
        OT["ORCHESTRATOR_GITLAB_TOKEN_*<br/>Operational reads<br/>Env vars only, never in YAML"]
    end
    subgraph tier2 [Tier 2: Workflow Action Tokens]
        AT["action_tokens in YAML<br/>Per-workflow bot identity<br/>Write scope (api)"]
    end
    subgraph tier3 [Tier 3: Model Tool Tokens]
        MT["GITLAB_TOKEN_RO, etc.<br/>Read scope (read_api)<br/>Env var NAMES in YAML"]
    end
    subgraph tier4 [Tier 4: Sandbox]
        SB["Zero credentials<br/>Zero network<br/>Zero SA token"]
    end

    OT -->|"used by"| OpsReads["actions.py<br/>(member access, push author,<br/>rate-limit notices)"]
    AT -->|"used by"| WfWrites["actions.py<br/>(notes, thread resolution)"]
    MT -->|"used by"| DataSources["data_sources/<br/>(GitLab, Konflux, TF)"]
    SB -->|"used by"| SandboxExec["sandbox_exec<br/>(sh -c commands)"]

    OpsReads -.-x|"NEVER"| DataSources
    WfWrites -.-x|"NEVER"| DataSources
    WfWrites -.-x|"NEVER"| SandboxExec
    DataSources -.-x|"NEVER"| OpsReads
    DataSources -.-x|"NEVER"| WfWrites

Tier 1 – Orchestrator tokens. Tokens used by actions.py for operational reads (member access checks, push author lookup, head pipeline queries) and infrastructure notes (access-denied replies, rate-limit notices). Resolved by convention from env vars: ORCHESTRATOR_GITLAB_TOKEN_<MANGLED_PROJECT> (per-project) or ORCHESTRATOR_GITLAB_TOKEN (global fallback). The ORCHESTRATOR_ prefix is a structural safeguard – these env var names can never appear in the YAML config’s data_sources, tokens, or action_tokens sections. These tokens can be scoped to read_api since all per-workflow writes moved to Tier 2.

Tier 2 – Workflow action tokens. Per-workflow bot identity tokens (api scope) used by actions.py for all workflow-attributable GitLab writes: placeholder notes, discussion notes, and thread resolution. Each workflow gets a dedicated bot user per project, providing clear audit trails for which agent produced each note. Declared in the YAML config as env var names at the project level:

projects:
  redhat/hummingbird/containers:
    action_tokens:
      gitlab: HUMMINGBIRD_AGENT_ACTION_CODE_REVIEW_GITLAB_TOKEN_CONTAINERS

Resolved by resolve_action_token(project_entry, ds_name) which reads the env var name from action_tokens and resolves it from os.environ. These tokens are passed as explicit token: str parameters to write functions in actions.py. They NEVER enter the ToolRegistry or model tool calls.

At startup, collect_workflow_bot_ids() authenticates each unique workflow token to build a set of all workflow bot user IDs. This set is used by handle_note for self-filtering (skipping notes from any workflow bot) and by find_session_for_reply for session marker scanning.

The naming convention groups tokens alphabetically: HUMMINGBIRD_AGENT_ACTION_<WORKFLOW>_GITLAB_TOKEN_<PROJECT>.

Tier 3 – Model tool tokens. Read-only tokens (read_api scope for GitLab) used by data source modules during tool execution. Declared in the YAML config as env var names (not values):

data_sources:
  gitlab:
    token_env: GITLAB_TOKEN_RO # name of the env var

Per-project overrides are possible:

projects:
  redhat/hummingbird/containers:
    tokens:
      gitlab: GITLAB_TOKEN_CONTAINERS_RO # overrides token_env for this project

The resolution chain in resolve_tool_token() is: project_entry.tokens[ds] -> workflow_cfg.data_sources[ds].token_env -> ""

Storing env var names (not values) in YAML means the config file is safe to commit and audit. Actual secret values live in environment variables, injected via K8s Secrets at deployment time.

Tier 4 – Sandbox. The sandbox container has zero credentials, zero network access, and automountServiceAccountToken: false (no K8s API access). Data enters the sandbox only via write_file() (stdin piping). The model cannot instruct the sandbox to reach external APIs – it must use the orchestrator’s data source tools.

4.3. Namespace separation

Sandbox pods run in a dedicated namespace (hummingbird--agent-sandbox), separate from the orchestrator namespace (hummingbird--internal). This limits blast radius: even if a sandbox pod is compromised, it has no visibility into the orchestrator’s Secrets, Pods, or ServiceAccount tokens.

RBAC setup:

The orchestrator’s ServiceAccount gets a Role in the sandbox namespace (not its own namespace) granting only:

  • pods: create, get, list, delete, patch – sandbox pod lifecycle and pool claims
  • pods/exec: create – command execution via kubectl exec

No CRDs, no custom runtimes, no cluster-scoped resources. The orchestrator needs only namespace-scoped permissions, so it works with standard OpenShift RBAC without requesting cluster-admin.

Konflux data is fetched via bearer token from kubeconfig credentials, not from inside the cluster. The orchestrator’s ServiceAccount does not need access to Konflux namespaces.

NetworkPolicy:

A deny-all-egress NetworkPolicy in the sandbox namespace uses podSelector: {} to match all pods and sets egress: []. The sandbox cannot reach the internet, the K8s API, or other pods in the cluster.

4.4. Security invariants

These must hold for the security model to be effective. Any change that violates one of these is a security regression:

  1. Orchestrator tokens must NEVER flow into ToolRegistry, data_sources, or model contents. They are resolved in actions.py only.

  2. Workflow action tokens must NEVER flow into ToolRegistry, data_sources, or model contents. They are passed as explicit token: str parameters within actions.py and runner.py only.

  3. Model tool tokens must NEVER flow into actions.py. They are resolved in workflow_config.py and consumed in data_sources/.

  4. The sandbox must NEVER have network access or credentials. No host mounts, no SA token, no env vars with secrets.

  5. The sandbox archive is NEVER extracted on the orchestrator. It is created inside one sandbox and restored inside another. The orchestrator only transports the bytes.

  6. No cluster-admin required. Only namespace-scoped resources (Role, RoleBinding, Pod, NetworkPolicy) are used.

  7. Agent-generated notes are skipped on re-processing. Notes containing SESSION_MARKER_PREFIX are filtered out in handle_note() to prevent infinite loops. Self-filtering checks against all workflow bot user IDs (collected at startup) plus the orchestrator bot ID.

  8. All auto-triggers require Developer access. handle_pipeline(), handle_merge_request(), and handle_note() each check check_member_access() on the event’s user before executing a workflow. Events from non-developers are silently skipped (or replied to with an access-denied message for slash commands). This ensures that the target MR’s work products – its diff, description, CI logs, and commit messages – originate from a trusted author. Fork MRs from external contributors are blocked because the MR author lacks Developer+ on the target project.

    Scope limitation: this gate only covers the target MR. Once the agent is running, the model controls tool arguments and can direct tools at content beyond the target MR – other MRs, other refs, other job IDs, even other projects reachable by the read-only token. The current mitigations are: (a) workflow prompts instruct the model to use the event’s {project, iid, sha}, (b) the model would need to be manipulated via prompt injection from already-trusted content to deviate, (c) the sandbox prevents the model from acting on manipulated reasoning beyond producing text output, and (d) the token is read-only with minimal scope. However, if a data source tool is added that returns user-authored prose from arbitrary resources (e.g. issue bodies, wiki pages), it should apply per-author trust filtering (invariant #10).

  9. Data source tools that accept URLs must validate them against an allowlist. tf_get_test_log restricts URLs to the Testing Farm artifacts prefix to prevent the LLM from directing the orchestrator to fetch arbitrary URLs (SSRF).

  10. Third-party commentary entering the model prompt must be trust-filtered. Data sources that feed text from users other than the event trigger into contents (discussion comments, issue bodies) must gate on project membership at Developer+ level per author. Content from untrusted authors must be redacted to a fixed placeholder, never sanitized or escaped – there is no reliable way to escape adversarial text for an LLM prompt. Discussions where all notes are untrusted must be dropped entirely. See §9.1 for the reference implementation in gitlab_get_mr_discussions.

This invariant covers commentary (what other people said about the MR), not work products (the diff, CI logs, commit messages). Work products are inherently the input to the agent’s analysis and cannot be content-filtered without defeating the agent’s purpose. For the target MR, their trust comes from invariant #8 (the event trigger is Developer+). For content the model fetches beyond the target MR, trust depends on the tool: discussion tools must filter per-author (#10), while code/log/metadata tools rely on the mitigations described in #7.

5. Configuration System

5.1. Two sources, strict separation

Configuration comes from exactly two sources with no overlap:

  • YAML config file (CONFIG_PATH): operational settings, workflow definitions, project allowlists, data source declarations, and token env var names. This file is safe to commit, review, and audit.
  • Environment variables: secrets only (API keys, GitLab tokens, kubeconfig paths). These are injected at deployment time via K8s Secrets.

This split exists because YAML provides structure, validation, and audit trails, while secrets must stay out of version control.

5.2. Config loading

Two config objects are built at startup:

Config (from config.py): loaded by Config.load(settings), where settings is the settings: section from the YAML file. Auth/bootstrap fields come from env vars (GOOGLE_API_KEY, GOOGLE_CLOUD_PROJECT, etc.). Operational fields come from the settings dict with sensible defaults.

AgentConfig (from workflow_config.py): loaded by workflow_config.load(path). Contains all workflow definitions, project entries, and a pre-built project_index.

The two are kept separate because Config is needed everywhere (model construction, sandbox creation, session storage), while AgentConfig is only needed for event routing and data source registration.

5.3. YAML config structure

settings:
  gitlab_url: https://gitlab.com # operational, not a secret
  sandbox:
    image: quay.io/.../image:tag
    namespace: my-namespace # K8s only
  model: gemini-3.1-pro-preview
  # model: claude-sonnet-4-5@20250929   # Anthropic via Vertex AI (alternative)
  max_iterations: 30
  max_runs_per_mr: 5
  internal_notes: true
  max_concurrent_agents: 4
  sqs_queue_url: "" # empty = no SQS (local dev)
  s3_session_bucket: "" # empty = no S3 (local dev)
  trigger_prefix: /hummingbird # slash command prefix
  session_marker_prefix: hummingbird-session # HTML comment marker ID
  auto_trigger: true # auto-run on failed pipelines

workflows:
  analyze-failures:
    prompt: workflows/analyze-failures.md # relative to config file dir
    action: post_gitlab_note
    model: gemini-3.1-pro-preview # per-workflow override
    max_iterations: 50 # per-workflow override

    data_sources:
      gitlab:
        token_env: GITLAB_TOKEN_RO # env var name, not the value
      konflux:
        cluster_url: https://example.com:6443/ns/my-tenant
        kubeconfig_env: KUBECONFIG
        kubearchive_url: https://kubearchive-api-server-product-kubearchive.apps.example.com
      testing_farm: {}

    projects:
      redhat/hummingbird/containers:
        tokens: # per-project token overrides
          gitlab: GITLAB_TOKEN_CONTAINERS_RO

Design choices in this structure:

Workflow-first organization. Each workflow owns its project list, not the other way around. This scopes data source permissions per workflow-project pair. A future code-review workflow can have different GitLab tokens (with different scopes) than the analyze-failures workflow, with no ambiguity.

Token env var names in YAML (not values). The YAML file is committed to the repo. It contains token_env: GITLAB_TOKEN_RO (a name), not the actual token. The actual secret value is resolved at runtime via os.environ.get(env_var). This allows the config to be reviewed and audited without exposing secrets.

Inline cluster_url only. The Konflux cluster URL is operational configuration (it identifies which cluster to talk to), not a secret. Putting it inline in YAML makes it visible and auditable. The URL must include the namespace path (e.g. /ns/my-tenant).

5.4. Project index

At load time, _build_project_index() constructs a reverse lookup:

project_index: dict[str, list[tuple[str, WorkflowConfig, ProjectEntry]]]
# e.g. {"redhat/hummingbird/containers": [("analyze-failures", wf_cfg, proj_entry)]}

This provides O(1) lookup when a pipeline event arrives with a project path. A single project can appear in multiple workflows (e.g. both analyze-failures and a future code-review), and each matching workflow will be triggered independently.

5.5. Token resolution chains

Model tool tokens (for data source API calls):

resolve_tool_token(project_entry, ds_name, workflow_cfg):
  1. project_entry.tokens[ds_name]           -> per-project override
  2. workflow_cfg.data_sources[ds_name].token_env  -> workflow default
  3. "" (empty string)                        -> no token
  Each step resolves the env var NAME, then reads os.environ[name].

Workflow action tokens (for per-workflow GitLab writes):

resolve_action_token(project_entry, ds_name):
  1. project_entry.action_tokens[ds_name]  -> env var NAME
  2. os.environ[name]                      -> resolved token value
  3. "" (empty)                            -> not configured
  No fallback chain. Missing tokens cause validate_env to fail at startup.

Orchestrator tokens (for operational reads and infrastructure notes):

resolve_orchestrator_token(project_path):
  1. ORCHESTRATOR_GITLAB_TOKEN_<MANGLED_PROJECT>  -> per-project
  2. ORCHESTRATOR_GITLAB_TOKEN                     -> global fallback
  Mangling: "/" -> "_", "-" -> "_", uppercase.
  e.g. "redhat/hummingbird/containers" -> ORCHESTRATOR_GITLAB_TOKEN_REDHAT_HUMMINGBIRD_CONTAINERS

Cluster URL (for Konflux):

resolve_cluster_url(ds_cfg):
  -> ds_cfg.cluster_url (inline value in YAML)

5.6. Environment validation

validate_env(agent_cfg) checks at startup that all referenced env vars exist. It walks every workflow’s data sources, project token overrides, and project action token entries, collecting missing vars into a single error message. This catches configuration errors early instead of failing mid-run when a specific data source or workflow action is first used. Missing action_tokens env vars are treated the same as missing model tool tokens: startup fails unconditionally.

See Also

3 - Hummingbird Agent Design — Engine

Agent loop, tool system, and sandbox backends for the Hummingbird Agent. This page covers the execution engine: how the agent iterates, dispatches tools, and isolates execution. For foundations (design philosophy, system architecture, modules, security, config), see Hummingbird Agent Design.

6. Agent Loop Design

For the wire-format walkthrough (what bytes go to the model API, what comes back), see Agent Model Loop. This section covers the design rationale behind the loop.

6.1. Full history replay

The Gemini and Anthropic APIs are stateless. Every call sends the complete contents list from the beginning of the conversation. This means every large tool output sitting in history inflates every subsequent API call.

This property is fundamental to why auto-spill exists (section 7). Without auto-spill, a single cat of a 32KB file early in the conversation adds ~8K tokens to every remaining API call. Over a 15-iteration run, that is ~120K wasted tokens.

The alternative – conversation compaction (replacing old tool results with summaries) – was considered and deferred. Each provider has strict requirements about content structure, and modifying history risks confusing the model or violating API constraints. Provider-specific format requirements are documented in Hummingbird AI. Auto-spill solves 90% of the problem with none of the risk.

6.2. Budget model: iterations + context limit

The agent uses a dual-limit approach rather than a cumulative token budget:

Iteration limit (max_iterations, default 30 per workflow config). Hard cap on the number of model round-trips. This is the primary cost control lever. With auto-spill keeping per-call context bounded, iteration count is roughly proportional to cost.

Per-call context limit (CONTEXT_LIMIT, default 60,000 tokens). Checked after each API call using response.usage.total_input_tokens. This is a safety net for cases where auto-spill is not sufficient (e.g., many small tool results that individually stay under the spill threshold but cumulatively fill the context).

Why not a cumulative token budget? Because with full history replay, each API call re-sends everything. “Cumulative billed tokens” double-counts: call 1 sends 5K, call 2 sends 10K (including the 5K again), so billed total is 15K but actual new content is only 10K. Iteration count is a simpler and more predictable proxy for cost.

6.3. Two-tier budget escalation

Both limits use the same escalation pattern:

flowchart LR
    Normal["Normal<br/>tools available"]
    Soft["Soft Warning (80%)<br/>ITERATION_WARNING or<br/>CONTEXT_WARNING<br/>tools still available"]
    Hard["Hard Stop (100%)<br/>FINAL_TURN_WARNING<br/>toolConfig: NONE"]

    Normal -->|"80% reached"| Soft
    Soft -->|"100% reached"| Hard

Soft warning (80%). An ephemeral user message (ITERATION_WARNING) is injected into contents for that turn only, then popped before the response is persisted. Tools remain available so the model can finish in-progress work.

Hard stop (100%). FINAL_TURN_WARNING is injected as an ephemeral user message AND tool_defs is set to [] (empty list) to physically prevent further tool calls. The model must produce text. This is more reliable than relying on provider-side tool disabling, which models may ignore.

6.3.1 Ephemeral messages and provider normalization

All per-turn warnings use ephemeral user messages: they are appended to contents before the API call and popped immediately after. This keeps the system prompt stable across all iterations and prevents warnings from polluting the conversation history saved in sessions.

Provider adapters normalize provider-specific conversation requirements while preserving the agent’s saved history; details are in Hummingbird AI.

6.4. Empty response handling

Models occasionally return unusable responses (no text and no valid tool calls). The agent retries up to MAX_EMPTY_RETRIES (2) times:

  1. Pop the empty response from contents. It adds nothing and may confuse the model on the next call.
  2. Inject EMPTY_RESPONSE_NUDGE as an ephemeral user message on the next turn: “Your previous response was empty. Please continue…”
  3. Continue the loop (consuming an iteration).

The nudge is delivered as an ephemeral user message (injected before the API call and popped after). This avoids mutating the system prompt and keeps the conversation history clean for session persistence.

The empty_retries counter resets to 0 after any successful iteration (one where tool calls were executed). This means the model gets fresh retries if it produces empty responses at different points in the conversation.

MALFORMED_FUNCTION_CALL handling. Models sometimes return a finishReason of MALFORMED_FUNCTION_CALL (Gemini) with no usable tool calls. This is treated as a special case of empty response: the retry mechanism kicks in, but the nudge is replaced with MALFORMED_CALL_NUDGE which tells the model to retry with simpler arguments and avoid large text payloads in tool call arguments.

6.5. Error recovery

Model API and transport errors. The hummingbird-ai library classifies provider and transport failures, applies the workflow’s retry budget, and records AI usage and request metrics. See Hummingbird AI for provider-specific behavior. The agent loop retains partial results when a terminal error reaches it.

Unexpected loop errors. run_agent_loop wraps the _generate_with_retry call in a try/except Exception that logs and breaks instead of propagating. This ensures the function always returns partial results (accumulated contents, transcript, sandbox state) even when an unexpected exception occurs (e.g. JSONDecodeError from a truncated API response). The caller saves the session normally – conversation history and sandbox archive are preserved for resumption. This follows the “partial over nothing” principle: 32 iterations of work are more valuable than a crash.

No failure-path session save. _execute_workflow does not save a session when run_workflow raises. With the agent loop catching unexpected errors, the failure path only fires for infrastructure errors (sandbox start, config) where there is no useful state. Not saving avoids overwriting a previous good session when a reply attempt fails.

Tool execution errors. _execute_tool_calls() wraps each tool_registry.execute() in a try/except. Unhandled exceptions are caught and returned to the model as {"error": "Tool X failed: ..."}. This prevents a single broken tool from crashing the entire session – the model sees the error and can adapt.

Sandbox exec timeout. If a command exceeds EXEC_TIMEOUT (120s), the TimeoutExpired exception is caught in the tool registry and returned as a structured error dict. The model can retry with a different command or proceed without the result.

6.6. Session resumption

When resuming from a previous session:

  1. initial_contents (the full contents list from the previous run) is prepended, followed by the user’s reply as a new user turn.
  2. CONTINUATION_PROMPT is appended to the system prompt.
  3. The sandbox is restored from the archived tarball.
  4. A fresh iteration budget starts from 0.

Saved sessions use a versioned JSON envelope (format_version: 1) whose contents are in a canonical message format independent of the active model. Unversioned legacy sessions are not supported. On resume, run_workflow asks the active adapter to load canonical history; after a successful run, it converts native contents back to canonical form before persistence. The library documents the canonical format and provider conversions in Hummingbird AI.

CONTINUATION_PROMPT is critical. Without it, the workflow prompt (e.g., analyze-failures.md) tells the model to follow a rigid multi-phase workflow: Data.1, Data.2, Analysis.1… The model would try to re-run the entire analysis. The continuation prompt overrides this: “Do NOT re-run the full workflow. Respond directly to the user’s question.”

The practical constraint on resumed sessions is the context window, not iterations. The restored history from an 8-iteration cold start uses ~50-100K input tokens. Each new tool call adds more. The existing CONTEXT_LIMIT check still applies and will force wrap-up if the context grows too large.

Canonical session format (rationale)

Storing sessions in canonical form decouples persisted history from any one provider’s JSON shape. The agent loop treats native contents as opaque; the library owns provider conversion at load and save boundaries.

6.7. Provider caching and pricing

Provider-specific prompt caching and token-cost estimation are owned by hummingbird-ai.

7. Tool System and Auto-Spill

7.1. Tool registry design

ToolRegistry is the single point of dispatch for all tool calls. The agent calls registry.execute(tool_call) and gets a dict back. It never calls sandbox methods or data source functions directly.

Three built-in tools are always available:

  • sandbox_exec – runs sh -c <command> in the sandbox container. Returns {exit_code, stdout, stderr}, with auto-spill for large outputs.
  • fetch_to_sandbox – calls a data source function and writes the result to a specified path in the sandbox. Returns metadata only (path, byte count, line count). Used when the model wants explicit path control.
  • fetch_batch_to_sandbox – calls multiple fetch_to_sandbox in one tool call. Saves iterations vs. sequential calls (e.g., fetching both pipelineruns and taskruns in one round-trip).

Data source tools are registered dynamically per workflow config. Each data source module provides a register() function that creates ToolDef objects and closures over credentials, then calls registry.register_data_source(tool_def, func, response_metadata).

7.2. Auto-spill architecture

Auto-spill is the key mechanism for keeping the LLM context bounded. Without it, large outputs accumulate in the contents list and inflate every subsequent API call (because both APIs replay the full history).

flowchart TD
    Output["Tool produces output"]
    SizeCheck{"size > OUTPUT_PREVIEW_BYTES<br/>(4 KB)?"}
    Inline["Return full output inline"]
    Spill["Write full output to<br/>/tmp/data/_out/N.txt"]
    Preview["Return preview<br/>(head + tail) +<br/>file path metadata"]

    Output --> SizeCheck
    SizeCheck -->|"<= 4 KB"| Inline
    SizeCheck -->|"> 4 KB"| Spill --> Preview

Three spill paths share the same _spill_counter to prevent filename collisions:

sandbox_exec spill. When stdout or stderr exceeds OUTPUT_PREVIEW_BYTES (4096 bytes), the full output is written to /tmp/data/_out/{counter}.txt via _spill_field(). The model receives:

{
  "exit_code": 0,
  "stdout": "<first 4KB>",
  "stdout_truncated": true,
  "stdout_file": "/tmp/data/_out/0.txt",
  "stdout_bytes": 32768,
  "stdout_lines": 1024,
  "stdout_tail": "<last 512 bytes>"
}

The preview (head + tail) gives the model enough context to decide whether to process the full file with jq/grep/head.

Data source inline spill. When a direct data source call returns text larger than MAX_INLINE_SIZE (4096 bytes), _spill_data_source() writes it to /tmp/data/_out/{name}_{counter}.txt and returns:

{
  "saved_to": "/tmp/data/_out/konflux_list_pipelineruns_1.txt",
  "bytes": 85432,
  "lines": 2100,
  "preview": "<first 4KB>"
}

Streaming responses (e.g., gitlab_get_repo_archive which returns a StreamingResponse with suffix .tar.gz) are written via _spill_streaming(). Text streams (binary=False, the default) capture a UTF-8 preview from the head while piping chunks to the sandbox. Binary streams (binary=True) skip preview capture entirely and return only {saved_to, bytes}, avoiding meaningless decoded output for formats like tar.gz. The StreamingResponse.suffix field controls the file extension (.jsonl, .tar.gz, .log).

Non-streaming binary responses are written to .bin files with no preview via _spill_binary().

Sandbox-optional mode. ToolRegistry accepts sb=None for use cases that do not require a sandbox (e.g. evaluation harnesses that only use native model tools like Google Search grounding). When no sandbox is available:

  • Sandbox tools (sandbox_exec, fetch_to_sandbox, fetch_batch_to_sandbox) are omitted from get_tool_defs() so the model never sees them.
  • Text data sources (inline and streaming) degrade to truncated previews (OUTPUT_PREVIEW_BYTES). The model receives {preview, truncated: true} instead of {saved_to, preview}. Streaming text is consumed into a memory buffer up to the preview limit and the rest is discarded.
  • Binary data (streaming and non-streaming) returns an error with the byte count ({error, bytes}). Binary content cannot be meaningfully previewed as text.

fetch_to_sandbox spill. Always writes to the caller-specified path (not /tmp/data/_out/). Returns metadata only (no preview). The model uses this when it wants a specific filename for later processing.

7.3. Why auto-spill instead of rejecting large outputs

The earlier design rejected large data source responses with “use fetch_to_sandbox instead.” This caused two wasted iterations per rejection: the model makes the call, gets rejected, then has to repeat with fetch_to_sandbox. With auto-spill, the data flows to a sandbox file transparently. Validation showed ~10% fewer iterations with auto-spill.

7.4. Why fetch_to_sandbox still exists alongside auto-spill

Auto-spill handles the common case, but fetch_to_sandbox provides:

  • Explicit path control. The model can choose meaningful filenames (/tmp/data/pipelineruns.json) instead of getting auto-generated names (/tmp/data/_out/konflux_list_pipelineruns_1.txt).
  • Batch fetching. fetch_batch_to_sandbox combines multiple fetches in one tool call, saving iterations.
  • No preview overhead. fetch_to_sandbox returns metadata only, which is useful when the model knows it will process the file with sandbox_exec anyway.

7.5. ToolDef notes

ToolDef has an optional notes field for domain knowledge that belongs in the system prompt but not in the tool’s JSON schema. Examples:

  • Konflux notes explain dual K8s/Kubearchive fetch, UID deduplication, and the two label selectors (BUILD vs. TEST).
  • Testing Farm notes explain XML result structure and usage patterns.

ToolRegistry.get_tool_notes() collects all non-None notes into a ## Data Source Notes section that is prepended to the workflow body in the system prompt:

BASE_SYSTEM_PROMPT + tool_notes + workflow_body

This keeps domain knowledge close to the tool definitions (in the data source module) rather than duplicated in every workflow .md file.

7.6. Response metadata

register_data_source() accepts optional response_metadata – a dict that is merged into every response from that tool (inline, auto-spill, and fetch_to_sandbox). Used for:

  • Konflux: {"konflux_ui": "https://..."} so the model can build reviewer-facing PipelineRun links.
  • Testing Farm: {"artifacts_base": "https://..."} so the model can build artifact links.

This avoids having the model ask “what is the Konflux UI URL?” – the information arrives with every tool response.

8. Sandbox Architecture

8.1. The Sandbox protocol

All backends implement an 8-method typing.Protocol:

class Sandbox(Protocol):
    def start(self) -> None                                                  # create container/pod, pre-create /tmp/data/
    def exec(command, *, stdin_data: bytes | None) -> ExecResult             # sh -c in sandbox
    def write_file(path, data: bytes) -> None                               # stdin pipe: cat > path
    def stream_to_file(path, chunks: Iterable[bytes]) -> tuple[int, int]    # Popen stdin pipe, returns (bytes, lines)
    def stream_exec(command, chunks: Iterable[bytes]) -> ExecResult         # Popen with streaming stdin
    def read_file_iter(path) -> Iterator[bytes]                             # Popen stdout pipe in chunks
    def cleanup(self) -> None                                               # rm -f container / delete pod
    def linger(session_id) -> None                                          # keep alive for reuse, or fall back to cleanup

ExecResult is (exit_code: int, stdout: str, stderr: str).

stream_to_file() and stream_exec() use subprocess.Popen with a stdin pipe, writing chunks incrementally. This avoids buffering large payloads in orchestrator memory (e.g., streaming a tar.gz archive into the sandbox). read_file_iter() reads from a Popen stdout pipe in 64 KB chunks.

write_file() uses stdin piping (cat > path), never host volume mounts. This is critical: it means data flows through the orchestrator process, not through a shared filesystem. The sandbox has no host mounts.

8.2. Backend selection

Five sandbox backends are available:

Backend --sandbox When to use
PodmanSandbox podman Local development (default for run)
K8sSandbox k8s Direct pod creation on a K8s cluster
K8sPoolSandbox k8spool Pre-warmed pool via Deployment (default for serve)
KubeVirtSandbox kubevirt Direct VMI creation with SSH exec
KubeVirtPoolSandbox kubevirtpool Pre-warmed VMI pool via VMIRS

create_sandbox() factory handles podman, k8s, and kubevirt. Pool backends (k8spool, kubevirtpool) are handled in __main__.py, which creates a SandboxPool or VmiPool and passes it to run_workflow().

The sandbox backend can be set per-workflow via the sandbox: field in the workflow config YAML. Resolution order: workflow.sandbox > CLI --sandbox

default. This allows different workflows to use different backends (e.g., lightweight code review uses pod pool, heavier analysis uses VM pool).

K8s is never auto-detected from the environment. It requires explicit CLI flags. This prevents accidental use of a K8s sandbox when developing locally.

For K8s namespace resolution (k8s and k8spool):

  1. If --namespace is provided, use it.
  2. If --context is provided, extract the namespace from the kubeconfig context. If the context has no default namespace, raise an error.
  3. If neither is provided (in-cluster), use sandbox.namespace from the config file.

8.3. PodmanSandbox

podman run -d --name hb-sandbox-{uuid8} --network=none --user 65532 \
    --workdir /tmp {image} sleep infinity
  • Unique container name with UUID suffix prevents collisions
  • sleep infinity keeps the container alive for repeated exec calls
  • --network=none provides complete network isolation
  • --user 65532 is a fixed non-root UID (matches nonroot in distroless)
  • Cleanup: podman rm -f (force, in case exec is still running)

8.4. K8sSandbox: the hybrid approach

The K8s sandbox uses a hybrid of two tools:

  • kubernetes Python library for pod lifecycle: create_namespaced_pod, read_namespaced_pod (poll for Running), delete_namespaced_pod.
  • kubectl exec subprocess for command execution.

Why not use the kubernetes Python library for exec too? Three problems discovered during development:

  1. No stdin EOF signal in WebSocket v1-v4. The Kubernetes exec protocol uses WebSocket channels (stdin=0, stdout=1, stderr=2). Protocol versions 1-4 have no mechanism to signal “stdin is done.” Commands like cat > /file hang forever waiting for more input. Python client v5 support does not exist.

  2. BrokenPipeError on large stdin. Sending more than ~1MB through the WebSocket stream() API causes pipe errors, breaking write_file() for large data source responses.

  3. Unbounded memory from stream(). The stream() function accumulates all stdout/stderr data in memory with no streaming control. A command producing megabytes of output would consume unbounded memory.

kubectl exec as a subprocess avoids all three problems and provides the same interface as podman exec – stdin via subprocess.PIPE, stdout/stderr captured, exit code from return code. The implementation in exec() is nearly identical between PodmanSandbox and K8sSandbox.

8.5. Pod manifest design

The K8s pod manifest is built by _build_pod_manifest():

metadata:
  labels:
    app.kubernetes.io/name: hummingbird-agent-sandbox
spec:
  automountServiceAccountToken: false # no K8s API from sandbox
  activeDeadlineSeconds: 1800 # 30min hard timeout, backstop
  restartPolicy: Never
  securityContext:
    runAsNonRoot: true
    seccompProfile: RuntimeDefault
  containers:
    - name: sandbox
      command: ["sleep", "infinity"]
      workingDir: /tmp
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
      resources:
        requests: { cpu: 100m, memory: 256Mi, ephemeral-storage: 256Mi }
        limits: { cpu: "1", memory: 1Gi, ephemeral-storage: 2Gi }

runAsUser is deliberately omitted. On OpenShift, the restricted-v2 SCC assigns a UID from the namespace UID range. On vanilla K8s, the image’s USER directive is used. Setting an explicit UID would conflict with OpenShift’s SCC admission.

8.6. Auth modes

  • Local development: --context flag passes the kubeconfig context to config.new_client_from_config(context=...), creating a per-instance ApiClient.
  • Production (in-cluster): context=None triggers config.load_incluster_config(), using the pod’s ServiceAccount token.

The kubectl exec commands include --context when running locally but omit it when in-cluster (kubectl uses the default in-cluster config).

8.7. Pre-created directories

Both backends run mkdir -p /tmp/data in start() after the container/pod is up. This provides a working directory for the model without consuming an iteration. /tmp/data/_out (the spill directory) is created on demand by mkdir -p $(dirname ...) in write_file().

8.8. K8sPoolSandbox: Deployment-backed pool

On-demand pod creation adds 5-30 seconds of latency per workflow (image pull, scheduling, container start). For interactive use (slash commands, reply-based resumption), this delay is user-facing. The pool eliminates it.

Mechanism. A Kubernetes Deployment maintains a set of pods labelled hummingbird/role: standby. When a sandbox is needed, SandboxPool.claim() finds a Running standby pod, patches it to role: active, clears its ownerReferences (detaching it from the ReplicaSet), and annotates it with hummingbird/reap-by (an absolute UTC deadline). The Deployment controller sees the ReplicaSet is under the desired replica count and creates a replacement.

Pod lifecycle.

  1. Deployment creates pod -> role: standby (managed by ReplicaSet)
  2. claim() patches -> role: active, ownerReferences: [], reap-by: now + max_active_seconds
  3. Agent uses the pod via inherited K8sSandbox.exec()
  4. On success: linger() patches -> reap-by: now + linger_seconds, session-id: <id> (pod stays alive)
  5. On reply within linger window: try_reclaim() patches -> reap-by: now + max_active_seconds, clears session-id
  6. On failure or linger expiry: pod is deleted (by cleanup() or the reaper)

Pod lingering. After a successful workflow, pool sandboxes are kept alive for linger_seconds (default 300) instead of being deleted immediately. The pod is annotated with hummingbird/session-id to link it back to the session. If a user reply arrives within the linger window, try_reclaim() finds the pod by session-id, clears the annotation (marking it as in-use), and resets reap-by. This skips pod creation and S3 archive restoration. If no reply arrives, the reaper deletes the pod when its reap-by deadline passes.

Concurrency safety. session-id is only present while lingering. Its absence during active execution prevents concurrent replies from adopting the same pod. try_reclaim() is serialized by a threading lock; the first caller wins, others fall back to S3 restore with a fresh pod.

Reaping. reap_expired() is called once per workflow execution (after sandbox acquisition, before the model loop). It performs a single sorted pass over all active pods:

  1. Non-lingering pods (no session-id) past their reap-by deadline are deleted immediately.
  2. Lingering pods (with session-id) are sorted by reap-by ascending. The loop deletes pods that are either expired (now > reap-by) or exceed max_lingering_pods (default 2, configurable), starting with those closest to their deadline. The loop stops when both conditions are satisfied (now <= reap-by and remaining count <= limit).

This replaces the previous design where _reap_expired() ran inside claim() on every poll iteration. Moving it to a once-per-workflow call reduces API load and centralizes cleanup. The max_lingering_pods cap prevents unbounded pod accumulation from many short-lived workflows.

Indefinite wait. claim() polls until a pod is available or the shutdown event fires. It logs at DEBUG level on each poll, escalating to WARNING after ~2 minutes. This is normal behavior when the pool (Deployment replicas) is smaller than max_concurrent_agents – the SQS semaphore caps concurrency, so demand will not permanently exceed supply.

Config simplification. In pool mode, the pod image, resources, metadata, and security context are defined solely in the Deployment template. The agent config only needs sandbox.namespace and sandbox.active_deadline_seconds. This eliminates duplication between the agent configmap and the Deployment.

Class design. K8sPoolSandbox inherits from K8sSandbox to reuse exec(), write_file(), read_file_iter(), and cleanup(). It overrides __init__() (does not call super().__init__() since that resolves kubeconfig), start() (claims from pool instead of creating a pod), and linger() (patches annotations instead of deleting). It adds start_from_existing() for the reclaim path. All sandbox backends implement linger(session_id): non-pool backends fall back to cleanup().

8.9. KubeVirtSandbox: SSH-based VM execution

KubeVirtSandbox provides full VM isolation using KubeVirt VirtualMachineInstances. It parallels K8sSandbox but uses CustomObjectsApi for VMI lifecycle and ssh for command execution instead of kubectl exec.

Why SSH? VMs have no kubectl exec equivalent. KubeVirt provides virtctl console (serial) and VNC, but these are not suitable for programmatic command execution. SSH provides a reliable, well-understood exec channel with stdin/stdout/stderr piping.

SSH key injection. The VM sandbox image includes a custom inject-ssh-keys.service that reads SSH public keys from a KubeVirt Secret volume attached as a virtio disk with serial ssh-pubkeys (visible at /dev/disk/by-id/virtio-ssh-pubkeys) and installs them to /root/.ssh/authorized_keys before sshd starts. This avoids the need for cloudInitNoCloud or qemu-guest-agent.

Single-VM lifecycle:

  1. Generate an ephemeral ed25519 key pair via ssh-keygen
  2. Create a K8s Secret with the public key
  3. Create the VMI referencing the containerDisk image and the SSH Secret
  4. Poll VMI status until Running with a pod network IP
  5. Poll SSH readiness (ssh sandbox@ip true)
  6. Execute workflow commands via SSH
  7. Cleanup: delete VMI, delete Secret, remove temp key files

Polling strategy. Two-phase wait: first poll status.phase via get_namespaced_custom_object until Running (same interval as pod polling), then poll SSH readiness with ConnectTimeout=5. Total timeout matches POD_START_TIMEOUT (300s) for VMI startup plus _SSH_READY_TIMEOUT (120s) for SSH.

8.10. KubeVirtPoolSandbox: VMIRS-backed pool

KubeVirtPoolSandbox + VmiPool parallel K8sPoolSandbox + SandboxPool, operating on VMI custom resources instead of Pods.

VirtualMachineInstanceReplicaSet (VMIRS). KubeVirt’s native controller maintains a desired number of identical VMIs. When a VMI is claimed (label flipped to active, ownerReferences cleared), the VMIRS automatically creates a replacement – the same pattern as a Deployment replenishing claimed pods.

Shared SSH key. Pool mode uses a single key pair for all standby VMIs. The public key is stored in a K8s Secret referenced by all VMIs in the VMIRS template. The private key path is provided via sandbox.vm_ssh_private_key in the config. On pool restart, a new key pair can be generated; existing VMIs drain naturally with the old key.

Claim/reap/linger semantics. Identical to the pod pool: claim() polls for Running VMIs with hummingbird/role=standby, patches to active, clears ownerReferences. reap_expired() deletes VMIs past their hummingbird/reap-by deadline. try_reclaim() finds lingering VMIs by hummingbird/session-id. The only difference is the API: all operations go through CustomObjectsApi with kubevirt.io/v1 virtualmachineinstances.

Class design. KubeVirtPoolSandbox inherits from KubeVirtSandbox to reuse exec(), write_file(), read_file_iter() (all SSH-based). It overrides __init__() (takes pool, skips key generation), start() (claims from pool), cleanup() (only deletes VMI, not Secret/keys which are pool-owned), and linger() (patches annotations).

4 - Hummingbird Agent Evals

Evaluation framework for scientific measurement of LLM output quality. For the agent itself, see Hummingbird Agent. For architecture details, see Hummingbird Agent Design.

What evals are

Evals measure the quality of LLM outputs under controlled conditions. They answer questions like “does prompt v5 produce better code reviews than v4?” or “can an LLM answer this question correctly using only public documentation?”

Evals are not unit tests. Unit tests verify deterministic behavior; evals quantify stochastic output quality through structured scoring.

General pattern

Evals generally follow a pattern of:

  1. Inputs – a set of test cases: golden dataset with scoring criteria, real MRs to review, or other structured inputs.
  2. Model execution – run one or more LLM configurations against the inputs, optionally with tools (web search, sandbox).
  3. Scoring – evaluate outputs via LLM-as-judge, comparison against reference answers, or human review.
  4. Iteration – adjust prompts or config, re-run, check for improvement.

Not all evals use every step. Baseline measurements may stop at step 3; comparison evals may skip the golden dataset entirely.

Shared infrastructure

hummingbird_agent.eval

The hummingbird_agent.eval module provides generic evaluation primitives used across all evals. Key components:

  • EvalJudge – LLM-as-judge wrapper over Vertex AI with explicit temperature and token limit control. Supports both strict judging (low temperature) and open-ended analysis (higher temperature).
  • Snapshot management – saves timestamped evaluation results with metadata, maintains a latest symlink.
  • Variant tracking – persists the best-performing prompt or config variant across iterations.
  • Convergence detection – checks whether scores have plateaued across consecutive iterations.
  • Holdout validation – compares holdout scores against dev scores using standard error of the mean.

See the module docstrings for the full API.

hummingbird_agent.evals.mr_helpers

MR-experiment-specific helpers built on top of hummingbird_agent.eval. Provides GitLab auth, MR data I/O, review generation orchestration, and CLI scaffolding. Used by the code-review and renovate-triage evals.

Running an eval

Prerequisites

  1. GCP Application Default Credentials:

    gcloud auth application-default login
    export GOOGLE_CLOUD_PROJECT=<your-gcp-project>
    
  2. The agent package installed in editable mode:

    cd hummingbird-agent
    pip install -e .
    

Execution

Each eval is a Python module under hummingbird_agent/evals/. Run from hummingbird-agent/:

python -m hummingbird_agent.evals.<name>.<script> [options]

Refer to each eval’s own README for specific options and configuration.

Existing evals

Writing a new eval

Create a new directory under hummingbird_agent/evals/ with:

  • README.md – what the eval measures and how to run it, linking back to this document for the general framework.
  • Golden dataset – inputs and scoring criteria. Can live outside the eval directory (e.g. in the documentation repo alongside the content being measured) and be passed via CLI.
  • run_*.py – orchestration script using hummingbird_agent.eval primitives (or hummingbird_agent.evals.mr_helpers for MR-based evals).

The eval’s run script should be importable as a module (python -m hummingbird_agent.evals.<name>.<script>). Add an __init__.py if needed. Do not hardcode GCP project IDs; VertexAuth resolves them at runtime from GOOGLE_CLOUD_PROJECT or ADC credentials.

Authentication

All evals use VertexAuth from hummingbird_ai.http, which resolves credentials and project in this order:

  1. Explicit project= parameter (if passed).
  2. GOOGLE_CLOUD_PROJECT environment variable.
  3. ADC default project (from gcloud auth application-default login).

Service account keys work automatically when GOOGLE_APPLICATION_CREDENTIALS is set.

5 - Documentation Quality Evaluation

Reference for the documentation quality evaluation, which measures whether public documentation is complete enough for LLMs with web search to answer questions correctly. For the general evaluation framework, see Hummingbird Agent Evals.

Config file

A single YAML file combines evaluation settings and the golden dataset. It is passed to the run script via -c:

python -m hummingbird_agent.evals.docs_eval.run_eval -c path/to/docs_eval.yml

The config file lives in the documentation repo alongside the content being measured.

Evaluation settings

Key Required Default Description
models yes – List of model configurations
models[].id yes – Vertex AI model identifier
models[].region yes – Vertex AI region
models[].google_search no false Enable Google Search grounding (Gemini)
models[].web_search no false Enable web search tool (Claude)
judge yes – Judge model configuration
judge.model yes – Judge model identifier
judge.region no global Judge model region
repetitions no 3 Runs per (question, model) pair
max_iterations no 5 Agent loop iteration cap per question

Golden dataset

The questions list defines what the evaluation measures:

Field Required Default Description
id yes – Unique identifier (used in output paths)
question yes – Question sent to each model
facts yes – List of fact checks
facts[].question yes – Yes/no question about the response
facts[].weight no 1 Relative importance
coherence no – Criterion for narrative quality scoring
coherence_weight no 1.5 Weight of coherence in the final score
threshold no 0.4 Minimum weighted score to pass

CLI options

Option Default Description
-c / --config required Path to config YAML
--phase all ask, judge, or all
--results-dir results/ next to config Output directory
--run-id UTC timestamp Subdirectory under results

Scoring

Each response is scored independently:

  • Fact scoring – the judge answers each fact question with yes or no. Passing facts contribute their weight to the numerator.
  • Coherence scoring – the judge rates narrative quality from 0.0 to 1.0. The coherence score is multiplied by its weight and added to the total.
  • Weighted score – sum(passed_fact_weights + coherence_contribution) / total_weight. A response passes if weighted_score >= threshold.

The summary aggregates per-model averages across all repetitions.

6 - Hummingbird Agent Design — Runtime

Data sources, event pipeline, sessions, workflows, and deployment for the Hummingbird Agent. This page covers the runtime: how the agent connects to external systems, processes events, manages sessions, and is deployed. For the execution engine (agent loop, tool system, sandbox), see Hummingbird Agent Design — Engine.

9. Data Sources

Data sources are external APIs wrapped as tool-calling functions. The model invokes them by name; the orchestrator executes them and returns results (inline or auto-spilled). Each data source module follows the same pattern:

  1. Define TOOL_DEFS – a list of ToolDef objects with names, descriptions, parameter schemas, and optional notes.
  2. Define implementation functions that take a pre-configured client as the first argument.
  3. Provide a register() function that creates the client, wraps implementations with functools.partial, and calls registry.register_data_source() for each tool.

Credentials are captured at registration time via functools.partial closures. They are never stored as global state and never leak into tool definitions or model contents.

9.1. GitLab

Uses python-gitlab library with retry_transient_errors=True for automatic retry on transient HTTP errors.

Tools:

  • gitlab_get_mr_details – MR metadata (title, author, SHA, labels, URLs)
  • gitlab_get_commit_statuses – all CI/CD statuses for a commit SHA (paginated automatically). Covers both Konflux external statuses and native GitLab CI job statuses. Each status includes target_url (job page link containing the job ID) and allow_failure.
  • gitlab_get_mr_diff – changed files in the MR
  • gitlab_get_file_at_ref – raw file content at a git ref
  • gitlab_get_repo_archive – repository tar.gz via repository_archive(iterator=True). Returns a StreamingResponse with chunked binary data (suffix .tar.gz), so the archive streams to the sandbox without buffering in orchestrator memory.
  • gitlab_get_job_log – job trace (log output) for a GitLab CI job. Uses lazy=True on the job object and trace(iterator=True) for streaming. Each chunk is decoded with errors='replace' and ANSI escape codes are stripped per-chunk before re-encoding. Returns a StreamingResponse (suffix .log). Per-chunk ANSI stripping is safe because escape sequences are <20 bytes and chunks are 1024+ bytes.

No response_metadata is set because GitLab commit statuses already contain target_url fields that the model uses for linking.

gitlab_get_mr_discussions – trust filtering and redaction.

MR discussions are the first data source that feeds user-generated free text into the model prompt. Unlike diffs and CI logs (which are code or machine output), discussion comments can contain arbitrary prose written by anyone who can post on the MR – including external contributors on public projects. This creates a prompt injection vector: an attacker posts a comment containing instructions that manipulate the model’s review output.

The discussions tool addresses this with a three-layer defence:

  1. Author trust gate. Each note’s author is checked for Developer+ access (level >= 30) on the project via members_all.get(author_id). Results are cached per author_id within a single call to avoid repeated API lookups. System notes (merge events, label changes) bypass the author check. Agent-authored notes are trusted via their author’s Developer+ access (the bot account holds Developer access on configured projects). Note: agent note detection for redaction purposes (stripping transcripts and footers) uses session marker presence, but trust is always based on the author’s access level, not the marker. This prevents marker injection from granting trust to non-member notes.

  2. Redaction, not sanitization. Untrusted notes in a discussion that also contains trusted notes are replaced with a fixed placeholder ([redacted: non-member comment]). The placeholder preserves the conversation structure (the model sees that someone replied, but not what they said). Discussions where all notes are untrusted are dropped entirely. No attempt is made to sanitize or escape untrusted content – there is no reliable escaping mechanism for LLM prompts, so the only safe option is to withhold the content entirely.

  3. Agent note stripping. Agent-authored notes contain large <details><summary>Agent transcript</summary>...</details> blocks (15K-90K chars of raw tool calls), metadata footers, and session markers. These are stripped before the note enters the model prompt, leaving only the review text. This serves dual purposes: token efficiency and avoiding feeding the model its own raw tool call history (which could cause degenerate self-referential loops).

Residual risks and scope limitations:

  • A compromised Developer+ account can inject adversarial content. This is accepted as equivalent to the existing risk of a compromised developer pushing malicious code (which the agent would also process).
  • The trust threshold is project-level (Developer role on the project), not MR-level. A developer on the project can influence any MR’s review.
  • Comments posted after the discussions are fetched but before the review is posted are not seen. This is a TOCTOU gap but has no security impact (the model simply misses late comments).

9.2. Konflux

Uses raw requests against K8s and Kubearchive APIs (not the kubernetes Python SDK). This avoids the heavy kubernetes client dependency for what is essentially bearer-token HTTP with label selectors.

Dual-fetch architecture:

flowchart LR
    subgraph fetch [Fetch Phase]
        KA["Kubearchive<br/>(historical)"]
        K8s["K8s API<br/>(live)"]
    end
    Combine["Combine items"]
    Dedup["Deduplicate<br/>by metadata.uid"]

    KA --> Combine
    K8s --> Combine
    Combine --> Dedup

For each resource type (PipelineRuns, TaskRuns), the client fetches from both Kubearchive (completed/historical resources) and the live K8s API, combines the results, and deduplicates by metadata.uid. This ensures no resources are missed regardless of whether they have been archived yet.

Two label selectors. Konflux uses different labels for BUILD and TEST PipelineRuns:

  • BUILD: pipelinesascode.tekton.dev/sha=<commit_sha>
  • TEST: pac.test.appstudio.openshift.io/sha=<commit_sha>

Each iter_pipelineruns()/iter_taskruns() call queries both selectors and combines the results. _iter_resources(endpoint, label_selectors) is the generic implementation: it creates one session, iterates each selector through _iter_both_sources(), and deduplicates across selectors by UID.

Per-PLR TaskRun fetching. iter_taskruns() accepts either commit_sha (two-selector path) or pipelinerun (single selector: tekton.dev/pipelineRun=<name>). When pipelinerun is set, only the ~15-21 TaskRuns for that PipelineRun are fetched (~600 KB) instead of all TaskRuns for the commit (~530 MB for large commits). This is server-side filtering — the K8s API and Kubearchive both support the tekton.dev/pipelineRun label selector.

Status filtering and streaming counts. _items_to_jsonl(items, status, stats) accepts an optional status: set[str] | None allowlist and a _StreamStats accumulator. For each item it increments stats.by_reason (keyed by reason) regardless of the filter (so counts reflect all items seen), then skips items whose status.conditions[0].status is not in the allowlist. The allowlist values are "True" (success), "False" (failure), or "Unknown" (in-progress) — only 3 possible values, covering every Tekton reason without enumeration. Items without conditions always pass. The _finalize() generator wrapper exhausts the chunks iterator, then writes counts, included, and excluded into the StreamingResponse.metadata dict — after the sandbox file is fully written. The filtering happens O(1) per item with no buffering.

StreamingResponse.metadata. StreamingResponse (in tools.py) has a mutable metadata: dict[str, Any] field (default empty dict). Both _exec_fetch_to_sandbox() and _spill_streaming() merge it into the response dict after the iterator is exhausted. The Konflux code populates it with counts, included, and excluded; other data sources can use it for their own post-streaming summaries without touching tools.py.

Pod log fetching. get_pod_log() accepts an optional container parameter. When specified, it fetches logs for that single container. When omitted, it discovers all containers from the pod spec and concatenates their logs with === container_name === headers. Each individual log fetch tries Kubearchive first, then falls back to the live K8s API. Logs may be unavailable from both sources if the pod has expired.

Streaming pagination. K8s list endpoints can return pages of 80+ MB when a commit touches many components (e.g. 500 PipelineRuns per page). iter_paginated() uses session.get(stream=True) and ijson.parse() to stream-parse each page: items are yielded one at a time via ObjectBuilder, and the metadata.continue pagination token is captured from the same parse pass. resp.raw.decode_content = True is set so urllib3 transparently decompresses gzip/deflate Content-Encoding inline (Kubearchive returns gzip-compressed responses). Only one item is in memory at a time – O(single_item) regardless of page size. Truncated or malformed streams raise IncompleteJSONError, which is caught and treated like a network error (log a warning, stop iterating that source).

Credential resolution. KonfluxClient.__init__() parses the kubeconfig file to extract the API server URL and bearer token for the cluster. The cluster domain is extracted from the cluster_url config value. This approach avoids depending on kubectl or the kubernetes Python library for API authentication.

Response metadata: Static metadata {"konflux_ui": "...", "konflux_pipelinerun_url_pattern": "..."} is merged into every response so the model can build reviewer-facing links. Dynamic per-call metadata (counts, included, excluded) is populated into StreamingResponse.metadata by _finalize() during streaming and merged by tools.py after the iterator is exhausted.

9.3. Testing Farm

Uses requests.Session with a module-level retry adapter (429, 5xx) for resilience against transient errors.

Tools:

  • tf_get_results – JUnit XML results for a request ID
  • tf_get_test_log – individual test log by URL (from results.xml); restricted to the ARTIFACTS_BASE prefix to prevent SSRF
  • tf_get_request_status – request state, queue/run times

ToolDef notes on tf_get_results document the XML structure: //testsuites/@overall-result, //testcase/@result, //testcase/logs/log with @name and @href. This domain knowledge goes into the system prompt so the model knows how to parse the XML with sandbox_exec using python3 xml.etree.ElementTree.

Response metadata: {"artifacts_base": "https://artifacts.osci.redhat.com/testing-farm"} is merged into every response so the model can build artifact links like [{request_id}]({artifacts_base}/{request_id}/).

9.4. Registration flow

data_sources.register_selected() is the entry point called by runner.py. It iterates the workflow config’s data_sources dict and registers only the declared sources:

for ds_name, ds_cfg in wf_cfg.data_sources.items():
    if ds_name == "gitlab":
        token = resolve_tool_token(proj_entry, "gitlab", wf_cfg)
        gitlab.register(registry, gitlab_url, token)
    elif ds_name == "konflux":
        cluster_url = resolve_cluster_url(ds_cfg)
        kubeconfig_path = os.environ.get(ds_cfg.kubeconfig_env, "")
        konflux.register(registry, cluster_url, kubeconfig_path, ds_cfg.kubearchive_url)
    elif ds_name == "testing_farm":
        testing_farm.register(registry)

This selective registration means a workflow with data_sources: {gitlab: {...}} only exposes GitLab tools to the model. Konflux and Testing Farm tools do not appear in the tool definitions, preventing the model from attempting to use unconfigured data sources.

9.5. Data flow: streaming vs buffered

Data moves from external APIs through the orchestrator to the model (inline or auto-spilled to sandbox). The memory profile of each tool depends on whether the HTTP response is consumed incrementally or buffered entirely.

Streaming (preferred for large/unbounded responses). The HTTP response is consumed incrementally – via ijson stream parsing, iterator=True in python-gitlab, or lazy pagination. Only one chunk or item is in memory at a time. Used by:

  • iter_paginated() (Konflux) – session.get(stream=True) + ijson
  • get_commit_statuses() (GitLab) – statuses.list(iterator=True) via _iter_commit_statuses(), yielding JSONL bytes one status at a time
  • get_repo_archive() (GitLab) – repository_archive(iterator=True), returns chunked tar.gz binary via StreamingResponse(suffix=".tar.gz", binary=True) – no text preview is generated
  • get_job_log() (GitLab) – trace(iterator=True) with per-chunk ANSI stripping, returns StreamingResponse(suffix=".log")

All produce StreamingResponse objects that flow through stream_to_file() into the sandbox without accumulating in orchestrator memory. StreamingResponse.suffix controls the auto-spill filename extension (.jsonl, .tar.gz, .log). StreamingResponse.binary controls whether a head preview is captured (False for text, True to skip for opaque binary formats).

Buffered (acceptable for small/bounded responses). The full response is loaded into memory as a string or dict. This is fine when the response size is bounded and small (typically < 1 MB). Used by:

  • get_mr_details() (GitLab) – single MR object, < 10 KB
  • get_file_at_ref() (GitLab) – single file, bounded by repo constraints
  • tf_get_results(), tf_get_test_log(), tf_get_request_status() (Testing Farm) – XML/text/JSON, typically < 1 MB

Buffered with risk (candidates for future streaming). Same buffered pattern but the response size is not bounded by design. Auto-spill mitigates the context growth problem (large outputs are written to sandbox files), but the orchestrator still spikes RSS during the fetch:

  • get_pod_log() (Konflux) – resp.text per container, concatenated. Multi-container pods accumulate all logs.
  • get_mr_diff() (GitLab) – full diff JSON, scales with MR size. GitLab truncates server-side but the result can still be large.

Invariant: paginated K8s list endpoints must always use streaming. Page sizes scale with the number of components in the commit – a single page can contain hundreds of PipelineRuns or TaskRuns (80+ MB JSON). Buffering these responses risks OOM under normal production workloads, especially with concurrent workflows.

10. Event Pipeline and Production Operations

10.1. Two-stage SQS pipeline (ingress router + FIFO worker)

Events flow through a two-stage pipeline to serialize per-session work:

flowchart LR
  STD["Standard SQS<br/>(ingress)"]
  R["Router"]
  FIFO["SQS FIFO<br/>(work queue)"]
  W["Worker<br/>(poll_loop)"]

  STD -->|ReceiveMessage| R
  R -->|"SendMessage<br/>MessageGroupId"| FIFO
  W -->|"continuation<br/>group=session_id"| FIFO
  FIFO -->|ReceiveMessage| W

Ingress router. events.router_loop() is a single-threaded loop that long-polls the standard (ingress) queue, peeks into each SNS envelope to assign a MessageGroupId, forwards the raw message body to the FIFO queue, then deletes from the standard queue.

  • Note events: MessageGroupId = discussion_id from the webhook body. This serializes replies to the same agent session – FIFO delivers at most one in-flight message per group.
  • Pipeline / MR events: MessageGroupId = SQS MessageId (unique per message). No serialization – each event is its own group.
  • MessageDeduplicationId: Always the SQS MessageId from the standard queue. Absorbs duplicate deliveries from standard SQS’s at-least-once semantics (5-minute dedup window).

If SendMessage to FIFO fails, the message is not deleted from the standard queue and retries via visibility timeout.

FIFO worker. events.poll_loop() long-polls the FIFO queue, decodes SNS envelopes, and dispatches to handle_event. The max_concurrent_agents semaphore caps parallel groups being processed. FIFO guarantees that within a group, only one message is in-flight.

Two-phase processing. When the FIFO worker picks up a webhook (Phase 1), the handler validates the event, resolves the session_id, then posts a continuation message back to the same FIFO with MessageGroupId = session_id. The Phase 1 message is deleted quickly (~1-5s). Phase 2 processes the continuation: loading session state, building WorkflowRequest, and calling _execute_workflow. Because Phase 2 continuations share a MessageGroupId per session, all work for a given session is serialized – even if it spans multiple GitLab discussion threads.

Uniform message format. Both webhook messages (from SNS) and continuation messages use the same SNS-style envelope with gzip+base64 compression. encode_envelope() is the inverse of decode_sns_message(). The consumer doesn’t need to distinguish between webhook and continuation messages at the transport layer.

Graceful shutdown. SIGTERM/SIGINT set a shutdown_event. Both loops exit. Messages in SQS become visible again after the visibility timeout for other consumers.

Why two queues. The standard queue receives events from SNS (via subscription). The FIFO queue serializes per-session work. The gitlab-event-forwarder and SNS subscription are unchanged – the routing logic lives entirely in the agent codebase.

10.2. SNS envelope decoding

Messages arrive as SNS notifications with:

  • MessageAttributes.source – event source (e.g., "gitlab")
  • MessageAttributes.event_type – event type (e.g., "pipeline", "merge_request", "note")
  • MessageAttributes.content_encoding – "gzip+base64" for gitlab-event-forwarder, absent for kubernetes-event-forwarder
  • Message – the actual event body (JSON string, or gzip+base64 encoded)

decode_sns_message() handles both encoding formats transparently.

10.3. Event routing

runner.handle_event() dispatches on (source, event_type):

flowchart TD
    Event["Event arrives"]
    Check{"source/type?"}
    Cont["handle_continuation()"]
    Pipeline["handle_pipeline()"]
    MRHandler["handle_merge_request()"]
    Note["handle_note()"]
    Ignore["Ignore"]

    Event --> Check
    Check -->|"agent/continuation"| Cont
    Check -->|"gitlab/pipeline"| Pipeline
    Check -->|"gitlab/merge_request"| MRHandler
    Check -->|"gitlab/note"| Note
    Check -->|"other"| Ignore

    Pipeline --> FilterSource{"source = merge_request_event?"}
    FilterSource -->|"yes"| AutoResolve["auto-resolve on success<br/>(unconditional)"]
    FilterSource -->|"no"| Skip1["Skip"]
    AutoResolve --> PipeRules["evaluate trigger_rules<br/>(status + user + branch + title)"]
    PipeRules -->|"allow"| Enqueue1["_enqueue_continuation()"]
    PipeRules -->|"deny"| Skip1a["Skip"]

    MRHandler --> FilterMRAction{"action in open/update?<br/>draft = false?"}
    FilterMRAction -->|"yes"| MRRules["evaluate trigger_rules<br/>(user + branch + title)"]
    FilterMRAction -->|"no"| Skip1b["Skip"]
    MRRules -->|"allow"| Enqueue2["_enqueue_continuation()"]
    MRRules -->|"deny"| Skip1c["Skip"]

    Note --> FilterMRNote{"MR note?<br/>action = create?"}
    FilterMRNote -->|"yes"| NoteType
    FilterMRNote -->|"no"| Skip3["Skip"]

    NoteType{"Note type?"}
    NoteType -->|"author_id == bot_id"| Skip4["Skip (own note)"]
    NoteType -->|"/hummingbird help"| Help["post_simple_reply (help)"]
    NoteType -->|"/hummingbird wf-name"| SlashCmd["_resolve_slash_workflow()<br/>→ _enqueue_continuation()"]
    NoteType -->|"DiscussionNote reply"| FindSession["find_session_for_reply()"]
    NoteType -->|"other"| Skip5["Skip"]

    FindSession --> Found{"session found?"}
    Found -->|"yes"| Enqueue3["_enqueue_continuation()<br/>(resume_session)"]
    Found -->|"no"| Skip6["Skip"]

    Cont --> ContType{"work_type?"}
    ContType -->|"new_session"| ExecNew["_execute_workflow()"]
    ContType -->|"resume_session"| LoadS3["load session from S3<br/>→ _execute_workflow()"]

Phase 1 handlers (handle_pipeline, handle_merge_request, handle_note) perform validation, auth checks, and session resolution, then post a WorkOrder continuation message via _enqueue_continuation(). They do not call _execute_workflow directly.

Phase 2 (handle_continuation) receives the WorkOrder, resolves the workflow config, builds a WorkflowRequest, and calls _execute_workflow. For resume_session orders, it loads the previous session from S3. _execute_workflow handles SHA dedup and per-workflow rate limiting via scan_agent_threads.

Slash commands dispatch to a single named workflow via _resolve_slash_workflow (exact match, then prefix match). Help and error replies are posted via post_simple_reply (no session marker, no rate limit impact).

10.4. Pipeline trigger design

Why gitlab::pipeline instead of kubernetes::PipelineRun: A GitLab pipeline event fires once when the pipeline completes. Since Konflux external stages are attached to the pipeline, the event naturally waits for all builds and tests to finish before triggering. This means the agent sees the full picture in one event, without needing to deduplicate or wait for stragglers.

Trigger filters:

  • status in {failed} – only failed pipelines trigger analysis
  • source == "merge_request_event" – only MR pipelines, not branch/tag

10.4a. Merge request trigger design

The handle_merge_request handler fires on MR open, reopen, and update events (action in {"open", "reopen", "update"}). Draft MRs are skipped (object_attributes.draft == True); marking a draft as ready triggers a review since the event arrives with draft: false.

Event classification is deliberately simple: the handler does not inspect oldrev or changes.draft fields. Instead, SHA-based deduplication in _execute_workflow (via JSON session markers) ensures each code revision is reviewed at most once. Metadata-only updates (title, label changes) on an already-reviewed SHA are silently skipped.

10.5. Note trigger design

Two sub-flows:

Slash command (/hummingbird <workflow-name>): Triggers a specific named workflow, bypassing rate limiting. _resolve_slash_workflow performs exact-match lookup first, then falls back to unique prefix matching. If the subcommand is missing or is “help”, _format_help returns a list of available workflows with descriptions. Ambiguous or unknown subcommands produce an error message via post_simple_reply. The note author must have Developer+ access (level >= 30) on the project, checked via check_member_access(). If denied, the agent replies in the same discussion thread with a short access-denied message.

Reply to agent note: When a user replies to an existing agent note (which contains a JSON session marker):

  1. find_session_for_reply() walks the discussion thread, filtering by bot author ID (get_bot_user_id()), and returns the first matching session marker. All bot markers in a thread share the same session ID.
  2. handle_note posts a resume_session continuation to the FIFO (grouped by session_id).
  3. handle_continuation loads the session from S3 (conversation history and sandbox archive), builds a WorkflowRequest, and calls _execute_workflow() to resume with the user’s reply.
  4. If the session is not found (expired/deleted), the agent replies with a message explaining the session has expired and suggesting to start a new run.

Reply authors are also subject to the Developer+ access check. If denied, the agent replies in the discussion thread with the same access-denied message.

Non-agent-directed notes: Regular comments that are neither slash commands nor replies to agent threads are silently ignored (debug-level log). No access check is performed for these.

Self-note filtering: Notes where author_id matches the bot’s own user ID (resolved via get_bot_user_id()) are skipped immediately. This prevents infinite loops where the agent’s own output triggers another agent run. The author-ID check replaces the previous substring check (marker_prefix_str in note_body), which was vulnerable to denial-of-service: a user including the marker prefix in their reply would cause the bot to silently ignore it.

Threading logic for replies: If the project requires internal notes (internal_notes: true) but the original discussion was public, the reply is posted as a new top-level internal note instead of replying in the public thread. This prevents leaking internal analysis into public threads.

10.6. Note lifecycle

1. create_placeholder_note()   -> "Running the <workflow> workflow..."
                                   + JSON session marker (id, wf, sha)
2. run_workflow()              -> agent loop
3. update_note()               -> replace placeholder with result
                                   + reply prompt + JSON session marker

Reply notes (from session resumption) omit the reply prompt since the
user is already engaged in the conversation.

On failure, the placeholder is updated to “Hummingbird analysis failed.” The JSON session marker is always present so the note can be identified as agent-generated (for rate limiting, SHA dedup, and self-note filtering). The marker embeds the workflow name and commit SHA, enabling per-workflow rate limiting and SHA-based deduplication.

10.7. Rate limiting and SHA deduplication

scan_agent_threads() performs a single pass over MR discussions, parsing JSON session markers (<!-- hummingbird-session: {"id":...,"wf":...,"sha":...} -->) to determine:

  1. Per-workflow thread count – how many threads the given workflow has created on this MR. If the count meets or exceeds max_runs_per_mr, the workflow is skipped.
  2. SHA dedup – whether the current commit SHA has already been reviewed by this workflow. If so, the workflow is skipped (prevents re-reviewing the same code on metadata-only MR updates).

Slash commands and replies bypass both checks entirely.

The rate limit is per-workflow per-MR: different workflows maintain independent thread counts on the same MR. JSON markers are backward compatible – old plain-UUID markers (<!-- hummingbird-session: UUID -->) are parsed as {"id": "UUID"} with no workflow or SHA information, so they are not counted toward any specific workflow’s limit.

11. Session System

Sessions enable conversation continuity: a user can reply to an agent note and the agent picks up where it left off, with full context and sandbox files restored.

11.1. What gets saved

Three artifacts are saved after each run:

  • context.json – the full contents list from the agent loop in the canonical provider-independent format. It is the minimum state needed to resume the conversation; format conversion is handled by the AI library.
  • transcript.md – a human-readable markdown rendering of the run for debugging and auditing. Not used for resumption.
  • sandbox.tar.gz – an archive of /tmp/data/ (the sandbox working directory, which includes the _out/ spill subdirectory). Contains PipelineRun JSONs, test logs, jq output, and any other files the model created during the run. Restored into the new sandbox on resumption so the model can reference its previous work.

11.2. Storage backends

S3 (production):

s3://{bucket}/sessions/{session_id}/context.json
s3://{bucket}/sessions/{session_id}/transcript.md
s3://{bucket}/sessions/{session_id}/sandbox.tar.gz

Local directory (development, --save-session):

{directory}/context.json
{directory}/transcript.md
{directory}/sandbox.tar.gz

Both backends have the same interface. S3 save is best-effort: wrapped in try/except so a transient S3 error does not prevent the MR note from being posted. The note always gets delivered first.

11.3. Sandbox archive transport

The sandbox archive requires special handling because the orchestrator cannot directly access the sandbox filesystem (no volume mounts):

Archive (inside sandbox):
  sb.exec("tar czf /tmp/_archive.tar.gz -C /tmp data")

Stream out (sandbox -> host temp file):
  for chunk in sb.read_file_iter("/tmp/_archive.tar.gz"):
      tmp.write(chunk)                     # 64 KB chunks, no base64

Restore (host temp file -> new sandbox):
  sb.stream_exec("tar xzf - -C /tmp", file_chunks())

Both directions use streaming binary I/O via Popen stdin/stdout pipes. No base64 encoding is needed: read_file_iter() yields raw bytes from the sandbox via a Popen stdout pipe, and stream_exec() feeds raw bytes into a Popen stdin pipe. The archive is never extracted on the orchestrator – it exists only as opaque bytes being transported between sandboxes. This is a security invariant: the orchestrator never parses or inspects the archive contents.

11.4. Resumption flow

flowchart TD
    Reply["User replies to agent note"]
    FindSession["find_session_for_reply()<br/>walks discussion thread"]
    LoadS3["load_s3(session_id)"]
    NotFound{"session found?"}
    Expired["Post session-expired reply"]
    NewSandbox["Start new sandbox"]
    Restore["restore_sandbox(archive)"]
    BuildContents["contents = old_contents + user_reply"]
    RunLoop["run_agent_loop(<br/>initial_contents, CONTINUATION_PROMPT)"]

    Reply --> FindSession --> LoadS3
    LoadS3 --> NotFound
    NotFound -->|"no"| Expired
    NotFound -->|"yes"| NewSandbox
    NewSandbox --> Restore --> BuildContents --> RunLoop

Key aspects:

  • Fresh iteration budget. The resumed session starts iteration 0 with the full max_iterations budget, regardless of how many iterations the previous session used.
  • Context window is the real limit. A typical 8-iteration cold start uses ~50-100K input tokens. The restored history is sent in full on every API call. The CONTEXT_LIMIT check applies and will force wrap-up if needed.
  • Session ID reuse. The resumed session keeps the original session ID. S3 state is overwritten in place with the updated conversation history. Since GitLab discussions are linear (not branching), there is no need for a tree of sessions – the conversation is a single sequential thread.
  • Graceful degradation. If the S3 session is not found (expired, deleted), the agent posts a reply explaining the session has expired and suggesting to start a new run. It does not fall back to a cold start.

11.5. Session markers

Every agent note contains a hidden HTML comment with a JSON payload:

<!-- hummingbird-session: {"id":"UUID","wf":"code-review","sha":"abc123"} -->

The JSON payload contains:

  • id – session UUID (always present)
  • wf – workflow name (present for new-format markers)
  • sha – commit SHA at the time of review (present for auto-triggered runs)

This marker serves four purposes:

  1. Resumption: find_session_for_reply() searches discussion threads for this marker to find the session ID. Only bot-authored notes are considered; the first matching marker wins.
  2. Per-workflow rate limiting: scan_agent_threads() counts discussion threads for a specific workflow using the wf field. Only bot-authored notes are scanned.
  3. SHA deduplication: scan_agent_threads() checks whether the current SHA has already been reviewed by the workflow using the sha field.
  4. Self-filtering: handle_note() compares the webhook event’s author_id against the bot’s own user ID (via get_bot_user_id()) to prevent infinite loops.

All marker-scanning functions resolve the bot user ID via gl.auth() (GET /user) on the orchestrator token. This ensures markers in non-bot notes (user replies, external contributors) are never parsed, preventing session hijacking via injected markers.

Old plain-UUID markers (<!-- hummingbird-session: UUID -->) are parsed as {"id": "UUID"} for backward compatibility. They are counted for resumption but not for per-workflow rate limiting or SHA dedup (no wf/sha information).

12. Workflow System

12.1. Separation of prompt and metadata

A workflow has two parts that live in different places:

  • Prompt (.md file): the LLM system prompt, verbatim. This is the investigation strategy, tool usage guidance, data patterns, and output format. Pure text, no code.
  • Metadata (YAML config): operational settings – action, model, max_iterations, data_sources, projects. This controls what the orchestrator does with the workflow, not what the model does.

This separation means changing the analysis strategy (e.g., adding a new investigation step) requires editing a markdown file. Changing operational parameters (e.g., which projects use this workflow, which model to use) requires editing the YAML config. Neither requires a code change.

12.2. System prompt layering

The system prompt sent to the model is built from three layers:

BASE_SYSTEM_PROMPT        # agent.py: sandbox rules, tool usage tips
+ tool_notes              # from ToolRegistry: per-data-source domain knowledge
+ workflow_body           # full content of e.g. workflows/analyze-failures.md

On session resumption, a fourth layer is appended:

+ CONTINUATION_PROMPT     # agent.py: "do not re-run the full workflow"

The system prompt is rebuilt every iteration (to allow warning suffixes to be appended), but the base content is stable. Warning suffixes are appended at the end so they override earlier instructions.

12.3. Design choice: strategy, not procedure

The workflow .md file describes strategy and guidance, not a rigid script. The model decides when and how to use each tool based on what it sees.

This matters because merge request failures are diverse. A fixed procedure would either miss edge cases (e.g., build failures before tests ran) or waste iterations on steps that don’t apply. By giving the model a strategy (“identify failing tests, fetch details, analyze root causes, group by similarity”), it can adapt to whatever it encounters.

The workflow does structure the investigation into phases (Data Collection, Analysis, Output) for clarity, but these are guidelines, not enforced checkpoints.

12.4. Workflow file anatomy (analyze-failures.md)

The primary workflow is structured as:

  1. Scope and approach – what this workflow analyzes and what it ignores
  2. Input – event JSON format (project, iid)
  3. Data Collection Phase:
    • Data.1: Fetch MR details (direct call, small response)
    • Data.2: Fetch commit statuses, identify failures
    • Data.3: Batch fetch PipelineRuns + TaskRuns (fetch_batch_to_sandbox)
    • Data.4: Process with jq, extract Testing Farm data in bulk
  4. Analysis Phase:
    • Analysis.1a: Investigate each failed PipelineRun individually
    • Analysis.1b: Summarize and group by root cause
  5. Output – markdown template with root causes, collapsible details, clickable links (PipelineRun, Testing Farm, test logs)
  6. Error Handling – partial report philosophy

Design considerations embedded in the workflow:

  • Efficient bulk fetching: PipelineRuns and TaskRuns are fetched in one fetch_batch_to_sandbox call, not individually.
  • Testing Farm data extracted in Data.4, analyzed in Analysis.1: All TF results.xml are fetched in bulk before analysis starts, enabling cross-failure pattern detection.
  • Log fetching is selective: Only 1-2 representative logs per failure pattern, not all logs. This keeps iteration count bounded.
  • Reviewer-facing URLs: The output template instructs the model to include clickable links using konflux_ui and artifacts_base from response metadata.

12.5. Prompt file resolution

get_prompt(workflow_cfg, config_dir) resolves the prompt: field relative to the config file’s directory:

prompt_path = config_dir / workflow_cfg.prompt
# e.g. /app/config.yaml with prompt: workflows/analyze-failures.md
# -> /app/workflows/analyze-failures.md

In the container image, workflows are baked in at /app/workflows/. A ConfigMap can override them at deploy time by mounting at the same path.

13. Deployment and Container

13.1. Container build strategy

The Containerfile uses an all-RPM builder+installroot+scratch pattern (no pip, no venv):

  1. Builder stage: Uses a Fedora-based builder image with dnf. Installs all dependencies as RPMs into a clean --installroot.
  2. Application code: Copies hummingbird_agent/ and workflows/ into the installroot at /app/.
  3. Final stage: FROM scratch, copies the entire installroot. No package manager, no shell beyond what RPMs provide.

RPM dependencies: python3-boto3, python3-google-auth+requests, python3-gitlab, python3-kubernetes, python3-pyyaml, python3-requests, python3-sentry-sdk, kubernetes1.35-client (for kubectl).

This approach was chosen over pip because:

  • All deps come from Fedora’s package repository – no PyPI supply chain risk
  • Smaller image (~22 MB saved vs. google-genai SDK alone)
  • Reproducible builds from known RPM versions
  • No compilation step (no gcc/python3-devel in the image)

13.2. Container runtime properties

CMD ["python3", "-m", "hummingbird_agent", "serve"]
WORKDIR /app
USER 65532

The default CMD runs serve mode for production. Local development uses run mode via explicit command override. USER 65532 matches the standard nonroot UID used by distroless images and the Podman sandbox.

13.3. K8s deployment manifests

Located in hummingbird-agent/kubernetes/:

deployment.yaml:

  • Single replica with rolling update (maxSurge: 1, maxUnavailable: 0)
  • terminationGracePeriodSeconds: 900 (15 minutes) to allow in-flight agent runs to complete on shutdown
  • ServiceAccount: hummingbird-agent
  • Secrets mounted from K8s Secret hummingbird-agent
  • Commented-out mounts for workflow ConfigMap and custom CA trust

rbac.yaml (applied in the sandbox namespace):

  • ServiceAccount hummingbird-agent
  • Role with pods: create, get, list, delete, patch and pods/exec: create
  • RoleBinding linking the SA to the Role
  • patch is required for pool mode (relabeling pods during claim)

sandbox-pool.yaml (applied in the sandbox namespace):

  • Deployment with replicas: 3 (tuned to balance latency vs cost)
  • Pods labelled hummingbird/role: standby for pool discovery
  • Same security context and resource limits as direct-creation pods
  • revisionHistoryLimit: 2, rolling update with maxSurge: 1, maxUnavailable: 0

networkpolicy.yaml:

  • podSelector: {} – applies to all pods in the namespace
  • egress: [] – deny all outbound traffic

secret.yaml:

  • Template with all required env vars (API keys, tokens, URLs)
  • Values must be populated per deployment

13.4. Workflow mounting

Workflows are baked into the image at /app/workflows/. To update workflows without rebuilding the image:

  1. Create a ConfigMap from the workflows directory
  2. Uncomment the volume mount in the deployment YAML
  3. The ConfigMap mount replaces the baked-in directory (all-or-nothing)

This is useful for rapid iteration in staging without waiting for a new image build.

13.5. Custom CA trust

For clusters with internal CA certificates (common in enterprise environments), the deployment supports OpenShift’s CA injection:

  1. Create a ConfigMap with the config.openshift.io/inject-trusted-cabundle label
  2. Mount it at /etc/pki/custom
  3. Set REQUESTS_CA_BUNDLE=/etc/pki/custom/ca-bundle.crt

OpenShift automatically injects the cluster’s CA bundle into the ConfigMap.

7 - Hummingbird Agent Design — Decisions

Design Decision Registry (DDR) for the Hummingbird Agent — every major architectural choice, the alternatives considered, why the chosen approach won, and what would break if reversed. For the full design context, see Hummingbird Agent Design. Provider adapter decisions are in Hummingbird AI Design.

Design Decisions

kubectl exec over kubernetes Python exec API

  • Chosen: kubectl exec as subprocess for sandbox command execution.
  • Alternative: kubernetes Python client stream() API.
  • Why: Three showstopper bugs in the Python client: (1) WebSocket v1-v4 has no stdin EOF signal, so cat > /file hangs forever; (2) BrokenPipeError on stdin larger than ~1MB; (3) stream() accumulates all stdout in memory with no control. kubectl avoids all three and provides the same subprocess interface as Podman.
  • If reversed: write_file() would hang or fail on large data. Pod log retrieval with large outputs would OOM. Sandbox reliability would drop significantly.

YAML config over env vars for operational settings

  • Chosen: Single YAML file for workflows, projects, limits, data source declarations.
  • Alternative: Everything in env vars (original design).
  • Why: Env vars cannot express structured data (workflow-project mappings, per-project token overrides, data source config with multiple fields). YAML provides structure, validation, and audit trails.
  • If reversed: Token scoping would be lost (no per-workflow-project token overrides). Project allowlists would be impossible. The config would be unauditable.

Token env var names in YAML, not values

  • Chosen: YAML contains token_env: GITLAB_TOKEN_RO (the env var name), not the actual token value.
  • Alternative: Inline secrets in YAML, or env vars for everything.
  • Why: The YAML file can be committed, reviewed, and audited. Actual secrets stay in env vars (injected via K8s Secrets). Inline secrets would make the config file a secret itself.
  • If reversed: The config file would become a secret, breaking audit trails and code review workflows.

Orchestrator token prefix convention (ORCHESTRATOR_*)

  • Chosen: Orchestrator tokens use ORCHESTRATOR_GITLAB_TOKEN_* env var names by convention.
  • Alternative: Same env var namespace as model tokens, distinguished by context.
  • Why: Structural separation makes it impossible to accidentally pass an orchestrator token to a model tool (or vice versa). The ORCHESTRATOR_ prefix is never used in YAML token_env fields.
  • If reversed: A misconfiguration could leak write-capable tokens to the model, which could then expose them via tool calls.

Auto-spill over conversation compaction

  • Chosen: Large outputs are saved to sandbox files with previews returned to the model.
  • Alternative: Replace old tool results in contents with compact summaries (conversation compaction).
  • Why: Compaction requires modifying the contents list, which risks violating provider conversation constraints. Auto-spill achieves ~90% of the token reduction without modifying model history. See Hummingbird AI for adapter format requirements.
  • If reversed: Token usage would increase ~15-30%. Per-call context would grow unbounded. Sessions would hit the context limit much sooner. Conversation compaction could be added on top of auto-spill in the future, but is not needed with current workloads.

Iteration count over cumulative token budget

  • Chosen: max_iterations as the primary cost control lever.
  • Alternative: Cumulative token budget (stop when total billed tokens exceed a threshold).
  • Why: With full history replay, each API call re-sends everything. Cumulative billed tokens double-count: call 1 = 5K, call 2 = 10K (including 5K again), total = 15K billed but only 10K new content. With auto-spill keeping per-call size bounded, iteration count is a much simpler and more predictable proxy for actual cost.
  • If reversed: The budget model would be confusing and inaccurate. Cost estimates would be wrong. The cumulative metric is still logged for observability, but it does not drive termination.

ToolDef notes in system prompt, not tool schema

  • Chosen: Domain knowledge (Konflux dual-fetch, TF XML structure) goes in ToolDef.notes, injected into the system prompt.
  • Alternative: Put everything in the tool schema description.
  • Why: Tool schemas have character limits and are sent in the tools field of every API call. Long descriptions waste tokens on tool defs. System prompt notes are sent once and can be arbitrarily detailed.
  • If reversed: Tool descriptions would be bloated. Domain knowledge would need to be duplicated in every workflow .md file.

Full history replay (no compaction)

  • Chosen: The contents list grows monotonically. Items are never removed or modified (except empty response retries).
  • Alternative: Compact old turns to reduce context size.
  • Why: Provider APIs impose strict conversation structure. Modifying or removing items risks invalid history. Auto-spill handles the growth problem at the source (preventing large items from entering history).
  • If reversed: Risk of provider API errors from malformed conversation structure and of confusing the model with summarized results.

fetch_to_sandbox kept alongside auto-spill

  • Chosen: Both fetch_to_sandbox and auto-spill coexist.
  • Alternative: Remove fetch_to_sandbox since auto-spill handles large outputs automatically.
  • Why: fetch_to_sandbox provides explicit path control (model can choose meaningful filenames), batch fetching (one tool call for multiple fetches), and no-preview responses (useful when the model will process with jq anyway).
  • If reversed: The model would lose path control and batch fetching would require multiple auto-spilled calls. The workflow would need more iterations to achieve the same result.

K8s library for lifecycle + kubectl for exec (hybrid)

  • Chosen: Use the kubernetes Python library for pod create/read/delete, kubectl subprocess for exec.
  • Alternative: All-kubectl (subprocess for everything) or all-library.
  • Why: The library provides typed pod status polling and clean error handling for lifecycle. kubectl provides reliable exec (see “kubectl exec over kubernetes Python exec API” above). Using the library for exec would require solving the three WebSocket bugs.
  • If reversed: Either lifecycle management would be fragile (parsing kubectl JSON output for pod status) or exec would be unreliable.

Single-file YAML config over multiple config files

  • Chosen: Everything in one config.yaml with settings and workflows sections.
  • Alternative: Separate files per workflow, or settings.yaml + workflows.yaml.
  • Why: Single source of truth. All project-workflow mappings visible in one place. Settings-level defaults flow down to all projects. No file discovery logic needed.
  • If reversed: Token resolution would need cross-file lookups. Project index would need multi-file aggregation. Config validation would be more complex.

Workflow .md as pure prompt, metadata in YAML

  • Chosen: The workflow .md file is pure system prompt text. Metadata (action, model, max_iterations, data_sources, projects) lives in the YAML config.
  • Alternative: YAML frontmatter in the .md file (original design).
  • Why: Separation of concerns. The .md file is the model’s instructions – it should be readable and editable by anyone writing prompts. The YAML config is the orchestrator’s instructions – it controls routing, limits, and credentials. Mixing them in one file conflates two audiences.
  • If reversed: Prompt authors would need to understand YAML config structure. Token/project config would be scattered across .md files instead of centralized.

Nudge via system prompt suffix, not user message

  • Chosen: Empty response nudge and budget warnings are appended to the system prompt as suffixes.
  • Alternative: Inject synthetic user messages into contents.
  • Why: User messages in contents must come from the actual user (the initial event, or a reply). Injecting synthetic messages pollutes the conversation history that is saved in sessions. System prompt suffixes are transient – they affect one API call without permanently modifying the conversation state.
  • If reversed: Saved sessions would contain synthetic user messages. Resumed conversations would be confusing. The model might respond to the synthetic messages instead of the user’s actual input.

Session ID reuse over new-ID-per-resume

  • Chosen: Resumed sessions keep the original session ID. S3 state is overwritten in place.
  • Alternative: Each resume generates a new UUID, saving alongside the original (append-only tree of sessions).
  • Why: GitLab discussions are linear, not branching. There is no scenario where two different sessions from the same thread are both valid. A new UUID per resume creates orphaned S3 snapshots that are never referenced again, since find_session_for_reply() always picks the latest marker. Reusing the ID is simpler, uses less storage, and matches the linear conversation model.
  • If reversed: S3 would accumulate orphaned session snapshots. Each resume would need a new note marker, but the old markers would still be in the thread, creating confusion about which session is current.

Reply to unauthorized agent-directed notes instead of silent skip

  • Chosen: When an unauthorized user sends a slash command or replies to an agent thread, the agent replies in the same discussion with a short access-denied message. Non-agent-directed notes are silently ignored (debug log, no access check).
  • Alternative: Log a warning and skip silently for all unauthorized notes (the original behavior).
  • Why: Silent skip gives no feedback to someone who intentionally tried to engage the agent, which is confusing. On the other hand, checking access and logging a warning for every random comment on a public project is noisy and pointless. Moving the access check after intent detection cleanly separates the two cases. The reply uses the discussions API so it automatically inherits confidentiality from the parent note/thread.
  • If reversed: Users without sufficient access who try /hummingbird would get no feedback. The agent log would be noisy with warnings for every comment on public MRs.

Deployment-backed pod pool over Python-thread pool

  • Chosen: A Kubernetes Deployment maintains pre-warmed standby pods. The agent claims a pod by relabeling it; the Deployment replaces it.
  • Alternative: A Python-side thread pool that pre-creates pods and queues them for use.
  • Why: The Deployment provides self-healing (restart crashed pods), native scaling (kubectl scale), rolling updates (image changes), and monitoring via standard K8s tooling. A Python pool would need to reimplement all of these.
  • If reversed: Pod replenishment, crash recovery, and image updates would all need custom code. Scaling would require agent redeployment.

Pod isolation: never reuse sandbox pods

  • Chosen: Each workflow run gets its own pod. After cleanup, the pod is deleted. Claimed pods are detached from the ReplicaSet.
  • Alternative: Return used pods to the pool and reset them.
  • Why: Residual state from a previous run (files, environment, running processes) could leak between MR investigations, creating security and correctness risks. Deletion is simple and foolproof.
  • If reversed: Would need a reliable pod-reset mechanism and auditing that no state survives between runs.

Absolute reap-by deadline over relative claimed-at age

  • Chosen: Active pods are annotated with hummingbird/reap-by (an absolute UTC timestamp). The reaper deletes pods where now > reap-by.
  • Alternative: Annotate with claimed-at and compute age relative to max_active_seconds; or use activeDeadlineSeconds on the pod spec.
  • Why: An absolute deadline simplifies the reaper to a single comparison. It also supports varying deadlines: claim sets reap-by = now + max_active_seconds, while linger sets reap-by = now + linger_seconds. With a relative timestamp, the reaper would need to know which mode the pod is in. activeDeadlineSeconds applies from pod creation, not from claim – standby pods would expire before being used.
  • If reversed: The reaper would need mode-aware age calculations. Lingering pods would require a separate annotation or reaper path.

Indefinite claim wait over timeout

  • Chosen: SandboxPool.claim() polls indefinitely (governed by shutdown event), logging at DEBUG then WARNING.
  • Alternative: Timeout after N seconds and raise an error.
  • Why: The pool is typically smaller than max_concurrent_agents for cost reasons. Waiting for a replacement pod is normal operational behavior, not an error. A timeout would cause spurious failures during burst traffic. The SQS semaphore already bounds concurrency.
  • If reversed: Burst traffic would cause avoidable failures instead of brief delays.

Pool config in Deployment only, not in agent configmap

  • Chosen: In pool mode (k8spool), the pod image, resources, labels, and security context are defined solely in the Deployment template. The agent config only needs namespace and active_deadline_seconds.
  • Alternative: Keep image/resources/metadata in the agent configmap too (as in k8s mode).
  • Why: Eliminates config duplication. The Deployment template is the single source of truth. Changes to pod resources or image only require updating one place and re-rolling the Deployment.
  • If reversed: Config drift between the Deployment and the agent configmap would be a constant risk.

Pod lingering over immediate cleanup for reply latency

  • Chosen: After a successful workflow, pool sandbox pods linger for linger_seconds (default 300) with a session-id annotation. Replies within the window reclaim the pod via try_reclaim(), skipping pod creation and S3 archive restoration.
  • Alternative: Always delete the pod immediately and restore from S3 on every reply.
  • Why: Reply latency drops from seconds (pod claim + S3 restore) to near-zero. The S3 archive is still saved as a fallback if the pod is gone. The reap-by annotation ensures lingering pods are cleaned up if no reply arrives. The session-id annotation doubles as a concurrency guard: it is only present while lingering, preventing active pods from being reclaimed.
  • If reversed: Every reply would pay full pod + restore latency, even for immediate follow-ups. User experience for conversational interactions would degrade noticeably.

linger() on Sandbox protocol over isinstance checks in runner

  • Chosen: All sandbox backends implement linger(session_id). Non-pool backends fall back to cleanup(). The runner calls sb.linger() without type checks.
  • Alternative: isinstance(sb, K8sPoolSandbox) in run_workflow().
  • Why: Keeps the runner backend-agnostic. Adding a new backend requires only implementing the protocol, not touching the runner. The fallback behavior is co-located with each backend.
  • If reversed: The runner would need to know about every backend type and their linger capabilities.

GCP SA key over Workload Identity Federation

  • Chosen: GCP service account key stored in Vault, rotated via cki-tools credential manager (prepare/switch/clean cycle).
  • Alternative: Workload Identity Federation (WIF) with projected SA token. Two sub-options: (a) automatic OIDC discovery if the cluster issuer is public, (b) manual JWKS upload for internal issuers.
  • Why: mpp-prod’s OIDC issuer is https://kubernetes.default.svc (internal, not publicly reachable), so GCP STS cannot discover it automatically. Manual JWKS upload works but requires re-upload after SRE-triggered signing key rotations. SA key integrates with the existing credential manager rotation infrastructure (same pattern as AWS keys and GitLab tokens), needs no OIDC reachability, and enables automated validate/update via google-auth in CI. The application code (hummingbird_ai.http.VertexAuth / google.auth.default()) works identically with both approaches – switching to WIF later requires only infrastructure changes.
  • If reversed: Replace SA key with WIF projected token + credential config ConfigMap. The gcp_service_account_key token type in cki-tools would no longer be needed for this use case.

Developer+ trust filtering over unfiltered or sanitized discussion comments

  • Chosen: gitlab_get_mr_discussions checks each note author’s project access level (Developer+ / >= 30). Untrusted notes are replaced with a fixed placeholder in mixed-trust discussions, or the entire discussion is dropped if all notes are untrusted. Agent notes are always trusted (detected by session marker) but have transcripts and metadata stripped.
  • Alternatives considered:
    • (a) Include all comments unfiltered. Simplest, but any external contributor can craft comments that manipulate the model’s review output (prompt injection).
    • (b) Sanitize/escape untrusted content (strip markdown, quote as code blocks, prefix with “[external]”). There is no reliable escaping mechanism for LLM prompts – the model interprets natural language regardless of formatting. Escaping gives a false sense of security.
    • (c) Only include agent-authored notes (skip all human comments). Safe, but defeats the purpose: the model would never see developer responses to its own findings.
    • (d) Include only discussions where the agent participated. Better, but still misses developer-initiated review threads that provide relevant context.
  • Why: Developer+ is the same threshold used for pipeline trigger authorization (invariant #8) and slash command access. It matches the trust boundary already established: people who can push code and approve MRs are trusted to provide review context. The placeholder approach preserves discussion structure (the model sees that someone replied) without exposing the content. Full-drop for all-untrusted discussions avoids noise from discussions that contain zero useful context.
  • If reversed: (a) opens a prompt injection vector on any project that accepts external MRs. (b) provides no actual protection. (c) makes follow-up reviews unable to see developer explanations, causing repeated false positives. (d) misses developer-initiated context.

Bot-author filtering for session marker parsing over unfiltered note scanning

  • Chosen: find_session_for_reply(), scan_agent_threads(), and the handle_note() self-filter all resolve the bot user ID via get_bot_user_id() (which calls gl.auth() on the orchestrator token) and only consider notes where author.id matches. In find_session_for_reply(), the first matching marker wins.
  • Alternative: Parse markers from any note and keep the last match; substring self-filter in handle_note (previous behavior).
  • Why: The previous unfiltered approach allowed session hijacking: a user reply or agent review prose containing an example marker (<!-- hummingbird-session: ... -->) was parsed as a real session, causing “session expired” errors in production. For handle_note, the substring check caused the reverse problem: a user embedding the marker prefix in a reply silently suppressed the event (denial of service). The gl.auth() call is one GET /user per invocation – negligible cost. _get_client_with_bot_id() returns both the authenticated client and the bot user ID, avoiding redundant client construction.
  • If reversed: Any MR participant can inject a marker to hijack the session ID or inflate rate-limit counts. The agent’s own review prose containing example markers causes spurious “session expired” messages.

ijson streaming over buffered JSON for K8s list responses

  • Chosen: session.get(stream=True) + ijson.parse(resp.raw) for iter_paginated(). Items are yielded one at a time via ObjectBuilder; the metadata.continue pagination token is captured in the same parse pass. resp.raw.decode_content = True is required because Kubearchive returns gzip Content-Encoding; without it ijson sees compressed bytes.
  • Alternative: session.get().json() loads the full page into memory (the original implementation).
  • Why: A single K8s list page can contain 500 PipelineRuns (80+ MB JSON). Parsing this with .json() creates a +247 MB RSS spike (raw bytes + decoded string + parsed dict coexist). With 4 concurrent workflows, this exceeds any reasonable pod memory limit. ijson stream-parsing yields items one at a time, reducing the peak to +12 MB for the same data – a 95% reduction. The continue token appears in the metadata object (before or after items depending on the server); the event-driven parser captures it regardless of order.
  • If reversed: Large MRs (100+ components) would OOM the agent pod. Concurrent workflows would multiply the problem. The pod memory limit would need to scale with the largest possible page size, which is unbounded.

Action token tier for per-workflow identity

  • Chosen: A dedicated Tier 2 of workflow action tokens, separate from both orchestrator tokens (Tier 1) and model tool tokens (Tier 3). Each workflow gets a dedicated GitLab bot user per project for all write operations (notes, thread resolution).
  • Alternative: Extend orchestrator tokens with a per-workflow fallback chain (ORCHESTRATOR_<WORKFLOW>_GITLAB_TOKEN_<PROJECT> → ORCHESTRATOR_GITLAB_TOKEN_<PROJECT> → ORCHESTRATOR_GITLAB_TOKEN).
  • Why: Orchestrator tokens serve a fundamentally different purpose (operational reads + infrastructure notes) than workflow output (analysis results, review comments). Mixing them in one fallback chain risks a single bot identity posting notes on behalf of multiple workflows, eliminating the audit trail. A separate tier with explicit token: str parameters makes the code path unambiguous: callers must choose which tier they use. The naming convention (HUMMINGBIRD_AGENT_ACTION_<WORKFLOW>_GITLAB_TOKEN_<PROJECT>) groups tokens alphabetically by workflow, making env var auditing straightforward.
  • If reversed: All workflow notes would appear from the same bot user, making it impossible to distinguish which agent (code-review vs analyze-failures vs renovate-babysit) produced a given note in the MR timeline. GitLab’s audit log would attribute all actions to one identity.

Per-project-only action token config (no workflow-level default)

  • Chosen: action_tokens is defined at the project level only in YAML config. No workflow-level action_tokens default is supported.
  • Alternative: Allow workflow-level action_tokens that projects inherit by default, with per-project overrides.
  • Why: GitLab project access tokens are project-scoped. A workflow-level default would be misleading – a single token cannot write to multiple projects. Forcing per-project declaration makes the mapping explicit and prevents configuration errors where a cross-project default silently fails.
  • If reversed: Operators could set a workflow-level token that only works for one project, causing silent 403 errors for other projects in the same workflow.

SSH over kubectl exec for KubeVirt VMs

  • Chosen: SSH (ssh -T sandbox@ip command) for VM command execution.
  • Alternative: virtctl console (serial console), KubeVirt websocket API, or qemu-guest-agent exec.
  • Why: VMs have no kubectl exec equivalent. virtctl console provides a serial terminal, not a programmable exec channel. The websocket API requires a VNC or serial connection. qemu-guest-agent is not installed in the VM image. SSH provides clean stdin/stdout/stderr separation, timeout control, and is already well-tested in subprocess pipelines.
  • If reversed: Would need qemu-guest-agent in the image and the exec endpoint, which has poorer error reporting and no streaming stdin.

Secret volume for SSH key injection

  • Chosen: Kubernetes Secret mounted as a virtio disk (serial ssh-pubkeys) read by the VM’s inject-ssh-keys.service at boot.
  • Alternative: cloudInitNoCloud or accessCredentials with qemu-guest-agent.
  • Why: The VM image already has inject-ssh-keys.service which reads from the virtio disk. cloudInitNoCloud requires cloud-init in the image. accessCredentials requires qemu-guest-agent. The Secret volume approach requires no additional software in the VM image.
  • If reversed: Would need cloud-init or guest agent in the bootc image, adding complexity and attack surface.

VMIRS for VM pool replenishment

  • Chosen: VirtualMachineInstanceReplicaSet (VMIRS) as the replenishment controller for the VM pool.
  • Alternative: Custom controller, or VirtualMachine with RunStrategy.
  • Why: VMIRS is KubeVirt’s native equivalent of a Deployment’s ReplicaSet – it maintains a desired count of identical VMIs. When a VMI is claimed (ownerReferences cleared), the VMIRS creates a replacement automatically. Same pattern as the pod pool with Deployment.
  • If reversed: Would need a custom controller to maintain the VMI pool, adding operational complexity.

Per-workflow sandbox backend

  • Chosen: sandbox: field in workflow config, resolved per-workflow with fallback to CLI --sandbox flag.
  • Alternative: Global-only sandbox selection via CLI.
  • Why: Different workflows have different isolation requirements. Code review needs only a lightweight container, while failure analysis may need a full VM with root access. Per-workflow selection avoids running all workflows in VMs (wasteful) or all in containers (insufficient isolation).
  • If reversed: All workflows would share the same sandbox backend, requiring either over-provisioning (VMs for everything) or accepting weaker isolation for some workflows.