# Hummingbird Agent Design — Runtime LLMS index: [llms.txt](https://hummingbird-project.io/llms.txt) | Full content: [llms-full.txt](https://hummingbird-project.io/llms-full.txt) --- Data sources, event pipeline, sessions, workflows, and deployment for the Hummingbird Agent. This page covers the runtime: how the agent connects to external systems, processes events, manages sessions, and is deployed. For the execution engine (agent loop, tool system, sandbox), see [Hummingbird Agent Design — Engine](https://hummingbird-project.io/docs/background/tools/hummingbird-agent-design-engine/). ## 9. Data Sources Data sources are external APIs wrapped as tool-calling functions. The model invokes them by name; the orchestrator executes them and returns results (inline or auto-spilled). Each data source module follows the same pattern: 1. Define `TOOL_DEFS` -- a list of `ToolDef` objects with names, descriptions, parameter schemas, and optional notes. 2. Define implementation functions that take a pre-configured client as the first argument. 3. Provide a `register()` function that creates the client, wraps implementations with `functools.partial`, and calls `registry.register_data_source()` for each tool. Credentials are captured at registration time via `functools.partial` closures. They are never stored as global state and never leak into tool definitions or model contents. ### 9.1. GitLab Uses `python-gitlab` library with `retry_transient_errors=True` for automatic retry on transient HTTP errors. **Tools:** - `gitlab_get_mr_details` -- MR metadata (title, author, SHA, labels, URLs) - `gitlab_get_commit_statuses` -- all CI/CD statuses for a commit SHA (paginated automatically). Covers both Konflux external statuses and native GitLab CI job statuses. Each status includes `target_url` (job page link containing the job ID) and `allow_failure`. - `gitlab_get_mr_diff` -- changed files in the MR - `gitlab_get_file_at_ref` -- raw file content at a git ref - `gitlab_get_repo_archive` -- repository tar.gz via `repository_archive(iterator=True)`. Returns a `StreamingResponse` with chunked binary data (suffix `.tar.gz`), so the archive streams to the sandbox without buffering in orchestrator memory. - `gitlab_get_job_log` -- job trace (log output) for a GitLab CI job. Uses `lazy=True` on the job object and `trace(iterator=True)` for streaming. Each chunk is decoded with `errors='replace'` and ANSI escape codes are stripped per-chunk before re-encoding. Returns a `StreamingResponse` (suffix `.log`). Per-chunk ANSI stripping is safe because escape sequences are <20 bytes and chunks are 1024+ bytes. No `response_metadata` is set because GitLab commit statuses already contain `target_url` fields that the model uses for linking. **`gitlab_get_mr_discussions` -- trust filtering and redaction.** MR discussions are the first data source that feeds _user-generated free text_ into the model prompt. Unlike diffs and CI logs (which are code or machine output), discussion comments can contain arbitrary prose written by anyone who can post on the MR -- including external contributors on public projects. This creates a prompt injection vector: an attacker posts a comment containing instructions that manipulate the model's review output. The discussions tool addresses this with a three-layer defence: 1. **Author trust gate.** Each note's author is checked for Developer+ access (level >= 30) on the project via `members_all.get(author_id)`. Results are cached per `author_id` within a single call to avoid repeated API lookups. System notes (merge events, label changes) bypass the author check. Agent-authored notes are trusted via their author's Developer+ access (the bot account holds Developer access on configured projects). Note: agent note detection for _redaction_ purposes (stripping transcripts and footers) uses session marker presence, but trust is always based on the author's access level, not the marker. This prevents marker injection from granting trust to non-member notes. 2. **Redaction, not sanitization.** Untrusted notes in a discussion that also contains trusted notes are replaced with a fixed placeholder (`[redacted: non-member comment]`). The placeholder preserves the conversation structure (the model sees that someone replied, but not what they said). Discussions where _all_ notes are untrusted are dropped entirely. No attempt is made to sanitize or escape untrusted content -- there is no reliable escaping mechanism for LLM prompts, so the only safe option is to withhold the content entirely. 3. **Agent note stripping.** Agent-authored notes contain large `
Agent transcript...
` blocks (15K-90K chars of raw tool calls), metadata footers, and session markers. These are stripped before the note enters the model prompt, leaving only the review text. This serves dual purposes: token efficiency and avoiding feeding the model its own raw tool call history (which could cause degenerate self-referential loops). **Residual risks and scope limitations:** - A compromised Developer+ account can inject adversarial content. This is accepted as equivalent to the existing risk of a compromised developer pushing malicious code (which the agent would also process). - The trust threshold is project-level (Developer role on the project), not MR-level. A developer on the project can influence any MR's review. - Comments posted _after_ the discussions are fetched but _before_ the review is posted are not seen. This is a TOCTOU gap but has no security impact (the model simply misses late comments). ### 9.2. Konflux Uses raw `requests` against K8s and Kubearchive APIs (not the kubernetes Python SDK). This avoids the heavy kubernetes client dependency for what is essentially bearer-token HTTP with label selectors. **Dual-fetch architecture:** ```mermaid flowchart LR subgraph fetch [Fetch Phase] KA["Kubearchive
(historical)"] K8s["K8s API
(live)"] end Combine["Combine items"] Dedup["Deduplicate
by metadata.uid"] KA --> Combine K8s --> Combine Combine --> Dedup ``` For each resource type (PipelineRuns, TaskRuns), the client fetches from both Kubearchive (completed/historical resources) and the live K8s API, combines the results, and deduplicates by `metadata.uid`. This ensures no resources are missed regardless of whether they have been archived yet. **Two label selectors.** Konflux uses different labels for BUILD and TEST PipelineRuns: - BUILD: `pipelinesascode.tekton.dev/sha=` - TEST: `pac.test.appstudio.openshift.io/sha=` Each `iter_pipelineruns()`/`iter_taskruns()` call queries both selectors and combines the results. `_iter_resources(endpoint, label_selectors)` is the generic implementation: it creates one session, iterates each selector through `_iter_both_sources()`, and deduplicates across selectors by UID. **Per-PLR TaskRun fetching.** `iter_taskruns()` accepts either `commit_sha` (two-selector path) or `pipelinerun` (single selector: `tekton.dev/pipelineRun=`). When `pipelinerun` is set, only the ~15-21 TaskRuns for that PipelineRun are fetched (~600 KB) instead of all TaskRuns for the commit (~530 MB for large commits). This is server-side filtering — the K8s API and Kubearchive both support the `tekton.dev/pipelineRun` label selector. **Status filtering and streaming counts.** `_items_to_jsonl(items, status, stats)` accepts an optional `status: set[str] | None` allowlist and a `_StreamStats` accumulator. For each item it increments `stats.by_reason` (keyed by reason) regardless of the filter (so counts reflect all items seen), then skips items whose `status.conditions[0].status` is not in the allowlist. The allowlist values are `"True"` (success), `"False"` (failure), or `"Unknown"` (in-progress) — only 3 possible values, covering every Tekton reason without enumeration. Items without conditions always pass. The `_finalize()` generator wrapper exhausts the chunks iterator, then writes `counts`, `included`, and `excluded` into the `StreamingResponse.metadata` dict — after the sandbox file is fully written. The filtering happens O(1) per item with no buffering. **`StreamingResponse.metadata`.** `StreamingResponse` (in `tools.py`) has a mutable `metadata: dict[str, Any]` field (default empty dict). Both `_exec_fetch_to_sandbox()` and `_spill_streaming()` merge it into the response dict after the iterator is exhausted. The Konflux code populates it with `counts`, `included`, and `excluded`; other data sources can use it for their own post-streaming summaries without touching `tools.py`. **Pod log fetching.** `get_pod_log()` accepts an optional `container` parameter. When specified, it fetches logs for that single container. When omitted, it discovers all containers from the pod spec and concatenates their logs with `=== container_name ===` headers. Each individual log fetch tries Kubearchive first, then falls back to the live K8s API. Logs may be unavailable from both sources if the pod has expired. **Streaming pagination.** K8s list endpoints can return pages of 80+ MB when a commit touches many components (e.g. 500 PipelineRuns per page). `iter_paginated()` uses `session.get(stream=True)` and `ijson.parse()` to stream-parse each page: items are yielded one at a time via `ObjectBuilder`, and the `metadata.continue` pagination token is captured from the same parse pass. `resp.raw.decode_content = True` is set so urllib3 transparently decompresses gzip/deflate Content-Encoding inline (Kubearchive returns gzip-compressed responses). Only one item is in memory at a time -- O(single\_item) regardless of page size. Truncated or malformed streams raise `IncompleteJSONError`, which is caught and treated like a network error (log a warning, stop iterating that source). **Credential resolution.** `KonfluxClient.__init__()` parses the kubeconfig file to extract the API server URL and bearer token for the cluster. The cluster domain is extracted from the `cluster_url` config value. This approach avoids depending on `kubectl` or the kubernetes Python library for API authentication. **Response metadata:** Static metadata `{"konflux_ui": "...", "konflux_pipelinerun_url_pattern": "..."}` is merged into every response so the model can build reviewer-facing links. Dynamic per-call metadata (`counts`, `included`, `excluded`) is populated into `StreamingResponse.metadata` by `_finalize()` during streaming and merged by `tools.py` after the iterator is exhausted. ### 9.3. Testing Farm Uses `requests.Session` with a module-level retry adapter (429, 5xx) for resilience against transient errors. **Tools:** - `tf_get_results` -- JUnit XML results for a request ID - `tf_get_test_log` -- individual test log by URL (from results.xml); restricted to the `ARTIFACTS_BASE` prefix to prevent SSRF - `tf_get_request_status` -- request state, queue/run times **ToolDef notes** on `tf_get_results` document the XML structure: `//testsuites/@overall-result`, `//testcase/@result`, `//testcase/logs/log` with `@name` and `@href`. This domain knowledge goes into the system prompt so the model knows how to parse the XML with `sandbox_exec` using `python3 xml.etree.ElementTree`. **Response metadata:** `{"artifacts_base": "https://artifacts.osci.redhat.com/testing-farm"}` is merged into every response so the model can build artifact links like `[{request_id}]({artifacts_base}/{request_id}/)`. ### 9.4. Registration flow `data_sources.register_selected()` is the entry point called by `runner.py`. It iterates the workflow config's `data_sources` dict and registers only the declared sources: ```python for ds_name, ds_cfg in wf_cfg.data_sources.items(): if ds_name == "gitlab": token = resolve_tool_token(proj_entry, "gitlab", wf_cfg) gitlab.register(registry, gitlab_url, token) elif ds_name == "konflux": cluster_url = resolve_cluster_url(ds_cfg) kubeconfig_path = os.environ.get(ds_cfg.kubeconfig_env, "") konflux.register(registry, cluster_url, kubeconfig_path, ds_cfg.kubearchive_url) elif ds_name == "testing_farm": testing_farm.register(registry) ``` This selective registration means a workflow with `data_sources: {gitlab: {...}}` only exposes GitLab tools to the model. Konflux and Testing Farm tools do not appear in the tool definitions, preventing the model from attempting to use unconfigured data sources. ### 9.5. Data flow: streaming vs buffered Data moves from external APIs through the orchestrator to the model (inline or auto-spilled to sandbox). The memory profile of each tool depends on whether the HTTP response is consumed incrementally or buffered entirely. **Streaming (preferred for large/unbounded responses).** The HTTP response is consumed incrementally -- via `ijson` stream parsing, `iterator=True` in python-gitlab, or lazy pagination. Only one chunk or item is in memory at a time. Used by: - `iter_paginated()` (Konflux) -- `session.get(stream=True)` + `ijson` - `get_commit_statuses()` (GitLab) -- `statuses.list(iterator=True)` via `_iter_commit_statuses()`, yielding JSONL bytes one status at a time - `get_repo_archive()` (GitLab) -- `repository_archive(iterator=True)`, returns chunked tar.gz binary via `StreamingResponse(suffix=".tar.gz", binary=True)` -- no text preview is generated - `get_job_log()` (GitLab) -- `trace(iterator=True)` with per-chunk ANSI stripping, returns `StreamingResponse(suffix=".log")` All produce `StreamingResponse` objects that flow through `stream_to_file()` into the sandbox without accumulating in orchestrator memory. `StreamingResponse.suffix` controls the auto-spill filename extension (`.jsonl`, `.tar.gz`, `.log`). `StreamingResponse.binary` controls whether a head preview is captured (`False` for text, `True` to skip for opaque binary formats). **Buffered (acceptable for small/bounded responses).** The full response is loaded into memory as a string or dict. This is fine when the response size is bounded and small (typically < 1 MB). Used by: - `get_mr_details()` (GitLab) -- single MR object, < 10 KB - `get_file_at_ref()` (GitLab) -- single file, bounded by repo constraints - `tf_get_results()`, `tf_get_test_log()`, `tf_get_request_status()` (Testing Farm) -- XML/text/JSON, typically < 1 MB **Buffered with risk (candidates for future streaming).** Same buffered pattern but the response size is not bounded by design. Auto-spill mitigates the context growth problem (large outputs are written to sandbox files), but the orchestrator still spikes RSS during the fetch: - `get_pod_log()` (Konflux) -- `resp.text` per container, concatenated. Multi-container pods accumulate all logs. - `get_mr_diff()` (GitLab) -- full diff JSON, scales with MR size. GitLab truncates server-side but the result can still be large. **Invariant: paginated K8s list endpoints must always use streaming.** Page sizes scale with the number of components in the commit -- a single page can contain hundreds of PipelineRuns or TaskRuns (80+ MB JSON). Buffering these responses risks OOM under normal production workloads, especially with concurrent workflows. ## 10. Event Pipeline and Production Operations ### 10.1. Two-stage SQS pipeline (ingress router + FIFO worker) Events flow through a two-stage pipeline to serialize per-session work: ```mermaid flowchart LR STD["Standard SQS
(ingress)"] R["Router"] FIFO["SQS FIFO
(work queue)"] W["Worker
(poll_loop)"] STD -->|ReceiveMessage| R R -->|"SendMessage
MessageGroupId"| FIFO W -->|"continuation
group=session_id"| FIFO FIFO -->|ReceiveMessage| W ``` **Ingress router.** `events.router_loop()` is a single-threaded loop that long-polls the standard (ingress) queue, peeks into each SNS envelope to assign a `MessageGroupId`, forwards the raw message body to the FIFO queue, then deletes from the standard queue. - **Note events:** `MessageGroupId = discussion_id` from the webhook body. This serializes replies to the same agent session -- FIFO delivers at most one in-flight message per group. - **Pipeline / MR events:** `MessageGroupId = SQS MessageId` (unique per message). No serialization -- each event is its own group. - **MessageDeduplicationId:** Always the SQS `MessageId` from the standard queue. Absorbs duplicate deliveries from standard SQS's at-least-once semantics (5-minute dedup window). If `SendMessage` to FIFO fails, the message is not deleted from the standard queue and retries via visibility timeout. **FIFO worker.** `events.poll_loop()` long-polls the FIFO queue, decodes SNS envelopes, and dispatches to `handle_event`. The `max_concurrent_agents` semaphore caps parallel groups being processed. FIFO guarantees that within a group, only one message is in-flight. **Two-phase processing.** When the FIFO worker picks up a webhook (Phase 1), the handler validates the event, resolves the `session_id`, then posts a **continuation message** back to the same FIFO with `MessageGroupId = session_id`. The Phase 1 message is deleted quickly (~1-5s). Phase 2 processes the continuation: loading session state, building `WorkflowRequest`, and calling `_execute_workflow`. Because Phase 2 continuations share a `MessageGroupId` per session, all work for a given session is serialized -- even if it spans multiple GitLab discussion threads. **Uniform message format.** Both webhook messages (from SNS) and continuation messages use the same SNS-style envelope with gzip+base64 compression. `encode_envelope()` is the inverse of `decode_sns_message()`. The consumer doesn't need to distinguish between webhook and continuation messages at the transport layer. **Graceful shutdown.** SIGTERM/SIGINT set a `shutdown_event`. Both loops exit. Messages in SQS become visible again after the visibility timeout for other consumers. **Why two queues.** The standard queue receives events from SNS (via subscription). The FIFO queue serializes per-session work. The gitlab-event-forwarder and SNS subscription are unchanged -- the routing logic lives entirely in the agent codebase. ### 10.2. SNS envelope decoding Messages arrive as SNS notifications with: - `MessageAttributes.source` -- event source (e.g., `"gitlab"`) - `MessageAttributes.event_type` -- event type (e.g., `"pipeline"`, `"merge_request"`, `"note"`) - `MessageAttributes.content_encoding` -- `"gzip+base64"` for gitlab-event-forwarder, absent for kubernetes-event-forwarder - `Message` -- the actual event body (JSON string, or gzip+base64 encoded) `decode_sns_message()` handles both encoding formats transparently. ### 10.3. Event routing `runner.handle_event()` dispatches on `(source, event_type)`: ```mermaid flowchart TD Event["Event arrives"] Check{"source/type?"} Cont["handle_continuation()"] Pipeline["handle_pipeline()"] MRHandler["handle_merge_request()"] Note["handle_note()"] Ignore["Ignore"] Event --> Check Check -->|"agent/continuation"| Cont Check -->|"gitlab/pipeline"| Pipeline Check -->|"gitlab/merge_request"| MRHandler Check -->|"gitlab/note"| Note Check -->|"other"| Ignore Pipeline --> FilterSource{"source = merge_request_event?"} FilterSource -->|"yes"| AutoResolve["auto-resolve on success
(unconditional)"] FilterSource -->|"no"| Skip1["Skip"] AutoResolve --> PipeRules["evaluate trigger_rules
(status + user + branch + title)"] PipeRules -->|"allow"| Enqueue1["_enqueue_continuation()"] PipeRules -->|"deny"| Skip1a["Skip"] MRHandler --> FilterMRAction{"action in open/update?
draft = false?"} FilterMRAction -->|"yes"| MRRules["evaluate trigger_rules
(user + branch + title)"] FilterMRAction -->|"no"| Skip1b["Skip"] MRRules -->|"allow"| Enqueue2["_enqueue_continuation()"] MRRules -->|"deny"| Skip1c["Skip"] Note --> FilterMRNote{"MR note?
action = create?"} FilterMRNote -->|"yes"| NoteType FilterMRNote -->|"no"| Skip3["Skip"] NoteType{"Note type?"} NoteType -->|"author_id == bot_id"| Skip4["Skip (own note)"] NoteType -->|"/hummingbird help"| Help["post_simple_reply (help)"] NoteType -->|"/hummingbird wf-name"| SlashCmd["_resolve_slash_workflow()
→ _enqueue_continuation()"] NoteType -->|"DiscussionNote reply"| FindSession["find_session_for_reply()"] NoteType -->|"other"| Skip5["Skip"] FindSession --> Found{"session found?"} Found -->|"yes"| Enqueue3["_enqueue_continuation()
(resume_session)"] Found -->|"no"| Skip6["Skip"] Cont --> ContType{"work_type?"} ContType -->|"new_session"| ExecNew["_execute_workflow()"] ContType -->|"resume_session"| LoadS3["load session from S3
→ _execute_workflow()"] ``` **Phase 1 handlers** (`handle_pipeline`, `handle_merge_request`, `handle_note`) perform validation, auth checks, and session resolution, then post a `WorkOrder` continuation message via `_enqueue_continuation()`. They do not call `_execute_workflow` directly. **Phase 2** (`handle_continuation`) receives the `WorkOrder`, resolves the workflow config, builds a `WorkflowRequest`, and calls `_execute_workflow`. For `resume_session` orders, it loads the previous session from S3. `_execute_workflow` handles SHA dedup and per-workflow rate limiting via `scan_agent_threads`. Slash commands dispatch to a single named workflow via `_resolve_slash_workflow` (exact match, then prefix match). Help and error replies are posted via `post_simple_reply` (no session marker, no rate limit impact). ### 10.4. Pipeline trigger design **Why `gitlab::pipeline` instead of `kubernetes::PipelineRun`:** A GitLab pipeline event fires once when the pipeline completes. Since Konflux external stages are attached to the pipeline, the event naturally waits for all builds and tests to finish before triggering. This means the agent sees the full picture in one event, without needing to deduplicate or wait for stragglers. **Trigger filters:** - `status` in `{failed}` -- only failed pipelines trigger analysis - `source == "merge_request_event"` -- only MR pipelines, not branch/tag ### 10.4a. Merge request trigger design The `handle_merge_request` handler fires on MR open, reopen, and update events (`action in {"open", "reopen", "update"}`). Draft MRs are skipped (`object_attributes.draft == True`); marking a draft as ready triggers a review since the event arrives with `draft: false`. Event classification is deliberately simple: the handler does not inspect `oldrev` or `changes.draft` fields. Instead, SHA-based deduplication in `_execute_workflow` (via JSON session markers) ensures each code revision is reviewed at most once. Metadata-only updates (title, label changes) on an already-reviewed SHA are silently skipped. ### 10.5. Note trigger design Two sub-flows: **Slash command** (`/hummingbird `): Triggers a specific named workflow, bypassing rate limiting. `_resolve_slash_workflow` performs exact-match lookup first, then falls back to unique prefix matching. If the subcommand is missing or is "help", `_format_help` returns a list of available workflows with descriptions. Ambiguous or unknown subcommands produce an error message via `post_simple_reply`. The note author must have Developer+ access (level >= 30) on the project, checked via `check_member_access()`. If denied, the agent replies in the same discussion thread with a short access-denied message. **Reply to agent note:** When a user replies to an existing agent note (which contains a JSON session marker): 1. `find_session_for_reply()` walks the discussion thread, filtering by bot author ID (`get_bot_user_id()`), and returns the first matching session marker. All bot markers in a thread share the same session ID. 2. `handle_note` posts a `resume_session` continuation to the FIFO (grouped by `session_id`). 3. `handle_continuation` loads the session from S3 (conversation history and sandbox archive), builds a `WorkflowRequest`, and calls `_execute_workflow()` to resume with the user's reply. 4. If the session is not found (expired/deleted), the agent replies with a message explaining the session has expired and suggesting to start a new run. Reply authors are also subject to the Developer+ access check. If denied, the agent replies in the discussion thread with the same access-denied message. **Non-agent-directed notes:** Regular comments that are neither slash commands nor replies to agent threads are silently ignored (debug-level log). No access check is performed for these. **Self-note filtering:** Notes where `author_id` matches the bot's own user ID (resolved via `get_bot_user_id()`) are skipped immediately. This prevents infinite loops where the agent's own output triggers another agent run. The author-ID check replaces the previous substring check (`marker_prefix_str in note_body`), which was vulnerable to denial-of-service: a user including the marker prefix in their reply would cause the bot to silently ignore it. **Threading logic for replies:** If the project requires internal notes (`internal_notes: true`) but the original discussion was public, the reply is posted as a new top-level internal note instead of replying in the public thread. This prevents leaking internal analysis into public threads. ### 10.6. Note lifecycle ```text 1. create_placeholder_note() -> "Running the workflow..." + JSON session marker (id, wf, sha) 2. run_workflow() -> agent loop 3. update_note() -> replace placeholder with result + reply prompt + JSON session marker Reply notes (from session resumption) omit the reply prompt since the user is already engaged in the conversation. ``` On failure, the placeholder is updated to "Hummingbird analysis failed." The JSON session marker is always present so the note can be identified as agent-generated (for rate limiting, SHA dedup, and self-note filtering). The marker embeds the workflow name and commit SHA, enabling per-workflow rate limiting and SHA-based deduplication. ### 10.7. Rate limiting and SHA deduplication `scan_agent_threads()` performs a single pass over MR discussions, parsing JSON session markers (``) to determine: 1. **Per-workflow thread count** -- how many threads the given workflow has created on this MR. If the count meets or exceeds `max_runs_per_mr`, the workflow is skipped. 2. **SHA dedup** -- whether the current commit SHA has already been reviewed by this workflow. If so, the workflow is skipped (prevents re-reviewing the same code on metadata-only MR updates). Slash commands and replies bypass both checks entirely. The rate limit is per-workflow per-MR: different workflows maintain independent thread counts on the same MR. JSON markers are backward compatible -- old plain-UUID markers (``) are parsed as `{"id": "UUID"}` with no workflow or SHA information, so they are not counted toward any specific workflow's limit. ## 11. Session System Sessions enable conversation continuity: a user can reply to an agent note and the agent picks up where it left off, with full context and sandbox files restored. ### 11.1. What gets saved Three artifacts are saved after each run: - **`context.json`** -- the full `contents` list from the agent loop in the canonical provider-independent format. It is the minimum state needed to resume the conversation; format conversion is handled by the AI library. - **`transcript.md`** -- a human-readable markdown rendering of the run for debugging and auditing. Not used for resumption. - **`sandbox.tar.gz`** -- an archive of `/tmp/data/` (the sandbox working directory, which includes the `_out/` spill subdirectory). Contains PipelineRun JSONs, test logs, jq output, and any other files the model created during the run. Restored into the new sandbox on resumption so the model can reference its previous work. ### 11.2. Storage backends **S3 (production):** ```text s3://{bucket}/sessions/{session_id}/context.json s3://{bucket}/sessions/{session_id}/transcript.md s3://{bucket}/sessions/{session_id}/sandbox.tar.gz ``` **Local directory (development, `--save-session`):** ```text {directory}/context.json {directory}/transcript.md {directory}/sandbox.tar.gz ``` Both backends have the same interface. S3 save is best-effort: wrapped in try/except so a transient S3 error does not prevent the MR note from being posted. The note always gets delivered first. ### 11.3. Sandbox archive transport The sandbox archive requires special handling because the orchestrator cannot directly access the sandbox filesystem (no volume mounts): ```text Archive (inside sandbox): sb.exec("tar czf /tmp/_archive.tar.gz -C /tmp data") Stream out (sandbox -> host temp file): for chunk in sb.read_file_iter("/tmp/_archive.tar.gz"): tmp.write(chunk) # 64 KB chunks, no base64 Restore (host temp file -> new sandbox): sb.stream_exec("tar xzf - -C /tmp", file_chunks()) ``` Both directions use streaming binary I/O via Popen stdin/stdout pipes. No base64 encoding is needed: `read_file_iter()` yields raw bytes from the sandbox via a Popen stdout pipe, and `stream_exec()` feeds raw bytes into a Popen stdin pipe. The archive is never extracted on the orchestrator -- it exists only as opaque bytes being transported between sandboxes. This is a security invariant: the orchestrator never parses or inspects the archive contents. ### 11.4. Resumption flow ```mermaid flowchart TD Reply["User replies to agent note"] FindSession["find_session_for_reply()
walks discussion thread"] LoadS3["load_s3(session_id)"] NotFound{"session found?"} Expired["Post session-expired reply"] NewSandbox["Start new sandbox"] Restore["restore_sandbox(archive)"] BuildContents["contents = old_contents + user_reply"] RunLoop["run_agent_loop(
initial_contents, CONTINUATION_PROMPT)"] Reply --> FindSession --> LoadS3 LoadS3 --> NotFound NotFound -->|"no"| Expired NotFound -->|"yes"| NewSandbox NewSandbox --> Restore --> BuildContents --> RunLoop ``` Key aspects: - **Fresh iteration budget.** The resumed session starts iteration 0 with the full `max_iterations` budget, regardless of how many iterations the previous session used. - **Context window is the real limit.** A typical 8-iteration cold start uses ~50-100K input tokens. The restored history is sent in full on every API call. The `CONTEXT_LIMIT` check applies and will force wrap-up if needed. - **Session ID reuse.** The resumed session keeps the original session ID. S3 state is overwritten in place with the updated conversation history. Since GitLab discussions are linear (not branching), there is no need for a tree of sessions -- the conversation is a single sequential thread. - **Graceful degradation.** If the S3 session is not found (expired, deleted), the agent posts a reply explaining the session has expired and suggesting to start a new run. It does not fall back to a cold start. ### 11.5. Session markers Every agent note contains a hidden HTML comment with a JSON payload: ```html ``` The JSON payload contains: - `id` -- session UUID (always present) - `wf` -- workflow name (present for new-format markers) - `sha` -- commit SHA at the time of review (present for auto-triggered runs) This marker serves four purposes: 1. **Resumption:** `find_session_for_reply()` searches discussion threads for this marker to find the session ID. Only bot-authored notes are considered; the first matching marker wins. 2. **Per-workflow rate limiting:** `scan_agent_threads()` counts discussion threads for a specific workflow using the `wf` field. Only bot-authored notes are scanned. 3. **SHA deduplication:** `scan_agent_threads()` checks whether the current SHA has already been reviewed by the workflow using the `sha` field. 4. **Self-filtering:** `handle_note()` compares the webhook event's `author_id` against the bot's own user ID (via `get_bot_user_id()`) to prevent infinite loops. All marker-scanning functions resolve the bot user ID via `gl.auth()` (`GET /user`) on the orchestrator token. This ensures markers in non-bot notes (user replies, external contributors) are never parsed, preventing session hijacking via injected markers. Old plain-UUID markers (``) are parsed as `{"id": "UUID"}` for backward compatibility. They are counted for resumption but not for per-workflow rate limiting or SHA dedup (no `wf`/`sha` information). ## 12. Workflow System ### 12.1. Separation of prompt and metadata A workflow has two parts that live in different places: - **Prompt** (`.md` file): the LLM system prompt, verbatim. This is the investigation strategy, tool usage guidance, data patterns, and output format. Pure text, no code. - **Metadata** (YAML config): operational settings -- `action`, `model`, `max_iterations`, `data_sources`, `projects`. This controls what the orchestrator does with the workflow, not what the model does. This separation means changing the analysis strategy (e.g., adding a new investigation step) requires editing a markdown file. Changing operational parameters (e.g., which projects use this workflow, which model to use) requires editing the YAML config. Neither requires a code change. ### 12.2. System prompt layering The system prompt sent to the model is built from three layers: ```text BASE_SYSTEM_PROMPT # agent.py: sandbox rules, tool usage tips + tool_notes # from ToolRegistry: per-data-source domain knowledge + workflow_body # full content of e.g. workflows/analyze-failures.md ``` On session resumption, a fourth layer is appended: ```text + CONTINUATION_PROMPT # agent.py: "do not re-run the full workflow" ``` The system prompt is rebuilt every iteration (to allow warning suffixes to be appended), but the base content is stable. Warning suffixes are appended at the end so they override earlier instructions. ### 12.3. Design choice: strategy, not procedure The workflow `.md` file describes strategy and guidance, not a rigid script. The model decides when and how to use each tool based on what it sees. This matters because merge request failures are diverse. A fixed procedure would either miss edge cases (e.g., build failures before tests ran) or waste iterations on steps that don't apply. By giving the model a strategy ("identify failing tests, fetch details, analyze root causes, group by similarity"), it can adapt to whatever it encounters. The workflow does structure the investigation into phases (Data Collection, Analysis, Output) for clarity, but these are guidelines, not enforced checkpoints. ### 12.4. Workflow file anatomy (analyze-failures.md) The primary workflow is structured as: 1. **Scope and approach** -- what this workflow analyzes and what it ignores 2. **Input** -- event JSON format (`project`, `iid`) 3. **Data Collection Phase:** - Data.1: Fetch MR details (direct call, small response) - Data.2: Fetch commit statuses, identify failures - Data.3: Batch fetch PipelineRuns + TaskRuns (fetch_batch_to_sandbox) - Data.4: Process with jq, extract Testing Farm data in bulk 4. **Analysis Phase:** - Analysis.1a: Investigate each failed PipelineRun individually - Analysis.1b: Summarize and group by root cause 5. **Output** -- markdown template with root causes, collapsible details, clickable links (PipelineRun, Testing Farm, test logs) 6. **Error Handling** -- partial report philosophy Design considerations embedded in the workflow: - **Efficient bulk fetching:** PipelineRuns and TaskRuns are fetched in one `fetch_batch_to_sandbox` call, not individually. - **Testing Farm data extracted in Data.4, analyzed in Analysis.1:** All TF results.xml are fetched in bulk before analysis starts, enabling cross-failure pattern detection. - **Log fetching is selective:** Only 1-2 representative logs per failure pattern, not all logs. This keeps iteration count bounded. - **Reviewer-facing URLs:** The output template instructs the model to include clickable links using `konflux_ui` and `artifacts_base` from response metadata. ### 12.5. Prompt file resolution `get_prompt(workflow_cfg, config_dir)` resolves the `prompt:` field relative to the config file's directory: ```python prompt_path = config_dir / workflow_cfg.prompt # e.g. /app/config.yaml with prompt: workflows/analyze-failures.md # -> /app/workflows/analyze-failures.md ``` In the container image, workflows are baked in at `/app/workflows/`. A ConfigMap can override them at deploy time by mounting at the same path. ## 13. Deployment and Container ### 13.1. Container build strategy The Containerfile uses an all-RPM builder+installroot+scratch pattern (no pip, no venv): 1. **Builder stage:** Uses a Fedora-based builder image with dnf. Installs all dependencies as RPMs into a clean `--installroot`. 2. **Application code:** Copies `hummingbird_agent/` and `workflows/` into the installroot at `/app/`. 3. **Final stage:** `FROM scratch`, copies the entire installroot. No package manager, no shell beyond what RPMs provide. RPM dependencies: `python3-boto3`, `python3-google-auth+requests`, `python3-gitlab`, `python3-kubernetes`, `python3-pyyaml`, `python3-requests`, `python3-sentry-sdk`, `kubernetes1.35-client` (for `kubectl`). This approach was chosen over pip because: - All deps come from Fedora's package repository -- no PyPI supply chain risk - Smaller image (~22 MB saved vs. google-genai SDK alone) - Reproducible builds from known RPM versions - No compilation step (no gcc/python3-devel in the image) ### 13.2. Container runtime properties ```dockerfile CMD ["python3", "-m", "hummingbird_agent", "serve"] WORKDIR /app USER 65532 ``` The default CMD runs `serve` mode for production. Local development uses `run` mode via explicit command override. `USER 65532` matches the standard nonroot UID used by distroless images and the Podman sandbox. ### 13.3. K8s deployment manifests Located in `hummingbird-agent/kubernetes/`: **`deployment.yaml`:** - Single replica with rolling update (`maxSurge: 1`, `maxUnavailable: 0`) - `terminationGracePeriodSeconds: 900` (15 minutes) to allow in-flight agent runs to complete on shutdown - ServiceAccount: `hummingbird-agent` - Secrets mounted from K8s Secret `hummingbird-agent` - Commented-out mounts for workflow ConfigMap and custom CA trust **`rbac.yaml`** (applied in the _sandbox_ namespace): - ServiceAccount `hummingbird-agent` - Role with `pods: create, get, list, delete, patch` and `pods/exec: create` - RoleBinding linking the SA to the Role - `patch` is required for pool mode (relabeling pods during claim) **`sandbox-pool.yaml`** (applied in the _sandbox_ namespace): - Deployment with `replicas: 3` (tuned to balance latency vs cost) - Pods labelled `hummingbird/role: standby` for pool discovery - Same security context and resource limits as direct-creation pods - `revisionHistoryLimit: 2`, rolling update with `maxSurge: 1`, `maxUnavailable: 0` **`networkpolicy.yaml`:** - `podSelector: {}` -- applies to all pods in the namespace - `egress: []` -- deny all outbound traffic **`secret.yaml`:** - Template with all required env vars (API keys, tokens, URLs) - Values must be populated per deployment ### 13.4. Workflow mounting Workflows are baked into the image at `/app/workflows/`. To update workflows without rebuilding the image: 1. Create a ConfigMap from the workflows directory 2. Uncomment the volume mount in the deployment YAML 3. The ConfigMap mount replaces the baked-in directory (all-or-nothing) This is useful for rapid iteration in staging without waiting for a new image build. ### 13.5. Custom CA trust For clusters with internal CA certificates (common in enterprise environments), the deployment supports OpenShift's CA injection: 1. Create a ConfigMap with the `config.openshift.io/inject-trusted-cabundle` label 2. Mount it at `/etc/pki/custom` 3. Set `REQUESTS_CA_BUNDLE=/etc/pki/custom/ca-bundle.crt` OpenShift automatically injects the cluster's CA bundle into the ConfigMap.