Hummingbird Agent Evals

Evaluation framework for scientific measurement of LLM output quality. For the agent itself, see Hummingbird Agent. For architecture details, see Hummingbird Agent Design.

What evals are

Evals measure the quality of LLM outputs under controlled conditions. They answer questions like “does prompt v5 produce better code reviews than v4?” or “can an LLM answer this question correctly using only public documentation?”

Evals are not unit tests. Unit tests verify deterministic behavior; evals quantify stochastic output quality through structured scoring.

General pattern

Evals generally follow a pattern of:

Inputs – a set of test cases: golden dataset with scoring criteria, real MRs to review, or other structured inputs.
Model execution – run one or more LLM configurations against the inputs, optionally with tools (web search, sandbox).
Scoring – evaluate outputs via LLM-as-judge, comparison against reference answers, or human review.
Iteration – adjust prompts or config, re-run, check for improvement.

Not all evals use every step. Baseline measurements may stop at step 3; comparison evals may skip the golden dataset entirely.

Shared infrastructure

`hummingbird_agent.eval`

The hummingbird_agent.eval module provides generic evaluation primitives used across all evals. Key components:

EvalJudge – LLM-as-judge wrapper over Vertex AI with explicit temperature and token limit control. Supports both strict judging (low temperature) and open-ended analysis (higher temperature).
Snapshot management – saves timestamped evaluation results with metadata, maintains a latest symlink.
Variant tracking – persists the best-performing prompt or config variant across iterations.
Convergence detection – checks whether scores have plateaued across consecutive iterations.
Holdout validation – compares holdout scores against dev scores using standard error of the mean.

See the module docstrings for the full API.

`hummingbird_agent.evals.mr_helpers`

MR-experiment-specific helpers built on top of hummingbird_agent.eval. Provides GitLab auth, MR data I/O, review generation orchestration, and CLI scaffolding. Used by the code-review and renovate-triage evals.

Running an eval

Prerequisites

GCP Application Default Credentials:

gcloud auth application-default login
export GOOGLE_CLOUD_PROJECT=<your-gcp-project>

The agent package installed in editable mode:
```
cd hummingbird-agent
pip install -e .
```

Execution

Each eval is a Python module under hummingbird_agent/evals/. Run from hummingbird-agent/:

python -m hummingbird_agent.evals.<name>.<script> [options]

Refer to each eval’s own README for specific options and configuration.

Existing evals

Documentation Quality Evaluation – measures whether public documentation answers contributor questions

Writing a new eval

Create a new directory under hummingbird_agent/evals/ with:

README.md – what the eval measures and how to run it, linking back to this document for the general framework.
Golden dataset – inputs and scoring criteria. Can live outside the eval directory (e.g. in the documentation repo alongside the content being measured) and be passed via CLI.
run_*.py – orchestration script using hummingbird_agent.eval primitives (or hummingbird_agent.evals.mr_helpers for MR-based evals).

The eval’s run script should be importable as a module (python -m hummingbird_agent.evals.<name>.<script>). Add an __init__.py if needed. Do not hardcode GCP project IDs; VertexAuth resolves them at runtime from GOOGLE_CLOUD_PROJECT or ADC credentials.

Authentication

All evals use VertexAuth from hummingbird_agent._http, which resolves credentials and project in this order:

Explicit project= parameter (if passed).
GOOGLE_CLOUD_PROJECT environment variable.
ADC default project (from gcloud auth application-default login).

Service account keys work automatically when GOOGLE_APPLICATION_CREDENTIALS is set.