# Hummingbird Agent Evals

LLMS index: [llms.txt](/llms.txt) | Full content: [llms-full.txt](/llms-full.txt)

---

Evaluation framework for scientific measurement of LLM output quality.
For the agent itself, see [Hummingbird Agent](hummingbird-agent.md).
For architecture details, see
[Hummingbird Agent Design](hummingbird-agent-design.md).

## What evals are

Evals measure the quality of LLM outputs under controlled conditions.
They answer questions like "does prompt v5 produce better code reviews
than v4?" or "can an LLM answer this question correctly using only
public documentation?"

Evals are **not unit tests**. Unit tests verify deterministic behavior;
evals quantify stochastic output quality through structured scoring.

## General pattern

Evals generally follow a pattern of:

1. **Inputs** -- a set of test cases: golden dataset with scoring
   criteria, real MRs to review, or other structured inputs.
2. **Model execution** -- run one or more LLM configurations against
   the inputs, optionally with tools (web search, sandbox).
3. **Scoring** -- evaluate outputs via LLM-as-judge, comparison
   against reference answers, or human review.
4. **Iteration** -- adjust prompts or config, re-run, check for
   improvement.

Not all evals use every step. Baseline measurements may stop at
step 3; comparison evals may skip the golden dataset entirely.

## Shared infrastructure

### `hummingbird_agent.eval`

The `hummingbird_agent.eval` module provides generic evaluation
primitives used across all evals. Key components:

- **`EvalJudge`** -- LLM-as-judge wrapper over Vertex AI with explicit
  temperature and token limit control. Supports both strict judging
  (low temperature) and open-ended analysis (higher temperature).
- **Snapshot management** -- saves timestamped evaluation results with
  metadata, maintains a `latest` symlink.
- **Variant tracking** -- persists the best-performing prompt or config
  variant across iterations.
- **Convergence detection** -- checks whether scores have plateaued
  across consecutive iterations.
- **Holdout validation** -- compares holdout scores against dev scores
  using standard error of the mean.

See the module docstrings for the full API.

### `hummingbird_agent.evals.mr_helpers`

MR-experiment-specific helpers built on top of `hummingbird_agent.eval`.
Provides GitLab auth, MR data I/O, review generation orchestration, and
CLI scaffolding. Used by the code-review and renovate-triage evals.

## Running an eval

### Prerequisites

1. GCP Application Default Credentials:

   ```bash
   gcloud auth application-default login
   export GOOGLE_CLOUD_PROJECT=<your-gcp-project>
   ```

2. The agent package installed in editable mode:

   ```bash
   cd hummingbird-agent
   pip install -e .
   ```

### Execution

Each eval is a Python module under `hummingbird_agent/evals/`. Run from
`hummingbird-agent/`:

```bash
python -m hummingbird_agent.evals.<name>.<script> [options]
```

Refer to each eval's own README for specific options and configuration.

### Existing evals

- [Documentation Quality Evaluation](hummingbird-agent-docs-eval.md) --
  measures whether public documentation answers contributor questions

## Writing a new eval

Create a new directory under `hummingbird_agent/evals/` with:

- **`README.md`** -- what the eval measures and how to run it, linking
  back to this document for the general framework.
- **Golden dataset** -- inputs and scoring criteria.  Can live outside
  the eval directory (e.g. in the documentation repo alongside the
  content being measured) and be passed via CLI.
- **`run_*.py`** -- orchestration script using `hummingbird_agent.eval`
  primitives (or `hummingbird_agent.evals.mr_helpers` for MR-based evals).

The eval's run script should be importable as a module
(`python -m hummingbird_agent.evals.<name>.<script>`). Add an `__init__.py` if needed.
Do not hardcode GCP project IDs; `VertexAuth` resolves them at
runtime from `GOOGLE_CLOUD_PROJECT` or ADC credentials.

## Authentication

All evals use `VertexAuth` from `hummingbird_agent._http`, which
resolves credentials and project in this order:

1. Explicit `project=` parameter (if passed).
2. `GOOGLE_CLOUD_PROJECT` environment variable.
3. ADC default project (from `gcloud auth application-default login`).

Service account keys work automatically when
`GOOGLE_APPLICATION_CREDENTIALS` is set.