# Documentation Quality Evaluation

LLMS index: [llms.txt](/llms.txt) | Full content: [llms-full.txt](/llms-full.txt)

---

Reference for the documentation quality evaluation, which measures whether
public documentation is complete enough for LLMs with web search to answer
questions correctly. For the general evaluation framework, see
[Hummingbird Agent Evals](hummingbird-agent-evals.md).

## Config file

A single YAML file combines evaluation settings and the golden dataset.
It is passed to the run script via `-c`:

```bash
python -m hummingbird_agent.evals.docs_eval.run_eval -c path/to/docs_eval.yml
```

The config file lives in the documentation repo alongside the content
being measured.

### Evaluation settings

| Key                      | Required | Default  | Description                             |
| ------------------------ | -------- | -------- | --------------------------------------- |
| `models`                 | yes      | --       | List of model configurations            |
| `models[].id`            | yes      | --       | Vertex AI model identifier              |
| `models[].region`        | yes      | --       | Vertex AI region                        |
| `models[].google_search` | no       | `false`  | Enable Google Search grounding (Gemini) |
| `models[].web_search`    | no       | `false`  | Enable web search tool (Claude)         |
| `judge`                  | yes      | --       | Judge model configuration               |
| `judge.model`            | yes      | --       | Judge model identifier                  |
| `judge.region`           | no       | `global` | Judge model region                      |
| `repetitions`            | no       | `3`      | Runs per (question, model) pair         |
| `max_iterations`         | no       | `5`      | Agent loop iteration cap per question   |

### Golden dataset

The `questions` list defines what the evaluation measures:

| Field              | Required | Default | Description                              |
| ------------------ | -------- | ------- | ---------------------------------------- |
| `id`               | yes      | --      | Unique identifier (used in output paths) |
| `question`         | yes      | --      | Question sent to each model              |
| `facts`            | yes      | --      | List of fact checks                      |
| `facts[].question` | yes      | --      | Yes/no question about the response       |
| `facts[].weight`   | no       | `1`     | Relative importance                      |
| `coherence`        | no       | --      | Criterion for narrative quality scoring  |
| `coherence_weight` | no       | `1.5`   | Weight of coherence in the final score   |
| `threshold`        | no       | `0.4`   | Minimum weighted score to pass           |

## CLI options

| Option            | Default                   | Description                |
| ----------------- | ------------------------- | -------------------------- |
| `-c` / `--config` | required                  | Path to config YAML        |
| `--phase`         | `all`                     | `ask`, `judge`, or `all`   |
| `--results-dir`   | `results/` next to config | Output directory           |
| `--run-id`        | UTC timestamp             | Subdirectory under results |

## Scoring

Each response is scored independently:

- **Fact scoring** -- the judge answers each fact question with yes or no.
  Passing facts contribute their weight to the numerator.
- **Coherence scoring** -- the judge rates narrative quality from 0.0 to
  1.0. The coherence score is multiplied by its weight and added to the
  total.
- **Weighted score** --
  `sum(passed_fact_weights + coherence_contribution) / total_weight`.
  A response passes if `weighted_score >= threshold`.

The summary aggregates per-model averages across all repetitions.