Documentation Quality Evaluation
Reference for the documentation quality evaluation, which measures whether public documentation is complete enough for LLMs with web search to answer questions correctly. For the general evaluation framework, see Hummingbird Agent Evals.
Config file
A single YAML file combines evaluation settings and the golden dataset.
It is passed to the run script via -c:
python -m hummingbird_agent.evals.docs_eval.run_eval -c path/to/docs_eval.yml
The config file lives in the documentation repo alongside the content being measured.
Evaluation settings
| Key | Required | Default | Description |
|---|---|---|---|
models |
yes | – | List of model configurations |
models[].id |
yes | – | Vertex AI model identifier |
models[].region |
yes | – | Vertex AI region |
models[].google_search |
no | false |
Enable Google Search grounding (Gemini) |
models[].web_search |
no | false |
Enable web search tool (Claude) |
judge |
yes | – | Judge model configuration |
judge.model |
yes | – | Judge model identifier |
judge.region |
no | global |
Judge model region |
repetitions |
no | 3 |
Runs per (question, model) pair |
max_iterations |
no | 5 |
Agent loop iteration cap per question |
Golden dataset
The questions list defines what the evaluation measures:
| Field | Required | Default | Description |
|---|---|---|---|
id |
yes | – | Unique identifier (used in output paths) |
question |
yes | – | Question sent to each model |
facts |
yes | – | List of fact checks |
facts[].question |
yes | – | Yes/no question about the response |
facts[].weight |
no | 1 |
Relative importance |
coherence |
no | – | Criterion for narrative quality scoring |
coherence_weight |
no | 1.5 |
Weight of coherence in the final score |
threshold |
no | 0.4 |
Minimum weighted score to pass |
CLI options
| Option | Default | Description |
|---|---|---|
-c / --config |
required | Path to config YAML |
--phase |
all |
ask, judge, or all |
--results-dir |
results/ next to config |
Output directory |
--run-id |
UTC timestamp | Subdirectory under results |
Scoring
Each response is scored independently:
- Fact scoring – the judge answers each fact question with yes or no. Passing facts contribute their weight to the numerator.
- Coherence scoring – the judge rates narrative quality from 0.0 to 1.0. The coherence score is multiplied by its weight and added to the total.
- Weighted score –
sum(passed_fact_weights + coherence_contribution) / total_weight. A response passes ifweighted_score >= threshold.
The summary aggregates per-model averages across all repetitions.