Vertex AI Cost Metrics

This document defines the shared cost-observability contract for all Hummingbird services that call Vertex AI. It covers two obligations:

1. Prometheus metrics — estimated cost counters scraped into Mimir.
2. Vertex request labels — billing metadata attached to every API call.

All services that make Vertex AI calls MUST implement both.

1. Prometheus metrics (estimated cost)

Every Vertex AI caller MUST emit the following counters on its scraped /metrics endpoint. Using identical names and label sets across services enables a single Grafana dashboard without per-app query logic.

Counters

Labels

Applications MUST set these labels. Additional labels are permitted if cardinality remains bounded.

Applications MUST NOT set cluster, namespace, app, service, or pod — Alloy adds these automatically at scrape time via relabel rules.

Cardinality

• model: bounded by deployed model set (currently 2-4)
• workflow: bounded by workflow config entries (~10 for agent, 1 for dashboard)
• direction: fixed set of 3
• status: fixed set of 3

Total series per app: model x workflow x max(direction, status) = 4 x 10 x 3 = 120. Well within Mimir per-metric limits.

2. Vertex request labels (billed cost)

Every Vertex API call MUST include billing labels so that GCP Cloud Billing rows can be attributed to a specific application and workflow once BigQuery export is enabled.

Required labels

Callers MUST set both keys on every generateContent, streamGenerateContent, or rawPredict request.

How to attach

For generateContent, add the labels object to the request body:

{
  "contents": [...],
  "labels": {"app": "agent", "workflow": "code-review"}
}

For rawPredict, base64-encode the labels JSON and pass as a header (shown decoded for clarity):

import base64
import json

labels = {"app": "dashboard", "workflow": "analyze-failures"}
headers["X-Vertex-AI-Labels"] = base64.b64encode(
    json.dumps(labels).encode(),
).decode()

Constraints (GCP requirements)

• Keys and values: lowercase letters, numbers, underscores, dashes only
• Max 63 characters each
• Keys MUST start with a letter
• Labels are only forwarded for PayGo consumption (Provisioned Throughput silently ignores them)

Reference

Example PromQL

Total estimated spend yesterday (all apps):

sum(increase(hummingbird_vertex_cost_dollars_total[1d]))

Per-app daily cost:

sum by (app)(increase(hummingbird_vertex_cost_dollars_total[1d]))

Per-workflow breakdown for the agent:

sum by (workflow, model)(
  increase(hummingbird_vertex_cost_dollars_total{app="hummingbird-agent"}[1d])
)

Token consumption by direction:

sum by (app, direction)(increase(hummingbird_vertex_tokens_total[1d]))

Implementing apps