Service metrics

Long-running exporter for Hummingbird AWS Cost Explorer spend, Kubernetes Metrics API CPU/memory gauges, OpenShift cluster resource quota usage, and PVC filesystem usage.

Features

  • Explicit collectors - nothing runs unless listed in METRICS_CONFIG
  • Kubernetes Metrics API - per-container CPU and memory gauges
  • Cluster resource quotas - AppliedClusterResourceQuota usage and limits
  • AWS Cost Explorer - yesterday’s NetAmortizedCost by team and service
  • Volume filesystem usage - PVC disk usage via shutil.disk_usage()
  • Prometheus - HTTP metrics on port 9090

Configuration

Collectors are off unless listed in METRICS_CONFIG:

enabled: [kubernetes]

Unknown names fail startup.

Variable Purpose
METRICS_CONFIG YAML with enabled collector list (required)
KUBERNETES_CONFIG YAML with namespaces for the kubernetes collector
VOLUMES_CONFIG YAML with volumes list for the volume collector
METRICS_PORT HTTP port (default 9090)
SENTRY_DSN Optional Sentry DSN
AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY Cost Explorer IAM user (hub)
AWS_DEFAULT_REGION Must be us-east-1 for Cost Explorer

Kubernetes Metrics API

hummingbird_k8s_resource_usage{namespace,pod,container,resource,app} with resource=cpu|memory and app sourced from the pod’s app.kubernetes.io/name label (empty string if absent). Instant gauges from /apis/metrics.k8s.io/v1beta1/namespaces/<ns>/pods. This is not cAdvisor: managed OpenShift does not grant nodes/metrics or cluster-monitoring-view, so kubelet /metrics/cadvisor and platform Prometheus federation are not available.

If KUBERNETES_CONFIG.namespaces is unset, the collector uses the pod’s ServiceAccount namespace.

Cluster resource quotas

The kubernetes collector also reads namespace-scoped AppliedClusterResourceQuota objects from the quota.openshift.io/v1 API and exposes two gauges that mirror the openshift-state-metrics schema with a hummingbird_ prefix:

  • hummingbird_clusterresourcequota_usage{name,resource,type} – cluster-wide totals from status.total (type=hard|used)
  • hummingbird_clusterresourcequota_namespace_usage{name,namespace,resource,type} – per-namespace breakdown from status.namespaces (type=hard|used)

resource values match Kubernetes quantity names: cpu, memory, requests.cpu, limits.memory, pods, etc. Quantities are converted to base units (cores, bytes, count).

The same ACRQ may appear in multiple namespaces; the global metric is idempotent across duplicates. Stale series are removed when ACRQs disappear.

Requires list on appliedclusterresourcequotas in quota.openshift.io (namespace-scoped Role – no ClusterRole needed).

Volume filesystem usage

The volume collector reports disk usage for PVC-backed volumes. It is designed to run as a sidecar on pods that mount PVCs, since kubelet_volume_stats_* metrics are inaccessible on managed clusters (no ClusterRole for nodes/metrics) and RWO volumes cannot be mounted cross-namespace.

Configure via VOLUMES_CONFIG:

VOLUMES_CONFIG: |
  volumes:
    - {name: my-postgres, path: /data}

Metrics (collected every 5 minutes):

  • hummingbird_volume_usage_bytes{name} – used bytes
  • hummingbird_volume_total_bytes{name} – total capacity
  • hummingbird_volume_available_bytes{name} – available bytes

Deploy with enabled: [volume] and a non-default METRICS_PORT (e.g. 9091) to avoid port conflicts when running as a sidecar alongside other containers. See the hummingbird-status StatefulSet in the infrastructure repo for an example.

AWS costs

hummingbird_aws_cost{group,type,key} is yesterday’s NetAmortizedCost. Queries run at process start and every 24 hours.

group=all is the shared account by app-code and SERVICE. group=hummingbird filters app-code=RPRM-001 and groups by SERVICE, OPERATION, USAGE_TYPE, Name tag, and SERVICE+OPERATION.

Untagged Cost Explorer keys such as app-code$ become key=untagged. Well-known SERVICE names are shortened (S3, EC2). Dual grouping uses S3|PutObject.

The SAM template creates an IAM user with ce:GetCostAndUsage only. Access keys are created after deploy and stored in Vault. Deployment lives in the infrastructure repo.

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.