# Service metrics

LLMS index: [llms.txt](/llms.txt) | Full content: [llms-full.txt](/llms-full.txt)

---

Long-running exporter for Hummingbird AWS Cost Explorer spend, Kubernetes
Metrics API CPU/memory gauges, OpenShift cluster resource quota usage, PVC
filesystem usage, and GitLab CI/CD schedule status.

## Features

- **Explicit collectors** - nothing runs unless listed in `METRICS_CONFIG`
- **Kubernetes Metrics API** - per-container CPU and memory gauges
- **Cluster resource quotas** - AppliedClusterResourceQuota usage and limits
- **AWS Cost Explorer** - yesterday's `NetAmortizedCost` by team and service
- **Volume filesystem usage** - PVC disk usage via `shutil.disk_usage()`
- **GitLab CI/CD schedules** - pipeline schedule health via GraphQL
- **Prometheus** - HTTP metrics on port 9090

## Configuration

Collectors are off unless listed in `METRICS_CONFIG`:

```yaml
enabled: [kubernetes]
```

Unknown names fail startup.

| Variable                                      | Purpose                                                                |
| --------------------------------------------- | ---------------------------------------------------------------------- |
| `METRICS_CONFIG`                              | YAML with `enabled` collector list (required)                          |
| `KUBERNETES_CONFIG`                           | YAML with `namespaces` for the kubernetes collector                    |
| `VOLUMES_CONFIG`                              | YAML with `volumes` list for the volume collector                      |
| `GITLAB_URL`                                  | GitLab instance URL (default `https://gitlab.com`)                     |
| `GITLAB_TOKEN`                                | GitLab API token with `read_api` scope (required for gitlab collector) |
| `GITLAB_CONFIG`                               | YAML with `namespaces` list for the gitlab collector                   |
| `METRICS_PORT`                                | HTTP port (default `9090`)                                             |
| `SENTRY_DSN`                                  | Optional Sentry DSN                                                    |
| `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` | Cost Explorer IAM user (hub)                                           |
| `AWS_DEFAULT_REGION`                          | Must be `us-east-1` for Cost Explorer                                  |

## Kubernetes Metrics API

`hummingbird_k8s_resource_usage{namespace,pod,container,resource,app}` with
`resource=cpu|memory` and `app` sourced from the pod's
`app.kubernetes.io/name` label (empty string if absent). Instant gauges from
`/apis/metrics.k8s.io/v1beta1/namespaces/<ns>/pods`. This is not cAdvisor:
managed OpenShift does not grant `nodes/metrics` or `cluster-monitoring-view`,
so kubelet `/metrics/cadvisor` and platform Prometheus federation are not
available.

If `KUBERNETES_CONFIG.namespaces` is unset, the collector uses the pod's
ServiceAccount namespace.

## Cluster resource quotas

The kubernetes collector also reads namespace-scoped
`AppliedClusterResourceQuota` objects from the `quota.openshift.io/v1` API
and exposes two gauges that mirror the
[openshift-state-metrics](https://github.com/openshift/openshift-state-metrics)
schema with a `hummingbird_` prefix:

- `hummingbird_clusterresourcequota_usage{name,resource,type}` --
  cluster-wide totals from `status.total` (`type=hard|used`)
- `hummingbird_clusterresourcequota_namespace_usage{name,namespace,resource,type}` --
  per-namespace breakdown from `status.namespaces` (`type=hard|used`)

`resource` values match Kubernetes quantity names: `cpu`, `memory`,
`requests.cpu`, `limits.memory`, `pods`, etc. Quantities are converted to
base units (cores, bytes, count).

The same ACRQ may appear in multiple namespaces; the global metric is
idempotent across duplicates. Stale series are removed when ACRQs disappear.

Requires `list` on `appliedclusterresourcequotas` in `quota.openshift.io`
(namespace-scoped Role -- no ClusterRole needed).

## Volume filesystem usage

The volume collector reports disk usage for PVC-backed volumes. It is
designed to run as a **sidecar** on pods that mount PVCs, since
`kubelet_volume_stats_*` metrics are inaccessible on managed clusters
(no ClusterRole for `nodes/metrics`) and RWO volumes cannot be mounted
cross-namespace.

Configure via `VOLUMES_CONFIG`:

```yaml
VOLUMES_CONFIG: |
  volumes:
    - {name: my-postgres, path: /data}
```

Metrics (collected every 5 minutes):

- `hummingbird_volume_usage_bytes{name}` -- used bytes
- `hummingbird_volume_total_bytes{name}` -- total capacity
- `hummingbird_volume_available_bytes{name}` -- available bytes

Deploy with `enabled: [volume]` and a non-default `METRICS_PORT` (e.g.
`9091`) to avoid port conflicts when running as a sidecar alongside other
containers. See the `hummingbird-status` StatefulSet in the
[infrastructure repo][infra-repo] for an example.

## GitLab CI/CD schedules

The gitlab collector monitors the health of pipeline schedules across all
projects in configured GitLab namespaces. It queries the GitLab GraphQL API
once per hour, paginating through projects to fetch each schedule's last
pipeline status.

Configure via `GITLAB_CONFIG`:

```yaml
GITLAB_CONFIG: |
  namespaces:
    - redhat/hummingbird
```

The `GITLAB_TOKEN` environment variable must contain a token with `read_api`
scope that has access to the monitored namespaces. Keep it in an OCP Secret
(not the ConfigMap).

Metric (collected every hour):

- `hummingbird_gitlab_schedule_status{instance,project,active,description,cron}`
  -- Enum reporting the last pipeline status of each CI/CD schedule

States: `created`, `waiting_for_resource`, `preparing`, `pending`, `running`,
`success`, `failed`, `canceled`, `skipped`, `manual`, `scheduled`, `unknown`.
Schedules with no last pipeline report `unknown`.

Stale series are removed automatically when schedules disappear.

## AWS costs

`hummingbird_aws_cost{group,type,key}` is yesterday's `NetAmortizedCost`.
Queries run at process start and every 24 hours.

`group=all` is the shared account by `app-code` and SERVICE.
`group=hummingbird` filters `app-code=RPRM-001` and groups by SERVICE,
OPERATION, USAGE_TYPE, Name tag, and SERVICE+OPERATION.

Untagged Cost Explorer keys such as `app-code$` become `key=untagged`.
Well-known SERVICE names are shortened (`S3`, `EC2`). Dual grouping uses
`S3|PutObject`.

The SAM template creates an IAM user with `ce:GetCostAndUsage` only. Access
keys are created after deploy and stored in Vault. Deployment lives in the
[infrastructure repo][infra-repo].

## License

This project is licensed under the GNU General Public License v3.0 or later -
see the [LICENSE][license] file for details.

[infra-repo]: https://gitlab.com/redhat/hummingbird/infrastructure
[license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE