# Upstream Monitor

LLMS index: [llms.txt](https://hummingbird-project.io/llms.txt) | Full content: [llms-full.txt](https://hummingbird-project.io/llms-full.txt)

---

Upstream release discovery and catalog API for Hummingbird. Application code
lives in `upstream-monitor/`. Project definitions belong to the separate
upstream-monitor-projects repository.

## Features

- Release discovery from Git, GitHub, GitLab, Anitya, and HTTP sources
- Version filtering, release streams, lifecycle dates, and Git provenance
- Catalog API and web interface
- File, PostgreSQL, or S3 storage
- Git catalog reconciliation and optional NATS JetStream events

## Prerequisites

- Go 1.27 or newer and a C compiler for local builds and tests
- Podman for container builds
- A project catalog and writable file storage, PostgreSQL, or S3 for results
- Deployment managed credentials for authenticated upstreams and API operations

## Build and Deployment

From the tools repository root:

```bash
podman build -t localhost/upstream-monitor:dev upstream-monitor/
```

Konflux builds ARM64 and AMD64 images as `upstream-monitor--tools` and releases
them to `quay.io/hummingbird-ci/upstream-monitor`. The image runs as UID `65532`
and contains these commands:

| Command                               | Purpose                                                     |
| ------------------------------------- | ----------------------------------------------------------- |
| `upstream-monitor`                    | Poll upstreams and persist results, the default entry point |
| `upstream-monitor-api`                | Serve the catalog API and web interface                     |
| `upstream-monitor-enricher`           | Resolve optional Git provenance                             |
| `upstream-monitor-project-reconciler` | Reconcile Git definitions into PostgreSQL                   |

The [infrastructure repository][infra] owns deployments, storage, messaging,
credentials, and image pins. Register the Konflux component before building,
and pin the released image digest in infrastructure and the catalog validator.
Tools does not deploy the service or bundle a project catalog.

## Usage

### Validate a catalog

From the tools repository root:

```bash
podman run --rm --pull=never --network=none --read-only \
  -e PROJECTS_DIR=/projects \
  -v "$PWD/upstream-monitor/examples/file-layout/config/projects:/projects:ro" \
  localhost/upstream-monitor:dev validate
```

Replace the mounted directory with a checkout of the deployment catalog to
validate it. Validation does not contact upstreams or write to a database.
An empty or missing directory validates zero projects, so check the reported
count before treating validation as successful.

### Run locally

From the application directory, start the monitor:

```bash
cd upstream-monitor
PROJECTS_DIR=examples/file-layout/config/projects \
  go run ./cmd/upstream-monitor
```

In another terminal, from the same directory, start the API:

```bash
PROJECTS_DIR=examples/file-layout/config/projects \
  go run ./cmd/upstream-monitor-api
```

Both processes must use the same project and results storage. File results
default to `observations/`, with provider cache data under `cache/`.
Without project definitions, the monitor has nothing to check.

For an upstream check without persisting results:

```bash
go run ./cmd/upstream-monitor dry-run \
  --projects-dir /path/to/projects --project example-widget
```

`dry-run` contacts the upstream and can consume API quota. It resolves credentials
through `source.credential_ref`, not the monitor's GitHub App configuration.

## Configuration

| Variable                     | Default           | Purpose                                               |
| ---------------------------- | ----------------- | ----------------------------------------------------- |
| `PROJECT_STORAGE_BACKEND`    | `file`            | Definitions: `file`, `postgres`, or `s3`              |
| `RESULTS_STORAGE_BACKEND`    | `file`            | Results: `file`, `postgres`, or `s3`                  |
| `PROJECT_MUTATIONS`          | `auto`            | Definition writes: `auto`, `enabled`, or `disabled`   |
| `PROJECTS_DIR`               | `config/projects` | External project JSON directory                       |
| `POSTGRES_DSN`               | empty             | Required when either backend is PostgreSQL            |
| `S3_BUCKET`                  | empty             | Required when either backend is S3                    |
| `S3_PREFIX`                  | empty             | Unique prefix for each installation sharing a bucket  |
| `S3_REGION`                  | `us-east-1`       | AWS region                                            |
| `S3_ENDPOINT`                | empty             | Optional S3 compatible endpoint                       |
| `S3_USE_PATH_STYLE`          | `false`           | Enable path style S3 addressing                       |
| `OBSERVATIONS_DIR`           | `observations`    | File results directory                                |
| `CACHE_DIR`                  | `cache`           | File backend provider cache                           |
| `LISTEN_ADDR`                | `:8080`           | API listen address                                    |
| `POLL_INTERVAL`              | `5m`              | Scheduler interval and default project check interval |
| `UPSTREAM_TIMEOUT`           | `5m`              | Maximum duration of an upstream check                 |
| `MAX_CONCURRENT_CHECKS`      | `4`               | Concurrent checks per worker                          |
| `MAX_CONCURRENT_GIT_FETCHES` | `1`               | Concurrent uncached Git fetches per worker            |
| `ENRICHMENT_ENABLED`         | `false`           | Include provenance in API responses                   |

File results support a single writer. Use PostgreSQL or S3 for shared worker
coordination. S3 must support conditional writes. Do not expire observation or
coordination objects through bucket lifecycle policies.

Use `GITHUB_TOKEN` for token based GitHub access. For a GitHub App, set
`GITHUB_APP_ID`, `GITHUB_APP_INSTALLATION_ID`, and
`GITHUB_APP_PRIVATE_KEY_FILE` together. A project's `source.credential_ref`
names an environment variable containing a credential, never the credential
itself. Keep secrets outside Git and container images.

See [configuration defaults][config] and [environment overrides][config-env]
for remaining cache, lease, retry, and concurrency settings.

## Project Definitions

Each JSON filename must match its `id`. IDs start with a lowercase letter,
contain only lowercase letters, digits, and hyphens, and have at most 128
characters. Do not supply `uid`, which the service assigns.

```json
{
  "id": "example-widget",
  "name": "Example widget",
  "homepage": "https://example.org/widget",
  "source": {
    "type": "github-releases",
    "location": "example/widget"
  },
  "version_policy": {
    "scheme": "semantic",
    "include_prereleases": false
  },
  "stream_policy": {
    "granularity": "minor"
  },
  "check_policy": {
    "mode": "fixed",
    "interval": "15m"
  }
}
```

| Source type                      | Location                                               |
| -------------------------------- | ------------------------------------------------------ |
| `git-tags`                       | Git remote URL                                         |
| `github-releases`                | GitHub `owner/repository`                              |
| `gitlab-releases`, `gitlab-tags` | GitLab numeric project ID or namespace path            |
| `anitya`                         | Numeric Anitya project ID                              |
| `http-regex`                     | HTTP(S) URL with a `source.extract` regular expression |

Set `source.base_url` for a separate GitLab or Anitya instance, and
`source.tag_pattern` to select a tag family. Upstream destinations are not host
allowlisted, so infrastructure must enforce the required network egress policy.

The default version scheme is `semantic`. Other choices are `calendar`, `rpm`,
`pep440` (`python`), and `lexical` (`generic`). Use `transforms` to normalize
versions, and `version_filters` and `prerelease_filters` to control release
selection. Streams group releases by `major` or `minor`, with optional semantic
numeric component buckets.

Optional lifecycle modes are `latest-release`, `latest-n-lines`, `date-based`,
`endoflife-date`, and `manual`, with per stream overrides. Fixed check policies
use `interval`. Adaptive policies back off between `min_interval` and
`max_interval`, resetting after a new release. The maximum interval is also the
worst polling detection delay.

## Catalog Reconciliation

The project reconciler validates an atomic git-sync checkout and replaces the
PostgreSQL definition set. Supply `POSTGRES_DSN` through the environment:

```bash
PROJECT_STORAGE_BACKEND=postgres \
PROJECT_MUTATIONS=disabled \
PROJECT_CHECKOUT_LINK=/git-sync/current \
PROJECT_CHECKOUT_SUBPATH=projects \
  go run ./cmd/upstream-monitor-project-reconciler
```

Run from `upstream-monitor/`. The `projects` subpath matches the separate
catalog repository. Set `PROJECT_MUTATIONS=disabled` on every process using
that catalog. Workers and API replicas read its database copy.

Invalid revisions leave the last valid catalog active. **Reconciliation removes
definitions absent from Git.** Verify completeness and back up the database
before enabling it. Git cannot restore deleted observation history.

The monitor initializes database schemas. The API does not. Schema and image
rollouts belong to infrastructure, not API startup.

## API and Authentication

Open `/` for the catalog browser. The generated API schema is available at
`/openapi.json` and `/openapi.yaml`. `/docs` redirects to the JSON schema.

| Endpoint                     | Purpose                                                      |
| ---------------------------- | ------------------------------------------------------------ |
| `/healthz`                   | Process health                                               |
| `/readyz`                    | Project store readability, not proof of successful polling   |
| `/v1/status`                 | Aggregate and per project check status                       |
| `/v1/project-reconciliation` | Attempted and active Git revisions and reconciliation errors |
| `/v1/projects`               | Project search and pagination                                |

Use `last_success_at` and project errors to assess freshness. Follow
`X-Next-Cursor` for paginated reads. Definition updates and deletes require the
current `ETag` in `If-Match`. A queued `scan` retains observations, while
**`rescan` clears that project's snapshot and release history before checking**.

Catalog reads are public. Definition reads, writes, inline checks, and scan
requests require authentication and Cedar authorization. `PROJECT_MUTATIONS=auto`
disables file definition writes but permits supported PostgreSQL and S3 writes.

| `API_AUTH_MODE`          | Required configuration                                          |
| ------------------------ | --------------------------------------------------------------- |
| `disabled`               | Default. Authenticated operations are unavailable.              |
| `static-token`           | `API_AUTH_TOKEN`, `API_AUTH_PRINCIPAL`, `API_AUTH_ROLES`        |
| `kubernetes-tokenreview` | `API_KUBERNETES_AUDIENCE` and a role mapping file               |
| `oidc`                   | `API_OIDC_ISSUER`, `API_OIDC_AUDIENCE`, and a role mapping file |
| `trusted-proxy`          | `API_TRUSTED_PROXY_CIDRS` and a role mapping file               |
| `insecure`               | Local development only. Never use in production.                |

`API_AUTH_ROLE_MAPPINGS_FILE` maps subjects and groups to Cedar roles:

```json
{
  "subjects": {
    "system:serviceaccount:ci:release-bot": ["ProjectChecker"]
  },
  "groups": {
    "editors": ["ProjectEditor"]
  }
}
```

The default policy grants definition reads to `ProjectReader`, inline checks
to `ProjectChecker`, editing and scan operations to `ProjectEditor`, and all
actions to `ProjectAdmin`. Use `API_CEDAR_POLICY_FILE` to replace that policy.

TokenReview requires a mounted reviewer token and permission to create
`tokenreviews.authentication.k8s.io`. Trust proxy identity headers only from an
isolated authenticating proxy that callers cannot bypass. Remove inbound
identity headers at the proxy, and prefer loopback CIDRs for a sidecar rather
than trusting a whole cluster network.

### Project status

Successful checks return to `scheduled`. `waiting` and `succeeded` are accepted
for compatibility with older stored records:

```mermaid
stateDiagram-v2
    [*] --> queued : definition due now
    [*] --> scheduled : definition with future check
    [*] --> waiting : load legacy record
    [*] --> succeeded : load older successful record
    waiting --> queued : restart with due check
    waiting --> scheduled : restart with future check
    succeeded --> queued : next check claimed
    scheduled --> queued : check due, scan, or definition changed
    queued --> processing : worker starts
    queued --> scheduled : repository lease held elsewhere
    processing --> scheduled : snapshot persisted
    processing --> failed : check or persistence error
    failed --> queued : retry or definition changed
    processing --> queued : recover interrupted file worker
```

Records include the project identity, definition fingerprint, pending scan
request, queue/start/finish/next check times, `last_success_at`, failure count,
last error, and release/stream counts.

## Events and Monitoring

Set `NATS_URL` to enable JetStream publishing and `NATS_CREDS_FILE` for mounted
credentials. Default subjects are `hummingbird.upstream.version.v1` and
`hummingbird.upstream.stream.v1`. Set `NATS_MANAGE_STREAM=false` if
infrastructure provisions the stream. Delivery is at least once, so consumers
must deduplicate using `event_key`. Publication failures retain events for retry.

Logs are JSON on stdout by default, configured through `LOG_LEVEL` and
`LOG_FORMAT`. Set `OTEL_METRICS_EXPORTER=otlp` with standard OpenTelemetry
endpoint settings to export metrics. Unset exporter configuration does not
connect to a local collector.

Monitor time since the last successful check, check failures, GitHub quota,
and outbox depth and oldest pending age. Quiet projects remain healthy when
checks succeed and adaptive intervals grow.

## Development

Run from `upstream-monitor/`:

```bash
go vet ./...
go test ./...
go build ./cmd/...
go run ./cmd/upstream-monitor validate --projects-dir examples/file-layout/config/projects
```

The GitLab job also rejects empty fixture directories. Container smoke tests
mount the examples and check API health through the [K8s test runner][k8s-tests].

Integration tests require disposable services. Set `TEST_POSTGRES_DSN` for
PostgreSQL storage, `TEST_POSTGRES_RECONCILE_DSN` for a separate catalog
replacement database, `TEST_S3_ENDPOINT` and `TEST_S3_BUCKET` for S3, or
`NATS_CLUSTER_TEST_URL` for a JetStream cluster with three stream replicas.
Never point these tests at production services.

## License

See the repository [LICENSE][license].

[infra]: https://gitlab.com/redhat/hummingbird/infrastructure
[config]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/upstream-monitor/internal/config/config.go
[config-env]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/upstream-monitor/internal/config/config_env.go
[k8s-tests]: https://hummingbird-project.io/docs/background/tools/k8s-integration-tests/
[license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE