Upstream Monitor

Upstream release discovery and catalog API for Hummingbird. Application code lives in upstream-monitor/. Project definitions belong to the separate upstream-monitor-projects repository.

Features

Release discovery from Git, GitHub, GitLab, Anitya, and HTTP sources
Version filtering, release streams, lifecycle dates, and Git provenance
Catalog API and web interface
File, PostgreSQL, or S3 storage
Git catalog reconciliation and optional NATS JetStream events

Prerequisites

Go 1.27 or newer and a C compiler for local builds and tests
Podman for container builds
A project catalog and writable file storage, PostgreSQL, or S3 for results
Deployment managed credentials for authenticated upstreams and API operations

Build and Deployment

From the tools repository root:

podman build -t localhost/upstream-monitor:dev upstream-monitor/

Konflux builds ARM64 and AMD64 images as upstream-monitor--tools and releases them to quay.io/hummingbird-ci/upstream-monitor. The image runs as UID 65532 and contains these commands:

Command	Purpose
`upstream-monitor`	Poll upstreams and persist results, the default entry point
`upstream-monitor-api`	Serve the catalog API and web interface
`upstream-monitor-enricher`	Resolve optional Git provenance
`upstream-monitor-project-reconciler`	Reconcile Git definitions into PostgreSQL

The infrastructure repository owns deployments, storage, messaging, credentials, and image pins. Register the Konflux component before building, and pin the released image digest in infrastructure and the catalog validator. Tools does not deploy the service or bundle a project catalog.

Usage

Validate a catalog

From the tools repository root:

podman run --rm --pull=never --network=none --read-only \
  -e PROJECTS_DIR=/projects \
  -v "$PWD/upstream-monitor/examples/file-layout/config/projects:/projects:ro" \
  localhost/upstream-monitor:dev validate

Replace the mounted directory with a checkout of the deployment catalog to validate it. Validation does not contact upstreams or write to a database. An empty or missing directory validates zero projects, so check the reported count before treating validation as successful.

Run locally

From the application directory, start the monitor:

cd upstream-monitor
PROJECTS_DIR=examples/file-layout/config/projects \
  go run ./cmd/upstream-monitor

In another terminal, from the same directory, start the API:

PROJECTS_DIR=examples/file-layout/config/projects \
  go run ./cmd/upstream-monitor-api

Both processes must use the same project and results storage. File results default to observations/, with provider cache data under cache/. Without project definitions, the monitor has nothing to check.

For an upstream check without persisting results:

go run ./cmd/upstream-monitor dry-run \
  --projects-dir /path/to/projects --project example-widget

dry-run contacts the upstream and can consume API quota. It resolves credentials through source.credential_ref, not the monitor’s GitHub App configuration.

Configuration

Variable	Default	Purpose
`PROJECT_STORAGE_BACKEND`	`file`	Definitions: `file`, `postgres`, or `s3`
`RESULTS_STORAGE_BACKEND`	`file`	Results: `file`, `postgres`, or `s3`
`PROJECT_MUTATIONS`	`auto`	Definition writes: `auto`, `enabled`, or `disabled`
`PROJECTS_DIR`	`config/projects`	External project JSON directory
`POSTGRES_DSN`	empty	Required when either backend is PostgreSQL
`S3_BUCKET`	empty	Required when either backend is S3
`S3_PREFIX`	empty	Unique prefix for each installation sharing a bucket
`S3_REGION`	`us-east-1`	AWS region
`S3_ENDPOINT`	empty	Optional S3 compatible endpoint
`S3_USE_PATH_STYLE`	`false`	Enable path style S3 addressing
`OBSERVATIONS_DIR`	`observations`	File results directory
`CACHE_DIR`	`cache`	File backend provider cache
`LISTEN_ADDR`	`:8080`	API listen address
`POLL_INTERVAL`	`5m`	Scheduler interval and default project check interval
`UPSTREAM_TIMEOUT`	`5m`	Maximum duration of an upstream check
`MAX_CONCURRENT_CHECKS`	`4`	Concurrent checks per worker
`MAX_CONCURRENT_GIT_FETCHES`	`1`	Concurrent uncached Git fetches per worker
`ENRICHMENT_ENABLED`	`false`	Include provenance in API responses

File results support a single writer. Use PostgreSQL or S3 for shared worker coordination. S3 must support conditional writes. Do not expire observation or coordination objects through bucket lifecycle policies.

Use GITHUB_TOKEN for token based GitHub access. For a GitHub App, set GITHUB_APP_ID, GITHUB_APP_INSTALLATION_ID, and GITHUB_APP_PRIVATE_KEY_FILE together. A project’s source.credential_ref names an environment variable containing a credential, never the credential itself. Keep secrets outside Git and container images.

See configuration defaults and environment overrides for remaining cache, lease, retry, and concurrency settings.

Project Definitions

Each JSON filename must match its id. IDs start with a lowercase letter, contain only lowercase letters, digits, and hyphens, and have at most 128 characters. Do not supply uid, which the service assigns.

{
  "id": "example-widget",
  "name": "Example widget",
  "homepage": "https://example.org/widget",
  "source": {
    "type": "github-releases",
    "location": "example/widget"
  },
  "version_policy": {
    "scheme": "semantic",
    "include_prereleases": false
  },
  "stream_policy": {
    "granularity": "minor"
  },
  "check_policy": {
    "mode": "fixed",
    "interval": "15m"
  }
}

Source type	Location
`git-tags`	Git remote URL
`github-releases`	GitHub `owner/repository`
`gitlab-releases`, `gitlab-tags`	GitLab numeric project ID or namespace path
`anitya`	Numeric Anitya project ID
`http-regex`	HTTP(S) URL with a `source.extract` regular expression

Set source.base_url for a separate GitLab or Anitya instance, and source.tag_pattern to select a tag family. Upstream destinations are not host allowlisted, so infrastructure must enforce the required network egress policy.

The default version scheme is semantic. Other choices are calendar, rpm, pep440 (python), and lexical (generic). Use transforms to normalize versions, and version_filters and prerelease_filters to control release selection. Streams group releases by major or minor, with optional semantic numeric component buckets.

Optional lifecycle modes are latest-release, latest-n-lines, date-based, endoflife-date, and manual, with per stream overrides. Fixed check policies use interval. Adaptive policies back off between min_interval and max_interval, resetting after a new release. The maximum interval is also the worst polling detection delay.

Catalog Reconciliation

The project reconciler validates an atomic git-sync checkout and replaces the PostgreSQL definition set. Supply POSTGRES_DSN through the environment:

PROJECT_STORAGE_BACKEND=postgres \
PROJECT_MUTATIONS=disabled \
PROJECT_CHECKOUT_LINK=/git-sync/current \
PROJECT_CHECKOUT_SUBPATH=projects \
  go run ./cmd/upstream-monitor-project-reconciler

Run from upstream-monitor/. The projects subpath matches the separate catalog repository. Set PROJECT_MUTATIONS=disabled on every process using that catalog. Workers and API replicas read its database copy.

Invalid revisions leave the last valid catalog active. Reconciliation removes definitions absent from Git. Verify completeness and back up the database before enabling it. Git cannot restore deleted observation history.

The monitor initializes database schemas. The API does not. Schema and image rollouts belong to infrastructure, not API startup.

API and Authentication

Open / for the catalog browser. The generated API schema is available at /openapi.json and /openapi.yaml. /docs redirects to the JSON schema.

Endpoint	Purpose
`/healthz`	Process health
`/readyz`	Project store readability, not proof of successful polling
`/v1/status`	Aggregate and per project check status
`/v1/project-reconciliation`	Attempted and active Git revisions and reconciliation errors
`/v1/projects`	Project search and pagination

Use last_success_at and project errors to assess freshness. Follow X-Next-Cursor for paginated reads. Definition updates and deletes require the current ETag in If-Match. A queued scan retains observations, while rescan clears that project’s snapshot and release history before checking.

Catalog reads are public. Definition reads, writes, inline checks, and scan requests require authentication and Cedar authorization. PROJECT_MUTATIONS=auto disables file definition writes but permits supported PostgreSQL and S3 writes.

`API_AUTH_MODE`	Required configuration
`disabled`	Default. Authenticated operations are unavailable.
`static-token`	`API_AUTH_TOKEN`, `API_AUTH_PRINCIPAL`, `API_AUTH_ROLES`
`kubernetes-tokenreview`	`API_KUBERNETES_AUDIENCE` and a role mapping file
`oidc`	`API_OIDC_ISSUER`, `API_OIDC_AUDIENCE`, and a role mapping file
`trusted-proxy`	`API_TRUSTED_PROXY_CIDRS` and a role mapping file
`insecure`	Local development only. Never use in production.

API_AUTH_ROLE_MAPPINGS_FILE maps subjects and groups to Cedar roles:

{
  "subjects": {
    "system:serviceaccount:ci:release-bot": ["ProjectChecker"]
  },
  "groups": {
    "editors": ["ProjectEditor"]
  }
}

The default policy grants definition reads to ProjectReader, inline checks to ProjectChecker, editing and scan operations to ProjectEditor, and all actions to ProjectAdmin. Use API_CEDAR_POLICY_FILE to replace that policy.

TokenReview requires a mounted reviewer token and permission to create tokenreviews.authentication.k8s.io. Trust proxy identity headers only from an isolated authenticating proxy that callers cannot bypass. Remove inbound identity headers at the proxy, and prefer loopback CIDRs for a sidecar rather than trusting a whole cluster network.

Project status

Successful checks return to scheduled. waiting and succeeded are accepted for compatibility with older stored records:

stateDiagram-v2
    [*] --> queued : definition due now
    [*] --> scheduled : definition with future check
    [*] --> waiting : load legacy record
    [*] --> succeeded : load older successful record
    waiting --> queued : restart with due check
    waiting --> scheduled : restart with future check
    succeeded --> queued : next check claimed
    scheduled --> queued : check due, scan, or definition changed
    queued --> processing : worker starts
    queued --> scheduled : repository lease held elsewhere
    processing --> scheduled : snapshot persisted
    processing --> failed : check or persistence error
    failed --> queued : retry or definition changed
    processing --> queued : recover interrupted file worker

Records include the project identity, definition fingerprint, pending scan request, queue/start/finish/next check times, last_success_at, failure count, last error, and release/stream counts.

Events and Monitoring

Set NATS_URL to enable JetStream publishing and NATS_CREDS_FILE for mounted credentials. Default subjects are hummingbird.upstream.version.v1 and hummingbird.upstream.stream.v1. Set NATS_MANAGE_STREAM=false if infrastructure provisions the stream. Delivery is at least once, so consumers must deduplicate using event_key. Publication failures retain events for retry.

Logs are JSON on stdout by default, configured through LOG_LEVEL and LOG_FORMAT. Set OTEL_METRICS_EXPORTER=otlp with standard OpenTelemetry endpoint settings to export metrics. Unset exporter configuration does not connect to a local collector.

Monitor time since the last successful check, check failures, GitHub quota, and outbox depth and oldest pending age. Quiet projects remain healthy when checks succeed and adaptive intervals grow.

Development

Run from upstream-monitor/:

go vet ./...
go test ./...
go build ./cmd/...
go run ./cmd/upstream-monitor validate --projects-dir examples/file-layout/config/projects

The GitLab job also rejects empty fixture directories. Container smoke tests mount the examples and check API health through the K8s test runner.

Integration tests require disposable services. Set TEST_POSTGRES_DSN for PostgreSQL storage, TEST_POSTGRES_RECONCILE_DSN for a separate catalog replacement database, TEST_S3_ENDPOINT and TEST_S3_BUCKET for S3, or NATS_CLUSTER_TEST_URL for a JetStream cluster with three stream replicas. Never point these tests at production services.

License

See the repository LICENSE.