RPM Tracker

RPM Tracker owns the rpm_tracker schema in the shared events PostgreSQL database. It stores upstream state and packaged version status for the dashboard to read. See the system proposal for the collection and UI design.

Features

  1. Independent Alembic migration chain with an initializer and revision gate.
  2. Transactional replacement of the current inventory with a generation check.
  3. Read-only dashboard access to tracker data in the shared database.
  4. Full RPM spec and upstream monitor collection with currency, staleness, packaged-stream lifecycle, and independent upstream scan health.
  5. Database-only update MR reconciliation with merge-time rollouts, independent open-MR listing, and creation-based ages with explicit uncertainty.
  6. A standalone Konflux image with native RPM support and independent collection and MR-reconciliation CronJob templates for staging and production.

Prerequisites

Python 3.11 or newer and access to the shared PostgreSQL events database. The migration user needs permission to create the rpm_tracker schema. Collection also requires Git, rpmspec, the system Python RPM bindings, and read access to the rpms repository and upstream monitor. On Fedora, install git-core, rpm-build, and python3-rpm. The Python dependencies needed to load the rpms version helper are included in the tracker package. Do not run collection in a virtual environment that hides the system RPM bindings. The helper also needs the packaged specfile dependency to include sudo’s upstream patchlevel from its extraver macro. MR reconciliation requires read access to public.gitlab_merge_requests and write access to tracker tables. It needs neither GitLab API credentials nor native RPM bindings.

Installation

cd rpm-tracker
pip install -e .
DATABASE_URL=postgresql://postgres@localhost:5432/events python -m rpm_tracker.init_db

The Alembic chain uses rpm_tracker.alembic_version, separate from other services’ migration histories and outside public. The migration files are bundled in the installed rpm_tracker package. The initializer creates the schema and applies pending migrations on each invocation; it is a no-op at the current revision. Run it before deploying the dashboard API, and run it as an init step before the collector starts. A failed migration must prevent collection. The dashboard read endpoint returns HTTP 503 until a migrated schema compatible with its response model is available. Later tracker revisions remain readable when they preserve the fields used by the dashboard.

Online initialization takes a PostgreSQL transaction advisory lock, so collection and reconciliation init containers can start together without racing migrations. The initializer delegates this database work to rpm_tracker.db.initialize(). That helper creates the schema, opens the transaction, and passes its connection to Alembic before applying runtime grants. Alembic’s migrations/env.py only configures offline or online migration execution and the tracker ORM metadata. When RPM_TRACKER_RUNTIME_PASSWORD is supplied, initialization also creates or updates the rpm_tracker login role. This requires the existing Status database administrator credential. The role receives tracker-table reads and writes, read-only access to the tracker revision and public.gitlab_merge_requests, and no schema-creation or administrator privileges. Role/password updates and migrations commit together. Omit this variable for migration-only initialization.

Standalone Alembic commands do not create the schema or configure runtime roles. Run the initializer first. Offline SQL likewise assumes that the tracker schema has already been provisioned.

Container image

The rpm-tracker/Containerfile installs Git, rpm-build, python3-rpm, CA certificates, and the tracker package from pyproject.toml. The installed package includes its migrations; the image runs as the base image’s non-root user. Konflux discovers the Containerfile through the existing tools renderer and builds both supported architectures. The release must be pinned by digest in the infrastructure repository’s vars.d/rpm-tracker.yml before deployment.

podman build -t localhost/rpm-tracker:dev rpm-tracker
podman run --rm localhost/rpm-tracker:dev rpm-tracker-collect --help

The infrastructure kubernetes/rpm-tracker/ templates initialize the database before each run, then use the restricted runtime role. Collection is scheduled at 0 */4 * * * UTC; reconciliation is scheduled independently every ten minutes. The staging and production variants use their own Status database and runtime password. The infrastructure operating runbook documents rollout and manual checks.

Usage

Collect an inventory

DATABASE_URL=postgresql://postgres@localhost:5432/events \
  python -m rpm_tracker.collect \
  --rpms-repository https://gitlab.com/org/group/rpms.git \
  --monitor-url https://upstream-monitor.example.org

The installed rpm-tracker-collect command accepts the same arguments. Each run checks the migrated schema and reads the expected generation before cloning the repository’s default branch into a temporary directory. It reads every package directory, queries spec versions through that checkout’s parse_spec_version helper, and records the checked commit SHA. Metadata version is not a fallback. Missing or malformed metadata and failed checkout enumeration stop collection; an unparseable spec does not omit a package. Clone failures log Git’s captured stderr before re-raising the original CalledProcessError, which retains the diagnostics in its stderr attribute.

The collector reads every monitor project-catalog and release-history page, following X-Next-Cursor even on an empty page. History pagination omits releases without dates, so collection also reads GET /v1/projects/{id} and merges its releases and observed_releases into the history by source and release ID. Snapshot records take precedence for duplicates, including corrected or newly unknown dates. Undated newer releases therefore affect currency and make staleness unavailable instead of disappearing from the comparison. Anitya’s latest-only date coverage is supported. A JSON null stream list is treated as empty; malformed non-list responses remain errors. A project without eligible releases produces unknown without preventing other packages from refreshing.

Requests have a 30-second timeout and are paced sequentially with a 0.2-second delay. All monitor redirects are rejected, including same-origin redirects, so bearer tokens are never forwarded to a redirect destination. Configure the canonical monitor API base URL; a redirect fails collection and retains the previous inventory. Definitions verify repository matches; shared projects are fetched only once. HTTP, response validation, or pagination failures, including a failed snapshot request, prevent any inventory write. Monitor APIs do not promise a single snapshot across requests; a later complete run reconciles changes.

Currency uses native RPM version comparison with no packaging or lexical fallback. A version equal to or ahead of the eligible upstream maximum is current; an eligible newer release makes it behind. No verified project mapping means unmonitored. A mapped package with an unusable spec, update target, or comparison is unknown; its reason is logged and monitor health is retained. Pinned lines use component-boundary matching. With no track_upstream, use upstream_stream or a uniquely matched packaged stream, never global latest. An explicit stream does not change a pinned or latest update target.

Monitor scheme names are trimmed and lowercased during definition validation, matching the monitor registry. Mixed-case or whitespace-padded names therefore use the same normalization and pinned-target filtering as their canonical names.

first_behind_at is recomputed as the earliest date of an eligible release newer than the packaged version. Later releases do not reset it. If any eligible newer release lacks a date, staleness is unavailable. Lifecycle always describes the packaged stream, independently of the update target. Non-numeric/custom stream names require an exact release match or an explicit upstream_stream override. Package suffix stripping and named transforms reuse the rpms helper; already normalized OpenJDK versions are not transformed twice. After explicit package transforms, semantic +build metadata is removed and prerelease hyphens are represented as RPM tildes. Build-only differences do not create upgrades or staleness, while OpenJDK’s build number is retained by its explicit transform. For monitor schemes pep440 and python, PEP 440 alpha, beta, and release-candidate versions are normalized to RPM tilde notation (3.15.0rc2 becomes 3.15.0~rc2), so equivalent packaged prereleases compare equal and final releases sort later. The standard packaging parser handles PEP 440 aliases and casing. Nonzero epochs, development releases, post-releases, local versions, and invalid PEP 440 values produce unknown rather than an unsupported comparison. CVE version_transform and source-artifact availability checks do not affect tracker currency.

Repository URLs are matched across HTTPS, SSH, and Git syntax with .git and trailing slashes removed. Multiple matching projects require the metadata field upstream_monitor_project_id. This is a monitor identifier, not an Anitya release_monitoring_project_id. Providers whose repository cannot be recovered from their definition, such as Anitya or numeric GitLab IDs, require that explicit verified mapping. A resolvable source repository must agree with upstream_repo.

A monitor status reporting a failed upstream scan is preserved even if readable history permits successful collection. Dashboard consumers can distinguish old collection data from failed scanning using sync timestamps and the separate monitor_state, monitor_last_error, and monitor_last_success_at fields. Write failures and generation conflicts propagate and leave the last successful inventory unchanged; they require a fresh collection rather than retrying an old payload. The collector does not migrate the database; the CronJob’s blocking init container runs the initializer first.

Reconcile update MRs

DATABASE_URL=postgresql://postgres@localhost:5432/events \
  python -m rpm_tracker.reconcile \
  --rpms-project org/group/rpms --target-branch main

The installed rpm-tracker-reconcile command accepts the same arguments. Run the initializer before reconciliation and schedule reconciliation periodically, independently of monitor collection. The first successful reconciliation stores the exact project and target branch in rpm_sync_state; subsequent runs reject a different scope to prevent historical rollout records being reassigned.

The reconciler reads the existing Status MR rows in the shared database. It recognizes chore(rpms): Update <package> to <version> and Update <package> to <version>, including Draft:, WIP:, and STALE PATCH: prefixes. Both chore/upstream-update-<package> and chore/upstream-update-failure-<package> corroborate package identity, including diagnostic MRs retitled to valid update titles, but never establish a version. Package names and versions are exact, not substring or prefix matches. Unsupported RPM version syntax and conflicting branch/title packages remain uncertain. Diagnostic failure MRs with title Draft: investigate failed upstream update for <package> remain visible as package-level candidates without a target version.

rpm_update_mrs preserves every qualifying MR by (project, iid), including multiple open MRs for one package and packages absent from the current inventory. Each successful run refreshes the candidate snapshot from the Status rows; closed and no-longer-qualifying MRs disappear from the open list. Previously observed merged identities survive unrecognized titles, changes of target branch, and temporarily missing Status rows. This retained evidence can reject a later version change, but only currently qualifying MRs can resolve a rollout. The tracker stores creation and merge timestamps from the raw webhook’s object_attributes.created_at and object_attributes.merged_at. Timestamps must include an offset, Z, or GitLab’s space-separated UTC suffix and are normalized to UTC. Neither mutable updated_at, observation time, nor MR-head-version dates are substitutes. Previously reliable timestamps survive later omitted fields. Conflicting timestamps remain unavailable, with their reason retained for investigation.

Rollouts are keyed by (package, version). Exactly one verified merged MR in the configured scope with an explicit reliable merge timestamp resolves a rollout. Missing timestamps, conflicting identities, and multiple matching merged MRs leave rolled_out_at null with an unavailable_reason. Merge times before creation or in the future are unavailable. Changing a verified version identity after merge cannot move its rollout to another version. The timestamp is a v1 merge-time proxy, not proof of successful publication.

Every run revisits current packaged versions, verified merged update targets, and historical rollout keys. Later MR evidence can therefore resolve an unavailable historical version after the inventory has advanced. New ambiguity invalidates an earlier resolution. A rollout’s resolved_at changes only when its resolution result changes. The MR snapshot, rollout changes, and last_mr_reconciled_at commit atomically under the existing sync-row lock; failure retains the previous snapshot and freshness timestamp. MR reconciliation does not change the inventory generation or collection freshness.

Read MR information

The dashboard exposes:

  • GET /api/rpm-upstream-state: inventory, rollout reasons, reconciliation scope/freshness, all open_update_mrs, and one blocking_mr per package row.
  • GET /api/rpm-update-mrs: all qualifying open MRs independently of package rows, plus last_mr_reconciled_at.
  • GET /api/rpm-update-mrs?stuck_only=true: MRs older than 24 hours and MRs with unknown ages. Exactly 24 hours is not older than the threshold.

MR ages are computed from creation time at read time. Missing, conflicting, or future creation times produce null age_seconds and null is_stuck, never zero or omission. created_at_issue explains invalid or conflicting evidence. The package-row MR prefers a verified match to the selected upstream version, then the newest known creation time, source update time, project, and IID for deterministic selection. A fallback MR link is a package-level candidate, not evidence of a target-version match. Selecting one package-row MR never removes other MRs from either listing. A null reconciliation timestamp means no MR snapshot has been collected yet.

Transactional storage

Create a SQLAlchemy engine with rpm_tracker.db.get_engine() for the storage module, and dispose of it when collection finishes. The tracker-owned ORM models mirror the Alembic tables; only the initializer migrates them. Call store.get_generation(engine) before collecting; both it and store.ingest(engine, request) require the bundled schema revision. Fetch the complete RPM checkout, every required monitor release page, and each mapped project’s snapshot before calling ingest() with one status per package and the checked checkout SHA. A failed fetch must not submit a partial inventory. ingest() locks the singleton sync row, checks the expected generation, and replaces upstream and status rows in one transaction. A failed write or conflicting generation leaves the previous inventory unchanged. Stored rollout times and MR information are separate from inventory replacement. The dashboard reads the sync, upstream, status, rollout, and MR tables in one read-only snapshot; it does not migrate or write tracker tables.

Configuration

Variable Description
DATABASE_URL Shared events PostgreSQL connection URL
POSTGRES_URL Connection URL fallback
RPMS_REPOSITORY_URL Repository URL, alternatively supplied by --rpms-repository
UPSTREAM_MONITOR_URL Monitor base URL, alternatively supplied by --monitor-url
UPSTREAM_MONITOR_TOKEN Optional bearer token for monitor API reads
RPM_TRACKER_RUNTIME_PASSWORD Initializer-only password for the restricted runtime role
PGPASSWORD PostgreSQL password supplied separately from the connection URL
SSL_CERT_FILE Trusted CA bundle for the monitor client’s stdlib HTTPS requests
GIT_SSL_CAINFO Trusted CA bundle for Git HTTPS requests
REQUESTS_CA_BUNDLE Trusted CA bundle for the rpms helper’s Requests dependency
Reconciliation option Description
--rpms-project Required exact GitLab project path, e.g. org/group/rpms
--target-branch Exact inventory target branch; defaults to main

Development

cd rpm-tracker
pip install -e '.[dev]'
pytest

To run the PostgreSQL migration, reconciliation, and atomic replacement tests, set DATABASE_URL to a disposable PostgreSQL database before running the tracker and dashboard tests. CI supplies test_events; the database-backed tests are skipped when DATABASE_URL is unset. The tracker tests refuse to replace an existing nonempty tracker inventory.

The initializer privilege test requires no existing rpm_tracker login role and uses a disposable database. It verifies concurrent initialization, quoted passwords, allowed tracker writes, and rejected public/revision/schema writes. Image smoke checks in rpm-tracker/tests-k8s.yml verify executable CLIs, native RPM ordering, spec parsing, specfile macro expansion, and bundled migrations through the existing Konflux K8s integration scenario.

Native RPM ordering and collector database integration checks require the system Python RPM bindings. Policy and collection-failure unit tests can run without them, but production collection fails immediately when the bindings are unavailable.

Check model and migration drift

After initializing a disposable database, compare its migrated schema with the tracker ORM models from the rpm-tracker/ directory:

DATABASE_URL=postgresql://postgres@localhost:5432/events \
  python -m alembic -c rpm_tracker/alembic.ini check

Alembic receives models.Base.metadata and reflects only the rpm_tracker schema. Supported schema differences, including unexpected tracker columns and tables, fail the check. Status tables in public and other service schemas are excluded. Checking does not run role provisioning or change the database.

License

This project is licensed under the GNU General Public License v3.0 or later. See the LICENSE.