RPM Tracker

RPM tracker owns the rpm_tracker schema in the shared events PostgreSQL database. It stores upstream state and packaged version status for the dashboard to read. See the system proposal for the collection and UI design.

Features

1. Independent Alembic migration chain with an initializer and revision gate.
2. Transactional replacement of the current inventory with a generation check.
3. Read-only dashboard access to tracker data in the shared database.
4. Full RPM spec and upstream monitor collection with currency, staleness, packaged-stream lifecycle, and independent upstream scan health.

Prerequisites

Python 3.11 or newer and access to the shared PostgreSQL events database. The migration user needs permission to create the rpm_tracker schema. Collection also requires Git, rpmspec, the system Python RPM bindings, and read access to the rpms repository and upstream monitor. On Fedora, install git-core, rpm-build, and python3-rpm. The Python dependencies needed to load the rpms version helper are included in the tracker package. Do not run collection in a virtual environment that hides the system RPM bindings.

Installation

cd rpm-tracker
pip install -e .
DATABASE_URL=postgresql://postgres@localhost:5432/events python -m rpm_tracker.init_db

The Alembic chain uses rpm_tracker.alembic_version, separate from other services’ migration histories and outside public. The migration files are bundled in the installed rpm_tracker package. The initializer creates the schema and applies pending migrations on each invocation; it is a no-op at the current revision. Run it before deploying the dashboard API, and run it as an init step before the collector starts. A failed migration must prevent collection. The dashboard read endpoint returns HTTP 503 until a migrated schema compatible with its response model is available. Later tracker revisions remain readable when they preserve the fields used by the dashboard.

Usage

Collect an inventory

DATABASE_URL=postgresql://postgres@localhost:5432/events \
  python -m rpm_tracker.collect \
  --rpms-repository https://gitlab.com/org/group/rpms.git \
  --monitor-url https://upstream-monitor.example.org

The installed rpm-tracker-collect command accepts the same arguments. Each run checks the migrated schema and reads the expected generation before cloning the repository’s default branch into a temporary directory. It reads every package directory, queries spec versions through that checkout’s parse_spec_version helper, and records the checked commit SHA. Metadata version is not a fallback. Missing or malformed metadata and failed checkout enumeration stop collection; an unparseable spec does not omit a package. Clone failures log Git’s captured stderr before re-raising the original CalledProcessError, which retains the diagnostics in its stderr attribute.

The collector reads every monitor project-catalog and release-history page, following X-Next-Cursor even on an empty page. History pagination omits releases without dates, so collection also reads GET /v1/projects/{id} and merges its releases and observed_releases into the history by source and release ID. Snapshot records take precedence for duplicates, including corrected or newly unknown dates. Undated newer releases therefore affect currency and make staleness unavailable instead of disappearing from the comparison. Anitya’s latest-only date coverage is supported. A JSON null stream list is treated as empty; malformed non-list responses remain errors. A project without eligible releases produces unknown without preventing other packages from refreshing.

Requests have a 30-second timeout and are paced sequentially with a 0.2-second delay. All monitor redirects are rejected, including same-origin redirects, so bearer tokens are never forwarded to a redirect destination. Configure the canonical monitor API base URL; a redirect fails collection and retains the previous inventory. Definitions verify repository matches; shared projects are fetched only once. HTTP, response validation, or pagination failures, including a failed snapshot request, prevent any inventory write. Monitor APIs do not promise a single snapshot across requests; a later complete run reconciles changes.

Currency uses native RPM version comparison with no packaging or lexical fallback. A version equal to or ahead of the eligible upstream maximum is current; an eligible newer release makes it behind. No verified project mapping means unmonitored. A mapped package with an unusable spec, update target, or comparison is unknown; its reason is logged and monitor health is retained. Pinned lines use component-boundary matching. With no track_upstream, use upstream_stream or a uniquely matched packaged stream, never global latest. An explicit stream does not change a pinned or latest update target.

Monitor scheme names are trimmed and lowercased during definition validation, matching the monitor registry. Mixed-case or whitespace-padded names therefore use the same normalization and pinned-target filtering as their canonical names.

first_behind_at is recomputed as the earliest date of an eligible release newer than the packaged version. Later releases do not reset it. If any eligible newer release lacks a date, staleness is unavailable. Lifecycle always describes the packaged stream, independently of the update target. Non-numeric/custom stream names require an exact release match or an explicit upstream_stream override. Package suffix stripping and named transforms reuse the rpms helper; already normalized OpenJDK versions are not transformed twice. After explicit package transforms, semantic +build metadata is removed and prerelease hyphens are represented as RPM tildes. Build-only differences do not create upgrades or staleness, while OpenJDK’s build number is retained by its explicit transform. For monitor schemes pep440 and python, PEP 440 alpha, beta, and release-candidate versions are normalized to RPM tilde notation (3.15.0rc2 becomes 3.15.0~rc2), so equivalent packaged prereleases compare equal and final releases sort later. The standard packaging parser handles PEP 440 aliases and casing. Nonzero epochs, development releases, post-releases, local versions, and invalid PEP 440 values produce unknown rather than an unsupported comparison. CVE version_transform and source-artifact availability checks do not affect tracker currency.

Repository URLs are matched across HTTPS, SSH, and Git syntax with .git and trailing slashes removed. Multiple matching projects require the metadata field upstream_monitor_project_id. This is a monitor identifier, not an Anitya release_monitoring_project_id. Providers whose repository cannot be recovered from their definition, such as Anitya or numeric GitLab IDs, require that explicit verified mapping. A resolvable source repository must agree with upstream_repo.

A monitor status reporting a failed upstream scan is preserved even if readable history permits successful collection. Dashboard consumers can distinguish old collection data from failed scanning using sync timestamps and the separate monitor_state, monitor_last_error, and monitor_last_success_at fields. Write failures and generation conflicts propagate and leave the last successful inventory unchanged; they require a fresh collection rather than retrying an old payload. The collector does not migrate the database. OpenShift scheduling is a subsequent part of the proposal.

Transactional storage

Create a SQLAlchemy engine with rpm_tracker.db.get_engine() for the storage module, and dispose of it when collection finishes. The tracker-owned ORM models mirror the Alembic tables; only the initializer migrates them. Call store.get_generation(engine) before collecting; both it and store.ingest(engine, request) require the bundled schema revision. Fetch the complete RPM checkout, every required monitor release page, and each mapped project’s snapshot before calling ingest() with one status per package and the checked checkout SHA. A failed fetch must not submit a partial inventory. ingest() locks the singleton sync row, checks the expected generation, and replaces upstream and status rows in one transaction. A failed write or conflicting generation leaves the previous inventory unchanged. Stored rollout times are separate from inventory replacement. The dashboard exposes GET /api/rpm-upstream-state to read the sync, upstream, status, and rollout tables in one read-only snapshot; it does not migrate or write tracker tables.

Configuration

Development

cd rpm-tracker
pip install -e '.[dev]'
pytest

To run the PostgreSQL migration and atomic replacement tests, set DATABASE_URL to a disposable PostgreSQL database before running the tracker and dashboard tests. CI supplies test_events; the database-backed tests are skipped when DATABASE_URL is unset. The tracker tests refuse to replace an existing nonempty tracker inventory.

Native RPM ordering and collector database integration checks require the system Python RPM bindings. Policy and collection-failure unit tests can run without them, but production collection fails immediately when the bindings are unavailable.

License

This project is licensed under the GNU General Public License v3.0 or later. See the LICENSE.