RPM Tracker
RPM Tracker owns the rpm_tracker schema in the shared events PostgreSQL
database. It stores upstream state and packaged version status for the dashboard
to read. See the system proposal for the collection and UI design.
Features
- Independent Alembic migration chain with an initializer and revision gate.
- Transactional replacement of the current inventory with a generation check.
- Read-only dashboard access to tracker data in the shared database.
- Full RPM spec and upstream monitor collection with currency, staleness, packaged-stream lifecycle, and independent upstream scan health.
- Database-only update MR reconciliation with merge-time rollouts, independent open-MR listing, and creation-based ages with explicit uncertainty.
- A standalone Konflux image with native RPM support and independent collection and MR-reconciliation CronJob templates for staging and production.
Prerequisites
Python 3.11 or newer and access to the shared PostgreSQL events database.
The migration user needs permission to create the rpm_tracker schema.
Collection also requires Git, rpmspec, the system Python RPM bindings, and
read access to the rpms repository and upstream monitor. On Fedora, install
git-core, rpm-build, and python3-rpm. The Python dependencies needed to
load the rpms version helper are included in the tracker package. Do not run
collection in a virtual environment that hides the system RPM bindings.
The helper also needs the packaged specfile dependency to include sudo’s
upstream patchlevel from its extraver macro.
MR reconciliation requires read access to public.gitlab_merge_requests and
write access to tracker tables. It needs neither GitLab API credentials nor
native RPM bindings.
Installation
cd rpm-tracker
pip install -e .
DATABASE_URL=postgresql://postgres@localhost:5432/events python -m rpm_tracker.init_db
The Alembic chain uses rpm_tracker.alembic_version, separate from other
services’ migration histories and outside public. The migration files are
bundled in the installed rpm_tracker package. The initializer creates the
schema and applies pending migrations on each invocation; it is a no-op at
the current revision. Run it before deploying the dashboard API, and run it as
an init step before the collector starts. A failed migration must prevent
collection. The dashboard read endpoint returns HTTP 503 until a migrated
schema compatible with its response model is available. Later tracker revisions
remain readable when they preserve the fields used by the dashboard.
Online initialization takes a PostgreSQL transaction advisory lock, so collection
and reconciliation init containers can start together without racing migrations.
The initializer delegates this database work to rpm_tracker.db.initialize().
That helper creates the schema, opens the transaction, and passes its connection
to Alembic before applying runtime grants. Alembic’s migrations/env.py only
configures offline or online migration execution and the tracker ORM metadata.
When RPM_TRACKER_RUNTIME_PASSWORD is supplied, initialization also creates or
updates the rpm_tracker login role. This requires the existing Status database
administrator credential. The role receives tracker-table reads and writes,
read-only access to the tracker revision and public.gitlab_merge_requests, and
no schema-creation or administrator privileges. Role/password updates and
migrations commit together. Omit this variable for migration-only initialization.
Standalone Alembic commands do not create the schema or configure runtime roles. Run the initializer first. Offline SQL likewise assumes that the tracker schema has already been provisioned.
Container image
The rpm-tracker/Containerfile installs Git, rpm-build, python3-rpm, CA
certificates, and the tracker package from pyproject.toml. The installed package
includes its migrations; the image runs as the base image’s non-root user.
Konflux discovers the Containerfile through the existing tools renderer and
builds both supported architectures. The release must be pinned by digest in the
infrastructure repository’s vars.d/rpm-tracker.yml before deployment.
podman build -t localhost/rpm-tracker:dev rpm-tracker
podman run --rm localhost/rpm-tracker:dev rpm-tracker-collect --help
The infrastructure kubernetes/rpm-tracker/ templates initialize the database
before each run, then use the restricted runtime role. Collection is scheduled at
0 */4 * * * UTC; reconciliation is scheduled independently every ten minutes.
The staging and production variants use their own Status database and runtime
password. The infrastructure operating runbook documents rollout and manual checks.
Usage
Collect an inventory
DATABASE_URL=postgresql://postgres@localhost:5432/events \
python -m rpm_tracker.collect \
--rpms-repository https://gitlab.com/org/group/rpms.git \
--monitor-url https://upstream-monitor.example.org
The installed rpm-tracker-collect command accepts the same arguments. Each run
checks the migrated schema and reads the expected generation before cloning the
repository’s default branch into a temporary directory. It reads every package
directory, queries spec versions through that checkout’s parse_spec_version
helper, and records the checked commit SHA. Metadata version is not a fallback.
Missing or malformed metadata and failed checkout enumeration stop collection;
an unparseable spec does not omit a package.
Clone failures log Git’s captured stderr before re-raising the original
CalledProcessError, which retains the diagnostics in its stderr attribute.
The collector reads every monitor project-catalog and release-history page,
following X-Next-Cursor even on an empty page. History pagination omits releases
without dates, so collection also reads GET /v1/projects/{id} and merges its
releases and observed_releases into the history by source and release ID.
Snapshot records take precedence for duplicates, including corrected or newly
unknown dates. Undated newer releases therefore affect currency and make
staleness unavailable instead of disappearing from the comparison. Anitya’s
latest-only date coverage is supported. A JSON null stream list is treated as
empty; malformed non-list responses remain errors. A project without eligible
releases produces unknown without preventing other packages from refreshing.
Requests have a 30-second timeout and are paced sequentially with a 0.2-second delay. All monitor redirects are rejected, including same-origin redirects, so bearer tokens are never forwarded to a redirect destination. Configure the canonical monitor API base URL; a redirect fails collection and retains the previous inventory. Definitions verify repository matches; shared projects are fetched only once. HTTP, response validation, or pagination failures, including a failed snapshot request, prevent any inventory write. Monitor APIs do not promise a single snapshot across requests; a later complete run reconciles changes.
Currency uses native RPM version comparison with no packaging or lexical
fallback. A version equal to or ahead of the eligible upstream maximum is
current; an eligible newer release makes it behind. No verified project
mapping means unmonitored. A mapped package with an unusable spec, update
target, or comparison is unknown; its reason is logged and monitor health is
retained. Pinned lines use component-boundary matching. With no track_upstream,
use upstream_stream or a uniquely matched packaged stream, never global latest.
An explicit stream does not change a pinned or latest update target.
Monitor scheme names are trimmed and lowercased during definition validation, matching the monitor registry. Mixed-case or whitespace-padded names therefore use the same normalization and pinned-target filtering as their canonical names.
first_behind_at is recomputed as the earliest date of an eligible release newer
than the packaged version. Later releases do not reset it. If any eligible newer
release lacks a date, staleness is unavailable. Lifecycle always describes the
packaged stream, independently of the update target. Non-numeric/custom stream
names require an exact release match or an explicit upstream_stream override.
Package suffix stripping and named transforms reuse the rpms helper; already
normalized OpenJDK versions are not transformed twice. After explicit package
transforms, semantic +build metadata is removed and prerelease hyphens are
represented as RPM tildes. Build-only differences do not create upgrades or
staleness, while OpenJDK’s build number is retained by its explicit transform.
For monitor schemes pep440 and python,
PEP 440 alpha, beta, and release-candidate versions are normalized to RPM tilde
notation (3.15.0rc2 becomes 3.15.0~rc2), so equivalent packaged prereleases
compare equal and final releases sort later. The standard packaging parser
handles PEP 440 aliases and casing. Nonzero epochs, development releases,
post-releases, local versions, and invalid PEP 440 values produce unknown
rather than an unsupported comparison. CVE version_transform and
source-artifact availability checks do not affect tracker currency.
Repository URLs are matched across HTTPS, SSH, and Git syntax with .git and
trailing slashes removed. Multiple matching projects require the metadata field
upstream_monitor_project_id. This is a monitor identifier, not an Anitya
release_monitoring_project_id. Providers whose repository cannot be recovered
from their definition, such as Anitya or numeric GitLab IDs, require that explicit
verified mapping. A resolvable source repository must agree with upstream_repo.
A monitor status reporting a failed upstream scan is preserved even if readable
history permits successful collection. Dashboard consumers can distinguish old
collection data from failed scanning using sync timestamps and the separate
monitor_state, monitor_last_error, and monitor_last_success_at fields.
Write failures and generation conflicts propagate and leave the last successful
inventory unchanged; they require a fresh collection rather than retrying an old
payload. The collector does not migrate the database; the CronJob’s blocking
init container runs the initializer first.
Reconcile update MRs
DATABASE_URL=postgresql://postgres@localhost:5432/events \
python -m rpm_tracker.reconcile \
--rpms-project org/group/rpms --target-branch main
The installed rpm-tracker-reconcile command accepts the same arguments. Run
the initializer before reconciliation and schedule reconciliation periodically,
independently of monitor collection. The first successful reconciliation stores
the exact project and target branch in rpm_sync_state; subsequent runs reject
a different scope to prevent historical rollout records being reassigned.
The reconciler reads the existing Status MR rows in the shared database. It
recognizes chore(rpms): Update <package> to <version> and
Update <package> to <version>, including Draft:, WIP:, and STALE PATCH:
prefixes. Both chore/upstream-update-<package> and
chore/upstream-update-failure-<package> corroborate package identity, including
diagnostic MRs retitled to valid update titles, but never establish a version.
Package names and versions are exact, not substring or prefix matches.
Unsupported RPM version syntax and conflicting
branch/title packages remain uncertain. Diagnostic failure MRs with title
Draft: investigate failed upstream update for <package> remain visible as
package-level candidates without a target version.
rpm_update_mrs preserves every qualifying MR by (project, iid), including
multiple open MRs for one package and packages absent from the current inventory.
Each successful run refreshes the candidate snapshot from the Status rows;
closed and no-longer-qualifying MRs disappear from the open list. Previously
observed merged identities survive unrecognized titles, changes of target
branch, and temporarily missing Status rows. This retained evidence can reject
a later version change, but only currently qualifying MRs can resolve a
rollout. The tracker stores creation and merge timestamps from the raw webhook’s
object_attributes.created_at and object_attributes.merged_at. Timestamps
must include an offset, Z, or GitLab’s space-separated UTC suffix and are normalized to
UTC. Neither mutable updated_at, observation time, nor MR-head-version dates
are substitutes. Previously reliable timestamps survive later omitted fields.
Conflicting timestamps remain unavailable, with their reason retained for
investigation.
Rollouts are keyed by (package, version). Exactly one verified merged MR in
the configured scope with an explicit reliable merge timestamp resolves a
rollout. Missing timestamps, conflicting identities, and multiple matching
merged MRs leave rolled_out_at null with an unavailable_reason. Merge times
before creation or in the future are unavailable. Changing a verified version
identity after merge cannot move its rollout to another version. The timestamp
is a v1 merge-time proxy, not proof of successful publication.
Every run revisits current packaged versions, verified merged update targets,
and historical rollout keys. Later MR evidence can therefore resolve an
unavailable historical version after the inventory has advanced. New ambiguity
invalidates an earlier resolution. A rollout’s resolved_at changes only when
its resolution result changes. The MR snapshot, rollout changes, and
last_mr_reconciled_at commit atomically under the existing sync-row lock;
failure retains the previous snapshot and freshness timestamp. MR reconciliation
does not change the inventory generation or collection freshness.
Read MR information
The dashboard exposes:
GET /api/rpm-upstream-state: inventory, rollout reasons, reconciliation scope/freshness, allopen_update_mrs, and oneblocking_mrper package row.GET /api/rpm-update-mrs: all qualifying open MRs independently of package rows, pluslast_mr_reconciled_at.GET /api/rpm-update-mrs?stuck_only=true: MRs older than 24 hours and MRs with unknown ages. Exactly 24 hours is not older than the threshold.
MR ages are computed from creation time at read time. Missing, conflicting, or
future creation times produce null age_seconds and null is_stuck, never
zero or omission. created_at_issue explains invalid or conflicting evidence.
The package-row MR prefers a verified match to the selected upstream version,
then the newest known creation time, source update time, project, and IID for
deterministic selection. A fallback MR link is a package-level candidate, not
evidence of a target-version match. Selecting one package-row MR never removes
other MRs from either listing. A null reconciliation timestamp means no MR
snapshot has been collected yet.
Transactional storage
Create a SQLAlchemy engine with rpm_tracker.db.get_engine() for the storage
module, and dispose of it when collection finishes. The tracker-owned ORM
models mirror the Alembic tables; only the initializer migrates them. Call
store.get_generation(engine) before collecting; both it and
store.ingest(engine, request) require the bundled schema revision. Fetch the
complete RPM checkout, every required monitor release page, and each mapped
project’s snapshot before calling
ingest() with one status per package and the checked checkout SHA. A failed
fetch must not submit a partial inventory. ingest() locks the singleton sync
row, checks the expected generation, and replaces upstream and status rows in
one transaction. A failed write or conflicting generation leaves the previous
inventory unchanged. Stored rollout times and MR information are separate from
inventory replacement. The dashboard reads the
sync, upstream, status, rollout, and MR tables in one read-only snapshot; it does
not migrate or write tracker tables.
Configuration
| Variable | Description |
|---|---|
DATABASE_URL |
Shared events PostgreSQL connection URL |
POSTGRES_URL |
Connection URL fallback |
RPMS_REPOSITORY_URL |
Repository URL, alternatively supplied by --rpms-repository |
UPSTREAM_MONITOR_URL |
Monitor base URL, alternatively supplied by --monitor-url |
UPSTREAM_MONITOR_TOKEN |
Optional bearer token for monitor API reads |
RPM_TRACKER_RUNTIME_PASSWORD |
Initializer-only password for the restricted runtime role |
PGPASSWORD |
PostgreSQL password supplied separately from the connection URL |
SSL_CERT_FILE |
Trusted CA bundle for the monitor client’s stdlib HTTPS requests |
GIT_SSL_CAINFO |
Trusted CA bundle for Git HTTPS requests |
REQUESTS_CA_BUNDLE |
Trusted CA bundle for the rpms helper’s Requests dependency |
| Reconciliation option | Description |
|---|---|
--rpms-project |
Required exact GitLab project path, e.g. org/group/rpms |
--target-branch |
Exact inventory target branch; defaults to main |
Development
cd rpm-tracker
pip install -e '.[dev]'
pytest
To run the PostgreSQL migration, reconciliation, and atomic replacement tests, set
DATABASE_URL to a disposable PostgreSQL database before running the tracker
and dashboard tests. CI supplies test_events; the database-backed tests are
skipped when DATABASE_URL is unset. The tracker tests refuse to replace an
existing nonempty tracker inventory.
The initializer privilege test requires no existing rpm_tracker login role and
uses a disposable database. It verifies concurrent initialization, quoted
passwords, allowed tracker writes, and rejected public/revision/schema writes.
Image smoke checks in rpm-tracker/tests-k8s.yml verify executable CLIs, native
RPM ordering, spec parsing, specfile macro expansion, and bundled migrations
through the existing Konflux K8s integration scenario.
Native RPM ordering and collector database integration checks require the system Python RPM bindings. Policy and collection-failure unit tests can run without them, but production collection fails immediately when the bindings are unavailable.
Check model and migration drift
After initializing a disposable database, compare its migrated schema with the
tracker ORM models from the rpm-tracker/ directory:
DATABASE_URL=postgresql://postgres@localhost:5432/events \
python -m alembic -c rpm_tracker/alembic.ini check
Alembic receives models.Base.metadata and reflects only the rpm_tracker
schema. Supported schema differences, including unexpected tracker columns and
tables, fail the check. Status tables in public and other service schemas are
excluded. Checking does not run role provisioning or change the database.
License
This project is licensed under the GNU General Public License v3.0 or later. See the LICENSE.