This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Hummingbird OSV Generator

A CLI tool that converts Red Hat vulnerability data into OSV records for Hummingbird RPMs, so that vulnerability scanners can match them against the packages installed in an image.

For how a run works end to end, see the workflow.

Features

  • Two sources: shipped fixes from the advisories repository and all known affects, including unfixed CVEs, from OSIDB
  • Binary RPM names: scanners match installed binary packages, so source RPMs are expanded to their binary RPMs
  • Fixed and unfixed states: records carry fixed, affected, will_not_fix or under_investigation in database_specific.status
  • RPM epochs: fixed versions are written as epoch:version-release
  • Incremental sync: content hashing means only changed inputs are parsed and only changed records are written
  • Embargo safe: embargoed or non-public OSIDB data is never published

Output format

The output follows OSV schema 1.9.0 with one record per source RPM per CVE and one affected entry per binary RPM.

Field Content
id Opaque, stable identifier: x_HBA prefix, CVE year and a hash
upstream The CVE the record covers
affected[].package.ecosystem Ecosystem scoped to one product stream by its CPE
affected[].package.purl Versionless package URL
affected[].ranges introduced: 0 and, once shipped, fixed: <epoch:version-release>
affected[].ecosystem_specific.advisory_id Advisory that supplied the fixed version, absent on unfixed entries
database_specific.status fixed, affected, under_investigation or will_not_fix
modified Time of the last content change

The x_ in the identifier prefix marks a local database. It is dropped once HBA is registered with the OSV schema project. The ecosystem name is also a placeholder until it is registered.

Prerequisites

  • Go: the version in hummingbird-osv-generator/go.mod
  • Advisories checkout: a clone of the advisories repository, passed as its data/ directory
  • OSIDB access: the OSIDB base URL, reachable from where the tool runs
  • Git: only when using -git or -git-remote

Installation

git clone https://gitlab.com/redhat/hummingbird/tools.git
cd tools/hummingbird-osv-generator
go build -o rhcve-osv-sync ./cmd/rhcve-osv-sync/

Usage

Full sync for one product stream, committing the output to git:

./rhcve-osv-sync \
  -advisories /path/to/advisories/data \
  -osidb-url https://osidb.example.com \
  -streams hummingbird-1 \
  -products hummingbird \
  -output /path/to/osv-output \
  -state /path/to/state.db \
  -full -git

Later runs without -full use an OSIDB delta query. Run a full reconciliation daily so that missed changes are corrected.

Binary RPM expansion

OSIDB tracks CVEs against source RPM names, while scanners need binary RPM names. -expansion selects how a source RPM is expanded:

Mode Source
osidb OSIDB subpackage_purls
advisory Binary RPM names derived from advisory purls
chain OSIDB first, then the advisory mapping

The default is chain.

Incremental sync

State lives in a SQLite database. Each run goes through these phases: advisory diff, OSIDB diff, convert, write output and commit state. The workflow describes each phase.

  • Advisory files are content-hashed and only changed files are parsed
  • OSIDB uses a delta query, plus a full reconciliation on demand
  • Output files are compared with a manifest and only changed files are written
  • Output is written before state is committed, so a crash is safe
  • If the manifest is lost, the next full run rebuilds it from the output and withdraws stale records
  • If OSIDB fails and the convert phase does not need it, advisory output is still published and the process exits non-zero so a scheduler can retry or alert

Unchanged input produces unchanged output, so duplicate scheduling wastes work but does not corrupt the output.

Configuration

Flag Required Description
-advisories Yes Path to the advisories repository data/ directory
-osidb-url Yes OSIDB base URL
-streams Yes Comma-separated ps_update_stream values, also filters advisories by product_stream
-output Yes Output directory for the OSV JSON files
-state Yes Path to the SQLite state database
-products No Filter advisory conversion by CPE product name
-full No Force a full OSIDB reconciliation
-expansion No Binary RPM expansion mode: osidb, advisory or chain
-git No Run git init, add and commit on the output after the sync
-git-remote No Git remote to push to after the commit (implies -git)
-log-json No Write structured JSON logs

Development

cd hummingbird-osv-generator

# Vet and test, including the OSV schema validation test
go vet ./...
go test ./...

The hummingbird-osv-generator CI job runs the same commands on merge requests.

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.

1 - Hummingbird OSV Generator Workflow

How a run of the Hummingbird OSV Generator turns its inputs into OSV records: what it reads, the phases of a sync, and what it writes.

Overview

flowchart LR
    adv[Advisories data directory] --> p1[Advisory diff]
    osidb[OSIDB] --> p2[OSIDB diff]
    db[(State database)] --> p1
    db --> p2
    p1 --> p3[Convert]
    p2 --> p3
    osidb --> p3
    p3 --> p4[Write output]
    p4 --> out[OSV JSON files]
    p4 --> p5[Commit state]
    p5 --> db
    out --> git[Optional git commit and push]

One run reads the advisories and OSIDB, converts what changed into OSV records, writes only the records whose content changed, and records what it did in the state database. Each run is a complete pass: nothing is kept in memory between runs.

Inputs

Input Flag What the generator reads
Advisories directory -advisories Advisory YAML files: shipped fixes with per-artifact package URLs and CVE fix lists
OSIDB -osidb-url Affects for the given update streams, and the flaws for their CVEs
Update streams -streams Which OSIDB update streams to read. Also filters advisories by product_stream
Products -products Optional filter on the CPE product name of an advisory
State database -state SQLite file from earlier runs, created on the first run
Output directory -output Read only when the record ledger was lost and has to be rebuilt from the published files

Advisories

  • Only security advisories (RHSA) are read. Bug fix and enhancement advisories are skipped, even if they mention CVEs.
  • Each advisory gives the product CPE, and each artifact gives a package URL and the CVEs it fixes. The package URL carries the binary RPM name, the version and release, and the RPM epoch.
  • From the advisories the generator also builds two lookups: the update stream to CPE map, and the source RPM to binary RPM mapping.

OSIDB

An affect is ignored when it has no CVE ID, when it is embargoed or not public, or when it is marked not affected (including a delegated not affected). An affect whose tracker is resolved as shipped (Done-Errata or Fixed) is also ignored, because the advisory supplies that fix.

Flaws supply the CVE title and description, severity, CWE and Red Hat statement. A flaw that is embargoed or not public is ignored, and any text cached from it is dropped. A rejected flaw marks its records as withdrawn.

State database

Table Content
input_hashes Content hashes of advisory files and of OSIDB affects, used to detect changes
sync_meta Cursors: the last sync, the last full reconciliation, and the last flaw change window
cve_meta Cached CVE text and severity, refreshed from the OSIDB flaw change feed
output_manifest One row per published record: path, content hash, time written and time last modified

The input hashes and the cursors are a cache that can be rebuilt. The manifest also holds each record’s last modification time, so it is durable state.

Phases

  1. Advisory diff. Hash every advisory file and compare with the stored hashes. The result is the list of added, changed and removed files.
  2. OSIDB diff. Decide between a delta and a full reconciliation. A delta asks OSIDB for affects updated since the last sync, minus a clock skew allowance, and compares their hashes. A full reconciliation fetches every affect and also detects deletions. By default a full reconciliation runs on the first run, when -full is given, and again once a day. If OSIDB fails here, the run goes on with advisory data when nothing needs OSIDB, and ends with an error (see Failure handling).
  3. Convert. Build records from both sources:
    • Fixed records come from the advisories.
    • Records for every other status come from OSIDB affects. Each affect is expanded from its source RPM to binary RPM names with the mode chosen by -expansion.
    • CVE text and severity come from the flaw cache, refreshed from the OSIDB flaw change feed. If the refresh fails, cached values are used.
    • When an advisory changed and output already exists, the complete OSIDB data is fetched, so that a record is never published from partial input.
  4. Write output. Compare each record’s content hash with the manifest and write only the records that changed. Remove records that are no longer produced (see Failure handling for when this is safe).
  5. Commit state. Store the new input hashes, the manifest and the cursors. The state is committed after the output is written, so a crash between the two repeats work and does not lose changes.

If -git or -git-remote is given, the output directory is then committed, and pushed when a remote is given.

Records

A record covers one source RPM in one update stream for one CVE. It has one affected entry for each binary RPM, and each entry keeps the fixed version of its own binary because subpackages can have different epochs.

When one source RPM has entries in different states, the record takes the status with the highest precedence:

Precedence Status Meaning
1 fixed An advisory shipped the fix
2 affected OSIDB says the product is affected and no fix has shipped
3 under_investigation OSIDB has a new affect with no resolution yet
4 will_not_fix OSIDB resolution is won’t fix or defer

A shipped fix always wins over an unfixed affect for the same record.

published is the earliest of the dates known for the record: when the flaw became public, when the affect was created, and when an advisory shipped. modified changes only when the content of the record changes: a new record takes its source update time, an unchanged record keeps its value, and a changed record gets the time of the write.

Status over time

The status is recomputed from the inputs on every run and is not stored. A typical record moves as follows.

stateDiagram-v2
    [*] --> UnderInvestigation: OSIDB affect created, no resolution
    [*] --> Affected: OSIDB affect with a resolution
    [*] --> WillNotFix: OSIDB resolution won't fix or defer

    UnderInvestigation --> Affected: triaged as affected
    UnderInvestigation --> WillNotFix: resolution won't fix or defer
    WillNotFix --> Affected: resolution changed back
    Affected --> Fixed: advisory ships the fix
    UnderInvestigation --> Fixed: advisory ships the fix
    WillNotFix --> Fixed: advisory ships the fix
    Affected --> UnderInvestigation: OSIDB affect reset to new

    UnderInvestigation --> Removed: marked not affected, embargoed or not public
    Affected --> Removed: marked not affected, embargoed or not public
    WillNotFix --> Removed: marked not affected, embargoed or not public
    Fixed --> Removed: advisory no longer lists the CVE and OSIDB has no affect
    Removed --> [*]

    state "under_investigation" as UnderInvestigation
    state "affected" as Affected
    state "will_not_fix" as WillNotFix
    state "fixed" as Fixed
    state "file removed from the output" as Removed

A removed record is deleted from the output directory and from the manifest. A record whose flaw was rejected is different: it stays in the output with a withdrawn timestamp.

Outputs

Output Location Content
OSV records <output>/<product path>/<ID>.json One JSON file per record. The product path comes from the CPE, for example redhat/hummingbird/1
State database The -state file Updated hashes, manifest, cursors and CVE cache (see State database)
Git commit (optional) The -output directory All changes committed, with a message that summarizes the run
Run summary standard error Counts of records created, updated, withdrawn and unchanged, and files written and removed

The git step initializes the repository if needed, skips the commit when nothing changed, and pushes the current branch when -git-remote is given. It runs only after a sync that ended without an error.

The process exits with 0 on success, 1 when the sync or the git step fails, and 2 for invalid flags.

Failure handling

  • OSIDB failure during the diff. If the convert phase does not need OSIDB data, advisory output is still written and the state is committed, and the process exits with 1 so a scheduler can retry or alert. The git step does not run in that case, so the written files are committed by the next successful run. Records that depend on OSIDB are not removed, because the record set is incomplete.
  • OSIDB unavailable when convert needs it. This happens when an advisory changed and output already exists, because the complete OSIDB data is needed to avoid a partial record. The run fails before writing anything.
  • No OSIDB changes in a delta run. The advisory-only result is not compared with existing records, and no record is removed, because OSIDB data is missing from it.
  • Lost state database. The next run behaves like a first run.
  • Lost manifest, output kept. A full run rebuilds the manifest from the published files and rewrites each record once, so stale records can still be removed.
  • A corrupt published file fails the run when the manifest is rebuilt, so it cannot hide stale state.
  • Repeated or duplicate runs. The same input produces the same output, so a duplicate run wastes work and does not corrupt anything.