This is the multi-page printable view of this section.
Click here to print.
Return to the regular view of this page.
Hummingbird OSV Generator
A CLI tool that converts Red Hat vulnerability data into OSV
records for Hummingbird RPMs, so that vulnerability scanners can match them
against the packages installed in an image.
For how a run works end to end, see the
workflow.
Features
- Two sources: shipped fixes from the advisories repository and all known
affects, including unfixed CVEs, from OSIDB
- Binary RPM names: scanners match installed binary packages, so source
RPMs are expanded to their binary RPMs
- Fixed and unfixed states: records carry
fixed, affected,
will_not_fix or under_investigation in database_specific.status
- RPM epochs: fixed versions are written as
epoch:version-release
- Incremental sync: content hashing means only changed inputs are parsed
and only changed records are written
- Embargo safe: embargoed or non-public OSIDB data is never published
The output follows OSV schema 1.9.0 with one record per source RPM per CVE and
one affected entry per binary RPM.
| Field |
Content |
id |
Opaque, stable identifier: x_HBA prefix, CVE year and a hash |
upstream |
The CVE the record covers |
affected[].package.ecosystem |
Ecosystem scoped to one product stream by its CPE |
affected[].package.purl |
Versionless package URL |
affected[].ranges |
introduced: 0 and, once shipped, fixed: <epoch:version-release> |
affected[].ecosystem_specific.advisory_id |
Advisory that supplied the fixed version, absent on unfixed entries |
database_specific.status |
fixed, affected, under_investigation or will_not_fix |
modified |
Time of the last content change |
The x_ in the identifier prefix marks a local database. It is dropped once
HBA is registered with the OSV schema project. The ecosystem name is also a
placeholder until it is registered.
Prerequisites
- Go: the version in
hummingbird-osv-generator/go.mod
- Advisories checkout: a clone of the advisories repository, passed as its
data/ directory
- OSIDB access: the OSIDB base URL, reachable from where the tool runs
- Git: only when using
-git or -git-remote
Installation
git clone https://gitlab.com/redhat/hummingbird/tools.git
cd tools/hummingbird-osv-generator
go build -o rhcve-osv-sync ./cmd/rhcve-osv-sync/
Usage
Full sync for one product stream, committing the output to git:
./rhcve-osv-sync \
-advisories /path/to/advisories/data \
-osidb-url https://osidb.example.com \
-streams hummingbird-1 \
-products hummingbird \
-output /path/to/osv-output \
-state /path/to/state.db \
-full -git
Later runs without -full use an OSIDB delta query. Run a full
reconciliation daily so that missed changes are corrected.
Binary RPM expansion
OSIDB tracks CVEs against source RPM names, while scanners need binary RPM
names. -expansion selects how a source RPM is expanded:
| Mode |
Source |
osidb |
OSIDB subpackage_purls |
advisory |
Binary RPM names derived from advisory purls |
chain |
OSIDB first, then the advisory mapping |
The default is chain.
Incremental sync
State lives in a SQLite database. Each run goes through these phases: advisory
diff, OSIDB diff, convert, write output and commit state. The
workflow describes each phase.
- Advisory files are content-hashed and only changed files are parsed
- OSIDB uses a delta query, plus a full reconciliation on demand
- Output files are compared with a manifest and only changed files are written
- Output is written before state is committed, so a crash is safe
- If the manifest is lost, the next full run rebuilds it from the output and
withdraws stale records
- If OSIDB fails and the convert phase does not need it, advisory output is
still published and the process exits non-zero so a scheduler can retry or
alert
Unchanged input produces unchanged output, so duplicate scheduling wastes work
but does not corrupt the output.
Configuration
| Flag |
Required |
Description |
-advisories |
Yes |
Path to the advisories repository data/ directory |
-osidb-url |
Yes |
OSIDB base URL |
-streams |
Yes |
Comma-separated ps_update_stream values, also filters advisories by product_stream |
-output |
Yes |
Output directory for the OSV JSON files |
-state |
Yes |
Path to the SQLite state database |
-products |
No |
Filter advisory conversion by CPE product name |
-full |
No |
Force a full OSIDB reconciliation |
-expansion |
No |
Binary RPM expansion mode: osidb, advisory or chain |
-git |
No |
Run git init, add and commit on the output after the sync |
-git-remote |
No |
Git remote to push to after the commit (implies -git) |
-log-json |
No |
Write structured JSON logs |
Development
cd hummingbird-osv-generator
# Vet and test, including the OSV schema validation test
go vet ./...
go test ./...
The hummingbird-osv-generator CI job runs the same commands on merge
requests.
License
This project is licensed under the GNU General Public License v3.0 or later -
see the LICENSE file for details.
1 - Hummingbird OSV Generator Workflow
How a run of the Hummingbird OSV Generator
turns its inputs into OSV records: what it reads, the phases of a sync, and
what it writes.
Overview
flowchart LR
adv[Advisories data directory] --> p1[Advisory diff]
osidb[OSIDB] --> p2[OSIDB diff]
db[(State database)] --> p1
db --> p2
p1 --> p3[Convert]
p2 --> p3
osidb --> p3
p3 --> p4[Write output]
p4 --> out[OSV JSON files]
p4 --> p5[Commit state]
p5 --> db
out --> git[Optional git commit and push]
One run reads the advisories and OSIDB, converts what changed into OSV records,
writes only the records whose content changed, and records what it did in the
state database. Each run is a complete pass: nothing is kept in memory between
runs.
| Input |
Flag |
What the generator reads |
| Advisories directory |
-advisories |
Advisory YAML files: shipped fixes with per-artifact package URLs and CVE fix lists |
| OSIDB |
-osidb-url |
Affects for the given update streams, and the flaws for their CVEs |
| Update streams |
-streams |
Which OSIDB update streams to read. Also filters advisories by product_stream |
| Products |
-products |
Optional filter on the CPE product name of an advisory |
| State database |
-state |
SQLite file from earlier runs, created on the first run |
| Output directory |
-output |
Read only when the record ledger was lost and has to be rebuilt from the published files |
Advisories
- Only security advisories (RHSA) are read. Bug fix and enhancement advisories
are skipped, even if they mention CVEs.
- Each advisory gives the product CPE, and each artifact gives a package URL
and the CVEs it fixes. The package URL carries the binary RPM name, the
version and release, and the RPM epoch.
- From the advisories the generator also builds two lookups: the update stream
to CPE map, and the source RPM to binary RPM mapping.
OSIDB
An affect is ignored when it has no CVE ID, when it is embargoed or not public,
or when it is marked not affected (including a delegated not affected). An
affect whose tracker is resolved as shipped (Done-Errata or Fixed) is also
ignored, because the advisory supplies that fix.
Flaws supply the CVE title and description, severity, CWE and Red Hat
statement. A flaw that is embargoed or not public is ignored, and any text
cached from it is dropped. A rejected flaw marks its records as withdrawn.
State database
| Table |
Content |
input_hashes |
Content hashes of advisory files and of OSIDB affects, used to detect changes |
sync_meta |
Cursors: the last sync, the last full reconciliation, and the last flaw change window |
cve_meta |
Cached CVE text and severity, refreshed from the OSIDB flaw change feed |
output_manifest |
One row per published record: path, content hash, time written and time last modified |
The input hashes and the cursors are a cache that can be rebuilt. The manifest
also holds each record’s last modification time, so it is durable state.
Phases
- Advisory diff. Hash every advisory file and compare with the stored
hashes. The result is the list of added, changed and removed files.
- OSIDB diff. Decide between a delta and a full reconciliation. A delta
asks OSIDB for affects updated since the last sync, minus a clock skew
allowance, and compares their hashes. A full reconciliation fetches every
affect and also detects deletions. By default a full reconciliation runs on
the first run, when
-full is given, and again once a day. If OSIDB fails
here, the run goes on with advisory data when nothing needs OSIDB, and ends
with an error (see Failure handling).
- Convert. Build records from both sources:
- Fixed records come from the advisories.
- Records for every other status come from OSIDB affects. Each affect is
expanded from its source RPM to binary RPM names with the mode chosen by
-expansion.
- CVE text and severity come from the flaw cache, refreshed from the OSIDB
flaw change feed. If the refresh fails, cached values are used.
- When an advisory changed and output already exists, the complete OSIDB data
is fetched, so that a record is never published from partial input.
- Write output. Compare each record’s content hash with the manifest and
write only the records that changed. Remove records that are no longer
produced (see Failure handling for when this is safe).
- Commit state. Store the new input hashes, the manifest and the cursors.
The state is committed after the output is written, so a crash between the
two repeats work and does not lose changes.
If -git or -git-remote is given, the output directory is then committed, and
pushed when a remote is given.
Records
A record covers one source RPM in one update stream for one CVE. It has one
affected entry for each binary RPM, and each entry keeps the fixed version of
its own binary because subpackages can have different epochs.
When one source RPM has entries in different states, the record takes the
status with the highest precedence:
| Precedence |
Status |
Meaning |
| 1 |
fixed |
An advisory shipped the fix |
| 2 |
affected |
OSIDB says the product is affected and no fix has shipped |
| 3 |
under_investigation |
OSIDB has a new affect with no resolution yet |
| 4 |
will_not_fix |
OSIDB resolution is won’t fix or defer |
A shipped fix always wins over an unfixed affect for the same record.
published is the earliest of the dates known for the record: when the flaw
became public, when the affect was created, and when an advisory shipped.
modified changes only when the content of the record changes: a new record
takes its source update time, an unchanged record keeps its value, and a
changed record gets the time of the write.
Status over time
The status is recomputed from the inputs on every run and is not stored. A
typical record moves as follows.
stateDiagram-v2
[*] --> UnderInvestigation: OSIDB affect created, no resolution
[*] --> Affected: OSIDB affect with a resolution
[*] --> WillNotFix: OSIDB resolution won't fix or defer
UnderInvestigation --> Affected: triaged as affected
UnderInvestigation --> WillNotFix: resolution won't fix or defer
WillNotFix --> Affected: resolution changed back
Affected --> Fixed: advisory ships the fix
UnderInvestigation --> Fixed: advisory ships the fix
WillNotFix --> Fixed: advisory ships the fix
Affected --> UnderInvestigation: OSIDB affect reset to new
UnderInvestigation --> Removed: marked not affected, embargoed or not public
Affected --> Removed: marked not affected, embargoed or not public
WillNotFix --> Removed: marked not affected, embargoed or not public
Fixed --> Removed: advisory no longer lists the CVE and OSIDB has no affect
Removed --> [*]
state "under_investigation" as UnderInvestigation
state "affected" as Affected
state "will_not_fix" as WillNotFix
state "fixed" as Fixed
state "file removed from the output" as Removed
A removed record is deleted from the output directory and from the manifest. A
record whose flaw was rejected is different: it stays in the output with a
withdrawn timestamp.
Outputs
| Output |
Location |
Content |
| OSV records |
<output>/<product path>/<ID>.json |
One JSON file per record. The product path comes from the CPE, for example redhat/hummingbird/1 |
| State database |
The -state file |
Updated hashes, manifest, cursors and CVE cache (see State database) |
| Git commit (optional) |
The -output directory |
All changes committed, with a message that summarizes the run |
| Run summary |
standard error |
Counts of records created, updated, withdrawn and unchanged, and files written and removed |
The git step initializes the repository if needed, skips the commit when nothing
changed, and pushes the current branch when -git-remote is given. It runs only
after a sync that ended without an error.
The process exits with 0 on success, 1 when the sync or the git step fails, and
2 for invalid flags.
Failure handling
- OSIDB failure during the diff. If the convert phase does not need OSIDB
data, advisory output is still written and the state is committed, and the
process exits with 1 so a scheduler can retry or alert. The git step does not
run in that case, so the written files are committed by the next successful
run. Records that depend on OSIDB are not removed, because the record set is
incomplete.
- OSIDB unavailable when convert needs it. This happens when an advisory
changed and output already exists, because the complete OSIDB data is needed
to avoid a partial record. The run fails before writing anything.
- No OSIDB changes in a delta run. The advisory-only result is not compared
with existing records, and no record is removed, because OSIDB data is missing
from it.
- Lost state database. The next run behaves like a first run.
- Lost manifest, output kept. A full run rebuilds the manifest from the
published files and rewrites each record once, so stale records can still be
removed.
- A corrupt published file fails the run when the manifest is rebuilt, so it
cannot hide stale state.
- Repeated or duplicate runs. The same input produces the same output, so a
duplicate run wastes work and does not corrupt anything.