Hummingbird OSV Generator Workflow
How a run of the Hummingbird OSV Generator turns its inputs into OSV records: what it reads, the phases of a sync, and what it writes.
Overview
flowchart LR
adv[Advisories data directory] --> p1[Advisory diff]
osidb[OSIDB] --> p2[OSIDB diff]
db[(State database)] --> p1
db --> p2
p1 --> p3[Convert]
p2 --> p3
osidb --> p3
p3 --> p4[Write output]
p4 --> out[OSV JSON files]
p4 --> p5[Commit state]
p5 --> db
out --> git[Optional git commit and push]
One run reads the advisories and OSIDB, converts what changed into OSV records, writes only the records whose content changed, and records what it did in the state database. Each run is a complete pass: nothing is kept in memory between runs.
Inputs
| Input | Flag | What the generator reads |
|---|---|---|
| Advisories directory | -advisories |
Advisory YAML files: shipped fixes with per-artifact package URLs and CVE fix lists |
| OSIDB | -osidb-url |
Affects for the given update streams, and the flaws for their CVEs |
| Update streams | -streams |
Which OSIDB update streams to read. Also filters advisories by product_stream |
| Products | -products |
Optional filter on the CPE product name of an advisory |
| State database | -state |
SQLite file from earlier runs, created on the first run |
| Output directory | -output |
Read only when the record ledger was lost and has to be rebuilt from the published files |
Advisories
- Only security advisories (RHSA) are read. Bug fix and enhancement advisories are skipped, even if they mention CVEs.
- Each advisory gives the product CPE, and each artifact gives a package URL and the CVEs it fixes. The package URL carries the binary RPM name, the version and release, and the RPM epoch.
- From the advisories the generator also builds two lookups: the update stream to CPE map, and the source RPM to binary RPM mapping.
OSIDB
An affect is ignored when it has no CVE ID, when it is embargoed or not public, or when it is marked not affected (including a delegated not affected). An affect whose tracker is resolved as shipped (Done-Errata or Fixed) is also ignored, because the advisory supplies that fix.
Flaws supply the CVE title and description, severity, CWE and Red Hat statement. A flaw that is embargoed or not public is ignored, and any text cached from it is dropped. A rejected flaw marks its records as withdrawn.
State database
| Table | Content |
|---|---|
input_hashes |
Content hashes of advisory files and of OSIDB affects, used to detect changes |
sync_meta |
Cursors: the last sync, the last full reconciliation, and the last flaw change window |
cve_meta |
Cached CVE text and severity, refreshed from the OSIDB flaw change feed |
output_manifest |
One row per published record: path, content hash, time written and time last modified |
The input hashes and the cursors are a cache that can be rebuilt. The manifest also holds each record’s last modification time, so it is durable state.
Phases
- Advisory diff. Hash every advisory file and compare with the stored hashes. The result is the list of added, changed and removed files.
- OSIDB diff. Decide between a delta and a full reconciliation. A delta
asks OSIDB for affects updated since the last sync, minus a clock skew
allowance, and compares their hashes. A full reconciliation fetches every
affect and also detects deletions. By default a full reconciliation runs on
the first run, when
-fullis given, and again once a day. If OSIDB fails here, the run goes on with advisory data when nothing needs OSIDB, and ends with an error (see Failure handling). - Convert. Build records from both sources:
- Fixed records come from the advisories.
- Records for every other status come from OSIDB affects. Each affect is
expanded from its source RPM to binary RPM names with the mode chosen by
-expansion. - CVE text and severity come from the flaw cache, refreshed from the OSIDB flaw change feed. If the refresh fails, cached values are used.
- When an advisory changed and output already exists, the complete OSIDB data is fetched, so that a record is never published from partial input.
- Write output. Compare each record’s content hash with the manifest and write only the records that changed. Remove records that are no longer produced (see Failure handling for when this is safe).
- Commit state. Store the new input hashes, the manifest and the cursors. The state is committed after the output is written, so a crash between the two repeats work and does not lose changes.
If -git or -git-remote is given, the output directory is then committed, and
pushed when a remote is given.
Records
A record covers one source RPM in one update stream for one CVE. It has one
affected entry for each binary RPM, and each entry keeps the fixed version of
its own binary because subpackages can have different epochs.
When one source RPM has entries in different states, the record takes the status with the highest precedence:
| Precedence | Status | Meaning |
|---|---|---|
| 1 | fixed |
An advisory shipped the fix |
| 2 | affected |
OSIDB says the product is affected and no fix has shipped |
| 3 | under_investigation |
OSIDB has a new affect with no resolution yet |
| 4 | will_not_fix |
OSIDB resolution is won’t fix or defer |
A shipped fix always wins over an unfixed affect for the same record.
published is the earliest of the dates known for the record: when the flaw
became public, when the affect was created, and when an advisory shipped.
modified changes only when the content of the record changes: a new record
takes its source update time, an unchanged record keeps its value, and a
changed record gets the time of the write.
Status over time
The status is recomputed from the inputs on every run and is not stored. A typical record moves as follows.
stateDiagram-v2
[*] --> UnderInvestigation: OSIDB affect created, no resolution
[*] --> Affected: OSIDB affect with a resolution
[*] --> WillNotFix: OSIDB resolution won't fix or defer
UnderInvestigation --> Affected: triaged as affected
UnderInvestigation --> WillNotFix: resolution won't fix or defer
WillNotFix --> Affected: resolution changed back
Affected --> Fixed: advisory ships the fix
UnderInvestigation --> Fixed: advisory ships the fix
WillNotFix --> Fixed: advisory ships the fix
Affected --> UnderInvestigation: OSIDB affect reset to new
UnderInvestigation --> Removed: marked not affected, embargoed or not public
Affected --> Removed: marked not affected, embargoed or not public
WillNotFix --> Removed: marked not affected, embargoed or not public
Fixed --> Removed: advisory no longer lists the CVE and OSIDB has no affect
Removed --> [*]
state "under_investigation" as UnderInvestigation
state "affected" as Affected
state "will_not_fix" as WillNotFix
state "fixed" as Fixed
state "file removed from the output" as Removed
A removed record is deleted from the output directory and from the manifest. A
record whose flaw was rejected is different: it stays in the output with a
withdrawn timestamp.
Outputs
| Output | Location | Content |
|---|---|---|
| OSV records | <output>/<product path>/<ID>.json |
One JSON file per record. The product path comes from the CPE, for example redhat/hummingbird/1 |
| State database | The -state file |
Updated hashes, manifest, cursors and CVE cache (see State database) |
| Git commit (optional) | The -output directory |
All changes committed, with a message that summarizes the run |
| Run summary | standard error | Counts of records created, updated, withdrawn and unchanged, and files written and removed |
The git step initializes the repository if needed, skips the commit when nothing
changed, and pushes the current branch when -git-remote is given. It runs only
after a sync that ended without an error.
The process exits with 0 on success, 1 when the sync or the git step fails, and 2 for invalid flags.
Failure handling
- OSIDB failure during the diff. If the convert phase does not need OSIDB data, advisory output is still written and the state is committed, and the process exits with 1 so a scheduler can retry or alert. The git step does not run in that case, so the written files are committed by the next successful run. Records that depend on OSIDB are not removed, because the record set is incomplete.
- OSIDB unavailable when convert needs it. This happens when an advisory changed and output already exists, because the complete OSIDB data is needed to avoid a partial record. The run fails before writing anything.
- No OSIDB changes in a delta run. The advisory-only result is not compared with existing records, and no record is removed, because OSIDB data is missing from it.
- Lost state database. The next run behaves like a first run.
- Lost manifest, output kept. A full run rebuilds the manifest from the published files and rewrites each record once, so stale records can still be removed.
- A corrupt published file fails the run when the manifest is rebuilt, so it cannot hide stale state.
- Repeated or duplicate runs. The same input produces the same output, so a duplicate run wastes work and does not corrupt anything.