# Hummingbird OSV Generator Workflow

LLMS index: [llms.txt](https://hummingbird-project.io/llms.txt) | Full content: [llms-full.txt](https://hummingbird-project.io/llms-full.txt)

---

How a run of the [Hummingbird OSV Generator](https://hummingbird-project.io/l/osv-generator)
turns its inputs into OSV records: what it reads, the phases of a sync, and
what it writes.

## Overview

```mermaid
flowchart LR
    adv[Advisories data directory] --> p1[Advisory diff]
    osidb[OSIDB] --> p2[OSIDB diff]
    db[(State database)] --> p1
    db --> p2
    p1 --> p3[Convert]
    p2 --> p3
    osidb --> p3
    p3 --> p4[Write output]
    p4 --> out[OSV JSON files]
    p4 --> p5[Commit state]
    p5 --> db
    out --> git[Optional git commit and push]
```

One run reads the advisories and OSIDB, converts what changed into OSV records,
writes only the records whose content changed, and records what it did in the
state database. Each run is a complete pass: nothing is kept in memory between
runs.

## Inputs

| Input                | Flag          | What the generator reads                                                                 |
| -------------------- | ------------- | ---------------------------------------------------------------------------------------- |
| Advisories directory | `-advisories` | Advisory YAML files: shipped fixes with per-artifact package URLs and CVE fix lists      |
| OSIDB                | `-osidb-url`  | Affects for the given update streams, and the flaws for their CVEs                       |
| Update streams       | `-streams`    | Which OSIDB update streams to read. Also filters advisories by `product_stream`          |
| Products             | `-products`   | Optional filter on the CPE product name of an advisory                                   |
| State database       | `-state`      | SQLite file from earlier runs, created on the first run                                  |
| Output directory     | `-output`     | Read only when the record ledger was lost and has to be rebuilt from the published files |

### Advisories

- Only security advisories (RHSA) are read. Bug fix and enhancement advisories
  are skipped, even if they mention CVEs.
- Each advisory gives the product CPE, and each artifact gives a package URL
  and the CVEs it fixes. The package URL carries the binary RPM name, the
  version and release, and the RPM epoch.
- From the advisories the generator also builds two lookups: the update stream
  to CPE map, and the source RPM to binary RPM mapping.

### OSIDB

An affect is ignored when it has no CVE ID, when it is embargoed or not public,
or when it is marked not affected (including a delegated not affected). An
affect whose tracker is resolved as shipped (Done-Errata or Fixed) is also
ignored, because the advisory supplies that fix.

Flaws supply the CVE title and description, severity, CWE and Red Hat
statement. A flaw that is embargoed or not public is ignored, and any text
cached from it is dropped. A rejected flaw marks its records as withdrawn.

### State database

| Table             | Content                                                                               |
| ----------------- | ------------------------------------------------------------------------------------- |
| `input_hashes`    | Content hashes of advisory files and of OSIDB affects, used to detect changes         |
| `sync_meta`       | Cursors: the last sync, the last full reconciliation, and the last flaw change window |
| `cve_meta`        | Cached CVE text and severity, refreshed from the OSIDB flaw change feed               |
| `output_manifest` | One row per published record: path, content hash, time written and time last modified |

The input hashes and the cursors are a cache that can be rebuilt. The manifest
also holds each record's last modification time, so it is durable state.

## Phases

1. **Advisory diff.** Hash every advisory file and compare with the stored
   hashes. The result is the list of added, changed and removed files.
2. **OSIDB diff.** Decide between a delta and a full reconciliation. A delta
   asks OSIDB for affects updated since the last sync, minus a clock skew
   allowance, and compares their hashes. A full reconciliation fetches every
   affect and also detects deletions. By default a full reconciliation runs on
   the first run, when `-full` is given, and again once a day. If OSIDB fails
   here, the run goes on with advisory data when nothing needs OSIDB, and ends
   with an error (see [Failure handling](#failure-handling)).
3. **Convert.** Build records from both sources:
   - Fixed records come from the advisories.
   - Records for every other status come from OSIDB affects. Each affect is
     expanded from its source RPM to binary RPM names with the mode chosen by
     `-expansion`.
   - CVE text and severity come from the flaw cache, refreshed from the OSIDB
     flaw change feed. If the refresh fails, cached values are used.
   - When an advisory changed and output already exists, the complete OSIDB data
     is fetched, so that a record is never published from partial input.
4. **Write output.** Compare each record's content hash with the manifest and
   write only the records that changed. Remove records that are no longer
   produced (see [Failure handling](#failure-handling) for when this is safe).
5. **Commit state.** Store the new input hashes, the manifest and the cursors.
   The state is committed after the output is written, so a crash between the
   two repeats work and does not lose changes.

If `-git` or `-git-remote` is given, the output directory is then committed, and
pushed when a remote is given.

## Records

A record covers one source RPM in one update stream for one CVE. It has one
`affected` entry for each binary RPM, and each entry keeps the fixed version of
its own binary because subpackages can have different epochs.

When one source RPM has entries in different states, the record takes the
status with the highest precedence:

| Precedence | Status                | Meaning                                                   |
| ---------- | --------------------- | --------------------------------------------------------- |
| 1          | `fixed`               | An advisory shipped the fix                               |
| 2          | `affected`            | OSIDB says the product is affected and no fix has shipped |
| 3          | `under_investigation` | OSIDB has a new affect with no resolution yet             |
| 4          | `will_not_fix`        | OSIDB resolution is won't fix or defer                    |

A shipped fix always wins over an unfixed affect for the same record.

`published` is the earliest of the dates known for the record: when the flaw
became public, when the affect was created, and when an advisory shipped.
`modified` changes only when the content of the record changes: a new record
takes its source update time, an unchanged record keeps its value, and a
changed record gets the time of the write.

### Status over time

The status is recomputed from the inputs on every run and is not stored. A
typical record moves as follows.

```mermaid
stateDiagram-v2
    [*] --> UnderInvestigation: OSIDB affect created, no resolution
    [*] --> Affected: OSIDB affect with a resolution
    [*] --> WillNotFix: OSIDB resolution won't fix or defer

    UnderInvestigation --> Affected: triaged as affected
    UnderInvestigation --> WillNotFix: resolution won't fix or defer
    WillNotFix --> Affected: resolution changed back
    Affected --> Fixed: advisory ships the fix
    UnderInvestigation --> Fixed: advisory ships the fix
    WillNotFix --> Fixed: advisory ships the fix
    Affected --> UnderInvestigation: OSIDB affect reset to new

    UnderInvestigation --> Removed: marked not affected, embargoed or not public
    Affected --> Removed: marked not affected, embargoed or not public
    WillNotFix --> Removed: marked not affected, embargoed or not public
    Fixed --> Removed: advisory no longer lists the CVE and OSIDB has no affect
    Removed --> [*]

    state "under_investigation" as UnderInvestigation
    state "affected" as Affected
    state "will_not_fix" as WillNotFix
    state "fixed" as Fixed
    state "file removed from the output" as Removed
```

A removed record is deleted from the output directory and from the manifest. A
record whose flaw was rejected is different: it stays in the output with a
`withdrawn` timestamp.

## Outputs

| Output                | Location                            | Content                                                                                           |
| --------------------- | ----------------------------------- | ------------------------------------------------------------------------------------------------- |
| OSV records           | `<output>/<product path>/<ID>.json` | One JSON file per record. The product path comes from the CPE, for example `redhat/hummingbird/1` |
| State database        | The `-state` file                   | Updated hashes, manifest, cursors and CVE cache (see [State database](#state-database))           |
| Git commit (optional) | The `-output` directory             | All changes committed, with a message that summarizes the run                                     |
| Run summary           | standard error                      | Counts of records created, updated, withdrawn and unchanged, and files written and removed        |

The git step initializes the repository if needed, skips the commit when nothing
changed, and pushes the current branch when `-git-remote` is given. It runs only
after a sync that ended without an error.

The process exits with 0 on success, 1 when the sync or the git step fails, and
2 for invalid flags.

## Failure handling

- **OSIDB failure during the diff.** If the convert phase does not need OSIDB
  data, advisory output is still written and the state is committed, and the
  process exits with 1 so a scheduler can retry or alert. The git step does not
  run in that case, so the written files are committed by the next successful
  run. Records that depend on OSIDB are not removed, because the record set is
  incomplete.
- **OSIDB unavailable when convert needs it.** This happens when an advisory
  changed and output already exists, because the complete OSIDB data is needed
  to avoid a partial record. The run fails before writing anything.
- **No OSIDB changes in a delta run.** The advisory-only result is not compared
  with existing records, and no record is removed, because OSIDB data is missing
  from it.
- **Lost state database.** The next run behaves like a first run.
- **Lost manifest, output kept.** A full run rebuilds the manifest from the
  published files and rewrites each record once, so stale records can still be
  removed.
- **A corrupt published file** fails the run when the manifest is rebuilt, so it
  cannot hide stale state.
- **Repeated or duplicate runs.** The same input produces the same output, so a
  duplicate run wastes work and does not corrupt anything.
