# Hummingbird OSV Generator

LLMS index: [llms.txt](https://hummingbird-project.io/llms.txt) | Full content: [llms-full.txt](https://hummingbird-project.io/llms-full.txt)

---

A CLI tool that converts Red Hat vulnerability data into [OSV][osv-schema]
records for Hummingbird RPMs, so that vulnerability scanners can match them
against the packages installed in an image.

For how a run works end to end, see the
[workflow](https://hummingbird-project.io/docs/background/tools/hummingbird-osv-generator/workflow/).

## Features

- **Two sources**: shipped fixes from the advisories repository and all known
  affects, including unfixed CVEs, from OSIDB
- **Binary RPM names**: scanners match installed binary packages, so source
  RPMs are expanded to their binary RPMs
- **Fixed and unfixed states**: records carry `fixed`, `affected`,
  `will_not_fix` or `under_investigation` in `database_specific.status`
- **RPM epochs**: fixed versions are written as `epoch:version-release`
- **Incremental sync**: content hashing means only changed inputs are parsed
  and only changed records are written
- **Embargo safe**: embargoed or non-public OSIDB data is never published

## Output format

The output follows OSV schema 1.9.0 with one record per source RPM per CVE and
one `affected` entry per binary RPM.

| Field                                       | Content                                                             |
| ------------------------------------------- | ------------------------------------------------------------------- |
| `id`                                        | Opaque, stable identifier: `x_HBA` prefix, CVE year and a hash      |
| `upstream`                                  | The CVE the record covers                                           |
| `affected[].package.ecosystem`              | Ecosystem scoped to one product stream by its CPE                   |
| `affected[].package.purl`                   | Versionless package URL                                             |
| `affected[].ranges`                         | `introduced: 0` and, once shipped, `fixed: <epoch:version-release>` |
| `affected[].ecosystem_specific.advisory_id` | Advisory that supplied the fixed version, absent on unfixed entries |
| `database_specific.status`                  | `fixed`, `affected`, `under_investigation` or `will_not_fix`        |
| `modified`                                  | Time of the last content change                                     |

The `x_` in the identifier prefix marks a local database. It is dropped once
`HBA` is registered with the OSV schema project. The ecosystem name is also a
placeholder until it is registered.

## Prerequisites

- **Go**: the version in `hummingbird-osv-generator/go.mod`
- **Advisories checkout**: a clone of the advisories repository, passed as its
  `data/` directory
- **OSIDB access**: the OSIDB base URL, reachable from where the tool runs
- **Git**: only when using `-git` or `-git-remote`

## Installation

```bash
git clone https://gitlab.com/redhat/hummingbird/tools.git
cd tools/hummingbird-osv-generator
go build -o rhcve-osv-sync ./cmd/rhcve-osv-sync/
```

## Usage

Full sync for one product stream, committing the output to git:

```bash
./rhcve-osv-sync \
  -advisories /path/to/advisories/data \
  -osidb-url https://osidb.example.com \
  -streams hummingbird-1 \
  -products hummingbird \
  -output /path/to/osv-output \
  -state /path/to/state.db \
  -full -git
```

Later runs without `-full` use an OSIDB delta query. Run a full
reconciliation daily so that missed changes are corrected.

## Binary RPM expansion

OSIDB tracks CVEs against source RPM names, while scanners need binary RPM
names. `-expansion` selects how a source RPM is expanded:

| Mode       | Source                                       |
| ---------- | -------------------------------------------- |
| `osidb`    | OSIDB `subpackage_purls`                     |
| `advisory` | Binary RPM names derived from advisory purls |
| `chain`    | OSIDB first, then the advisory mapping       |

The default is `chain`.

## Incremental sync

State lives in a SQLite database. Each run goes through these phases: advisory
diff, OSIDB diff, convert, write output and commit state. The
[workflow](https://hummingbird-project.io/docs/background/tools/hummingbird-osv-generator/workflow/) describes each phase.

- Advisory files are content-hashed and only changed files are parsed
- OSIDB uses a delta query, plus a full reconciliation on demand
- Output files are compared with a manifest and only changed files are written
- Output is written before state is committed, so a crash is safe
- If the manifest is lost, the next full run rebuilds it from the output and
  withdraws stale records
- If OSIDB fails and the convert phase does not need it, advisory output is
  still published and the process exits non-zero so a scheduler can retry or
  alert

Unchanged input produces unchanged output, so duplicate scheduling wastes work
but does not corrupt the output.

## Configuration

| Flag          | Required | Description                                                                            |
| ------------- | -------- | -------------------------------------------------------------------------------------- |
| `-advisories` | Yes      | Path to the advisories repository `data/` directory                                    |
| `-osidb-url`  | Yes      | OSIDB base URL                                                                         |
| `-streams`    | Yes      | Comma-separated `ps_update_stream` values, also filters advisories by `product_stream` |
| `-output`     | Yes      | Output directory for the OSV JSON files                                                |
| `-state`      | Yes      | Path to the SQLite state database                                                      |
| `-products`   | No       | Filter advisory conversion by CPE product name                                         |
| `-full`       | No       | Force a full OSIDB reconciliation                                                      |
| `-expansion`  | No       | Binary RPM expansion mode: `osidb`, `advisory` or `chain`                              |
| `-git`        | No       | Run git init, add and commit on the output after the sync                              |
| `-git-remote` | No       | Git remote to push to after the commit (implies `-git`)                                |
| `-log-json`   | No       | Write structured JSON logs                                                             |

## Development

```bash
cd hummingbird-osv-generator

# Vet and test, including the OSV schema validation test
go vet ./...
go test ./...
```

The `hummingbird-osv-generator` CI job runs the same commands on merge
requests.

## License

This project is licensed under the GNU General Public License v3.0 or later -
see the [LICENSE][license] file for details.

[osv-schema]: https://ossf.github.io/osv-schema/
[license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE

---

Section pages:

- [Hummingbird OSV Generator Workflow](https://hummingbird-project.io/docs/background/tools/hummingbird-osv-generator/workflow/)
