# Hummingbird Status

LLMS index: [llms.txt](https://hummingbird-project.io/llms.txt) | Full content: [llms-full.txt](https://hummingbird-project.io/llms-full.txt)

---

SQS worker and database management for ingesting Konflux pipeline events.

## Features

- **SQS Worker** - Real-time database updates from SNS pipeline events
- **GitLab Event Ingestion** - Push events and merge request events with version tracking
- **Database Init** - Populate from pg_dump, S3 archive, or local filesystem mirror
- **Prometheus Metrics** - Built-in metrics endpoint for monitoring
- **MR Reconciliation** - Fix stale MR state by checking GitLab API
- **Schema Management** - Idempotent database schema creation

Pipeline timing and lineage are ingested by the separate [Pipeline Timing](https://hummingbird-project.io/docs/background/tools/pipeline-timing/)
service into a dedicated schema in the same events database. The Status worker
ignores pipeline, job, and TaskRun events it does not store in `public`.

## Prerequisites

- Python 3.11+
- PostgreSQL 16
- AWS credentials (for SQS worker and S3 initialization)

## Installation

```bash
cd hummingbird-status
pip install -e .
```

## Usage

### Local Development

```bash
cd hummingbird-status

# Database management
./dev.sh db-start              # Start PostgreSQL container
./dev.sh db-init               # Initialize from available source
./dev.sh db-shell              # PostgreSQL interactive shell
./dev.sh db-dump               # Create pg_dump file
./dev.sh db-stop               # Stop PostgreSQL
./dev.sh db-reset              # Stop and delete volume

# With data source
SNS_MIRROR=/path/to/mirror ./dev.sh db-init   # From filesystem
S3_BUCKET=bucket-name ./dev.sh db-init        # From S3

# SQS worker (requires credentials)
SQS_QUEUE_URL=https://... ./dev.sh worker
```

### Container Deployment

The container runs the SQS worker by default:

```bash
podman build -f Containerfile -t hummingbird-status .
podman run -e DATABASE_URL=... -e SQS_QUEUE_URL=... hummingbird-status
```

For database initialization:

```bash
podman run -e DATABASE_URL=... -e S3_BUCKET=... \
    hummingbird-status python3 -m hummingbird_status.worker.init_db
```

### MR Reconciliation

MR state is updated via GitLab webhooks. If a webhook is lost, an MR can appear
as "opened" in the dashboard when it is actually merged or closed. The reconcile
command checks all open MRs against the GitLab API and corrects stale entries:

```bash
# Using environment variables
GITLAB_TOKEN=glpat-... python -m hummingbird_status.reconcile

# Using CLI arguments
python -m hummingbird_status.reconcile \
    --database-url postgresql://... \
    --gitlab-token glpat-...
```

In a container:

```bash
podman run -e DATABASE_URL=... -e GITLAB_TOKEN=... \
    hummingbird-status python3 -m hummingbird_status.reconcile
```

## Configuration

### Environment Variables

| Variable            | Default               | Description                     |
| ------------------- | --------------------- | ------------------------------- |
| `DATABASE_URL`      | -                     | PostgreSQL connection URL       |
| `SQS_QUEUE_URL`     | -                     | SQS queue URL (worker)          |
| `S3_BUCKET`         | -                     | S3 bucket for init              |
| `S3_PREFIX`         | `sns/`                | S3 key prefix                   |
| `LOCAL_MIRROR_PATH` | `/data/sns-mirror`    | Local S3 mirror path            |
| `INIT_DUMP_PATH`    | `/data/init/dump.sql` | pg_dump file path               |
| `GITLAB_URL`        | `https://gitlab.com`  | GitLab instance URL (reconcile) |
| `GITLAB_TOKEN`      | -                     | GitLab API token (reconcile)    |
| `METRICS_PORT`      | `9090`                | Prometheus metrics port         |

### Database Initialization Priority

1. **pg_dump file** - `$INIT_DUMP_PATH` if exists
2. **Local S3 mirror** - `$LOCAL_MIRROR_PATH/sns/*.json.gz` if exists
3. **S3 bucket** - `$S3_BUCKET/$S3_PREFIX` with AWS credentials

## Database Schema

The database stores Konflux pipeline events and GitLab notifications (pushes and MRs).

```mermaid
flowchart TD
    push[["<b>gitlab_pushes</b><br/>commit sha, changed files"]]
    mr[["<b>gitlab_merge_requests</b><br/>current MR state"]]
    mrv[["<b>gitlab_mr_versions</b><br/>head commit history"]]
    comp[["<b>components</b><br/>git_context → component mapping"]]
    build[["<b>pipelineruns</b> (type=build)<br/>one per affected component"]]
    snap[["<b>snapshots</b><br/>image digest, links via source_plr"]]
    test[["<b>pipelineruns</b> (type=test)<br/>integration tests per snapshot"]]
    rel[["<b>releases</b><br/>publish to registry, links to snapshot"]]
    relplr[["<b>pipelineruns</b> (type=release)<br/>executes release, linked via release_plr"]]

    push -- "+ affected" --> build
    mr -- "sha" --> mrv
    mrv -- "sha joins" --> build
    comp -- "components" --> build
    build -- "success<br/>creates" --> snap
    snap -- "triggers" --> test
    test -- "success<br/>creates" --> rel
    rel -- "managed<br/>by" --> relplr
```

### Tables

| Table                   | Primary Key     | Description                             |
| ----------------------- | --------------- | --------------------------------------- |
| `gitlab_pushes`         | sha,repo,ref    | GitLab push events to main branch       |
| `gitlab_merge_requests` | project,iid     | Current state of merge requests         |
| `gitlab_mr_versions`    | project,iid,sha | Head commit history for each MR         |
| `components`            | name            | Konflux Component resources             |
| `pipelineruns`          | name            | Build, test, and release pipelines      |
| `snapshots`             | name            | Image snapshots after successful builds |
| `releases`              | name            | Published releases to target registry   |

### Merge Request Tracking

The MR tables enable tracking build status across MR versions:

- **`gitlab_merge_requests`** - Stores current MR metadata (title, state, branches,
  author, latest head SHA). Updated via `ON CONFLICT ... WHERE updated_at <` to keep
  the most recent state. `author_name` captures the webhook `user.name` alongside
  `author` (the username) so the dashboard can show a bot/access-token account's
  real GitLab name (e.g. "chore-mr") instead of its generated username -- like
  `author`, it's set once and preserved via `COALESCE` on conflict, so it isn't
  backfilled for MRs that predate this column.

- **`gitlab_mr_versions`** - Records each unique head commit SHA for an MR. When
  force-pushing the same SHA, `created_at` is updated to the latest event timestamp
  via `GREATEST()`, ensuring correct ordering even after force pushes.

## Development

### Running Tests

```bash
cd hummingbird-status
pip install -e ".[dev]"
pytest
```

### Project Structure

```text
hummingbird-status/
├── Containerfile
├── dev.sh
├── template.yaml         # SAM template for AWS resources
├── hummingbird_status/
│   ├── db.py             # Database schema and utilities
│   ├── ingest.py         # SNS event parsing and ingestion
│   ├── reconcile.py      # MR state reconciliation with GitLab
│   └── worker/
│       ├── sqs.py        # SQS consumer
│       └── init_db.py    # Database initializer
└── tests/
```

## AWS Resources

Deploy the SQS queue using SAM:

```bash
cd hummingbird-status
make build     # Build SAM application
make deploy    # First deployment (guided)
make redeploy  # Subsequent deployments
```

### Parameters

| Parameter        | Description                       | Default      |
| ---------------- | --------------------------------- | ------------ |
| `ResourcePrefix` | Prefix for all resource names     | `myapp-prod` |
| `SnsTopicArn`    | ARN of the SNS topic to subscribe | (required)   |

**Prerequisites:** Requires an existing SNS topic. Deploy
[hummingbird-events-topic][het-docs] first.

See the main [README][readme] for development workflows.

[het-docs]: https://hummingbird-project.io/docs/background/tools/hummingbird-events-topic/

## License

This project is licensed under the GNU General Public License v3.0 or later -
see the [LICENSE][license] file for details.

[readme]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/README.md
[license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE