Hummingbird Status

SQS worker and database management for ingesting Konflux pipeline events.

Features

• SQS Worker - Real-time database updates from SNS pipeline events
• GitLab Event Ingestion - Push events and merge request events with version tracking
• Database Init - Populate from pg_dump, S3 archive, or local filesystem mirror
• Prometheus Metrics - Built-in metrics endpoint for monitoring
• MR Reconciliation - Fix stale MR state by checking GitLab API
• Schema Management - Idempotent database schema creation

Prerequisites

• Python 3.11+
• PostgreSQL 16
• AWS credentials (for SQS worker and S3 initialization)

Installation

cd hummingbird-status
pip install -e .

Usage

Local Development

cd hummingbird-status

# Database management
./dev.sh db-start              # Start PostgreSQL container
./dev.sh db-init               # Initialize from available source
./dev.sh db-shell              # PostgreSQL interactive shell
./dev.sh db-dump               # Create pg_dump file
./dev.sh db-stop               # Stop PostgreSQL
./dev.sh db-reset              # Stop and delete volume

# With data source
SNS_MIRROR=/path/to/mirror ./dev.sh db-init   # From filesystem
S3_BUCKET=bucket-name ./dev.sh db-init        # From S3

# SQS worker (requires credentials)
SQS_QUEUE_URL=https://... ./dev.sh worker

Container Deployment

The container runs the SQS worker by default:

podman build -f Containerfile -t hummingbird-status .
podman run -e DATABASE_URL=... -e SQS_QUEUE_URL=... hummingbird-status

For database initialization:

podman run -e DATABASE_URL=... -e S3_BUCKET=... \
    hummingbird-status python3 -m hummingbird_status.worker.init_db

MR Reconciliation

MR state is updated via GitLab webhooks. If a webhook is lost, an MR can appear as “opened” in the dashboard when it is actually merged or closed. The reconcile command checks all open MRs against the GitLab API and corrects stale entries:

# Using environment variables
GITLAB_TOKEN=glpat-... python -m hummingbird_status.reconcile

# Using CLI arguments
python -m hummingbird_status.reconcile \
    --database-url postgresql://... \
    --gitlab-token glpat-...

In a container:

podman run -e DATABASE_URL=... -e GITLAB_TOKEN=... \
    hummingbird-status python3 -m hummingbird_status.reconcile

Configuration

Environment Variables

Database Initialization Priority

1. pg_dump file - $INIT_DUMP_PATH if exists
2. Local S3 mirror - $LOCAL_MIRROR_PATH/sns/*.json.gz if exists
3. S3 bucket - $S3_BUCKET/$S3_PREFIX with AWS credentials

Database Schema

The database stores Konflux pipeline events and GitLab notifications (pushes and MRs).

flowchart TD
    push[["<b>gitlab_pushes</b><br/>commit sha, changed files"]]
    mr[["<b>gitlab_merge_requests</b><br/>current MR state"]]
    mrv[["<b>gitlab_mr_versions</b><br/>head commit history"]]
    comp[["<b>components</b><br/>git_context → component mapping"]]
    build[["<b>pipelineruns</b> (type=build)<br/>one per affected component"]]
    snap[["<b>snapshots</b><br/>image digest, links via source_plr"]]
    test[["<b>pipelineruns</b> (type=test)<br/>integration tests per snapshot"]]
    rel[["<b>releases</b><br/>publish to registry, links to snapshot"]]
    relplr[["<b>pipelineruns</b> (type=release)<br/>executes release, linked via release_plr"]]

    push -- "+ affected" --> build
    mr -- "sha" --> mrv
    mrv -- "sha joins" --> build
    comp -- "components" --> build
    build -- "success<br/>creates" --> snap
    snap -- "triggers" --> test
    test -- "success<br/>creates" --> rel
    rel -- "managed<br/>by" --> relplr

Tables

Merge Request Tracking

The MR tables enable tracking build status across MR versions:

• gitlab_merge_requests - Stores current MR metadata (title, state, branches, author, latest head SHA). Updated via ON CONFLICT ... WHERE updated_at < to keep the most recent state.

• gitlab_mr_versions - Records each unique head commit SHA for an MR. When force-pushing the same SHA, created_at is updated to the latest event timestamp via GREATEST(), ensuring correct ordering even after force pushes.

Development

Running Tests

cd hummingbird-status
pip install -e ".[dev]"
pytest

Project Structure

hummingbird-status/
├── Containerfile
├── dev.sh
├── template.yaml         # SAM template for AWS resources
├── hummingbird_status/
│   ├── db.py             # Database schema and utilities
│   ├── ingest.py         # SNS event parsing and ingestion
│   ├── reconcile.py      # MR state reconciliation with GitLab
│   └── worker/
│       ├── sqs.py        # SQS consumer
│       └── init_db.py    # Database initializer
└── tests/

AWS Resources

Deploy the SQS queue using SAM:

cd hummingbird-status
make build     # Build SAM application
make deploy    # First deployment (guided)
make redeploy  # Subsequent deployments

Parameters

Prerequisites: Requires an existing SNS topic. Deploy hummingbird-events-topic first.

See the main README for development workflows.

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.