Hummingbird Status

SQS worker and database management for ingesting Konflux pipeline events.

Features

  • SQS Worker - Real-time database updates from SNS pipeline events
  • GitLab Event Ingestion - Push events and merge request events with version tracking
  • Database Init - Populate from pg_dump, S3 archive, or local filesystem mirror
  • Prometheus Metrics - Built-in metrics endpoint for monitoring
  • Schema Management - Idempotent database schema creation

Prerequisites

  • Python 3.11+
  • PostgreSQL 16
  • AWS credentials (for SQS worker and S3 initialization)

Installation

cd hummingbird-status
pip install -e .

Usage

Local Development

cd hummingbird-status

# Database management
./dev.sh db-start              # Start PostgreSQL container
./dev.sh db-init               # Initialize from available source
./dev.sh db-shell              # PostgreSQL interactive shell
./dev.sh db-dump               # Create pg_dump file
./dev.sh db-stop               # Stop PostgreSQL
./dev.sh db-reset              # Stop and delete volume

# With data source
SNS_MIRROR=/path/to/mirror ./dev.sh db-init   # From filesystem
S3_BUCKET=bucket-name ./dev.sh db-init        # From S3

# SQS worker (requires credentials)
SQS_QUEUE_URL=https://... ./dev.sh worker

Container Deployment

The container runs the SQS worker by default:

podman build -f Containerfile -t hummingbird-status .
podman run -e DATABASE_URL=... -e SQS_QUEUE_URL=... hummingbird-status

For database initialization:

podman run -e DATABASE_URL=... -e S3_BUCKET=... \
    hummingbird-status python3 -m hummingbird_status.worker.init_db

Configuration

Environment Variables

Variable Default Description
DATABASE_URL - PostgreSQL connection URL
SQS_QUEUE_URL - SQS queue URL (worker)
S3_BUCKET - S3 bucket for init
S3_PREFIX sns/ S3 key prefix
LOCAL_MIRROR_PATH /data/sns-mirror Local S3 mirror path
INIT_DUMP_PATH /data/init/dump.sql pg_dump file path
METRICS_PORT 9090 Prometheus metrics port

Database Initialization Priority

  1. pg_dump file - $INIT_DUMP_PATH if exists
  2. Local S3 mirror - $LOCAL_MIRROR_PATH/sns/*.json.gz if exists
  3. S3 bucket - $S3_BUCKET/$S3_PREFIX with AWS credentials

Database Schema

The database stores Konflux pipeline events and GitLab notifications (pushes and MRs).

flowchart TD
    push[["<b>gitlab_pushes</b><br/>commit sha, changed files"]]
    mr[["<b>gitlab_merge_requests</b><br/>current MR state"]]
    mrv[["<b>gitlab_mr_versions</b><br/>head commit history"]]
    comp[["<b>components</b><br/>git_context → component mapping"]]
    build[["<b>pipelineruns</b> (type=build)<br/>one per affected component"]]
    snap[["<b>snapshots</b><br/>image digest, links via source_plr"]]
    test[["<b>pipelineruns</b> (type=test)<br/>integration tests per snapshot"]]
    rel[["<b>releases</b><br/>publish to registry, links to snapshot"]]
    relplr[["<b>pipelineruns</b> (type=release)<br/>executes release, linked via release_plr"]]

    push -- "+ affected" --> build
    mr -- "sha" --> mrv
    mrv -- "sha joins" --> build
    comp -- "components" --> build
    build -- "success<br/>creates" --> snap
    snap -- "triggers" --> test
    test -- "success<br/>creates" --> rel
    rel -- "managed<br/>by" --> relplr

Tables

Table Primary Key Description
gitlab_pushes sha,repo,ref GitLab push events to main branch
gitlab_merge_requests project,iid Current state of merge requests
gitlab_mr_versions project,iid,sha Head commit history for each MR
components name Konflux Component resources
pipelineruns name Build, test, and release pipelines
snapshots name Image snapshots after successful builds
releases name Published releases to target registry

Merge Request Tracking

The MR tables enable tracking build status across MR versions:

  • gitlab_merge_requests - Stores current MR metadata (title, state, branches, author, latest head SHA). Updated via ON CONFLICT ... WHERE updated_at < to keep the most recent state.

  • gitlab_mr_versions - Records each unique head commit SHA for an MR. When force-pushing the same SHA, created_at is updated to the latest event timestamp via GREATEST(), ensuring correct ordering even after force pushes.

Development

Running Tests

cd hummingbird-status
pip install -e ".[dev]"
pytest

Project Structure

hummingbird-status/
├── Containerfile
├── dev.sh
├── template.yaml         # SAM template for AWS resources
├── hummingbird_status/
│   ├── db.py             # Database schema and utilities
│   ├── ingest.py         # SNS event parsing and ingestion
│   └── worker/
│       ├── sqs.py        # SQS consumer
│       └── init_db.py    # Database initializer
└── tests/

AWS Resources

Deploy the SQS queue using SAM:

cd hummingbird-status
make build     # Build SAM application
make deploy    # First deployment (guided)
make redeploy  # Subsequent deployments

Parameters

Parameter Description Default
ResourcePrefix Prefix for all resource names myapp-prod
SnsTopicArn ARN of the SNS topic to subscribe (required)

Prerequisites: Requires an existing SNS topic. Deploy hummingbird-events-topic first.

See the main README for development workflows.

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.