Message Bus Architecture

The Hummingbird message bus is an event-driven architecture built on AWS SNS. Events from multiple sources flow through a central topic, enabling subscribers to filter and process only the events they need.

Architecture

flowchart TD
    subgraph Publishers
        GL[GitLab Webhooks]
        K8S[Kubernetes Clusters]
    end

    subgraph MessageBus [Message Bus]
        SNS[(SNS Topic)]
    end

    subgraph Archiver
        ARCH[SNS S3 Archiver]
        S3[(S3 Bucket)]
    end

    subgraph StatusDB [Status Database]
        SQS1[SQS Queue]
        WORKER[hummingbird-status]
        PG[(PostgreSQL)]
    end

    subgraph ConsumerQueue [Consumer Queue]
        SQS2[SQS Queue]
    end

    subgraph Consumers
        CONSUMER[Event Consumer]
    end

    GL -->|gitlab-event-forwarder| SNS
    K8S -->|kubernetes-event-forwarder| SNS
    SNS --> ARCH
    ARCH --> S3
    SNS --> SQS1
    SQS1 --> WORKER
    WORKER --> PG
    SNS --> SQS2
    SQS2 -->|new events| CONSUMER
    S3 -.->|historic events| CONSUMER
    PG -.->|aggregated status| CONSUMER

Components

Message Format

All messages include standard attributes for filtering:

Additional attributes vary by source - see individual publisher docs for details.

GitLab Events

Published by gitlab-event-forwarder:

Kubernetes Events

Published by kubernetes-event-forwarder:

Subscription Filtering

SNS filter policies enable subscribers to receive only relevant events:

{
  "source": ["gitlab"],
  "event_type": ["push", "merge_request"]
}

{
  "source": ["kubernetes"],
  "kind": ["Deployment"],
  "event_type": ["MODIFIED"]
}

See hummingbird-events-topic for subscription setup instructions.

Event Flow Example

1. Developer pushes to GitLab repository
2. GitLab sends webhook to gitlab-event-forwarder Lambda
3. Lambda validates token, extracts metadata, publishes to SNS
4. SNS delivers to all matching subscribers:
   • sns-s3-archiver stores event in S3
   • Event consumers process new events in real-time

Consuming Events

Consumers can receive events from multiple sources:

• New events: Subscribe to SNS topic for real-time processing
• Historic events: Download from S3 archive for replay or catch-up
• Structured queries: Query PostgreSQL via hummingbird-status for pipeline status, component state, and cross-referenced data

The S3 archive enables consumers to bootstrap state, then switch to live SNS events. The PostgreSQL database provides a queryable view of pipeline status with relationships between pushes, builds, snapshots, and releases.

Replaying Events

The sns-s3-archiver stores complete SNS records with decoded payloads, enabling easy replay:

# Download archived events
aws s3 sync s3://bucket-name/2025/12/13/ ./local-events/

# Browse events
zcat ./local-events/2025/12/13/12/34/*.json.gz | jq

import gzip
import json
from pathlib import Path

# Replay archived events to handler
for path in sorted(Path("./local-events").rglob("*.json.gz")):
    with gzip.open(path, "rt") as f:
        sns_record = json.load(f)
    event = {"Records": [{"Sns": sns_record}]}
    my_handler.lambda_handler(event, None)

See sns-s3-archiver documentation for details on storage format and the _decode_message pattern for handlers.