Hummingbird Monitoring

Shared observability resources for all Hummingbird AWS stacks. Currently provides a central SNS alerts topic for CloudWatch alarm notifications; future additions may include CloudWatch dashboards, composite alarms, and additional notification channels.

This stack handles infrastructure alerting (Lambda errors, DLQ depth, API 5xx). For SLO alerting (CVE exposure burn-rate), see Error Budgets.

Features

  • Central Alerts Topic: Single SNS topic shared by all Hummingbird stacks
  • Email Subscription: Optional email subscription with automatic confirmation workflow
  • Opt-in per Stack: Consumer stacks receive the topic ARN as a parameter; alarms are wired only when the ARN is non-empty

Prerequisites

  • AWS CLI configured with appropriate credentials (IAM permissions for SNS, CloudFormation)
  • Podman or Docker (for containerized SAM build/deploy)

Deployment

Build and deploy using containerized AWS SAM CLI:

cd hummingbird-monitoring
make build     # Build SAM application
make deploy    # First deployment (interactive/guided)
make redeploy  # Subsequent deployments (non-interactive)

Deployment outputs:

  • AlertsTopicArn - SNS topic ARN (for consumer stacks)
  • AlertsTopicName - SNS topic name

Parameters

Parameter Description Default
TopicName SNS topic name myapp-prod-alerts
AlertEmail Email for notifications (empty=skip) ""

When AlertEmail is non-empty, an SNS email subscription is created. The recipient must confirm via a link in the confirmation email before notifications start flowing.

Consumer Stacks

Each consumer stack accepts an AlertTopicArn parameter. When non-empty, CloudWatch alarms publish to the shared topic on state transitions. When empty (the default), alarms still fire but do not send notifications.

Stack Alarms
container-catalog Lambda errors, API 5xx, DLQ depth, metrics iterator age
hummingbird-agent DLQ depth (events, work)
hummingbird-status DLQ depth (events)

Wiring Pattern

Consumer stacks use this pattern to conditionally wire alarms:

Parameters:
  AlertTopicArn:
    Type: String
    Default: ""

Conditions:
  HasAlertTopicArn: !Not [!Equals [!Ref AlertTopicArn, ""]]

Resources:
  MyAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      # ... alarm configuration ...
      AlarmActions: !If [HasAlertTopicArn, [!Ref AlertTopicArn], !Ref "AWS::NoValue"]
      OKActions: !If [HasAlertTopicArn, [!Ref AlertTopicArn], !Ref "AWS::NoValue"]

Adding a New Alarm

  1. Define the AWS::CloudWatch::Alarm resource in the stack’s template.yaml with AlarmActions / OKActions using the pattern above.
  2. The stack must already have the AlertTopicArn parameter and HasAlertTopicArn condition (all three consumer stacks already do).

Adding a New Consumer Stack

  1. Add AlertTopicArn parameter and HasAlertTopicArn condition to the stack’s template.yaml.
  2. Add ALERT_TOPIC_ARN to the stack’s vars.sh in the infrastructure repo.
  3. Add AlertTopicArn to parameter_overrides in the stack’s samconfig.toml.j2.

Development

This is a pure infrastructure project (no application code). See the main README for SAM build/deploy commands.

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.