Kubernetes Event Forwarder

A Kubernetes deployment that watches resource changes (ADDED/MODIFIED/DELETED) across multiple clusters and forwards them to an SNS topic with structured metadata for filtering. Uses kubeconfig contexts as the source of truth for which clusters and namespaces to watch.

The full Kubernetes object JSON is forwarded as the SNS message body, compressed with gzip and base64-encoded.

Features

• Multi-Cluster Support: Watch resources across multiple Kubernetes clusters using kubeconfig contexts
• Dynamic Resource Watching: Configure any namespaced resource type using standard Kubernetes apiVersion and kind
• SNS Integration: Publish events with structured message attributes for precise filtering
• Compression: Events are compressed (gzip+base64) to reduce SNS message size
• Automatic Reconnection: Handles watch connection failures and reconnects automatically

Architecture

Threading Model

The forwarder uses a multi-threaded architecture:

• Watcher threads (one per context + resource type): Each runs an independent LIST+Watch loop. Isolation ensures one slow/failing cluster doesn’t affect others.

• Publisher thread (single, shared): Reads events from a queue and publishes to SNS. Decouples K8s API interaction from SNS latency (~150ms per publish), preventing watch loop stalls that could cause resourceVersion staleness.

Memory optimization: When SPOOL_DIR is set, messages are written to temporary files instead of being held in memory. This reduces memory pressure during event bursts (e.g., LIST operations or many resources created at once) and enables recovery of unsent messages after restarts (e.g., after OOM kills).

LIST + Watch Pattern

The forwarder uses explicit LIST followed by Watch rather than resource_version="0":

1. LIST retrieves all current objects and a snapshot resourceVersion
2. Watch starts from the LIST’s resourceVersion (not object RVs)
3. On graceful watch timeout (300s), restart watch from last known RV (no re-list)
4. On errors, fresh LIST to ensure current state

This is necessary because synthetic ADDED events from resource_version="0" have unsorted, potentially-stale resourceVersions (they reflect when objects were last modified). If the watch disconnects mid-stream, the tracked RV could be arbitrarily old and may already be compacted by etcd, causing a 410 error cascade.

Prerequisites

• Target cluster: ServiceAccount with watch permissions on the resources you want to monitor
• Deployment cluster: Kubernetes cluster to run the forwarder
• AWS credentials: SNS publish permissions (optional - logs events if not configured)

Deployment

The container image is built via Konflux CI/CD and published to quay.io/hummingbird-ci/kubernetes-event-forwarder:latest.

Example Kubernetes manifests are provided in the kubernetes/ directory:

• rbac.yaml - ServiceAccount, Role, RoleBinding for the target cluster
• secret.yaml - Kubeconfig and AWS credentials
• configmap.yaml - Resource watch configuration
• deployment.yaml - Forwarder deployment

Quick Start

1. On the target cluster (the one you want to watch), apply RBAC and create a token:

   kubectl apply -f kubernetes/rbac.yaml
   kubectl create token kubernetes-event-forwarder --duration=8760h
   

2. Update the example manifests with your values:

   • secret.yaml: cluster URL, token, AWS credentials
   • configmap.yaml: resources to watch
   • deployment.yaml: SNS topic ARN, AWS region

3. Apply the manifests to your deployment cluster:

   kubectl apply -f kubernetes/secret.yaml
   kubectl apply -f kubernetes/configmap.yaml
   kubectl apply -f kubernetes/deployment.yaml
   

Prerequisites: Deploy hummingbird-events-topic first to create the SNS topic, then deploy the AWS resources (see below).

AWS Resources

The SAM template (template.yaml) provisions IAM resources for SNS publishing:

• IAM User (${ResourcePrefix}-user) - Service account for the forwarder
• IAM Policy (${ResourcePrefix}-policy) - Grants sns:Publish to the SNS topic

Deploy using containerized AWS SAM CLI:

cd kubernetes-event-forwarder
sam build
sam deploy --guided  # First deployment (interactive)
sam deploy           # Subsequent deployments

After deployment, create access keys for the IAM user and store them securely.

SAM Parameters

Resource naming: IAM resources follow {ResourcePrefix}-{type} pattern (e.g., myapp-prod-user, myapp-prod-policy).

Usage

Configure resources to watch in config.yaml:

resources:
  - apiVersion: v1
    kind: Pod
  - apiVersion: apps/v1
    kind: Deployment
  - apiVersion: v1
    kind: ConfigMap

SNS Subscription Filter Examples:

Pod events in a specific namespace:

{
  "source": ["kubernetes"],
  "kind": ["Pod"],
  "namespace": ["production"]
}

All deployment changes:

{
  "source": ["kubernetes"],
  "kind": ["Deployment"]
}

Deleted resources across all clusters:

{
  "source": ["kubernetes"],
  "event_type": ["DELETED"]
}

Development

See the main README for development workflows.

make kubernetes-event-forwarder/setup  # Install dependencies
make check                             # Lint code (ruff)
make fmt                               # Format code
make test                              # Run unit tests
make coverage                          # Run tests with coverage

Configuration

The deployment is configured via environment variables:

Event Metadata

The forwarder extracts metadata from Kubernetes events and adds them as SNS message attributes:

Security & Limitations

Security:

• AWS credentials stored in Kubernetes Secret
• Kubeconfig credentials stored in Kubernetes Secret
• SNS topic follows least privilege principle (publish-only)
• Sentry integration for error tracking

Limitations:

• Only namespaced resources supported
• One thread per context + resource type combination
• SNS message size limit: 256 KB (after compression)

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.