Kubernetes Event Forwarder

A Kubernetes deployment that watches resource changes (ADDED/MODIFIED/DELETED) across multiple clusters and forwards them to an SNS topic with structured metadata for filtering. Uses kubeconfig contexts as the source of truth for which clusters and namespaces to watch.

The full Kubernetes object JSON is forwarded as the SNS message body, compressed with gzip and base64-encoded.

For TaskRun events only, Kubernetes metadata.managedFields is removed before compression. All other object data, including identity, owner references, labels, annotations, task specification, conditions, results, pod name, and timing is retained.

Features

• Multi-Cluster Support: Watch resources across multiple Kubernetes clusters using kubeconfig contexts
• Dynamic Resource Watching: Configure any namespaced resource type using standard Kubernetes apiVersion and kind
• SNS Integration: Publish events with structured message attributes for precise filtering
• Compression: Events are compressed (gzip+base64) to reduce SNS message size
• Selectors: Resource-specific Kubernetes label and field selectors limit LIST and Watch requests
• Automatic Reconnection: Handles watch connection failures and reconnects automatically

Architecture

Threading Model

The forwarder uses a multi-threaded architecture:

• Watcher threads (one per context + resource type): Each runs an independent LIST+Watch loop. Isolation ensures one slow/failing cluster doesn’t affect others.

• Publisher thread (single, shared): Reads events from a queue and publishes to SNS. Decouples K8s API interaction from SNS latency (~150ms per publish), preventing watch loop stalls that could cause resourceVersion staleness.

Memory optimization: When SPOOL_DIR is set, messages are written to temporary files instead of being held in memory. This reduces memory pressure during event bursts (e.g., LIST operations or many resources created at once) and enables recovery of unsent messages after restarts (e.g., after OOM kills).

LIST + Watch Pattern

The forwarder uses explicit LIST followed by Watch rather than resource_version="0":

1. LIST retrieves all current objects and a snapshot resourceVersion
2. Watch starts from the LIST’s resourceVersion (not object RVs)
3. On graceful watch timeout (300s), restart watch from last known RV (no re-list)
4. On errors, fresh LIST to ensure current state

This is necessary because synthetic ADDED events from resource_version="0" have unsorted, potentially-stale resourceVersions (they reflect when objects were last modified). If the watch disconnects mid-stream, the tracked RV could be arbitrarily old and may already be compacted by etcd, causing a 410 error cascade.

Prerequisites

• Target cluster: ServiceAccount with watch permissions on the resources you want to monitor
• Deployment cluster: Kubernetes cluster to run the forwarder
• AWS credentials: SNS publish permissions (optional - logs events if not configured)

Deployment

The container image is built via Konflux CI/CD and published to quay.io/hummingbird-ci/kubernetes-event-forwarder:latest.

Example Kubernetes manifests are provided in the kubernetes/ directory:

• rbac.yaml - ServiceAccount, Role, RoleBinding for the target cluster
• secret.yaml - Kubeconfig and AWS credentials
• configmap.yaml - Resource watch configuration
• deployment.yaml - Forwarder deployment

Quick Start

1. On the target cluster (the one you want to watch), apply RBAC and create a token:

   kubectl apply -f kubernetes/rbac.yaml
   kubectl create token kubernetes-event-forwarder --duration=8760h
   

2. Update the example manifests with your values:

   • secret.yaml: cluster URL, token, AWS credentials
   • configmap.yaml: resources to watch
   • deployment.yaml: SNS topic ARN, AWS region

3. Apply the manifests to your deployment cluster:

   kubectl apply -f kubernetes/secret.yaml
   kubectl apply -f kubernetes/configmap.yaml
   kubectl apply -f kubernetes/deployment.yaml
   

Prerequisites: Deploy hummingbird-events-topic first to create the SNS topic, then deploy the AWS resources (see below).

AWS Resources

The SAM template (template.yaml) provisions IAM resources for SNS publishing:

• IAM User (${ResourcePrefix}-user) - Service account for the forwarder
• IAM Policy (${ResourcePrefix}-policy) - Grants sns:Publish to the SNS topic

Deploy using containerized AWS SAM CLI:

cd kubernetes-event-forwarder
sam build
sam deploy --guided  # First deployment (interactive)
sam deploy           # Subsequent deployments

After deployment, create access keys for the IAM user and store them securely.

SAM Parameters

Resource naming: IAM resources follow {ResourcePrefix}-{type} pattern (e.g., myapp-prod-user, myapp-prod-policy).

Usage

Configure resources to watch in config.yaml:

resources:
  - apiVersion: v1
    kind: Pod
    labelSelector: app=example
    fieldSelector: status.phase=Running
  - apiVersion: apps/v1
    kind: Deployment
  - apiVersion: v1
    kind: ConfigMap

labelSelector and fieldSelector are optional per-resource settings and use standard Kubernetes selector syntax. Field-selector support is resource and API server dependent; do not use a TaskRun status field selector. Use a stable RPM application label for TaskRuns only after confirming its propagation in the target environment. Source activation is owned by HUM-7533 and environment-specific configuration by HUM-7535. Managed Release namespace access is owned by HUM-7532; this example RBAC does not grant that access.

SNS Subscription Filter Examples:

Pod events in a specific namespace:

{
  "source": ["kubernetes"],
  "kind": ["Pod"],
  "namespace": ["production"]
}

All deployment changes:

{
  "source": ["kubernetes"],
  "kind": ["Deployment"]
}

Deleted resources across all clusters:

{
  "source": ["kubernetes"],
  "event_type": ["DELETED"]
}

Development

See the main README for development workflows.

make kubernetes-event-forwarder/setup  # Install dependencies
make check                             # Lint code (ruff)
make fmt                               # Format code
make test                              # Run unit tests
make coverage                          # Run tests with coverage

Configuration

The deployment is configured via environment variables:

Event Metadata

The forwarder extracts metadata from Kubernetes events and adds them as SNS message attributes:

cluster is the context.cluster value for the kubeconfig context that the forwarder uses to watch the resource. It is not the API server URL.

Security & Limitations

Security:

• AWS credentials stored in Kubernetes Secret
• Kubeconfig credentials stored in Kubernetes Secret
• SNS topic follows least privilege principle (publish-only)
• Sentry integration for error tracking

Limitations:

• Only namespaced resources supported
• One thread per context + resource type combination
• SNS message size limit: 256 KB (after compression)
• SNS measures the UTF-8 gzip/base64 message plus attribute names, types, and values against its 262,144-byte limit. The forwarder logs each computed size and remaining headroom and exports hummingbird_event_forwarder_sns_publish_size_bytes. Review representative RPM build, test, and tenant-release TaskRun measurements before enabling the source; do not remove further fields without an approved projection.
• The source-tree fixtures in kubernetes-event-forwarder/tests/fixtures/ are synthetic, redacted RPM build, test, and tenant-release TaskRun regression fixtures. Their unit test measures the complete publish body and attributes and requires at least 32 KiB of headroom. HUM-7533 owns source-activation measurement review.

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.