# Kubernetes Event Forwarder LLMS index: [llms.txt](https://hummingbird-project.io/llms.txt) | Full content: [llms-full.txt](https://hummingbird-project.io/llms-full.txt) --- A Kubernetes deployment that watches resource changes (ADDED/MODIFIED/DELETED) across multiple clusters and forwards them to an SNS topic with structured metadata for filtering. Uses kubeconfig contexts as the source of truth for which clusters and namespaces to watch. The full Kubernetes object JSON is forwarded as the SNS message body, compressed with gzip and base64-encoded. For `TaskRun` events only, Kubernetes `metadata.managedFields` is removed before compression. All other object data, including identity, owner references, labels, annotations, task specification, conditions, results, pod name, and timing is retained. ## Features - **Multi-Cluster Support**: Watch resources across multiple Kubernetes clusters using kubeconfig contexts - **Dynamic Resource Watching**: Configure any namespaced resource type using standard Kubernetes `apiVersion` and `kind` - **SNS Integration**: Publish events with structured message attributes for precise filtering - **Compression**: Events are compressed (gzip+base64) to reduce SNS message size - **Selectors**: Resource-specific Kubernetes label and field selectors limit LIST and Watch requests - **Automatic Reconnection**: Handles watch connection failures and reconnects automatically ## Architecture ### Threading Model The forwarder uses a multi-threaded architecture: - **Watcher threads** (one per context + resource type): Each runs an independent LIST+Watch loop. Isolation ensures one slow/failing cluster doesn't affect others. - **Publisher thread** (single, shared): Reads events from a queue and publishes to SNS. Decouples K8s API interaction from SNS latency (~150ms per publish), preventing watch loop stalls that could cause resourceVersion staleness. **Memory optimization:** When `SPOOL_DIR` is set, messages are written to temporary files instead of being held in memory. This reduces memory pressure during event bursts (e.g., LIST operations or many resources created at once) and enables recovery of unsent messages after restarts (e.g., after OOM kills). ### LIST + Watch Pattern The forwarder uses explicit LIST followed by Watch rather than `resource_version="0"`: 1. **LIST** retrieves all current objects and a snapshot `resourceVersion` 2. **Watch** starts from the LIST's `resourceVersion` (not object RVs) 3. On graceful watch timeout (300s), restart watch from last known RV (no re-list) 4. On errors, fresh LIST to ensure current state This is necessary because synthetic ADDED events from `resource_version="0"` have unsorted, potentially-stale resourceVersions (they reflect when objects were last modified). If the watch disconnects mid-stream, the tracked RV could be arbitrarily old and may already be compacted by etcd, causing a 410 error cascade. ## Prerequisites - **Target cluster**: ServiceAccount with watch permissions on the resources you want to monitor - **Deployment cluster**: Kubernetes cluster to run the forwarder - **AWS credentials**: SNS publish permissions (optional - logs events if not configured) ## Deployment The container image is built via Konflux CI/CD and published to `quay.io/hummingbird-ci/kubernetes-event-forwarder:latest`. Example Kubernetes manifests are provided in the [`kubernetes/`][k8s-dir] directory: - `rbac.yaml` - ServiceAccount, Role, RoleBinding for the **target** cluster - `secret.yaml` - Kubeconfig and AWS credentials - `configmap.yaml` - Resource watch configuration - `deployment.yaml` - Forwarder deployment ### Quick Start 1. On the **target** cluster (the one you want to watch), apply RBAC and create a token: ```bash kubectl apply -f kubernetes/rbac.yaml kubectl create token kubernetes-event-forwarder --duration=8760h ``` 2. Update the example manifests with your values: - `secret.yaml`: cluster URL, token, AWS credentials - `configmap.yaml`: resources to watch - `deployment.yaml`: SNS topic ARN, AWS region 3. Apply the manifests to your deployment cluster: ```bash kubectl apply -f kubernetes/secret.yaml kubectl apply -f kubernetes/configmap.yaml kubectl apply -f kubernetes/deployment.yaml ``` **Prerequisites:** Deploy [hummingbird-events-topic][het-docs] first to create the SNS topic, then deploy the AWS resources (see below). [k8s-dir]: https://gitlab.com/redhat/hummingbird/tools/-/tree/main/kubernetes-event-forwarder/kubernetes [het-docs]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/documentation/hummingbird-events-topic.md ### AWS Resources The SAM template (`template.yaml`) provisions IAM resources for SNS publishing: - **IAM User** (`${ResourcePrefix}-user`) - Service account for the forwarder - **IAM Policy** (`${ResourcePrefix}-policy`) - Grants `sns:Publish` to the SNS topic Deploy using containerized AWS SAM CLI: ```bash cd kubernetes-event-forwarder sam build sam deploy --guided # First deployment (interactive) sam deploy # Subsequent deployments ``` After deployment, create access keys for the IAM user and store them securely. ### SAM Parameters | Parameter | Description | Default | | ---------------- | -------------------- | ------------ | | `ResourcePrefix` | Prefix for resources | `myapp-prod` | | `SnsTopicArn` | SNS topic ARN | (required) | **Resource naming:** IAM resources follow `{ResourcePrefix}-{type}` pattern (e.g., `myapp-prod-user`, `myapp-prod-policy`). ## Usage Configure resources to watch in `config.yaml`: ```yaml resources: - apiVersion: v1 kind: Pod labelSelector: app=example fieldSelector: status.phase=Running - apiVersion: apps/v1 kind: Deployment - apiVersion: v1 kind: ConfigMap ``` `labelSelector` and `fieldSelector` are optional per-resource settings and use standard Kubernetes selector syntax. Field-selector support is resource and API server dependent; do not use a TaskRun status field selector. Use a stable RPM application label for TaskRuns only after confirming its propagation in the target environment. Source activation is owned by HUM-7533 and environment-specific configuration by HUM-7535. Managed Release namespace access is owned by HUM-7532; this example RBAC does not grant that access. **SNS Subscription Filter Examples:** Pod events in a specific namespace: ```json { "source": ["kubernetes"], "kind": ["Pod"], "namespace": ["production"] } ``` All deployment changes: ```json { "source": ["kubernetes"], "kind": ["Deployment"] } ``` Deleted resources across all clusters: ```json { "source": ["kubernetes"], "event_type": ["DELETED"] } ``` ## Development See the main [README][readme] for development workflows. ```bash make kubernetes-event-forwarder/setup # Install dependencies make check # Lint code (ruff) make fmt # Format code make test # Run unit tests make coverage # Run tests with coverage ``` ## Configuration The deployment is configured via environment variables: | Variable | Description | | ----------------------- | ----------------------------------------------------- | | `CONFIG_PATH` | Path to config YAML file | | `CONFIG` | Inline config YAML (alternative) | | `SNS_TOPIC_ARN` | SNS topic ARN (optional - logs if unset) | | `SPOOL_DIR` | Optional spool directory for file-based message queue | | `AWS_ACCESS_KEY_ID` | AWS access key ID | | `AWS_SECRET_ACCESS_KEY` | AWS secret access key | | `AWS_DEFAULT_REGION` | AWS region | | `METRICS_PORT` | Prometheus metrics port (default: `9090`) | | `SENTRY_DSN` | Optional Sentry DSN | ## Event Metadata The forwarder extracts metadata from Kubernetes events and adds them as SNS message attributes: | Attribute | Description | Example | | ------------------ | ------------------------------------- | ------------------------------ | | `source` | Always `"kubernetes"` | `kubernetes` | | `cluster` | Logical kubeconfig cluster entry name | `konflux-rh03` | | `namespace` | Object namespace | `production` | | `api_version` | Resource API version | `v1`, `apps/v1` | | `kind` | Resource kind | `Pod`, `Deployment` | | `event_type` | Event type | `ADDED`, `MODIFIED`, `DELETED` | | `object_name` | Name of the object | `nginx-7d8c4c9d6f` | | `content_encoding` | Message encoding | `gzip+base64` | `cluster` is the `context.cluster` value for the kubeconfig context that the forwarder uses to watch the resource. It is not the API server URL. ## Security & Limitations **Security:** - AWS credentials stored in Kubernetes Secret - Kubeconfig credentials stored in Kubernetes Secret - SNS topic follows least privilege principle (publish-only) - Sentry integration for error tracking **Limitations:** - Only namespaced resources supported - One thread per context + resource type combination - SNS message size limit: 256 KB (after compression) - SNS measures the UTF-8 gzip/base64 message plus attribute names, types, and values against its 262,144-byte limit. The forwarder logs each computed size and remaining headroom and exports `hummingbird_event_forwarder_sns_publish_size_bytes`. Review representative RPM build, test, and tenant-release TaskRun measurements before enabling the source; do not remove further fields without an approved projection. - The source-tree fixtures in `kubernetes-event-forwarder/tests/fixtures/` are synthetic, redacted RPM build, test, and tenant-release TaskRun regression fixtures. Their unit test measures the complete publish body and attributes and requires at least 32 KiB of headroom. HUM-7533 owns source-activation measurement review. ## License This project is licensed under the GNU General Public License v3.0 or later - see the [LICENSE][license] file for details. [readme]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/README.md [license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE