CloudWatch Log Forwarder

A Python-based Kubernetes pod log forwarder that collects logs via the Kubernetes API and forwards them to AWS CloudWatch Logs. Runs as a single deployment per cluster, collecting logs from configured namespaces without requiring cluster admin privileges or sidecar containers.

Features

  • API-Based Collection: Uses Kubernetes API (pods/log) - no DaemonSet or privileged access required
  • Per-Cluster Log Groups: Each cluster gets its own CloudWatch Log Group
  • Per-Container Log Streams: Dynamic log streams per container instance
  • Automatic Recovery: Reconnects on API disconnects, resumes from last timestamp
  • Auto-Create Resources: Log groups and streams created automatically
  • Log Retention: Configurable retention policy (default: 30 days)

Prerequisites

  • AWS CLI configured with appropriate credentials
  • Podman or Docker (for containerized SAM build/deploy)
  • Target namespace for forwarder deployment on each cluster

Deployment

AWS Resources

Build and deploy using containerized AWS SAM CLI:

cd cloudwatch-log-forwarder
sam build
sam deploy --guided  # First deployment (interactive)
sam deploy           # Subsequent deployments

After deployment, create access keys for the IAM user and store them securely.

Parameters

Parameter Description Default
ResourcePrefix Prefix for all resource names myapp-prod

Resource naming: IAM resources follow {ResourcePrefix}-{type} pattern (e.g., myapp-prod-user, myapp-prod-policy).

Kubernetes Resources

Deploy the forwarder with:

  • ServiceAccount, Role, RoleBinding per namespace
  • Secret with AWS credentials
  • ConfigMap with forwarder configuration
  • Deployment running the Python forwarder

CloudWatch Log Structure

Component Pattern
Log Group /{ResourcePrefix}-{cluster}
Log Stream {namespace}/{pod}/{container}/{restart_count}

Log groups are auto-created by the forwarder when first log is written.

Example Query

fields @timestamp, @message
| filter @logStream like /hummingbird--runner-jobs/
| sort @timestamp desc
| limit 100

Configuration

The forwarder is configured via a YAML config file:

# config.yaml
prefix: arr-hummingbird-prod-cloudwatch-log-forwarder
cluster: mpp-prod
namespaces:
  - hummingbird--runner
  - hummingbird--runner-jobs
exclude_labels:
  app.kubernetes.io/name: cloudwatch-log-forwarder
retention_days: 30

Config File Reference

Field Required Description
prefix Yes CloudWatch resource prefix (used in log group name)
cluster Yes Cluster identifier (used in log group name)
namespaces Yes List of namespaces to collect logs from
exclude_labels No Label key/value pairs to exclude pods from collection
retention_days No Log retention in days (default: 30)

Environment Variables

  • CONFIG_PATH: Path to YAML config file (required)
  • AWS_ACCESS_KEY_ID: AWS access key
  • AWS_SECRET_ACCESS_KEY: AWS secret key
  • AWS_DEFAULT_REGION: AWS region

Security & Limitations

Security:

  • AWS credentials stored in Kubernetes Secret
  • RBAC follows least privilege (read-only pods/log access)
  • IAM policy scoped to specific log group prefix

Limitations:

  • Historical logs not collected (only from forwarder start time)
  • Restart recovery depends on CloudWatch timestamp availability

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.