Lambda S3 Cache

An AWS Lambda setup that caches files from public URLs in S3, supporting both HTTP file caching (RPMs, repository metadata) and OCI Distribution API v2 (container image manifests and blobs). When a URL is requested, the service returns a 302 redirect to either the cached S3 copy (via presigned URL) or the original source. Cache misses trigger asynchronous downloads to S3, ensuring future requests are served from the cache.

Caching uses the original URL’s host and path as the S3 key for HTTP content, and a flat content-addressed layout (cache/oci/<digest>) for OCI artifacts. Frequently accessed content automatically extends its cache lifetime on each access.

Note: Cached content behavior depends on file type:

• Immutable content (matching immutable extension filter, e.g., .rpm): Changes at origin won’t be reflected until cache expires
• Mutable content (non-matching extension, e.g., repomd.xml): Cache is revalidated on each request via HEAD check; stale behavior is configurable

Cache Revalidation: Mutable content uses HTTP ETags for efficient validation. The uploader stores the origin’s ETag in S3 metadata when caching content. On subsequent requests, the handler sends a HEAD request with If-None-Match header containing the stored ETag. If the origin responds with 304 Not Modified, the cache is fresh and served directly. If the ETag differs, the cache is stale and an async refresh is triggered; response behavior depends on StaleCacheBehavior.

Note: If the origin doesn’t provide ETags, the cache cannot validate freshness. In this case, mutable content always redirects to origin without triggering cache updates (caching would be ineffective since every request would redirect anyway).

Features

• Streaming Upload: Handles files efficiently by streaming directly from source to S3 without loading into memory
• Presigned S3 URLs: Returns short-lived signed URLs for cached content
• Automatic Expiration: Cached content expires after a certain time, with lifetime extended on each access
• URL Prefix Allowlist: Only caches content matching explicitly allowed host+path prefixes
• Immutable Extension Filter: Identifies immutable content that doesn’t require revalidation
• Cache Revalidation: Mutable content (non-matching extensions) is validated on each request; stale cache behavior is configurable (origin or cache)
• OCI Image Caching: Caches OCI container image manifests and blobs via the OCI Distribution API v2, protecting against upstream digest garbage collection
• Multi-Registry Support: OCI caching supports multiple upstream registries by encoding the registry hostname in the request path
• Custom Domain: Optional custom domain with automatic TLS certificate management via ACM and Route53

Architecture

Three Lambda functions handle the caching workflow:

1. Handler - API Gateway endpoint that checks cache, returns 302 redirects, and triggers async operations. Implements touch cooldown to prevent S3 throttling by only touching objects after a configurable time period has elapsed since last modification.
2. Uploader - Downloads from origin and streams to S3 on cache misses (invoked asynchronously)
3. Touch - Updates S3 object timestamps to extend cache lifetime on cache hits (invoked asynchronously only when cooldown period has elapsed)

Request Flow

flowchart TD
    Start([Request]) --> CheckURL{URL matches<br/>AllowedPrefixes?}

    CheckURL -->|No| RedirectOrigin
    CheckURL -->|Yes| HeadS3[HEAD S3]:::network
    HeadS3 --> CheckCache{Cache exists?}

    CheckCache -->|No: Cache Miss| InvokeUploaderMiss[Invoke Uploader async]:::async
    CheckCache -->|Yes: Cache Hit| CheckImmutable{Immutable extension?}

    CheckImmutable -->|No: Mutable| HeadOrigin[HEAD Origin<br/>If-None-Match: stored ETag]:::network
    HeadOrigin --> CheckChanged{Origin changed?}

    CheckChanged -->|No ETag from origin| RedirectOrigin
    CheckChanged -->|ETag differs| InvokeUploaderStale[Invoke Uploader async]:::async
    CheckChanged -->|ETag matches| CheckCooldown
    CheckChanged -->|304 Not Modified| CheckCooldown
    CheckChanged -->|"Error/timeout"| CheckCooldown

    CheckImmutable -->|Yes| CheckCooldown

    CheckCooldown{Touch cooldown<br/>elapsed?} -->|Yes| InvokeTouch[Invoke Touch async]:::async

    InvokeUploaderMiss --> RedirectOrigin
    InvokeUploaderStale -->|"STALE_CACHE_BEHAVIOR=origin"| RedirectOrigin

    InvokeUploaderStale -->|"STALE_CACHE_BEHAVIOR=cache"| RedirectCache
    CheckCooldown -->|No| RedirectCache
    InvokeTouch --> RedirectCache

    subgraph redirectGroup [ ]
        RedirectOrigin[302 to Origin URL]:::redirect
        RedirectCache[302 to S3 Presigned URL]:::redirect
    end
    style redirectGroup fill:none,stroke:none

    classDef network fill:#10b981,color:#000
    classDef async fill:#ff9900,color:#000
    classDef redirect fill:#3b82f6,color:#fff

Legend: Green = network call, Orange = async Lambda invocation, Blue = 302 redirect response

Prerequisites

• AWS CLI configured with appropriate credentials (IAM permissions for Lambda, API Gateway, S3, CloudFormation, CloudWatch Logs, and optionally Route53/ACM for custom domain)
• Podman or Docker (for containerized SAM build/deploy)
• Python 3.11 or later (for development)

Deployment

Build and deploy using containerized AWS SAM CLI:

make lambda-s3-cache/build     # Build Lambda package
make lambda-s3-cache/deploy    # First deployment (interactive/guided)
make lambda-s3-cache/redeploy  # Subsequent deployments (non-interactive)

Deployment output: ApiEndpoint - the API Gateway URL to use for requests

Custom Domain

Optional custom domain with automatic TLS certificate management (ACM + Route53). Requires a Route53 hosted zone. Deploy with CustomDomainName and HostedZoneId parameters - CloudFormation handles certificate creation, DNS validation, and configuration. Certificate validation takes 5-30 minutes; allow up to 1 hour for DNS propagation.

Parameters

Resource naming: All AWS resources follow {ResourcePrefix}-{type}-{name} pattern (e.g., myapp-prod-bucket, myapp-prod-lambda-handler).

Usage

Make GET requests to the API endpoint (or custom domain if configured). The URL to cache is encoded in the path (without the https:// prefix):

curl -L "https://<api-endpoint>/example.com/path/to/file.rpm"

Behavior:

• First request (cache miss): Redirects to original URL while triggering async S3 upload
• Subsequent requests (cache hit):
  • Immutable content (matching immutable extension filter): Redirects to presigned S3 URL and resets cache expiration
  • Mutable content (non-matching extension): HEAD request validates cache freshness (10s timeout). If stale, behavior depends on StaleCacheBehavior; if fresh, serves from cache
• Disallowed prefixes: URLs not matching allowed prefixes are transparently redirected to the original URL (302 pass-through)

Immutable Extension Filter Behavior

The ImmutableExtensions parameter determines caching behavior:

• Matching extension (e.g., .rpm): Treated as immutable content. Cached without revalidation - changes at origin won’t be reflected until cache expires. Served directly from cache on all requests.

• Non-matching extension (e.g., .xml, .gz): Treated as mutable content. Cached with revalidation - HEAD request on each cache hit validates freshness using ETags. If stale, behavior depends on StaleCacheBehavior:

  • origin (default): Redirects to origin for fresh content, async refresh
  • cache: Serves stale cache immediately (faster), async refresh in background

Important: All files matching AllowedPrefixes are cached, regardless of extension. The extension filter only determines whether revalidation is performed.

Usage as Repository Proxy

The cache can be used as a DNF/yum baseurl for RPM repositories. Configure ImmutableExtensions to include .rpm:

• .rpm files (matching extension): Cached as immutable - fast, no revalidation
• Metadata files (non-matching extension, e.g., repomd.xml, primary.xml.gz): Cached with revalidation - ensures fresh metadata while benefiting from cache

[myrepo]
name=My Repository
baseurl=https://koji-s3-cache.example.com/download.example.org/pub/repo/$basearch/
enabled=1

This provides caching benefits for RPM downloads while ensuring repository metadata stays fresh.

OCI Image Caching

The cache also supports the OCI Distribution API v2 for caching container image manifests and blobs. This prevents deployment failures caused by upstream registries garbage-collecting digest-pinned images when tags move.

Requests with a v2/ path prefix are handled as OCI requests. The upstream registry is extracted from the first path segment after v2/, enabling multi-registry support through a single cache endpoint.

OCI request format:

GET /v2/<registry>/<image-name>/manifests/<digest>
GET /v2/<registry>/<image-name>/blobs/<digest>

Example:

# Health check
curl https://cache.example.com/v2/

# Manifest request (quay.io/openshift/origin-oauth-proxy)
curl -L https://cache.example.com/v2/quay.io/openshift/origin-oauth-proxy/manifests/sha256:504c036d...

OCI cache strategy (origin-first):

The OCI cache uses an origin-first strategy, matching the proven CKI dependency-archive pattern. This minimizes S3 egress — the cache is only consulted when the upstream registry is unavailable:

1. HEAD upstream registry — check if the digest is still available
2. Origin available + not cached → redirect to upstream, trigger async S3 upload
3. Origin available + cached → redirect to upstream (touch if cooldown elapsed)
4. Origin unavailable + cached manifest → serve inline from S3 (200 with body, Content-Type from metadata, Docker-Content-Digest header)
5. Origin unavailable + cached blob → 302 to presigned S3 URL
6. Origin unavailable + not cached → redirect to upstream as last resort
7. Any error → redirect to upstream (fail-open)

S3 key layout: cache/oci/sha256:<digest> — flat, content-addressed. The same blob shared by different images is stored only once.

OCI request flow:

flowchart TD
    Start([OCI Request]) --> CheckPath{Path starts<br/>with v2/?}
    CheckPath -->|No| RPMHandler[RPM cache handler]
    CheckPath -->|Yes| V2Health{v2 or v2/?}
    V2Health -->|Yes| Return200[200 OK<br/>registry/2.0]:::redirect
    V2Health -->|No| ParsePath[Parse path:<br/>registry/name/kind/digest]
    ParsePath --> ValidDigest{Valid digest?<br/>manifests or blobs?}
    ValidDigest -->|No| ForwardUpstream[302 to upstream]:::redirect
    ValidDigest -->|Yes| CheckAllow{In OCI<br/>AllowedPrefixes?}
    CheckAllow -->|No| ForwardUpstream
    CheckAllow -->|Yes| HeadOrigin[HEAD upstream registry]:::network
    HeadOrigin --> OriginUp{Origin available?}
    OriginUp -->|Yes| CheckOciCache{Cached in S3?}
    CheckOciCache -->|No| InvokeUploader[Invoke Uploader async]:::async
    CheckOciCache -->|Yes| TouchCheck{Touch cooldown?}
    TouchCheck -->|Elapsed| InvokeTouchOci[Invoke Touch async]:::async
    TouchCheck -->|Not elapsed| RedirectUpstream
    InvokeTouchOci --> RedirectUpstream[302 to upstream]:::redirect
    InvokeUploader --> RedirectUpstream
    OriginUp -->|No| CheckFallback{Cached in S3?}
    CheckFallback -->|Yes, manifest| ServeInline[200 inline<br/>from S3]:::redirect
    CheckFallback -->|Yes, blob| PresignedRedirect[302 to<br/>presigned S3]:::redirect
    CheckFallback -->|No| ForwardUpstream

    classDef network fill:#10b981,color:#000
    classDef async fill:#ff9900,color:#000
    classDef redirect fill:#3b82f6,color:#fff

Note: Tag-based pulls (/manifests/latest), tag listings (/tags/list), and images not in OciAllowedPrefixes are transparently forwarded to the upstream registry without caching.

Development

See the main README for development workflows.

make lambda-s3-cache/setup  # Install dependencies
make check                  # Lint code
make fmt                    # Format code
make test                   # Run unit tests
make coverage               # Run tests with coverage

Configuration

Lambda functions receive configuration via environment variables (automatically set by CloudFormation):

Security & Limitations

Security:

• S3 bucket has public access blocked; all objects encrypted at rest (AES256)
• Presigned URLs expire after a certain time
• IAM policies follow least privilege principle
• URL prefix allowlist prevents caching arbitrary URLs
• OCI image allowlist (OCI_ALLOWED_PREFIXES) controls which images are cached
• Immutable extension filter distinguishes immutable vs mutable content for revalidation
• OCI caching only handles public images (no auth forwarding)

Limitations:

• Lambda timeout: 15 min (uploader), 30 sec (handler, touch)
• Lambda memory: 1024 MB (uploader), 256 MB (handler, touch)
• S3 object size: Up to 5 TB (AWS limit)
• API Gateway 6 MB response limit for inline manifest responses (manifests are typically under 100 KB)
• OCI caching only supports digest-based pulls (sha256:...); tag-based pulls are forwarded to the upstream registry

Design Decisions Record

DDR-1: S3 Lambda OCI cache vs ECR pull-through cache

Decision: Build the OCI cache as a new code path in the existing lambda-s3-cache Lambda, rather than using AWS ECR pull-through cache.

Context: Third-party container images pinned by digest vanish when upstream registries garbage-collect old manifests after tag re-pushes (e.g., quay.io/openshift/origin-oauth-proxy:5.1 digests disappear within hours). This causes ImagePullBackOff failures on pod restarts.

Alternatives: (a) ECR pull-through cache — managed service, no custom code, retains cached digests when upstream is unavailable. (b) images.paas.redhat.com — Red Hat internal registry with existing pull-through proxy orgs for some registries. (c) Artifactory — supports any upstream registry as a remote repository.

Rationale: ECR pull-through cache requires private ECR registries, which need IAM authentication — tokens expire every 12 hours, requiring a CronJob for credential rotation on OCP pods. The S3 Lambda approach serves content via a public HTTPS URL with zero authentication overhead. images.paas.redhat.com was ruled out due to dependency on Red Hat IT (rug-pulling risk if they decommission the proxy orgs). Artifactory was not in use and would add a new service dependency. The S3 Lambda approach reuses existing infrastructure (same Lambda, API Gateway, S3 bucket, custom domain) and adds ~100 lines of OCI handler code ported from the proven CKI dependency-archive.

DDR-2: Origin-first (HEAD-first) strategy vs cache-first

Decision: Use an origin-first strategy: always check the upstream registry first via HEAD request, redirect to upstream when available, and serve from S3 only when upstream returns 404 or errors.

Context: The existing RPM cache uses a cache-first strategy (check S3 first, revalidate with origin). For OCI artifacts, the design choice affects S3 egress costs and cache freshness.

Alternatives: (a) Cache-first — check S3 first, serve from cache if present. (b) Origin-first — check upstream first, fall back to S3.

Rationale: OCI digests are content-addressed and immutable — there is no staleness concern (unlike RPM repo metadata). The origin-first approach minimizes S3 egress costs: when upstream is healthy (the common case), every pull redirects directly to the upstream registry with zero S3 data transfer. S3 is only read when upstream has garbage-collected the digest. This matches the CKI dependency-archive pattern, which is proven in production. The cache populates asynchronously on first access and serves as a safety net, not a performance layer.

DDR-3: Registry hostname in URL path vs per-registry subdomains

Decision: Encode the upstream registry hostname as the first path segment after v2/ (e.g., koji-s3-cache.example.com/quay.io/openshift/origin-oauth-proxy@sha256:...).

Context: The CKI dependency-archive assumes a single upstream registry (OCI_REGISTRY env var). Hummingbird pulls from 4+ registries (quay.io, docker.io, ghcr.io, registry.gitlab.com).

Alternatives: (a) Per-registry subdomains (quay-io.cache.example.com/...) — standard OCI URL format but requires wildcard TLS certificates and DNS records, potentially separate Lambda deployments. (b) Registry hostname in the URL path — single endpoint, handler extracts the registry from the first path segment.

Rationale: Encoding the registry in the path allows a single Lambda, single API Gateway, single custom domain, and single TLS certificate to serve all registries. The /{path+} catch-all route already captures everything. The v2/ prefix naturally separates OCI traffic from RPM cache traffic. The kubelet constructs the correct path automatically from the image reference. This approach requires zero new infrastructure beyond the existing lambda-s3-cache stack.

DDR-4: Jinja macro rewrite vs registries.conf

Decision: Rewrite image URLs at Jinja template render time using a cached_image() macro, rather than configuring registries.conf on OCP nodes.

Context: OCP pods need to pull from the cache instead of directly from upstream registries. The standard approach is registries.conf mirror configuration on the cluster nodes.

Alternatives: (a) Node-level registries.conf via MachineConfig. (b) Jinja macro rewriting image refs at deploy time.

Rationale: Hummingbird has namespace-admin access only on OCP clusters — no MachineConfig, no node-level changes. The Jinja macro approach works entirely at the deployment manifest level: vars.d/*.yml files continue to contain upstream refs (Renovate keeps working unchanged), and the macro rewrites them to cache URLs at render time. This is transparent to Renovate, requires no cluster-admin privileges, and can be disabled with a single variable change.

DDR-5: Scope limited to third-party images

Decision: Only cache third-party images. Hummingbird-owned images (quay.io/hummingbird-ci/* and registry.access.redhat.com/hi/*) are excluded.

Context: The cache exists to protect against upstream registries garbage-collecting digest-pinned images.

Alternatives: Cache all images uniformly.

Rationale: Hummingbird-owned images have additional sha-based tags (commit SHA, build ID) that persist independently of the floating :latest tag. Old digests are not garbage-collected because these immutable tags keep them reachable. Caching them would add S3 storage and Lambda invocations with no benefit. The cached_image() Jinja macro skips rewriting for quay.io/hummingbird-ci/ prefixes.

DDR-6: Separate OCI handler function vs extending RPM cache logic

Decision: Implement _handle_oci_cacheable() as a separate function from the existing RPM cache logic (lambda_handler → _handle_rpm).

Context: Both OCI and RPM caching share the same S3 bucket, uploader Lambda, and touch mechanism. The code could be unified.

Alternatives: Extend the existing RPM handler to handle OCI requests by adding OCI-specific branches.

Rationale: The RPM cache has ETag-based revalidation and configurable stale-cache behavior (origin vs cache) that do not apply to OCI — digests are immutable by definition. Keeping the handlers separate avoids accidentally changing RPM behavior and keeps each code path focused. The OCI handler was ported from the CKI dependency-archive, which is proven in production; staying close to the CKI code structure reduces the risk of introducing untested edge cases.

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.