# Lambda S3 Cache

LLMS index: [llms.txt](https://hummingbird-project.io/llms.txt) | Full content: [llms-full.txt](https://hummingbird-project.io/llms-full.txt)

---

An AWS Lambda setup that caches files from public URLs in S3, supporting both
HTTP file caching (RPMs, repository metadata) and OCI Distribution API v2
(container image manifests and blobs). When a URL is requested, the service
returns a 302 redirect to either the cached S3 copy (via presigned URL) or the
original source. Cache misses trigger asynchronous downloads to S3, ensuring
future requests are served from the cache.

Caching uses the original URL's host and path as the S3 key for HTTP content,
and a flat content-addressed layout (`cache/oci/<digest>`) for OCI artifacts.
Frequently accessed content automatically extends its cache lifetime on each
access.

**Note:** Cached content behavior depends on file type:

- **Immutable content** (matching immutable extension filter, e.g., `.rpm`): Changes at origin
  won't be reflected until cache expires
- **Mutable content** (non-matching extension, e.g., `repomd.xml`): Cache is
  revalidated on each request via HEAD check; stale behavior is configurable

**Cache Revalidation:** Mutable content uses HTTP ETags for efficient validation.
The uploader stores the origin's ETag in S3 metadata when caching content. On
subsequent requests, the handler sends a HEAD request with `If-None-Match` header
containing the stored ETag. If the origin responds with `304 Not Modified`, the
cache is fresh and served directly. If the ETag differs, the cache is stale and
an async refresh is triggered; response behavior depends on `StaleCacheBehavior`.

**Note:** If the origin doesn't provide ETags, the cache cannot validate freshness.
In this case, mutable content always redirects to origin without triggering cache
updates (caching would be ineffective since every request would redirect anyway).

## Features

- **Streaming Upload**: Handles files efficiently by streaming directly from
  source to S3 without loading into memory
- **Presigned S3 URLs**: Returns short-lived signed URLs for cached content
- **Automatic Expiration**: Cached content expires after a certain time, with
  lifetime extended on each access
- **URL Prefix Allowlist**: Only caches content matching explicitly allowed host+path prefixes
- **Immutable Extension Filter**: Identifies immutable content that doesn't require revalidation
- **Cache Revalidation**: Mutable content (non-matching extensions) is validated
  on each request; stale cache behavior is configurable (origin or cache)
- **OCI Image Caching**: Caches OCI container image manifests and blobs via the
  OCI Distribution API v2, protecting against upstream digest garbage collection
- **Multi-Registry Support**: OCI caching supports multiple upstream registries
  by encoding the registry hostname in the request path
- **Custom Domain**: Optional custom domain with automatic TLS certificate
  management via ACM and Route53

## Architecture

Three Lambda functions handle the caching workflow:

1. **Handler** - API Gateway endpoint that checks cache, returns 302 redirects,
   and triggers async operations. Implements touch cooldown to prevent S3
   throttling by only touching objects after a configurable time period has
   elapsed since last modification.
2. **Uploader** - Downloads from origin and streams to S3 on cache misses
   (invoked asynchronously)
3. **Touch** - Updates S3 object timestamps to extend cache lifetime on cache
   hits (invoked asynchronously only when cooldown period has elapsed)

### Request Flow

```mermaid
flowchart TD
    Start([Request]) --> CheckURL{URL matches<br/>AllowedPrefixes?}

    CheckURL -->|No| RedirectOrigin
    CheckURL -->|Yes| HeadS3[HEAD S3]:::network
    HeadS3 --> CheckCache{Cache exists?}

    CheckCache -->|No: Cache Miss| InvokeUploaderMiss[Invoke Uploader async]:::async
    CheckCache -->|Yes: Cache Hit| CheckImmutable{Immutable extension?}

    CheckImmutable -->|No: Mutable| HeadOrigin[HEAD Origin<br/>If-None-Match: stored ETag]:::network
    HeadOrigin --> CheckChanged{Origin changed?}

    CheckChanged -->|No ETag from origin| RedirectOrigin
    CheckChanged -->|ETag differs| InvokeUploaderStale[Invoke Uploader async]:::async
    CheckChanged -->|ETag matches| CheckCooldown
    CheckChanged -->|304 Not Modified| CheckCooldown
    CheckChanged -->|"Error/timeout"| CheckCooldown

    CheckImmutable -->|Yes| CheckCooldown

    CheckCooldown{Touch cooldown<br/>elapsed?} -->|Yes| InvokeTouch[Invoke Touch async]:::async

    InvokeUploaderMiss --> RedirectOrigin
    InvokeUploaderStale -->|"STALE_CACHE_BEHAVIOR=origin"| RedirectOrigin

    InvokeUploaderStale -->|"STALE_CACHE_BEHAVIOR=cache"| RedirectCache
    CheckCooldown -->|No| RedirectCache
    InvokeTouch --> RedirectCache

    subgraph redirectGroup [ ]
        RedirectOrigin[302 to Origin URL]:::redirect
        RedirectCache[302 to S3 Presigned URL]:::redirect
    end
    style redirectGroup fill:none,stroke:none

    classDef network fill:#10b981,color:#000
    classDef async fill:#ff9900,color:#000
    classDef redirect fill:#3b82f6,color:#fff
```

**Legend:** Green = network call, Orange = async Lambda invocation, Blue = 302 redirect response

## Prerequisites

- AWS CLI configured with appropriate credentials (IAM permissions for Lambda,
  API Gateway, S3, CloudFormation, CloudWatch Logs, and optionally Route53/ACM
  for custom domain)
- Podman or Docker (for containerized SAM build/deploy)
- Python 3.11 or later (for development)

## Deployment

Build and deploy using containerized AWS SAM CLI:

```bash
make lambda-s3-cache/build     # Build Lambda package
make lambda-s3-cache/deploy    # First deployment (interactive/guided)
make lambda-s3-cache/redeploy  # Subsequent deployments (non-interactive)
```

**Deployment output:** `ApiEndpoint` - the API Gateway URL to use for requests

### Custom Domain

Optional custom domain with automatic TLS certificate management (ACM +
Route53). Requires a Route53 hosted zone. Deploy with `CustomDomainName` and
`HostedZoneId` parameters - CloudFormation handles certificate creation, DNS
validation, and configuration. Certificate validation takes 5-30 minutes; allow
up to 1 hour for DNS propagation.

### Parameters

| Parameter                | Description                                               | Default             |
| ------------------------ | --------------------------------------------------------- | ------------------- |
| `ResourcePrefix`         | Prefix for all resource names                             | `myapp-prod`        |
| `PresignedUrlExpiration` | Presigned URL expiration (seconds)                        | `3600`              |
| `AllowedPrefixes`        | Whitespace-separated list of allowed URL prefixes         | `example.com/path/` |
| `ImmutableExtensions`    | File extensions for immutable content (no revalidation)   | `.rpm`              |
| `CacheExpirationDays`    | Days to keep cached content (minimum: 1)                  | `14`                |
| `TouchCooldownMinutes`   | Minimum minutes between touch operations                  | `60`                |
| `StaleCacheBehavior`     | When cache is stale: `origin` or `cache`                  | `origin`            |
| `OciAllowedPrefixes`     | Whitespace-separated OCI image prefixes (`registry/name`) | ``                  |
| `OciOriginHeadTimeout`   | Timeout (seconds) for HEAD requests to OCI registries     | `5`                 |
| `CustomDomainName`       | Optional custom domain name                               | ``                  |
| `HostedZoneId`           | Route53 hosted zone ID (required if custom domain)        | ``                  |
| `SentryDsn`              | Optional Sentry DSN for error tracking                    | ``                  |

**Resource naming:** All AWS resources follow `{ResourcePrefix}-{type}-{name}`
pattern (e.g., `myapp-prod-bucket`, `myapp-prod-lambda-handler`).

## Usage

Make GET requests to the API endpoint (or custom domain if configured). The URL
to cache is encoded in the path (without the `https://` prefix):

```bash
curl -L "https://<api-endpoint>/example.com/path/to/file.rpm"
```

**Behavior:**

- **First request** (cache miss): Redirects to original URL while triggering
  async S3 upload
- **Subsequent requests** (cache hit):
  - **Immutable content** (matching immutable extension filter): Redirects to presigned S3
    URL and resets cache expiration
  - **Mutable content** (non-matching extension): HEAD request validates cache
    freshness (10s timeout). If stale, behavior depends on `StaleCacheBehavior`;
    if fresh, serves from cache
- **Disallowed prefixes**: URLs not matching allowed prefixes are transparently
  redirected to the original URL (302 pass-through)

### Immutable Extension Filter Behavior

The `ImmutableExtensions` parameter determines caching behavior:

- **Matching extension** (e.g., `.rpm`): Treated as **immutable content**. Cached
  without revalidation - changes at origin won't be reflected until cache expires.
  Served directly from cache on all requests.

- **Non-matching extension** (e.g., `.xml`, `.gz`): Treated as **mutable content**.
  Cached with revalidation - HEAD request on each cache hit validates freshness
  using ETags. If stale, behavior depends on `StaleCacheBehavior`:
  - `origin` (default): Redirects to origin for fresh content, async refresh
  - `cache`: Serves stale cache immediately (faster), async refresh in background

**Important:** All files matching `AllowedPrefixes` are cached, regardless of
extension. The extension filter only determines whether revalidation is performed.

### Usage as Repository Proxy

The cache can be used as a DNF/yum `baseurl` for RPM repositories. Configure
`ImmutableExtensions` to include `.rpm`:

- **`.rpm` files** (matching extension): Cached as immutable - fast, no revalidation
- **Metadata files** (non-matching extension, e.g., `repomd.xml`, `primary.xml.gz`):
  Cached with revalidation - ensures fresh metadata while benefiting from cache

```ini
[myrepo]
name=My Repository
baseurl=https://koji-s3-cache.example.com/download.example.org/pub/repo/$basearch/
enabled=1
```

This provides caching benefits for RPM downloads while ensuring repository
metadata stays fresh.

### OCI Image Caching

The cache also supports the OCI Distribution API v2 for caching container image
manifests and blobs. This prevents deployment failures caused by upstream
registries garbage-collecting digest-pinned images when tags move.

Requests with a `v2/` path prefix are handled as OCI requests. The upstream
registry is extracted from the first path segment after `v2/`, enabling
multi-registry support through a single cache endpoint.

**OCI request format:**

```text
GET /v2/<registry>/<image-name>/manifests/<digest>
GET /v2/<registry>/<image-name>/blobs/<digest>
```

**Example:**

```bash
# Health check
curl https://cache.example.com/v2/

# Manifest request (quay.io/openshift/origin-oauth-proxy)
curl -L https://cache.example.com/v2/quay.io/openshift/origin-oauth-proxy/manifests/sha256:504c036d...
```

**OCI cache strategy (origin-first):**

The OCI cache uses an origin-first strategy, matching the proven CKI
dependency-archive pattern. This minimizes S3 egress — the cache is only
consulted when the upstream registry is unavailable:

1. **HEAD upstream registry** — check if the digest is still available
2. **Origin available + not cached** → redirect to upstream, trigger async S3
   upload
3. **Origin available + cached** → redirect to upstream (touch if cooldown
   elapsed)
4. **Origin unavailable + cached manifest** → serve inline from S3 (200 with
   body, `Content-Type` from metadata, `Docker-Content-Digest` header)
5. **Origin unavailable + cached blob** → 302 to presigned S3 URL
6. **Origin unavailable + not cached** → redirect to upstream as last resort
7. **Any error** → redirect to upstream (fail-open)

**S3 key layout:** `cache/oci/sha256:<digest>` — flat, content-addressed. The
same blob shared by different images is stored only once.

**OCI request flow:**

```mermaid
flowchart TD
    Start([OCI Request]) --> CheckPath{Path starts<br/>with v2/?}
    CheckPath -->|No| RPMHandler[RPM cache handler]
    CheckPath -->|Yes| V2Health{v2 or v2/?}
    V2Health -->|Yes| Return200[200 OK<br/>registry/2.0]:::redirect
    V2Health -->|No| ParsePath[Parse path:<br/>registry/name/kind/digest]
    ParsePath --> ValidDigest{Valid digest?<br/>manifests or blobs?}
    ValidDigest -->|No| ForwardUpstream[302 to upstream]:::redirect
    ValidDigest -->|Yes| CheckAllow{In OCI<br/>AllowedPrefixes?}
    CheckAllow -->|No| ForwardUpstream
    CheckAllow -->|Yes| HeadOrigin[HEAD upstream registry]:::network
    HeadOrigin --> OriginUp{Origin available?}
    OriginUp -->|Yes| CheckOciCache{Cached in S3?}
    CheckOciCache -->|No| InvokeUploader[Invoke Uploader async]:::async
    CheckOciCache -->|Yes| TouchCheck{Touch cooldown?}
    TouchCheck -->|Elapsed| InvokeTouchOci[Invoke Touch async]:::async
    TouchCheck -->|Not elapsed| RedirectUpstream
    InvokeTouchOci --> RedirectUpstream[302 to upstream]:::redirect
    InvokeUploader --> RedirectUpstream
    OriginUp -->|No| CheckFallback{Cached in S3?}
    CheckFallback -->|Yes, manifest| ServeInline[200 inline<br/>from S3]:::redirect
    CheckFallback -->|Yes, blob| PresignedRedirect[302 to<br/>presigned S3]:::redirect
    CheckFallback -->|No| ForwardUpstream

    classDef network fill:#10b981,color:#000
    classDef async fill:#ff9900,color:#000
    classDef redirect fill:#3b82f6,color:#fff
```

**Note:** Tag-based pulls (`/manifests/latest`), tag listings
(`/tags/list`), and images not in `OciAllowedPrefixes` are transparently
forwarded to the upstream registry without caching.

## Development

See the main [README][readme] for development workflows.

```bash
make lambda-s3-cache/setup  # Install dependencies
make check                  # Lint code
make fmt                    # Format code
make test                   # Run unit tests
make coverage               # Run tests with coverage
```

## Configuration

Lambda functions receive configuration via environment variables (automatically
set by CloudFormation):

| Variable                   | Handler | Uploader | Touch | Description                      |
| -------------------------- | ------- | -------- | ----- | -------------------------------- |
| `S3_BUCKET_NAME`           | ✅      | ✅       | ✅    | S3 bucket name                   |
| `PRESIGNED_URL_EXPIRATION` | ✅      |          |       | Presigned URL expiration (sec)   |
| `UPLOADER_LAMBDA_ARN`      | ✅      |          |       | Uploader Lambda ARN              |
| `TOUCH_LAMBDA_ARN`         | ✅      |          |       | Touch Lambda ARN                 |
| `TOUCH_COOLDOWN_MINUTES`   | ✅      |          |       | Min minutes between touch ops    |
| `ALLOWED_PREFIXES`         | ✅      |          |       | Allowed URL prefixes             |
| `IMMUTABLE_EXTENSIONS`     | ✅      |          |       | Extensions for immutable content |
| `STALE_CACHE_BEHAVIOR`     | ✅      |          |       | Stale behavior: origin or cache  |
| `OCI_ALLOWED_PREFIXES`     | ✅      |          |       | OCI image prefixes to cache      |
| `ORIGIN_HEAD_TIMEOUT`      | ✅      |          |       | OCI origin HEAD timeout (sec)    |
| `SENTRY_DSN`               | ✅      | ✅       | ✅    | Optional Sentry DSN              |

## Security & Limitations

**Security:**

- S3 bucket has public access blocked; all objects encrypted at rest (AES256)
- Presigned URLs expire after a certain time
- IAM policies follow least privilege principle
- URL prefix allowlist prevents caching arbitrary URLs
- OCI image allowlist (`OCI_ALLOWED_PREFIXES`) controls which images are cached
- Immutable extension filter distinguishes immutable vs mutable content for revalidation
- OCI caching only handles public images (no auth forwarding)

**Limitations:**

- Lambda timeout: 15 min (uploader), 30 sec (handler, touch)
- Lambda memory: 1024 MB (uploader), 256 MB (handler, touch)
- S3 object size: Up to 5 TB (AWS limit)
- API Gateway 6 MB response limit for inline manifest responses (manifests are
  typically under 100 KB)
- OCI caching only supports digest-based pulls (`sha256:...`); tag-based pulls
  are forwarded to the upstream registry

## Design Decisions Record

### DDR-1: S3 Lambda OCI cache vs ECR pull-through cache

**Decision**: Build the OCI cache as a new code path in the existing
lambda-s3-cache Lambda, rather than using AWS ECR pull-through cache.

**Context**: Third-party container images pinned by digest vanish when
upstream registries garbage-collect old manifests after tag re-pushes
(e.g., `quay.io/openshift/origin-oauth-proxy:5.1` digests disappear
within hours). This causes `ImagePullBackOff` failures on pod restarts.

**Alternatives**: (a) ECR pull-through cache — managed service, no custom
code, retains cached digests when upstream is unavailable. (b)
`images.paas.redhat.com` — Red Hat internal registry with existing
pull-through proxy orgs for some registries. (c) Artifactory — supports
any upstream registry as a remote repository.

**Rationale**: ECR pull-through cache requires private ECR registries,
which need IAM authentication — tokens expire every 12 hours, requiring a
CronJob for credential rotation on OCP pods. The S3 Lambda approach
serves content via a public HTTPS URL with zero authentication overhead.
`images.paas.redhat.com` was ruled out due to dependency on Red Hat IT
(rug-pulling risk if they decommission the proxy orgs). Artifactory was
not in use and would add a new service dependency. The S3 Lambda approach
reuses existing infrastructure (same Lambda, API Gateway, S3 bucket, custom
domain) and adds ~100 lines of OCI handler code ported from the proven CKI
dependency-archive.

### DDR-2: Origin-first (HEAD-first) strategy vs cache-first

**Decision**: Use an origin-first strategy: always check the upstream
registry first via HEAD request, redirect to upstream when available, and
serve from S3 only when upstream returns 404 or errors.

**Context**: The existing RPM cache uses a cache-first strategy (check S3
first, revalidate with origin). For OCI artifacts, the design choice
affects S3 egress costs and cache freshness.

**Alternatives**: (a) Cache-first — check S3 first, serve from cache if
present. (b) Origin-first — check upstream first, fall back to S3.

**Rationale**: OCI digests are content-addressed and immutable — there is
no staleness concern (unlike RPM repo metadata). The origin-first approach
minimizes S3 egress costs: when upstream is healthy (the common case), every
pull redirects directly to the upstream registry with zero S3 data transfer.
S3 is only read when upstream has garbage-collected the digest. This matches
the CKI dependency-archive pattern, which is proven in production. The cache
populates asynchronously on first access and serves as a safety net, not a
performance layer.

### DDR-3: Registry hostname in URL path vs per-registry subdomains

**Decision**: Encode the upstream registry hostname as the first path
segment after `v2/` (e.g.,
`koji-s3-cache.example.com/quay.io/openshift/origin-oauth-proxy@sha256:...`).

**Context**: The CKI dependency-archive assumes a single upstream registry
(`OCI_REGISTRY` env var). Hummingbird pulls from 4+ registries (quay.io,
docker.io, ghcr.io, registry.gitlab.com).

**Alternatives**: (a) Per-registry subdomains
(`quay-io.cache.example.com/...`) — standard OCI URL format but requires
wildcard TLS certificates and DNS records, potentially separate Lambda
deployments. (b) Registry hostname in the URL path — single endpoint,
handler extracts the registry from the first path segment.

**Rationale**: Encoding the registry in the path allows a single Lambda,
single API Gateway, single custom domain, and single TLS certificate to
serve all registries. The `/{path+}` catch-all route already captures
everything. The `v2/` prefix naturally separates OCI traffic from RPM
cache traffic. The kubelet constructs the correct path automatically from
the image reference. This approach requires zero new infrastructure beyond
the existing lambda-s3-cache stack.

### DDR-4: Jinja macro rewrite vs `registries.conf`

**Decision**: Rewrite image URLs at Jinja template render time using a
`cached_image()` macro, rather than configuring `registries.conf` on OCP
nodes.

**Context**: OCP pods need to pull from the cache instead of directly from
upstream registries. The standard approach is `registries.conf` mirror
configuration on the cluster nodes.

**Alternatives**: (a) Node-level `registries.conf` via MachineConfig. (b)
Jinja macro rewriting image refs at deploy time.

**Rationale**: Hummingbird has namespace-admin access only on OCP
clusters — no MachineConfig, no node-level changes. The Jinja macro
approach works entirely at the deployment manifest level:
`vars.d/*.yml` files continue to contain upstream refs (Renovate keeps
working unchanged), and the macro rewrites them to cache URLs at render
time. This is transparent to Renovate, requires no cluster-admin
privileges, and can be disabled with a single variable change.

### DDR-5: Scope limited to third-party images

**Decision**: Only cache third-party images. Hummingbird-owned images
(`quay.io/hummingbird-ci/*` and `registry.access.redhat.com/hi/*`) are
excluded.

**Context**: The cache exists to protect against upstream registries
garbage-collecting digest-pinned images.

**Alternatives**: Cache all images uniformly.

**Rationale**: Hummingbird-owned images have additional sha-based tags
(commit SHA, build ID) that persist independently of the floating
`:latest` tag. Old digests are not garbage-collected because these
immutable tags keep them reachable. Caching them would add S3 storage
and Lambda invocations with no benefit. The `cached_image()` Jinja macro
skips rewriting for `quay.io/hummingbird-ci/` prefixes.

### DDR-6: Separate OCI handler function vs extending RPM cache logic

**Decision**: Implement `_handle_oci_cacheable()` as a separate function
from the existing RPM cache logic (`lambda_handler` → `_handle_rpm`).

**Context**: Both OCI and RPM caching share the same S3 bucket, uploader
Lambda, and touch mechanism. The code could be unified.

**Alternatives**: Extend the existing RPM handler to handle OCI requests
by adding OCI-specific branches.

**Rationale**: The RPM cache has ETag-based revalidation and configurable
stale-cache behavior (origin vs cache) that do not apply to OCI — digests
are immutable by definition. Keeping the handlers separate avoids
accidentally changing RPM behavior and keeps each code path focused.
The OCI handler was ported from the CKI dependency-archive, which is
proven in production; staying close to the CKI code structure reduces
the risk of introducing untested edge cases.

## License

This project is licensed under the GNU General Public License v3.0 or later -
see the [LICENSE][license] file for details.

[readme]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/README.md
[license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE