# Work Queue Service

LLMS index: [llms.txt](/llms.txt) | Full content: [llms-full.txt](/llms-full.txt)

---

Generic work queue service for Hummingbird. Manages work items through
type-specific lifecycles with a shared processing framework. Currently
handles MR creation; designed to support CVE remediation, advisory
tracking, and other work item types.

For design decisions and architectural rationale, see
[Work Queue Service Design](workqueue-service-design.md).

## Architecture

The service runs as a Kubernetes deployment sharing the same PostgreSQL
database as hummingbird-status and hummingbird-dashboard (with a
service-specific Alembic migration chain).

```mermaid
flowchart LR
    clients["Internal services"] -- "POST /work-items/mr" --> svc["Work Queue Service"]
    gl_events["GitLab\n(events)"] -- webhook --> fwd["gitlab-event-forwarder"]
    fwd --> sns["SNS"]
    sns -- "lifecycle events" --> svc
    svc -- "Commits API,\ncreate MR" --> gl["GitLab\n(API)"]
    svc -- "shared DB" --> pg[("PostgreSQL")]
```

## Design

### Generic + detail tables

The schema separates queue machinery from type-specific data:

- **`work_items`** — generic: id, type, phase, pending_action, attempts,
  leased_until, error, timestamps. Shared by all work item types.
- **`mr_details`** — MR-specific: source, package, project, file_actions,
  commit_message, mr_iid, etc. Joined to work_items via foreign key.
- **`work_item_statuses`** — independent readiness dimensions with
  composite key (work_item_id, dimension, variant).

New work item types add their own detail table (e.g. `cve_details`) —
no changes to the generic layer. SQLAlchemy joined table inheritance
loads the correct subclass automatically via `polymorphic_load="selectin"`.

### Phases

Four generic phases track coarse lifecycle status:

| Phase       | Description                |
| ----------- | -------------------------- |
| `active`    | Being processed            |
| `completed` | All lifecycle steps done   |
| `failed`    | Error occurred (retryable) |
| `cancelled` | Cancelled                  |

Items start as `active`. Type-specific milestones (MR created, CI
passed, MR merged) are tracked via dimensions and detail-table columns,
not phases. Items remain `active` after MR creation -- the `active`
phase covers the entire lifecycle from submission through CI, approval,
merge, and post-merge tracking. The decider returns no further actions
until event-driven dimension updates (future) trigger the next step.

### Dimensions

Independent readiness signals stored in `work_item_statuses`. Each
dimension has a status (pending/running/passed/failed) and an optional
variant for multi-instance signals (e.g. per-component Konflux builds).

### Decider

A pure function registered per type that derives the next action from
the work item's current state:

```python
def _mr_decide(item: MRWorkItem) -> str | None:
    if item.mr_iid is None:
        return "create_mr"
    return None
```

The decider only determines the next `pending_action` — it does not set
phases or modify state.

### Processing loop

The `ProcessingManager` runs per-item threads:

- Claims items where `pending_action IS NOT NULL` using
  `SELECT ... FOR UPDATE SKIP LOCKED`
- **Serial per item** — one action at a time per work item
- **Parallel across items** — independent items processed concurrently
- Heartbeat context manager extends `leased_until` during execution
- Executors apply state updates directly (phase, MR fields) and must
  not call `session.commit()` or `session.rollback()`
- After execution, the decider determines the next `pending_action`
- Periodic sweep reclaims items with expired leases

### Adding a new work item type

1. Create a detail table and SQLAlchemy model (e.g. `CveWorkItem`)
2. Register a decider function: `decider.register("cve", _cve_decide)`
3. Register executor functions for each action
4. Add a submission endpoint: `POST /api/work-items/cve`

No changes needed to the processing loop, claiming, or generic API.

## API

### Endpoints

| Method | Path                         | Description                            |
| ------ | ---------------------------- | -------------------------------------- |
| `POST` | `/api/work-items/mr`         | Submit an MR work item                 |
| `GET`  | `/api/work-items/mr`         | List MR items (source, project, etc.)  |
| `GET`  | `/api/work-items`            | List all items (type, phase filters)   |
| `GET`  | `/api/work-items/{id}`       | Get item with nested details           |
| `POST` | `/api/work-items/{id}/retry` | Retry a failed item                    |
| `GET`  | `/api/work-items/me`         | Exchange OAuth cookie for Bearer token |

Responses include a nested `details` dict with type-specific fields:

```json
{
  "id": "...",
  "type": "mr",
  "phase": "active",
  "details": {
    "source": "rpms-ci",
    "mr_iid": 42,
    "mr_url": "https://gitlab.com/org/group/project/-/merge_requests/42"
  }
}
```

### Authentication

All callers authenticate via `Authorization: Bearer` tokens. The
service validates each token in one of two ways:

| Caller          | Token type         | Validation                                  |
| --------------- | ------------------ | ------------------------------------------- |
| K8s SA          | JWT (projected)    | JWKS signature verification                 |
| GitLab CI       | JWT (OIDC)         | JWKS signature verification                 |
| Human (web/CLI) | `sha256~...` (OCP) | OCP `users/~` API                           |
| Local dev       | none               | `WORKQUEUE_SERVICE_LOCAL_AUTH_USER` env var |

`X-Forwarded-*` headers are never trusted on API paths. The proxy
protects only the OAuth login flow (`/oauth/start`, `/oauth/callback`)
and the `/me` endpoint, which bridges cookie sessions to Bearer tokens
for browser and CLI users.

### OAuth proxy deployment requirements

The oauth-proxy sidecar handles only the OAuth login flow. All `/api/`
paths bypass the proxy — the app handles auth itself. The proxy must be
configured with these flags:

| Flag                                              | Purpose                                                                                                                                                 |
| ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--bypass-auth-except-for=^/(oauth/\|api/.*/me$)` | Proxy only protects OAuth flow + `/me`                                                                                                                  |
| `--pass-access-token`                             | Sets `X-Forwarded-Access-Token` on cookie-authenticated requests (required for `/me`). Cookie secret must be exactly 16, 24, or 32 bytes (AES key size) |
| `--pass-user-headers`                             | Sets `X-Forwarded-User` on protected paths                                                                                                              |
| `--cookie-refresh=1h`                             | Refreshes the OCP token inside the cookie before it expires (see below)                                                                                 |
| `--cookie-expire=24h`                             | Hard session limit, forces re-login                                                                                                                     |
| `--cookie-samesite=lax`                           | CSRF defense-in-depth for `/me`                                                                                                                         |

#### Why `--cookie-refresh` is critical

The proxy manages two independent lifetimes that are easy to confuse:

- **Session cookie** — controlled by the proxy (`--cookie-expire`,
  defaults to browser session). Contains the encrypted OCP access token.
- **OCP access token** — controlled by OCP (default 24h). The actual
  credential the app uses to call `users/~`.

The proxy does not know when the OCP token inside its cookie expires.
It only checks whether the _cookie_ is valid (signature, expiry).
Without `--cookie-refresh`, the cookie outlives the token:

| Time  | What happens                                                                                                                                                                                  |
| ----- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| T+0h  | User logs in. Proxy creates cookie with fresh OCP token (expires T+24h)                                                                                                                       |
| T+24h | OCP token expires. Cookie still valid.                                                                                                                                                        |
| T+25h | User makes request. Proxy sets `X-Forwarded-User` (valid cookie) and `X-Forwarded-Access-Token` (expired token). App calls `users/~` with expired token → 401 → anonymous → "not authorized". |

`--cookie-refresh=1h` tells the proxy to silently refresh the OCP token
inside the cookie every hour, keeping it well within the 24h expiry.
The workqueue-service also uses a TTL-bounded cache (5 minutes) for
`users/~` results, so expired-token failures are never permanent.

### Authorization

Access rules are defined in the config file. Each rule maps a remote
identity to permissions (`read`, `retry`, `submit`), optionally scoped
to work item types. Rules use either `groups` (OCP group membership) or
`issuer`+`claims` (JWT bearer). Claim matching follows Vault's
`bound_claims` pattern: all conditions AND, list values OR.

All permissions (including `read`) must be granted by a matching rule.
Authenticated callers with no matching rules get 403. Multiple matching
rules are merged (union of permissions).

See [Configuration](#configuration) for the config file format.

### API Documentation

Interactive API documentation available without authentication:
`/docs` (Swagger UI), `/redoc` (ReDoc), `/openapi.json`.

## CLI

```bash
export WORKQUEUE_SERVICE_URL=https://mr.apps.cluster.example.com
export WORKQUEUE_SERVICE_TOKEN=$(oc whoami -t)

# Update an existing file (local_path:repo_path)
workqueue-service submit \
  --package kernel \
  --project org/group/project \
  --change-kind sync \
  --file /path/to/sources.spec:sources.spec \
  --commit-message "Sync sources from upstream" \
  --source-branch mr-service/sync-1 --title "Sync sources" \
  --idempotency-key kernel-sync-1

# Create a new file
workqueue-service submit \
  --package kernel \
  --project org/group/project \
  --change-kind sync \
  --create /path/to/new-file.txt:new-file.txt \
  --commit-message "Add new file" \
  --source-branch mr-service/add-1 --title "Add file" \
  --idempotency-key kernel-add-1

# Check status
workqueue-service status --id <uuid>

# List active MR items
workqueue-service status --phase active
```

`--url` overrides `WORKQUEUE_SERVICE_URL`. Token is read exclusively
from `WORKQUEUE_SERVICE_TOKEN` (not on the command line).

## Configuration

### Config file

The service reads its configuration from a YAML file at startup. The
path defaults to `/etc/workqueue/config.yml` and can be overridden with
`WORKQUEUE_SERVICE_CONFIG`. The service refuses to start if the file is missing
or invalid (fail-closed).

```yaml
authorization:
  rules:
    # Any in-cluster SA with the right audience can read
    - issuer: k8s
      claims:
        aud: workqueue-service
      permissions: [read]

    # Human users -- OCP group membership
    - groups:
        - konflux-hummingbird-admin-access
      permissions: [read, submit, retry]

    # Specific K8s SA for CI automation
    - issuer: k8s
      claims:
        aud: workqueue-service
        sub: system:serviceaccount:hummingbird--internal:rpms-ci
      permissions: [submit, retry]
      item_types: [mr]

    # GitLab CI OIDC
    - issuer: https://gitlab.com
      claims:
        aud: workqueue-service
        project_path: redhat/hummingbird/rpms
      permissions: [read, submit, retry]
      item_types: [mr]

handled_types:
  mr:
    gitlab_url: https://gitlab.com
```

#### Authorization rules

Each rule has either `groups` (OCP group membership from `users/~`
response) or `issuer`+`claims` (JWT bearer), not both.

- `issuer` determines which JWKS keys validate the token signature.
  Use `k8s` as shorthand for the auto-detected cluster OIDC issuer.
- `claims` matches JWT payload fields. All conditions must match (AND);
  list values match any (OR). Values are compared case-insensitively
  with string coercion (handles numeric `project_id`, boolean
  `ref_protected`).
- `permissions` grants `read`, `retry`, and/or `submit`.
- `item_types` optionally restricts the rule to specific work item types.
  Omit to apply to all types.
- Every issuer rule must include `aud` in `claims` to prevent
  cross-service token reuse.

#### Handled types

`handled_types` declares which work item types this instance serves.
Only listed types get their routers and executors registered. The
processing sweep only picks up items matching these types. Empty or
omitted = handle nothing.

Each type can have per-type operational config:

| Type | Field        | Description                                         |
| ---- | ------------ | --------------------------------------------------- |
| `mr` | `gitlab_url` | GitLab instance URL (default: `https://gitlab.com`) |

#### Adding a new issuer

Add a rule with the issuer's OIDC discovery URL. The service discovers
the JWKS endpoint lazily on the first token from that issuer:

```yaml
- issuer: https://token.actions.githubusercontent.com
  claims:
    aud: workqueue-service
    repository: org/repo
  permissions: [submit]
  types: [mr]
```

### Environment variables

| Variable                            | Default                     | Description                             |
| ----------------------------------- | --------------------------- | --------------------------------------- |
| `DATABASE_URL`                      | —                           | PostgreSQL connection URL (required)    |
| `SENTRY_DSN`                        | —                           | Sentry DSN for error reporting          |
| `WORKQUEUE_SERVICE_CONFIG`          | `/etc/workqueue/config.yml` | Config file path                        |
| `WORKQUEUE_SERVICE_LOCAL_AUTH_USER` | —                           | Dev-only auth bypass (warns at startup) |
| `WORKQUEUE_SERVICE_URL`             | —                           | CLI: service URL                        |
| `WORKQUEUE_SERVICE_TOKEN`           | —                           | CLI: Bearer token                       |
| `WORKQUEUE_SERVICE_GITLAB_TOKEN`    | —                           | GitLab API token for MR executor        |

## Development

### Local Development

```bash
cd workqueue-service
./dev.sh db-start    # Start local PostgreSQL
./dev.sh db-migrate  # Run Alembic migrations
./dev.sh start       # Start the service
./dev.sh db-shell    # PostgreSQL shell
./dev.sh db-stop     # Stop PostgreSQL
./dev.sh db-reset    # Stop and delete volume
```

### Running Tests

```bash
cd workqueue-service
pip install -e ".[dev]"
python -m unittest discover tests
```

Tests use `testcontainers[postgres]` locally (auto-detects Podman) or a
CI-provided PostgreSQL sidecar via `DATABASE_URL`.

## License

This project is licensed under the GNU General Public License v3.0 or
later — see the [LICENSE][license] file for details.

[license]: https://gitlab.com/redhat/hummingbird/tools/-/blob/main/LICENSE