Work Queue Service

Generic work queue service for Hummingbird. Manages work items through type-specific lifecycles with a shared processing framework. Currently handles MR creation; designed to support CVE remediation, advisory tracking, and other work item types.

For design decisions and architectural rationale, see Work Queue Service Design.

Architecture

The service runs as a Kubernetes deployment sharing the same PostgreSQL database as hummingbird-status and hummingbird-dashboard (with a service-specific Alembic migration chain).

flowchart LR
    clients["Internal services"] -- "POST /work-items/mr" --> svc["Work Queue Service"]
    gl_events["GitLab\n(events)"] -- webhook --> fwd["gitlab-event-forwarder"]
    fwd --> sns["SNS"]
    sns -- "lifecycle events" --> svc
    svc -- "Commits API,\ncreate MR" --> gl["GitLab\n(API)"]
    svc -- "shared DB" --> pg[("PostgreSQL")]

Design

Generic + detail tables

The schema separates queue machinery from type-specific data:

• work_items — generic: id, type, phase, pending_action, attempts, leased_until, error, timestamps. Shared by all work item types.
• mr_details — MR-specific: source, package, project, file_actions, commit_message, mr_iid, etc. Joined to work_items via foreign key.
• work_item_statuses — independent readiness dimensions with composite key (work_item_id, dimension, variant).

New work item types add their own detail table (e.g. cve_details) — no changes to the generic layer. SQLAlchemy joined table inheritance loads the correct subclass automatically via polymorphic_load="selectin".

Phases

Four generic phases track coarse lifecycle status:

Items start as active. Type-specific milestones (MR created, CI passed, MR merged) are tracked via dimensions and detail-table columns, not phases. Items remain active after MR creation – the active phase covers the entire lifecycle from submission through CI, approval, merge, and post-merge tracking. The decider returns no further actions until event-driven dimension updates (future) trigger the next step.

Dimensions

Independent readiness signals stored in work_item_statuses. Each dimension has a status (pending/running/passed/failed) and an optional variant for multi-instance signals (e.g. per-component Konflux builds).

Decider

A pure function registered per type that derives the next action from the work item’s current state:

def _mr_decide(item: MRWorkItem) -> str | None:
    if item.mr_iid is None:
        return "create_mr"
    return None

The decider only determines the next pending_action — it does not set phases or modify state.

Processing loop

The ProcessingManager runs per-item threads:

• Claims items where pending_action IS NOT NULL using SELECT ... FOR UPDATE SKIP LOCKED
• Serial per item — one action at a time per work item
• Parallel across items — independent items processed concurrently
• Heartbeat context manager extends leased_until during execution
• Executors apply state updates directly (phase, MR fields) and must not call session.commit() or session.rollback()
• After execution, the decider determines the next pending_action
• Periodic sweep reclaims items with expired leases

Adding a new work item type

1. Create a detail table and SQLAlchemy model (e.g. CveWorkItem)
2. Register a decider function: decider.register("cve", _cve_decide)
3. Register executor functions for each action
4. Add a submission endpoint: POST /api/work-items/cve

No changes needed to the processing loop, claiming, or generic API.

API

Endpoints

Responses include a nested details dict with type-specific fields:

{
  "id": "...",
  "type": "mr",
  "phase": "active",
  "details": {
    "source": "rpms-ci",
    "mr_iid": 42,
    "mr_url": "https://gitlab.com/org/group/project/-/merge_requests/42"
  }
}

Authentication

All callers authenticate via Authorization: Bearer tokens. The service validates each token in one of two ways:

X-Forwarded-* headers are never trusted on API paths. The proxy protects only the OAuth login flow (/oauth/start, /oauth/callback) and the /me endpoint, which bridges cookie sessions to Bearer tokens for browser and CLI users.

OAuth proxy deployment requirements

The oauth-proxy sidecar handles only the OAuth login flow. All /api/ paths bypass the proxy — the app handles auth itself. The proxy must be configured with these flags:

Why --cookie-refresh is critical

The proxy manages two independent lifetimes that are easy to confuse:

• Session cookie — controlled by the proxy (--cookie-expire, defaults to browser session). Contains the encrypted OCP access token.
• OCP access token — controlled by OCP (default 24h). The actual credential the app uses to call users/~.

The proxy does not know when the OCP token inside its cookie expires. It only checks whether the cookie is valid (signature, expiry). Without --cookie-refresh, the cookie outlives the token:

--cookie-refresh=1h tells the proxy to silently refresh the OCP token inside the cookie every hour, keeping it well within the 24h expiry. The workqueue-service also uses a TTL-bounded cache (5 minutes) for users/~ results, so expired-token failures are never permanent.

Authorization

Access rules are defined in the config file. Each rule maps a remote identity to permissions (read, retry, submit), optionally scoped to work item types. Rules use either groups (OCP group membership) or issuer+claims (JWT bearer). Claim matching follows Vault’s bound_claims pattern: all conditions AND, list values OR.

All permissions (including read) must be granted by a matching rule. Authenticated callers with no matching rules get 403. Multiple matching rules are merged (union of permissions).

See Configuration for the config file format.

API Documentation

Interactive API documentation available without authentication: /docs (Swagger UI), /redoc (ReDoc), /openapi.json.

CLI

export WORKQUEUE_SERVICE_URL=https://mr.apps.cluster.example.com
export WORKQUEUE_SERVICE_TOKEN=$(oc whoami -t)

# Update an existing file (local_path:repo_path)
workqueue-service submit \
  --package kernel \
  --project org/group/project \
  --change-kind sync \
  --file /path/to/sources.spec:sources.spec \
  --commit-message "Sync sources from upstream" \
  --source-branch mr-service/sync-1 --title "Sync sources" \
  --idempotency-key kernel-sync-1

# Create a new file
workqueue-service submit \
  --package kernel \
  --project org/group/project \
  --change-kind sync \
  --create /path/to/new-file.txt:new-file.txt \
  --commit-message "Add new file" \
  --source-branch mr-service/add-1 --title "Add file" \
  --idempotency-key kernel-add-1

# Check status
workqueue-service status --id <uuid>

# List active MR items
workqueue-service status --phase active

--url overrides WORKQUEUE_SERVICE_URL. Token is read exclusively from WORKQUEUE_SERVICE_TOKEN (not on the command line).

Configuration

Config file

The service reads its configuration from a YAML file at startup. The path defaults to /etc/workqueue/config.yml and can be overridden with WORKQUEUE_SERVICE_CONFIG. The service refuses to start if the file is missing or invalid (fail-closed).

authorization:
  rules:
    # Any in-cluster SA with the right audience can read
    - issuer: k8s
      claims:
        aud: workqueue-service
      permissions: [read]

    # Human users -- OCP group membership
    - groups:
        - konflux-hummingbird-admin-access
      permissions: [read, submit, retry]

    # Specific K8s SA for CI automation
    - issuer: k8s
      claims:
        aud: workqueue-service
        sub: system:serviceaccount:hummingbird--internal:rpms-ci
      permissions: [submit, retry]
      item_types: [mr]

    # GitLab CI OIDC
    - issuer: https://gitlab.com
      claims:
        aud: workqueue-service
        project_path: redhat/hummingbird/rpms
      permissions: [read, submit, retry]
      item_types: [mr]

handled_types:
  mr:
    gitlab_url: https://gitlab.com

Authorization rules

Each rule has either groups (OCP group membership from users/~ response) or issuer+claims (JWT bearer), not both.

• issuer determines which JWKS keys validate the token signature. Use k8s as shorthand for the auto-detected cluster OIDC issuer.
• claims matches JWT payload fields. All conditions must match (AND); list values match any (OR). Values are compared case-insensitively with string coercion (handles numeric project_id, boolean ref_protected).
• permissions grants read, retry, and/or submit.
• item_types optionally restricts the rule to specific work item types. Omit to apply to all types.
• Every issuer rule must include aud in claims to prevent cross-service token reuse.

Handled types

handled_types declares which work item types this instance serves. Only listed types get their routers and executors registered. The processing sweep only picks up items matching these types. Empty or omitted = handle nothing.

Each type can have per-type operational config:

Adding a new issuer

Add a rule with the issuer’s OIDC discovery URL. The service discovers the JWKS endpoint lazily on the first token from that issuer:

- issuer: https://token.actions.githubusercontent.com
  claims:
    aud: workqueue-service
    repository: org/repo
  permissions: [submit]
  types: [mr]

Environment variables

Development

Local Development

cd workqueue-service
./dev.sh db-start    # Start local PostgreSQL
./dev.sh db-migrate  # Run Alembic migrations
./dev.sh start       # Start the service
./dev.sh db-shell    # PostgreSQL shell
./dev.sh db-stop     # Stop PostgreSQL
./dev.sh db-reset    # Stop and delete volume

Running Tests

cd workqueue-service
pip install -e ".[dev]"
python -m unittest discover tests

Tests use testcontainers[postgres] locally (auto-detects Podman) or a CI-provided PostgreSQL sidecar via DATABASE_URL.

License

This project is licensed under the GNU General Public License v3.0 or later — see the LICENSE file for details.