Work Queue Service Design

Architectural design document for workqueue-service. Covers the reasoning behind every major design choice so that future changes can be made safely. For operational usage, see Work Queue Service.

1. Design Philosophy

Four principles shaped the service’s architecture:

Security boundary. The service sits between untrusted callers (AI agents, CI jobs, automation scripts) and trusted write operations (GitLab pushes, MR creation). All changes are inspectable structured data, not opaque blobs. Path blocklists and action validation run before any GitLab write.

Type-agnostic queue, type-specific behaviour. The processing loop, claiming, heartbeat, and lease management are generic. Type-specific logic (MR creation, CVE remediation) lives in pluggable subpackages that register deciders and executors at import time. Adding a new work item type requires no changes to the generic layer.

Single pending action. Only one action per work item at a time. After each action, the decider re-evaluates with current state. This avoids queuing actions that assume a future state which may not hold.

Phase + dimensions, not a linear state machine. The coarse lifecycle (active/completed/failed/cancelled) is separate from independent readiness signals (CI, approval, Konflux build). Dimensions can regress (force-push resets CI), which a linear state machine cannot express.

2. Architecture

Callers submit work items via REST API. The ProcessingManager drains pending actions in per-item threads, dispatching to type-specific executors. Currently the only executor creates GitLab MRs via the Commits API.

flowchart LR
    callers["Callers\n(CI, CLI, agents)"] --> API["REST API\n+ Auth"]
    API --> DB[("PostgreSQL")]
    Processing["ProcessingManager"] --> DB
    Processing --> GL["GitLab\nCommits API"]
    API -.-> Processing

3. Design Decisions Record

DDR-1: Phase + dimensions vs linear state machine

Decision: Replace the 12-state linear state machine with a phase + dimensions model.

Context: The initial design used a single state column: pending → in_progress → created → mr_ci → approved → merged → post_merge_ci → building → releasing → completed. This broke because post-creation concerns (CI, approval, Konflux build) are independent — CI can pass before approval, approval can be revoked, force-push resets CI but not approval.

Alternatives: (a) Composite states (ci_passed_approval_pending) — exponential explosion. (b) Multiple boolean columns — rigid, schema migration for each new signal.

Rationale: Phase tracks coarse lifecycle; independent dimensions in a separate table track readiness signals with variant support for multi-instance cases (per-component Konflux builds). Dimensions can regress naturally. New dimension types need no schema migration.

DDR-2: Four generic phases (active/completed/failed/cancelled)

Decision: Narrow from 6 MR-specific phases to 4 generic phases.

Context: The intermediate design had pending, active, merged, completed, failed, closed — mixing generic concepts with MR-specific milestones.

Alternatives: Keep MR-specific phases and add more for each type.

Rationale: Phase answers “is this item done, and how?” Type-specific milestones (MR merged, CI passed) are tracked via dimensions and detail-table columns. Items start as active — there is no pending phase because the decider immediately assigns an action at creation time.

DDR-3: Dimensions in a separate table with variant support

Decision: Store dimensions in work_item_statuses table, not as columns on work_items.

Context: Initially considered a column per dimension (ci_status, approval_status, etc.).

Alternatives: (a) Column per dimension — schema migration for each new one. (b) JSONB column — flexible but loses CHECK constraints and complicates queries. (c) Separate table with composite PK (work_item_id, dimension, variant).

Rationale: Option (c). New dimensions need no migration. Variants handle multi-instance signals (multiple Konflux components). Each work item only has rows for dimensions that apply to it. The “are all dimensions satisfied?” query is a natural GROUP BY/HAVING.

DDR-4: Decider as pure function, not state machine transitions

Decision: The decider is a pure function decide(item) → action | None registered per type. It only determines the next pending_action.

Context: The original design had a TRANSITIONS dict mapping state → frozenset[state] with validate_transition().

Alternatives: (a) Transition table — rigid, can’t express “depends on which dimensions passed”. (b) Decider function — flexible, testable.

Rationale: The decider is trivially testable (given state, assert action). State transitions come from executors (they apply phase changes directly) and events (they update dimensions). The decider just picks the next action based on current state.

DDR-5: Single pending action, not a queue

Decision: One pending_action per work item at a time.

Context: Considered queuing multiple actions (“merge, then trigger Konflux build”).

Alternatives: Action queue table per work item.

Rationale: Queued actions assume future state that may not hold. If “merge” fails, the queued “trigger Konflux build” is nonsense. After each action, the decider re-evaluates with current state. The attempts counter tracks consecutive failures, not total actions.

DDR-6: Claiming via leased_until (reusable)

Decision: Time-based lease (leased_until column) for work item claiming, reusable across all phases.

Context: The original design used state = 'in_progress' as a one-time claiming mechanism.

Alternatives: (a) claimed_by + claimed_at columns — identifies the worker but adds complexity. (b) state = 'in_progress' — ties claiming to phase, not reusable. (c) leased_until timestamp — simple, reusable, self-recovering.

Rationale: Option (c). Items can be reclaimed repeatedly throughout their lifecycle. Expired leases are automatically reclaimable by other workers. The heartbeat context manager extends the lease during processing.

DDR-7: Generic + detail tables (joined table inheritance)

Decision: Split work_items into generic queue table + per-type detail tables (mr_details, future cve_details).

Context: The initial design had all MR-specific columns on the work_items table.

Alternatives: (a) Single table with nullable type-specific columns — sparse, doesn’t enforce type constraints. (b) JSONB payload column — flexible but loses type safety. (c) Joined table inheritance.

Rationale: Option (c). SQLAlchemy’s polymorphic_load="selectin" automatically loads the correct subclass. New types add their own detail table — no changes to the generic layer. Type-specific columns have proper types and constraints.

DDR-8: Type-dispatched decider/executor registries

Decision: Registry pattern for deciders and executors. Each type registers at import time via decider.register("mr", _mr_decide).

Context: Needed a way to dispatch to type-specific logic without the generic layer knowing about types.

Alternatives: (a) if/elif chains in the processing loop. (b) Class-based dispatch (strategy pattern). (c) Registry functions.

Rationale: Option (c). Simple, explicit, no class hierarchy needed. The mr/__init__.py import triggers registration. New types just add their own register() calls.

DDR-9: Structured file actions vs git patch_data

Decision: Replace opaque patch_data (BYTEA, git format-patch output) with structured file_actions (JSONB) and commit_message.

Context: The MR service sits on the security boundary — it writes to GitLab repos on behalf of untrusted callers. HUM-851 and the AI Investigation design rules require pre-commit inspection (path blocklists, content validation).

Alternatives: (a) Keep patch_data, clone repo, apply patch, then inspect diff — inspection happens after creating a working tree with credentials. (b) Parse patches with unidiff library — gives diffs not full content, still needs original files. (c) Structured file actions.

Rationale: Option (c). Inspection is a pure function over data the service already has — no clone, no git binary, no temp dirs. Path blocklists and action validation run before any API call. The file_actions JSONB column is also a queryable audit record.

DDR-10: GitLab Commits API vs git subprocess

Decision: Use the GitLab Commits API for branch creation and file commits. Delete git_ops.py.

Context: The original executor shelled out to git (clone, am, push), requiring: git binary in the container, subprocess timeout handling, GIT_ASKPASS credential management, temp directory lifecycle.

Alternatives: (a) Keep git subprocess with GIT_ASKPASS and timeout. (b) GitLab Commits API.

Rationale: Option (b). gitlab_sync.py in the same repo already proves the pattern. The Commits API’s start_branch parameter creates the branch automatically. No subprocess, no credentials in URLs, no temp dirs. Callers have full file contents (not patches), so git am semantics aren’t needed.

DDR-11: OIDC issuer from discovery, not hardcoded

Decision: Read the OIDC issuer from the /.well-known/openid-configuration discovery response. Rewrite the JWKS URI to use kubernetes.default.svc.

Context: The initial implementation hardcoded issuer = "https://kubernetes.default.svc". This failed on EaaS clusters where --service-account-issuer is https://oidc.op1.openshiftapps.com/.... The JWKS URI from discovery pointed to an IP:6443 that was unreachable from pods.

Alternatives: (a) Environment variable override (KUBE_OIDC_ISSUER). (b) Read from discovery.

Rationale: Option (b). No configuration needed — works on any cluster automatically. The JWKS fetch uses the path from discovery but rewrites the host to kubernetes.default.svc (always reachable on port 443). SA bearer token authenticates both requests (OpenShift blocks anonymous JWKS access).

DDR-12: `item_type` not `type`

Decision: Name the discriminator column item_type everywhere (DB, ORM, API, schemas).

Context: type is a Python builtin. Using it as a function parameter triggers ruff A002.

Alternatives: (a) Use type with alias in different contexts. (b) Rename to item_type everywhere.

Rationale: Option (b). Clean rename, no aliases, no special casing. Consistent at every layer.

DDR-13: `source` field removed

Decision: Remove the source column from mr_details. The caller_identity field (set automatically from authentication) serves the same purpose.

Context: source was a user-declared label (“rpms-ci”, “dependency-updater”) while caller_identity is the authenticated identity. Having both was redundant.

Alternatives: Keep both for cases where a shared SA submits on behalf of different logical sources.

Rationale: Removed until needed. caller_identity captures who submitted. If per-source filtering is needed later, it can be re-introduced as a field on MRWorkItemCreate.

DDR-14: `attempts` counts consecutive failures, not total actions

Decision: attempts increments only on executor failure and resets to 0 on success.

Context: The initial implementation incremented attempts before every action execution. A work item chaining 4 actions (create_mr → merge → trigger_build → complete) would exhaust MAX_ATTEMPTS on the third action.

Alternatives: (a) Separate total_actions counter. (b) Only count failures.

Rationale: Option (b). attempts answers “how many consecutive times has this action failed?” — the signal for escalation. Total actions executed is an audit concern, not a processing concern.

DDR-15: Executor applies state directly; decider only picks next action

Decision: Executors directly modify item.phase, item.mr_iid, etc. The decider only returns the next pending_action.

Context: Initially the processing loop set the phase based on ExecutorResult.success. This mixed two concerns.

Alternatives: (a) Processing loop sets phase. (b) Decider sets phase. (c) Executor sets phase.

Rationale: Option (c). The executor knows what happened (“I created an MR, so phase stays active”). Events that update dimensions also directly set state. The pattern is the same: something happens → state is updated → decider evaluates for the next action. Symmetry with event-driven dimension updates.

DDR-16: Items stay `active` after MR creation

Decision: After the create_mr executor succeeds, the item remains in active phase with pending_action = None.

Context: After MR creation, the decider returns None (no action). The item sits in active indefinitely.

Alternatives: (a) Set phase to completed after MR creation. (b) Keep active for future event-driven lifecycle tracking.

Rationale: Option (b). The active phase covers the entire lifecycle — from submission through CI, approval, merge, and post-merge tracking. The decider returns no actions until event-driven dimension updates (future SQS consumer) trigger the next step.

DDR-17: Audit log to S3, not database

Decision: Service-level audit (state transitions, decider decisions, actions taken) goes to S3 as immutable objects.

Context: Raw webhook events are already archived by sns-s3-archiver. The service needs its own audit trail for state transitions and decider evaluations.

Alternatives: (a) Database table (work_item_events). (b) S3 immutable objects. (c) Both.

Rationale: Option (b). S3 is immutable and cheap. The operational DB stays lean. Raw events are already in S3 via sns-s3-archiver; service-level audit captures the service’s reaction to those events.

DDR-18: Agent direction — dumber, not smarter

Decision: The hummingbird-agent should become a pure compute engine. The work queue service is the MR lifecycle orchestrator. The agent never calls the work queue service; the work queue service calls the agent.

Context: HUM-857 proposed adding create_merge_request as a new action in the agent’s actions.py. This was the wrong direction.

Alternatives: (a) Agent gains more actions (push, merge, MR management). (b) Agent becomes stateless compute; work queue service orchestrates.

Rationale: Option (b). The agent keeps LLM loop, tool registry, and sandbox — nothing else. Event routing, rate limiting, placeholder notes, result posting, session management, and all GitLab writes move to the work queue service. HUM-857 closed as Obsolete.

DDR-19: Shared database

Decision: The work queue service uses the same events PostgreSQL database as hummingbird-status and hummingbird-dashboard.

Context: Cross-service joins (e.g. work_items JOIN gitlab_merge_requests JOIN pipelineruns) are useful for the dashboard.

Alternatives: Separate database per service.

Rationale: Shared DB with service-specific Alembic migration chains (alembic_version_mr_service). Each service manages its own tables independently. The migration chain is isolated so one service can evolve its schema without affecting others.

DDR-20: Pydantic config over cki-lib/jsonschema

Decision: Use Pydantic for config validation. Load config from a YAML file at startup.

Context: Needed structured config for authorization rules and per-type operational settings. Alternatives: cki-lib config helpers, jsonschema, or plain dicts.

Alternatives: (a) cki-lib — adds a dependency, the service stays self-contained without it. (b) jsonschema — separate schema file, no type-safe access. (c) Pydantic — already a dependency (API schemas).

Rationale: Option (c). Pydantic provides type-safe attribute access, defaults, cross-field validators (e.g. mutual exclusivity of groups vs issuer+claims), and optional JSON Schema export. No new dependencies.

DDR-21: Claim-based rules (Vault bound_claims pattern)

Decision: Authorization rules match on arbitrary JWT claims, not just sub. All claim conditions AND; list values OR.

Context: Studied JWT trust/authz config from AWS API Gateway, HashiCorp Vault, Kubernetes Structured Auth, Istio, Traefik Hub, and oauth2-proxy. All support matching on arbitrary claims.

Alternatives: (a) Subject allowlist — rigid, requires exact sub values. (b) Claim-based matching — flexible, supports project_path, ref, groups, etc.

Rationale: Option (b). Vault’s bound_claims pattern is proven and simple. Values are coerced to lowercase strings before comparison to handle int/string mismatches (GitLab’s numeric project_id) and bool/string mismatches (ref_protected: true).

DDR-22: Issuer per rule, audience as claim

Decision: issuer is the only special field on a token rule — it determines which JWKS keys to use. All other JWT conditions (including aud) are standard claim matches in the claims dict.

Context: Considered a separate jwt.issuers section and/or treating aud as a first-class field alongside issuer.

Alternatives: (a) Global jwt section with trusted issuers + separate rules section. (b) aud as a dedicated field on each rule. (c) Issuer per rule, audience as a claim.

Rationale: Option (c). Keeps each rule self-contained and self-documenting. The Pydantic validator enforces that every issuer rule includes aud in claims to prevent cross-service token reuse.

DDR-23: Unified rule model for groups and tokens

Decision: A single rules list where every rule maps a remote identity (groups or issuer+claims) to local permissions.

Context: The original design had admin_groups/retry_groups (local permission → remote groups) alongside rules (remote identity → local permissions) — opposite mapping directions.

Alternatives: Keep separate group and token rule formats.

Rationale: Unified model. Every rule follows the same structure: remote identity → local permissions. Pydantic cross-field validators enforce mutual exclusivity.

DDR-24: Bearer-only auth with users/~ self-authentication

Decision: All API auth uses Authorization: Bearer tokens. Opaque OCP tokens (sha256~...) are validated by calling the OCP users/~ API with the user’s own token (self-authenticating).

Context: The oauth-proxy cannot simultaneously forward cookie-session tokens AND pass through non-OCP Bearer tokens on the same path. The TokenReview API would be cleaner but requires system:auth-delegator ClusterRoleBinding — a cluster-scoped permission not available on managed OCP clusters.

Alternatives: (a) Trust X-Forwarded-User from the proxy on all paths — injection risk on bypassed paths. (b) TokenReview API — requires cluster RBAC. (c) users/~ self-authentication.

Rationale: Option (c). No special RBAC needed. The users/~ endpoint returns username and groups. Results are cached with a thread-safe TTL cache (5 minutes). The /me endpoint bridges OAuth cookie sessions to Bearer tokens.

DDR-25: handled_types as explicit opt-in

Decision: handled_types declares which work item types this instance serves. Empty means handle nothing.

Context: Different deployments may serve different types from the same container image. Types carry per-type config (e.g. gitlab_url for MR).

Alternatives: (a) Handle all types by default. (b) Explicit opt-in.

Rationale: Option (b). Prevents accidental catch-all deployments. The sweep returns early when handled_types is empty — no items stolen from other instances in a shared-database deployment.

DDR-26: k8s shorthand skipped when not in-cluster

Decision: Rules with issuer: k8s are silently skipped when the in-cluster CA bundle is absent.

Context: The K8s OIDC issuer URL varies by cluster and is auto-detected at runtime. Local development doesn’t have a cluster.

Alternatives: (a) Error on startup if k8s issuer can’t be resolved. (b) Skip silently.

Rationale: Option (b). Allows using a production config file for local development and testing without startup errors.

Decision: The oauth-proxy sidecar handles only the OAuth login flow. API endpoints bypass the proxy. A proxy-protected /me endpoint bridges cookie sessions to Bearer tokens. The proxy must be configured with --cookie-refresh=1h to prevent stale-token failures.

Context: The OCP oauth-proxy cannot simultaneously forward cookie-session tokens AND pass through non-OCP Bearer tokens on the same path. Protected paths validate Bearer tokens via TokenReview (requires cluster RBAC we cannot obtain) or reject non-OCP tokens. Bypassed paths pass all Bearer tokens through but ignore cookies. Additionally, the OCP access token embedded in the proxy’s session cookie has its own expiry (default 24h), independent of the cookie lifetime. Without --cookie-refresh, the cookie outlives the token, causing the proxy to forward expired credentials while still reporting the user as authenticated — the “logged in but not authorized” failure observed in the hummingbird-dashboard deployment.

Alternatives: (a) All paths protected — blocks non-OCP Bearer tokens. (b) --openshift-delegate-urls — requires system:auth-delegator ClusterRoleBinding. (c) App reads proxy cookies directly — tight coupling to proxy internals. (d) Login-only proxy with /me bridge.

Rationale: Option (d). The /me endpoint is the only proxy-protected API path. It reads X-Forwarded-Access-Token (set by the proxy from the cookie session) and returns the OCP access token to the caller. Subsequent API calls use Authorization: Bearer sha256~... on bypassed paths, where the app validates via the OCP users/~ API. --cookie-refresh=1h keeps the embedded token fresh. TTL-bounded caching (5 minutes) on users/~ results ensures expired-token failures are never permanent (unlike @functools.cache which caches failures indefinitely).

4. Future Work

Event-driven dimension updates

Subscribe to the existing SNS topic (same pattern as hummingbird-agent’s SQS subscription). CI pipeline events update the ci dimension, MR approval events update approval, Konflux PipelineRun events update konflux_build/konflux_release. Each event upserts one (dimension, variant) row and triggers the decider.

CVE work item type

Add cve/ subpackage with CveWorkItem, CVE-specific decider (analyze → label → create advisory MR → check VEX → close), and CVE-specific executors. No changes to the generic layer.

Content validation and size limits

Add content-level inspection to the pre-commit checks: maximum file size, binary detection, and content pattern matching. Currently only path blocklists and action validation are enforced.

Reconciliation sweep

Periodic job queries GitLab API for items in active phase, compares actual state with dimension rows, corrects drift from missed events.

Service rename

The service was renamed from hummingbird-mr-service to workqueue-service to reflect its generalized purpose. A further rename to hummingbird-work-queue or similar is deferred until a second work item type is implemented.