# Work Queue Service Design

LLMS index: [llms.txt](/llms.txt) | Full content: [llms-full.txt](/llms-full.txt)

---

Architectural design document for `hummingbird-mr-service`. Covers the
reasoning behind every major design choice so that future changes can be
made safely. For operational usage, see
[Hummingbird MR Service](hummingbird-mr-service.md).

## 1. Design Philosophy

Four principles shaped the service's architecture:

**Security boundary.** The service sits between untrusted callers (AI
agents, CI jobs, automation scripts) and trusted write operations (GitLab
pushes, MR creation). All changes are inspectable structured data, not
opaque blobs. Path blocklists and action validation run before any
GitLab write.

**Type-agnostic queue, type-specific behaviour.** The processing loop,
claiming, heartbeat, and lease management are generic. Type-specific
logic (MR creation, CVE remediation) lives in pluggable subpackages that
register deciders and executors at import time. Adding a new work item
type requires no changes to the generic layer.

**Single pending action.** Only one action per work item at a time.
After each action, the decider re-evaluates with current state. This
avoids queuing actions that assume a future state which may not hold.

**Phase + dimensions, not a linear state machine.** The coarse lifecycle
(active/completed/failed/cancelled) is separate from independent
readiness signals (CI, approval, Konflux build). Dimensions can regress
(force-push resets CI), which a linear state machine cannot express.

## 2. Architecture

Callers submit work items via REST API. The `ProcessingManager` drains
pending actions in per-item threads, dispatching to type-specific
executors. Currently the only executor creates GitLab MRs via the
Commits API.

```mermaid
flowchart LR
    callers["Callers\n(CI, CLI, agents)"] --> API["REST API\n+ Auth"]
    API --> DB[("PostgreSQL")]
    Processing["ProcessingManager"] --> DB
    Processing --> GL["GitLab\nCommits API"]
    API -.-> Processing
```

## 3. Design Decisions Record

### DDR-1: Phase + dimensions vs linear state machine

**Decision**: Replace the 12-state linear state machine with a phase +
dimensions model.

**Context**: The initial design used a single `state` column:
`pending → in_progress → created → mr_ci → approved → merged →
post_merge_ci → building → releasing → completed`. This broke because
post-creation concerns (CI, approval, Konflux build) are independent —
CI can pass before approval, approval can be revoked, force-push resets
CI but not approval.

**Alternatives**: (a) Composite states (`ci_passed_approval_pending`) —
exponential explosion. (b) Multiple boolean columns — rigid, schema
migration for each new signal.

**Rationale**: Phase tracks coarse lifecycle; independent dimensions in a
separate table track readiness signals with variant support for
multi-instance cases (per-component Konflux builds). Dimensions can
regress naturally. New dimension types need no schema migration.

### DDR-2: Four generic phases (active/completed/failed/cancelled)

**Decision**: Narrow from 6 MR-specific phases to 4 generic phases.

**Context**: The intermediate design had `pending`, `active`, `merged`,
`completed`, `failed`, `closed` — mixing generic concepts with
MR-specific milestones.

**Alternatives**: Keep MR-specific phases and add more for each type.

**Rationale**: Phase answers "is this item done, and how?" Type-specific
milestones (MR merged, CI passed) are tracked via dimensions and
detail-table columns. Items start as `active` — there is no `pending`
phase because the decider immediately assigns an action at creation time.

### DDR-3: Dimensions in a separate table with variant support

**Decision**: Store dimensions in `work_item_statuses` table, not as
columns on `work_items`.

**Context**: Initially considered a column per dimension
(`ci_status`, `approval_status`, etc.).

**Alternatives**: (a) Column per dimension — schema migration for each
new one. (b) JSONB column — flexible but loses CHECK constraints and
complicates queries. (c) Separate table with composite PK
`(work_item_id, dimension, variant)`.

**Rationale**: Option (c). New dimensions need no migration. Variants
handle multi-instance signals (multiple Konflux components). Each work
item only has rows for dimensions that apply to it. The "are all
dimensions satisfied?" query is a natural GROUP BY/HAVING.

### DDR-4: Decider as pure function, not state machine transitions

**Decision**: The decider is a pure function
`decide(item) → action | None` registered per type. It only determines
the next `pending_action`.

**Context**: The original design had a `TRANSITIONS` dict mapping
`state → frozenset[state]` with `validate_transition()`.

**Alternatives**: (a) Transition table — rigid, can't express "depends
on which dimensions passed". (b) Decider function — flexible, testable.

**Rationale**: The decider is trivially testable (given state, assert
action). State transitions come from executors (they apply phase changes
directly) and events (they update dimensions). The decider just picks
the next action based on current state.

### DDR-5: Single pending action, not a queue

**Decision**: One `pending_action` per work item at a time.

**Context**: Considered queuing multiple actions ("merge, then trigger
Konflux build").

**Alternatives**: Action queue table per work item.

**Rationale**: Queued actions assume future state that may not hold. If
"merge" fails, the queued "trigger Konflux build" is nonsense. After
each action, the decider re-evaluates with current state. The `attempts`
counter tracks consecutive failures, not total actions.

### DDR-6: Claiming via leased_until (reusable)

**Decision**: Time-based lease (`leased_until` column) for work item
claiming, reusable across all phases.

**Context**: The original design used `state = 'in_progress'` as a
one-time claiming mechanism.

**Alternatives**: (a) `claimed_by` + `claimed_at` columns — identifies
the worker but adds complexity. (b) `state = 'in_progress'` — ties
claiming to phase, not reusable. (c) `leased_until` timestamp — simple,
reusable, self-recovering.

**Rationale**: Option (c). Items can be reclaimed repeatedly throughout
their lifecycle. Expired leases are automatically reclaimable by other
workers. The heartbeat context manager extends the lease during
processing.

### DDR-7: Generic + detail tables (joined table inheritance)

**Decision**: Split `work_items` into generic queue table + per-type
detail tables (`mr_details`, future `cve_details`).

**Context**: The initial design had all MR-specific columns on the
`work_items` table.

**Alternatives**: (a) Single table with nullable type-specific columns —
sparse, doesn't enforce type constraints. (b) JSONB payload column —
flexible but loses type safety. (c) Joined table inheritance.

**Rationale**: Option (c). SQLAlchemy's `polymorphic_load="selectin"`
automatically loads the correct subclass. New types add their own detail
table — no changes to the generic layer. Type-specific columns have
proper types and constraints.

### DDR-8: Type-dispatched decider/executor registries

**Decision**: Registry pattern for deciders and executors. Each type
registers at import time via `decider.register("mr", _mr_decide)`.

**Context**: Needed a way to dispatch to type-specific logic without
the generic layer knowing about types.

**Alternatives**: (a) if/elif chains in the processing loop. (b)
Class-based dispatch (strategy pattern). (c) Registry functions.

**Rationale**: Option (c). Simple, explicit, no class hierarchy needed.
The `mr/__init__.py` import triggers registration. New types just add
their own `register()` calls.

### DDR-9: Structured file actions vs git patch_data

**Decision**: Replace opaque `patch_data` (BYTEA, git format-patch
output) with structured `file_actions` (JSONB) and `commit_message`.

**Context**: The MR service sits on the security boundary — it writes to
GitLab repos on behalf of untrusted callers. HUM-851 and the AI
Investigation design rules require pre-commit inspection (path
blocklists, content validation).

**Alternatives**: (a) Keep `patch_data`, clone repo, apply patch, then
inspect diff — inspection happens after creating a working tree with
credentials. (b) Parse patches with `unidiff` library — gives diffs not
full content, still needs original files. (c) Structured file actions.

**Rationale**: Option (c). Inspection is a pure function over data the
service already has — no clone, no git binary, no temp dirs. Path
blocklists and action validation run before any API call. The
`file_actions` JSONB column is also a queryable audit record.

### DDR-10: GitLab Commits API vs git subprocess

**Decision**: Use the GitLab Commits API for branch creation and file
commits. Delete `git_ops.py`.

**Context**: The original executor shelled out to `git` (clone, am,
push), requiring: git binary in the container, subprocess timeout
handling, GIT_ASKPASS credential management, temp directory lifecycle.

**Alternatives**: (a) Keep git subprocess with GIT_ASKPASS and timeout.
(b) GitLab Commits API.

**Rationale**: Option (b). `gitlab_sync.py` in the same repo already
proves the pattern. The Commits API's `start_branch` parameter creates
the branch automatically. No subprocess, no credentials in URLs, no temp
dirs. Callers have full file contents (not patches), so `git am`
semantics aren't needed.

### DDR-11: OIDC issuer from discovery, not hardcoded

**Decision**: Read the OIDC issuer from the `/.well-known/openid-configuration`
discovery response. Rewrite the JWKS URI to use `kubernetes.default.svc`.

**Context**: The initial implementation hardcoded
`issuer = "https://kubernetes.default.svc"`. This failed on EaaS clusters
where `--service-account-issuer` is
`https://oidc.op1.openshiftapps.com/...`. The JWKS URI from discovery
pointed to an IP:6443 that was unreachable from pods.

**Alternatives**: (a) Environment variable override
(`KUBE_OIDC_ISSUER`). (b) Read from discovery.

**Rationale**: Option (b). No configuration needed — works on any cluster
automatically. The JWKS fetch uses the path from discovery but rewrites
the host to `kubernetes.default.svc` (always reachable on port 443). SA
bearer token authenticates both requests (OpenShift blocks anonymous
JWKS access).

### DDR-12: `item_type` not `type`

**Decision**: Name the discriminator column `item_type` everywhere (DB,
ORM, API, schemas).

**Context**: `type` is a Python builtin. Using it as a function
parameter triggers ruff A002.

**Alternatives**: (a) Use `type` with `alias` in different contexts.
(b) Rename to `item_type` everywhere.

**Rationale**: Option (b). Clean rename, no aliases, no special casing.
Consistent at every layer.

### DDR-13: `source` field removed

**Decision**: Remove the `source` column from `mr_details`. The
`caller_identity` field (set automatically from authentication) serves
the same purpose.

**Context**: `source` was a user-declared label ("rpms-ci",
"dependency-updater") while `caller_identity` is the authenticated
identity. Having both was redundant.

**Alternatives**: Keep both for cases where a shared SA submits on
behalf of different logical sources.

**Rationale**: Removed until needed. `caller_identity` captures who
submitted. If per-source filtering is needed later, it can be
re-introduced as a field on `MRWorkItemCreate`.

### DDR-14: `attempts` counts consecutive failures, not total actions

**Decision**: `attempts` increments only on executor failure and resets
to 0 on success.

**Context**: The initial implementation incremented `attempts` before
every action execution. A work item chaining 4 actions (create_mr →
merge → trigger_build → complete) would exhaust MAX_ATTEMPTS on the
third action.

**Alternatives**: (a) Separate `total_actions` counter. (b) Only count
failures.

**Rationale**: Option (b). `attempts` answers "how many consecutive
times has this action failed?" — the signal for escalation. Total
actions executed is an audit concern, not a processing concern.

### DDR-15: Executor applies state directly; decider only picks next action

**Decision**: Executors directly modify `item.phase`, `item.mr_iid`,
etc. The decider only returns the next `pending_action`.

**Context**: Initially the processing loop set the phase based on
`ExecutorResult.success`. This mixed two concerns.

**Alternatives**: (a) Processing loop sets phase. (b) Decider sets
phase. (c) Executor sets phase.

**Rationale**: Option (c). The executor knows what happened ("I created
an MR, so phase stays active"). Events that update dimensions also
directly set state. The pattern is the same: something happens → state
is updated → decider evaluates for the next action. Symmetry with
event-driven dimension updates.

### DDR-16: Items stay `active` after MR creation

**Decision**: After the `create_mr` executor succeeds, the item remains
in `active` phase with `pending_action = None`.

**Context**: After MR creation, the decider returns `None` (no action).
The item sits in `active` indefinitely.

**Alternatives**: (a) Set phase to `completed` after MR creation.
(b) Keep `active` for future event-driven lifecycle tracking.

**Rationale**: Option (b). The `active` phase covers the entire lifecycle
— from submission through CI, approval, merge, and post-merge tracking.
The decider returns no actions until event-driven dimension updates
(future SQS consumer) trigger the next step.

### DDR-17: Audit log to S3, not database

**Decision**: Service-level audit (state transitions, decider decisions,
actions taken) goes to S3 as immutable objects.

**Context**: Raw webhook events are already archived by
`sns-s3-archiver`. The service needs its own audit trail for state
transitions and decider evaluations.

**Alternatives**: (a) Database table (`work_item_events`). (b) S3
immutable objects. (c) Both.

**Rationale**: Option (b). S3 is immutable and cheap. The operational
DB stays lean. Raw events are already in S3 via `sns-s3-archiver`;
service-level audit captures the service's reaction to those events.

### DDR-18: Agent direction — dumber, not smarter

**Decision**: The hummingbird-agent should become a pure compute engine.
The work queue service is the MR lifecycle orchestrator. The agent never
calls the work queue service; the work queue service calls the agent.

**Context**: HUM-857 proposed adding `create_merge_request` as a new
action in the agent's `actions.py`. This was the wrong direction.

**Alternatives**: (a) Agent gains more actions (push, merge, MR
management). (b) Agent becomes stateless compute; work queue service
orchestrates.

**Rationale**: Option (b). The agent keeps LLM loop, tool registry, and
sandbox — nothing else. Event routing, rate limiting, placeholder notes,
result posting, session management, and all GitLab writes move to the
work queue service. HUM-857 closed as Obsolete.

### DDR-19: Shared database

**Decision**: The work queue service uses the same `events` PostgreSQL
database as hummingbird-status and hummingbird-dashboard.

**Context**: Cross-service joins (e.g. `work_items` JOIN
`gitlab_merge_requests` JOIN `pipelineruns`) are useful for the dashboard.

**Alternatives**: Separate database per service.

**Rationale**: Shared DB with service-specific Alembic migration chains
(`alembic_version_mr_service`). Each service manages its own tables
independently. The migration chain is isolated so one service can evolve
its schema without affecting others.

## 4. Future Work

### Event-driven dimension updates

Subscribe to the existing SNS topic (same pattern as
hummingbird-agent's SQS subscription). CI pipeline events update the
`ci` dimension, MR approval events update `approval`, Konflux
PipelineRun events update `konflux_build`/`konflux_release`. Each event
upserts one `(dimension, variant)` row and triggers the decider.

### CVE work item type

Add `cve/` subpackage with `CveWorkItem`, CVE-specific decider
(analyze → label → create advisory MR → check VEX → close), and
CVE-specific executors. No changes to the generic layer.

### Authorization

Add authorization layer: OpenShift group check for humans (reusing
dashboard pattern), application-level SA identity allowlist for
in-cluster services. Read endpoints open to all authenticated callers,
write endpoints gated per type.

### Content validation and size limits

Add content-level inspection to the pre-commit checks: maximum file
size, binary detection, and content pattern matching. Currently only
path blocklists and action validation are enforced.

### Reconciliation sweep

Periodic job queries GitLab API for items in `active` phase, compares
actual state with dimension rows, corrects drift from missed events.

### Service rename

The service is currently named `hummingbird-mr-service` from its
origins as an MR-only service. A rename to `hummingbird-work-queue` or
similar is deferred until a second work item type is implemented.