This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Work Queue Service

Generic work queue service for Hummingbird. Manages work items through type-specific lifecycles with a shared processing framework. Currently handles MR creation and updates; designed to support CVE remediation, advisory tracking, and other work item types.

For design decisions and architectural rationale, see Work Queue Service Design.

Architecture

The service runs as a Kubernetes deployment sharing the same PostgreSQL database as hummingbird-status and hummingbird-dashboard (with a service-specific Alembic migration chain).

flowchart TD
    clients["Internal services"] -- "POST /work-items/mr" --> svc["Work Queue Service"]
    gl_events["GitLab\n(events)"] -- webhook --> gl_fwd["gitlab-event-forwarder"]
    gl_fwd --> sns["SNS"]
    k8s_events["Konflux K8s\n(PipelineRun events)"] -- watch --> k8s_fwd["k8s-event-forwarder"]
    k8s_fwd --> sns
    sns -->|"filter: source=gitlab,\nevent_type=[merge_request, build]"| sqs["SQS Queue"]
    sns -->|"filter: source=kubernetes,\nkind=PipelineRun"| sqs
    sqs --> svc
    svc -- "Commits API\n(create + update)\nJobs API\n(trace + retry)" --> gl["GitLab\n(API)"]
    svc -- "PLR fallback fetch\n(label selector on SHA)" --> k8s_api["Konflux K8s API"]
    svc -- "archived PLR fetch" --> ka["Kubearchive"]
    svc -- "shared DB" --> pg[("PostgreSQL")]

The SQS consumer thread receives GitLab merge_request and pipeline events and Kubernetes PipelineRun events from the SNS topic via one filtered SQS subscription with a two-branch $or filter policy covering both source=gitlab and source=kubernetes. Events are stored in the pending_events table and processed by existing work-item threads, which map MR state changes to phase transitions, track Konflux build status, and trigger CI job retries on pipeline failure.

Design

Generic + detail tables

The schema separates queue machinery from type-specific data:

  • work_items — generic: id, type, phase, actionable, attempts, error, timestamps. Shared by all work item types.
  • mr_details — MR-specific: package, project, file_actions, commit_message, content_hash, mr_iid, etc. Joined to work_items via foreign key.
  • work_item_statuses — independent readiness dimensions with composite key (work_item_id, dimension, variant). Includes pending_until for deferred re-evaluation.

New work item types add their own detail table (e.g. cve_details) — no changes to the generic layer. SQLAlchemy joined table inheritance loads the correct subclass automatically via polymorphic_load="selectin".

Phases

Four generic phases track coarse lifecycle status:

Phase Description
active Being processed — the only mutable phase
completed All lifecycle steps done (terminal)
failed Error occurred (terminal, retryable)
cancelled Cancelled (terminal, retryable)

Items start as active. Only active items accept event-driven phase transitions and executor actions. The three terminal phases (completed, failed, cancelled) are immutable to events — recovery requires explicit action via the retry API. GitLab merge_request events drive transitions from active: merge -> completed, close -> cancelled.

Dimensions

Independent readiness signals stored in work_item_statuses. All dimensions follow a uniform needs_* trigger model: status is "true" (action needed) or "false" (handled). Absent dimensions are treated as actionable by the decider (!= "false"). Each dimension has an optional variant for multi-instance signals (e.g. per-component Konflux builds) and an optional pending_until timestamp for deferred re-evaluation (see Deferred dimensions below).

Dimensions are set by executors or event handlers via EventResult.dimensions; the generic handle_event layer applies event-driven dimensions and emits audit entries.

Exception — needs_plr: Unlike other dimensions, needs_plr requires explicit arming to "true" to be actionable. Absent or "false" means inactive (not awaiting PLR events). The decider checks dims.get("needs_plr") == "true", so a missing row is treated as inactive, not actionable.

Dimension schema

work_item_statuses:
  work_item_id: UUID  (PK, FK)
  dimension:    Text  (PK)
  variant:      Text  (PK, default="")
  status:       Text  (NOT NULL)       -- "true" or "false"
  updated_at:   DateTime(tz)
  detail:       JSONB (nullable)
  pending_until: DateTime(tz, nullable)

Unified state diagram

All dimensions share the same state machine:

stateDiagram-v2
    [*] --> absent : item created
    absent --> true : decider treats absent as actionable
    true --> true_deferred : executor (defer for later)
    true --> false : executor (done, no further checks)
    true_deferred --> true : sweep notifies (pending_until expired)
    true_deferred --> true : webhook/handler (clears pending_until)
    false --> true : webhook/handler (re-activates)
    note right of absent : no dimension row exists
    note right of true : status="true", pending_until=NULL
    note right of true_deferred : status="true", pending_until=future
    note right of false : status="false", decider ignores

Per-dimension transitions

Dimension On success (→ "false") On error / re-trigger (→ "true") Deferred?
needs_mr MR created or already exists — No
needs_expected_plrs Expected components fetched GitLab API error, unexpected error (malformed manifest permanently completes with error: "parse_error") Yes — 5 min error backoff
needs_plr All PLRs terminal or fetched PLR event or PLR fallback fetch (non-terminal) Yes — 30 min fallback poll
needs_dashboard Dashboard note posted or updated PLR event, PLR fallback fetch, transient GitLab 5xx No — always set “false”; re-armed by PLR handler/executor
needs_reconcile Reconciler completed Webhook (approved, pipeline, update), sweep expiry Yes — 30 min cycle / 5 min error backoff / 15 s→5 min recheck backoff for unchecked/checking/unknown
needs_empty_check Empty-diff check completed Sweep expiry Yes — 30 min cycle / 5 min error backoff
needs_ci_retry All jobs processed Build event (job failed), sweep expiry Yes — per-job not_before delay

Dimension detail schemas

The detail JSONB column carries executor-specific state. Dimensions without detail (needs_mr, needs_empty_check) store null; needs_reconcile stores null except during a merge-recheck streak. Frozen dataclass definitions live in workqueue_service/mr/dimension_types.py; typed accessor functions (get_plr_detail, get_dashboard_detail, get_expected_plrs_detail) deserialize the stored JSONB into the dataclass, or return None when absent. JSON arrays round-trip as tuples — NeedsCiRetryDetail.jobs and NeedsExpectedPlrsDetail.components are immutable.

Detail preservation. DimensionUpdate.detail defaults to a _PRESERVE sentinel. Event handlers that return a DimensionUpdate without explicit detail automatically keep whatever is stored in the database (via SQL COALESCE). Pass detail=None explicitly to clear, or a detail dataclass to replace. See Dimension update helpers below.

Updating detail. Event handlers update the existing detail instead of rebuilding it: read it via the typed accessor, then apply a mutation helper — NeedsPlrDetail.with_component() adds or replaces one component entry, NeedsCiRetryDetail.with_job() appends one job. Both return a new frozen instance; the original is untouched. Handlers pass the dataclass to DimensionUpdate, and executors that snapshot authoritative state (ensure_plr_status, ensure_expected_plrs, retry_ci_jobs) construct a fresh dataclass. crud.upsert_dimension serializes dataclasses via dataclasses.asdict() before the DB write, so callers never handle raw dicts.

needs_expected_plrs detail

Stored after fetching the Konflux trigger manifest for a commit SHA. Set by ensure_expected_plrs; read by ensure_plr_status, ensure_dashboard_note, and the PLR event handler.

Field Type Required Description
sha str | null yes Commit SHA the manifest was fetched for. null for items without a SHA.
components list[str] yes Expected component names (e.g. "caddy--hummingbird--default"). Empty when the repo has no trigger manifest (404).
error str no Set on parse failure; empty components list accompanies it.
needs_plr detail

Tracks observed PipelineRun state per component. Updated by the PLR SNS handler (merge per event) and ensure_plr_status (full snapshot from K8s/Kubearchive). Read by ensure_dashboard_note.

Field Type Required Description
components dict[str, PlrComponentInfo] yes Keyed by component name. Empty dict on init.

Each PlrComponentInfo value:

Field Type Required Description
status str yes PLR condition: "Running", "Succeeded", "Failed", "Unknown"
plr_name str yes Kubernetes PipelineRun resource name
start_time str yes ISO 8601 timestamp from status.startTime (pipeline start, not resource creation)
needs_dashboard detail

Stored after posting or updating the MR dashboard note. Set by ensure_dashboard_note; read on subsequent runs to decide whether to update or post. The PLR handler and ensure_plr_status re-arm needs_dashboard to "true" without passing detail, which preserves this detail.

Field Type Required Description
note_id int yes GitLab note ID for in-place updates
url str yes Dashboard URL included in the note body
note_body str no Last rendered note body; compared to skip no-op GitLab API calls
needs_reconcile detail

Optional — only present during a merge-recheck streak. When reconcile_mr observes detailed_merge_status in the fast family (unchecked, checking, unknown — recheck in flight), it re-arms the dimension at now + MERGE_RECHECK_SCHEDULE.delay(retry_step) (15 s doubling to a 5-minute cap) and bumps the step via advance_retry. The converge outcomes (merge attempted, real blocker, idle) clear the detail, so a fresh streak starts at 15 s again; the early exits (no MR yet, GitLab fetch error) leave the stored step untouched. Webhook re-arms preserve the stored step through the _PRESERVE sentinel.

Field Type Required Description
retry_step int yes Consecutive fast-family recheck count (starts at 1)
needs_ci_retry detail

Accumulates failed CI jobs eligible for retry. Set by the build event handler (appends per job) and retry_ci_jobs executor (removes processed jobs). Each entry represents one failed GitLab CI job.

Field Type Required Description
jobs list[CiRetryJobEntry] yes List of job entries (see below)

Each job entry:

Field Type Required Description
build_id int yes GitLab job ID
build_name str yes Job name (e.g. "test", "check")
build_failure_reason str yes GitLab failure reason (e.g. "script_failure")
build_finished_at str yes ISO 8601 timestamp when the job finished
retries_count int yes Number of retries already attempted
pipeline_id int yes GitLab pipeline ID the job belongs to
matcher_name str no Name of the retry matcher rule that matched this job
not_before str no ISO 8601 timestamp — earliest time the retry may be attempted

Deferred dimensions

When an executor or handler sets pending_until to a future timestamp alongside status="true", the decider treats the dimension as "false" (not actionable) until that time. The periodic sweep finds items with expired pending_until and notifies them — no synthetic events are inserted. The sweep wakes exactly at the earliest armed deadline — or after ~1 s when a deferred timer is already due, letting the next pass’s expired scan pick it up (see Processing loop) — so even short timers are honored precisely rather than rounded up to a sweep tick.

status pending_until Decider sees Meaning
(absent) — actionable No row — needs action
"true" NULL actionable Needs action now
"true" past actionable Sweep notified, ready to act
"true" future not actionable Deferred — sweep will notify
"false" any not actionable Done

Dimension update helpers

crud.upsert_dimension provides safe-by-default mutation patterns via its detail parameter:

  • Omitted — keep existing detail (SQL COALESCE on conflict); e.g. re-triggering needs_dashboard after a PLR event without clobbering note_id
  • Detail dataclass — replace with a full snapshot; e.g. initialising a dimension or completing an executor action
  • None — clear the stored detail explicitly

Partial detail updates do not use a merge helper — event handlers apply the frozen dataclass mutation helpers (with_component(), with_job()) to the existing detail instead, and executors construct a fresh dataclass for a full snapshot. crud.upsert_dimension takes a detail dataclass (or None) and serializes it via dataclasses.asdict() internally; callers never handle raw dicts.

See Dimension detail schemas for the _PRESERVE sentinel, frozen dataclass definitions, and typed accessors.

Dimension state (status plus optional deferred re-arm) is expressed via the DimensionState enum in workqueue_service/mr/constants.py. Passing a bare status string or timedelta to the state parameter is a type error; always use an enum member:

Member Status Delay Used for
DONE false — Dimension fulfilled, no action needed
PROCESS_IMMEDIATELY true — Needs action now
PROCESS_AFTER_TIMEOUT true 30 min PLR silence fallback, reconcile/empty-check cycle
PROCESS_AFTER_ERROR true 5 min Transient API error backoff, CI job not_before delay

For computed absolute times (e.g. the CI retry not_before), use DimensionState.process_at(when). It returns a ProcessAt wrapper that the CRUD layer resolves to status="true" with pending_until set to the given time.

Decider

A pure function registered per type that derives the next action from the work item’s current state. The decider reads the selectin-loaded item.statuses list (in-memory, no DB call) to check dimensions, pre-filtering deferred dimensions (where pending_until is in the future). See workqueue_service/mr/decider.py for the MR decider implementation.

The decider scans dimensions in priority order (DDR-45, DDR-48). Absent dimensions are treated as actionable (!= "false"). Deferred dimensions (pending_until in the future) are excluded from the scan. Adding a new concern = new dimension + handler, no decider changes.

The decider determines the next action to execute — it does not set phases or modify state.

MR upsert semantics

The MR submission endpoint uses upsert logic keyed on (project, source_branch). Callers always submit the current desired state; the service decides what to do based on existing work items:

stateDiagram-v2
    [*] --> created : no active item
    [*] --> unchanged : active, same hash
    [*] --> updated : active, different hash
    [*] --> retried : failed item

    created --> active : new work item
    unchanged --> active : no-op
    updated --> active : refresh fields
    retried --> active : reset phase
Upsert action Condition HTTP status Effect
created No active/failed item for (project, branch) 201 New work item, actionable = true
unchanged Active item exists, same content_hash 200 No-op, return existing item
updated Active item exists, different content_hash 200 Refresh file_actions/title/description, set actionable = true
retried Failed item exists for (project, branch) 200 Reset to active, clear error, update content, set actionable = true

Content hash. A SHA-256 of the normalized file_actions JSON (sorted by file_path + action, compact separators, sorted keys). Enables cheap same-content detection without comparing the full payload.

Idempotency key. Derived server-side as mr:{project}:{source_branch}:{generation} where generation is a counter per (project, source_branch) pair. A partial unique index ensures uniqueness only among non-terminal phases — completed and cancelled items can share a key prefix without blocking new items for the same branch.

Race handling. If two concurrent inserts race for the same idempotency key, the loser catches the IntegrityError, rolls back, and retries the full upsert — the winner is now visible and the correct action (unchanged/updated) is taken.

MR action chain

The MR work item lifecycle chains executor actions. After upsert, the processing loop calls the decider to select the first action; subsequent actions are also selected by the decider after each executor completes:

  • ensure_mr — creates or updates the MR. If mr_iid is not set, creates the branch, commit, and MR via the GitLab Commits API and applies the managed-by::workqueue scoped label. If mr_iid is already set, force-pushes updated content to the existing MR branch and syncs MR metadata (title, description). Detects empty diffs (before creation) or closes an empty MR after force-push. Captures the commit SHA as current_sha, resets the needs_plr dimension (clearing stale component state), and re-arms needs_expected_plrs. Sets needs_mr to "false" on success.
  • ensure_expected_plrs — fetches the Konflux trigger manifest (.tekton/trigger-manifest-pull-request.json) from the target project via the GitLab repository files API, parses the component-to-path mappings, and derives the set of affected components from the MR’s changed file paths. Stores the component list in the needs_expected_plrs dimension detail. If the manifest is missing (404), stores an empty list. Retries on transient API errors with a 5-minute backoff.
  • ensure_plr_status — fallback executor that fetches current PLR state from the Konflux K8s API (with Kubearchive fallback) using label selectors on the commit SHA. Fires on 30-minute pending_until re-arm when PipelineRun events have been silent. Writes the complete component snapshot into the needs_plr dimension detail and re-arms needs_dashboard to trigger a note update. Per component, the newest PipelineRun by status.startTime wins — after a retrigger, the older archived run does not overwrite the newer live one. If any component is still non-terminal, re-arms needs_plr with another 30-minute delay. Requires KONFLUX_KUBECONFIG_PATH and optionally KUBEARCHIVE_URL environment variables. The KubeArchive client reuses the bearer token from the kubeconfig’s auth settings.
  • ensure_dashboard_note — posts or updates an internal note on the MR with the Hummingbird dashboard link and a per-component Konflux build status table. The note uses the :robot: **Hummingbird Status** header shared with the dashboard-mr-linker Lambda. On first run, it creates the note and stores note_id in the dimension detail. On subsequent runs (triggered by PipelineRun events or the PLR fallback fetch), updates the existing note in place if the rendered body differs from the stored note_body. Always sets needs_dashboard to "false" on completion — re-arming is handled by the PLR SNS handler and the ensure_plr_status executor, which set needs_dashboard = "true" whenever PLR state changes.
  • check_empty_diff — checks whether an active MR’s diff became empty (target branch moved). Triggered by the needs_empty_check dimension via pending_until expiry (~every 30 minutes per item). If the diff is empty, closes the MR with an explanatory note, deletes the branch, and cancels the item. Sets needs_empty_check to "false" on completion. Transient GitLab API errors are logged but do not fail the item.
  • reconcile_mr — reconciles desired vs actual MR state. Triggered by the needs_reconcile dimension, set by webhooks (approved, unapproved, update, pipeline success/failure) and pending_until expiry. Merges directly when detailed_merge_status is mergeable and auto_merge is enabled. Syncs title and description if drifted. Re-arms needs_reconcile according to the merge outcome: a failed merge PUT retries immediately; unchecked/checking/unknown (a mergeability recheck in flight, completing seconds later) retry on an exponential schedule — 15 s doubling to a 5-minute cap, with the step tracked in the dimension detail; every other status keeps the 30-minute cycle.

Draft state. The draft submission option controls initial MR creation only. After creation, GitLab owns Ready/Draft state; update_mr and reconcile_mr preserve the current GitLab state while synchronizing title text. The details.draft response field reports the initial creation setting, not the live GitLab status.

After all actions complete, the decider returns None and the item sits in active phase waiting for event-driven lifecycle tracking (merge → completed, close → cancelled).

Label coordination with dashboard-mr-linker. The managed-by::workqueue label causes the dashboard-mr-linker Lambda to skip the MR, so the workqueue-service creates the dashboard note for its MRs. When dashboard_base_url is not configured, ensure_dashboard_note returns immediately without posting a note.

head_pipeline_id on mr_details

The current head pipeline ID is stored as a typed integer column (head_pipeline_id) on the mr_details table rather than as a dimension. Build events and the reconciler update it directly. The CI retry executor uses it to filter stale jobs from previous pipelines.

Processing loop

The ProcessingManager uses a bounded ThreadPoolExecutor (10 workers):

  • One loop per item — a loop takes a session-level advisory lock pg_try_advisory_lock(0x5751, item_id & 0x7FFFFFFF) on a pinned connection and holds it for its whole lifetime. Another loop (another pod, the sweep, or the local rerun path) returns immediately when the lock is held; duplicate submissions are cheap. Losers increment hummingbird_workqueue_processing_lock_skipped_total, labelled by item type. The key is the item UUID’s low 31 bits, so two unrelated items can rarely collide; the skipped item is re-spawned by the sweep once the winner finishes — delayed, never lost.
  • Serial per item — one action at a time per work item, across pods.
  • Parallel across items — independent items processed concurrently.
  • Per-item deadline — processing loops exit after 60 seconds to prevent stuck items from holding pool workers.
  • Executors apply state updates directly (phase, MR fields) and must not call session.commit() or session.rollback().
  • After execution, the loop re-derives the next action via the decider.
  • Crash recovery — the lock lives on a pinned database connection; if the pod or connection dies, PostgreSQL releases it. A death at any loop step (inside an executor or at the post-execution state write) aborts the loop; it never continues on a reconnected connection without the lock. The item stays actionable, so the periodic sweep re-spawns it. There is no lease, heartbeat, or TTL.
  • The periodic sweep submits actionable items and items with pending events or due pending_until dimensions (no synthetic events). It is deadline-driven: after each pass it sleeps until the earliest armed pending_until deadline (plus ~1 s so the deadline is already expired on wake), or for ~1 s when a deferred timer is already due (a single query returns the earliest future deadline and a past-due flag over one snapshot, so a timer expiring between the sweep’s scans cannot be missed by both), keeping the former 5-minute interval as a backstop cap for the un-timed legs — actionable crash recovery, pending events, and deadlines armed by other pods. A wake event cuts a long wait short whenever an enqueue or an executor commit arms an earlier deadline in-process; sweep-initiated submissions do not wake it, as they arm no timer themselves.

Connection pinning. Each loop checks out one connection and binds every session it opens to that connection, so the session lock survives the per-action commits. The engine pool is sized to the worker count with overflow headroom, and libpq keepalives plus tcp_user_timeout detect a dead server instead of leaving a lock on a half-open socket. The loop releases the lock explicitly before returning the connection to the pool; if the release fails, it invalidates the DBAPI connection, because a pooled close() alone would keep the session lock alive.

MR labeling

The executor always adds the managed-by::workqueue scoped label to every MR it creates, merged with any caller-provided labels. This label is a coordination signal: other services (e.g. dashboard-mr-linker) check for this label and skip managed MRs, since the workqueue-service handles dashboard note creation itself via the ensure_dashboard_note action.

Event processing

The SQS consumer thread receives events from the SNS topic — GitLab webhooks (merge_request, build) and Kubernetes resource events (PipelineRun) — and stores them in the pending_events table. Work-item threads drain and process these events as part of their main loop.

Consumer thread. A single daemon thread (poll_loop) long-polls SQS with MaxNumberOfMessages=10 and WaitTimeSeconds=20. For each message it decodes the SNS envelope (handles gzip+base64), resolves matching work items via the resolver registry, INSERTs into pending_events, commits, deletes the SQS message, and calls manager.spawn_loop() for each affected work item. The consumer is started only when WORKQUEUE_SERVICE_EVENTS_QUEUE_URL is set.

State mapping. MR state changes are mapped to phases:

GitLab state Phase Notes
merged completed Terminal
closed cancelled Terminal (recover via retry API)
opened — Ignored (only active items accept events)
locked — Ignored (transient during merge)

Only items in active phase accept event-driven transitions. Events for terminal items (completed, failed, cancelled) are silently discarded.

PipelineRun events. Kubernetes PipelineRun events (forwarded via the event forwarder → SNS) are resolved to MR work items by matching the commit SHA in the PipelineRun’s labels against mr_details.current_sha. Only type=build PipelineRuns are processed; test PipelineRuns are ignored. The handler also rejects events whose SHA label no longer matches the item’s current_sha (stale SNS delivery after a force-push). Each accepted event merges the component’s status into the needs_plr dimension’s JSONB detail and re-arms needs_dashboard to trigger a note update. Events are attributed to a PipelineRun by metadata.name: every event for the same PipelineRun carries the same status.startTime, so matching names always apply the newer status (a terminal run never regresses to a non-terminal one). For a different PipelineRun — a retrigger — the event is accepted only when its status.startTime is strictly newer (compared as parsed instants, so fractional-second formats order correctly). An event without a startTime cannot overwrite a component already recorded as Succeeded or Failed, nor an in-progress component that already has a stored startTime.

Konflux SHA labels. Build and test PipelineRuns carry different SHA labels — using the wrong label returns zero results:

PLR type SHA label Type label value
Build pipelinesascode.tekton.dev/sha build
Test pac.test.appstudio.openshift.io/sha test

The event resolver checks pipelinesascode.tekton.dev/sha first (build PLRs), falling back to pac.test.appstudio.openshift.io/sha for edge cases. The K8s/Kubearchive fallback fetcher (list_build_plrs_for_sha) queries only pipelinesascode.tekton.dev/sha with type=build, matching the label that build PLRs actually carry.

Out-of-order guard. The last_event_at column on work_items rejects events with updated_at older than the last processed event. Same-timestamp events pass the <= guard (processing is idempotent).

Exit coordination. The work-item exit decision runs under a SELECT ... FOR UPDATE row lock on work_items, so an enqueue that updates the item in the same transaction is either visible to the exit or blocks until the exit commits and then leaves actionable = true. The former pg_advisory_xact_lock insert/exit handshake was removed; the row lock covers every writer, including the API enqueue paths the handshake never covered. See DDR-51.

After a no-work exit commits, the loop releases the session lock and re-checks pending_events and the decider on the pinned connection. If a writer committed in that gap, the loop re-acquires the session lock and continues instead of returning. A writer that arrives after the re-check is caught by the sweep; the item remains actionable.

Sweep extension. The deadline-driven periodic sweep submits actionable items and items with unprocessed pending_events, providing startup recovery and a safety net behind the row-lock exit and the unlock/re-check. It wakes exactly at the earliest armed pending_until deadline; the 5-minute backstop interval covers everything un-timed (see DDR-52).

Concurrent drain tolerance. In steady state the per-item session lock prevents two pods from draining the same item. During the rolling update that introduced the session lock, an old pod still used leased_until while a new pod used the session lock, so both could briefly drain the same item. The drain therefore reads each listed event’s (id, event_type, event_body) before processing the first one and deletes by id — a row another pod already deleted is a no-op. An event may be applied by both pods; application is idempotent (dimension upserts, last_event_at guard), matching the at-least-once semantics of crash recovery. See DDR-50.

The same coexistence applied to executor actions, not just the drain: the two-key session lock deliberately could not conflict with the old single-key/lease path, so an old pod’s leased loop and a new pod’s locked loop could both run the same action for one item during the overlap. Read-and-upsert actions (reconcile_mr, ensure_plr_status, ensure_expected_plrs) converge on the same state. Creation actions (ensure_mr, ensure_dashboard_note) and retry_ci_jobs check their result field (e.g. mr_iid) before acting, so a duplicate requires both loops to pass that check before either commits — possible in the worst case, producing one duplicate external mutation (a second MR or note, or a second CI retry) for an item that was mid-action during the pod replacement. New pods deliberately ignored leased_until rather than skipping rows with an unexpired value: a lease written by a pod that died mid-rollout must not hide the item from the sweep. That coexistence window ended when the locking redesign finished rolling out; the contract migration drops the leased_until column, so the lease mechanism no longer exists. See DDR-51.

Adding a new work item type

  1. Create a detail table and SQLAlchemy model (e.g. CveWorkItem)
  2. Register a decider function: decider.register("cve", _cve_decide)
  3. Register executor functions for each action
  4. Add a submission endpoint: POST /api/work-items/cve

No changes needed to the processing loop, claiming, or generic API.

API

Endpoints

Method Path Description
POST /api/work-items/mr Upsert an MR work item (create, update, or skip)
GET /api/work-items/mr List MR items (source, project, etc.)
GET /api/work-items List all items (type, phase filters)
GET /api/work-items/{id} Get item with nested details
POST /api/work-items/{id}/retry Retry a failed item
GET /api/whoami Introspect caller identity and permissions
GET /oauth/start Initiate OAuth login (handled by oauth-proxy)
GET /oauth/callback OAuth provider callback (handled by oauth-proxy)
POST /oauth/token Exchange OAuth session cookie for Bearer token
GET /healthz Health check
GET /metrics Prometheus metrics

Responses include a nested details dict with type-specific fields:

id: "..."
type: mr
phase: active
details:
  source: rpms-ci
  mr_iid: 42
  mr_url: https://gitlab.com/org/group/project/-/merge_requests/42

Authentication

All callers authenticate via Authorization: Bearer tokens. The service validates each token in one of two ways:

Caller Token type Validation
K8s SA JWT (projected) JWKS signature verification
GitLab CI JWT (OIDC) JWKS signature verification
Human (web/CLI) sha256~... (OCP) OCP users/~ API
Local dev none WORKQUEUE_SERVICE_LOCAL_AUTH_USER env var

X-Forwarded-* headers are never trusted on API paths. The proxy protects only the OAuth login flow (/oauth/start, /oauth/callback) and the POST /oauth/token endpoint, which bridges cookie sessions to Bearer tokens for browser and CLI users. The /oauth/token endpoint uses POST (not GET) to prevent CSRF — browser-initiated GETs via <img> or <script> tags cannot trigger it.

OAuth proxy deployment requirements

The oauth-proxy sidecar handles only the OAuth login flow. All /api/ paths bypass the proxy — the app handles auth itself. The proxy must be configured with these flags:

Flag Purpose
--bypass-auth-except-for=^/oauth/ Proxy only protects the OAuth flow (login + token exchange)
--pass-access-token Sets X-Forwarded-Access-Token on cookie-authenticated requests (required for /oauth/token). Cookie secret must be exactly 16, 24, or 32 bytes (AES key size)
--pass-user-headers Sets X-Forwarded-User on protected paths
--cookie-refresh=1h Refreshes the OCP token inside the cookie before it expires (see below)
--cookie-expire=24h Hard session limit, forces re-login
--cookie-samesite=lax CSRF defense-in-depth for /oauth/token (POST already blocks the main vectors)

The proxy manages two independent lifetimes that are easy to confuse:

  • Session cookie — controlled by the proxy (--cookie-expire, defaults to browser session). Contains the encrypted OCP access token.
  • OCP access token — controlled by OCP (default 24h). The actual credential the app uses to call users/~.

The proxy does not know when the OCP token inside its cookie expires. It only checks whether the cookie is valid (signature, expiry). Without --cookie-refresh, the cookie outlives the token:

Time What happens
T+0h User logs in. Proxy creates cookie with fresh OCP token (expires T+24h)
T+24h OCP token expires. Cookie still valid.
T+25h User makes request. Proxy sets X-Forwarded-User (valid cookie) and X-Forwarded-Access-Token (expired token). App calls users/~ with expired token → 401 → anonymous → “not authorized”.

--cookie-refresh=1h tells the proxy to silently refresh the OCP token inside the cookie every hour, keeping it well within the 24h expiry. The workqueue-service also uses a TTL-bounded cache (5 minutes) for users/~ results, so expired-token failures are never permanent.

Authorization

Access rules are defined in the config file. Each rule maps a remote identity to permissions (read, retry, submit), optionally scoped to work item types. Rules use either groups (OCP group membership) or issuer+claims (JWT bearer). Claim matching follows Vault’s bound_claims pattern: all conditions AND, list values OR.

All permissions (including read) must be granted by a matching rule. Authenticated callers with no matching rules get 403. Multiple matching rules are merged (union of permissions).

See Configuration for the config file format.

Whoami introspection

GET /api/whoami returns the caller’s identity and effective permissions. Example response:

authentication:
  tier: ocp-user
  identity: admin-user
  groups:
    - konflux-hummingbird-admin-access
authorization:
  effective_permissions:
    mr:
      - read
      - retry
      - submit
  matching_rules:
    - groups:
        - konflux-hummingbird-admin-access
      permissions:
        - read
        - submit
        - retry
      item_types: []

API Documentation

Interactive API documentation available without authentication: /docs (Swagger UI), /redoc (ReDoc), /openapi.json.

Audit Log

State-changing operations (processing, API calls, auth decisions) emit audit entries. Every entry is logged at INFO level. When WORKQUEUE_SERVICE_AUDIT_S3_BUCKET is set, entries are additionally archived to S3 as gzipped JSON objects for long-term querying via Athena.

Audited events

Event type Source Description
action_started Worker Action execution began
action_completed Worker Action finished (success or failure)
action_decided Decider Decider assigned next action
action_no_op Decider Decider returned no next action
phase_change Worker / API / Events Phase transition (max attempts, submit, event)
item_updated API Work item content updated via upsert
dimension_change Events / Executor Dimension upsert (needs_dashboard posted, etc.)
retry_requested API Retry endpoint reset a failed item
auth_failed Auth Anonymous request rejected (401)
auth_denied Auth Authenticated caller lacks permission (403)

S3 key format

When S3 archival is enabled, entries are stored as gzipped JSON with an Athena-partitionable key layout:

{prefix}/YYYY/MM/DD/HH/MM/{timestamp}#{work_item_id}::{event_type}.json.gz

The ContentEncoding: gzip header is set so S3 API consumers get transparent decompression.

Field conventions

Follow these conventions when adding new audit calls:

  • actor: caller identity for API/auth events; thread name for worker events; "system" for decider/system-triggered transitions.
  • old_state: state before the change. Include phase and action when both are relevant. Omit on creation (no prior state) and on auth events (no work item state).
  • new_state: state after the change. Same keys as old_state. Omit fields that are None (filtered by to_dict()).
  • trigger: what caused this event. Always include source ("api", "worker", or "decider"). Add action for deliberate operations, reason for error/system transitions, error for failure details.
  • metadata: context orthogonal to the state transition — request info on auth events, type-specific data on submit. Not for duplicating state already in old_state/new_state.

CI Job Retry

When a CI pipeline on a managed MR fails, the service can automatically retry jobs that match known transient failure patterns. This is ported from the CKI pipeline-herder’s proven regex-based job log matching engine.

How it works

  1. GitLab sends a build (job) webhook when a CI job finishes
  2. The event handler appends the failed job to the needs_ci_retry dimension’s jobs list and sets status to "true"
  3. The decider selects the retry_ci_jobs executor
  4. The executor processes all jobs in the list, matching each against the config tree and retrying matching jobs via the GitLab Jobs API
  5. Jobs with non-zero delay set pending_until to the earliest per-job not_before timestamp — the sweep notifies when it expires
  6. Multiple failed jobs in the same pipeline accumulate in the list

needs_ci_retry dimension

The needs_ci_retry dimension follows the unified state machine (see Unified state diagram). Jobs with non-zero delay use the deferred path: status="true" with pending_until set to the earliest per-job not_before timestamp. The sweep notifies when any job’s delay elapses.

The true → true self-loop fires when another job in the same pipeline fails — the build event handler appends the new job to the existing list (clearing pending_until so the decider picks it up immediately).

The detail JSONB stores a jobs list. Each entry carries build_id, build_name, build_failure_reason, build_finished_at, retries_count, and pipeline_id — all from the build webhook payload. The executor deduplicates by build_id and skips jobs whose pipeline_id does not match the MR’s current head_pipeline_id. When a job has a non-zero delay, the executor stamps a per-job not_before ISO timestamp based on build_finished_at + retry_delays[retries_count]. The dimension’s pending_until is set to the earliest per-job not_before so the sweep fires as soon as any job is ready.

Matcher configuration

CI job retry is configured per type under handled_types.mr.job_retry. The config uses a recursive matcher tree where parent fields are inherited by children:

handled_types:
  mr:
    job_retry:
      retry_delays: ["0", "5m", "10m"]
      retry_limit: 5
      matchers:
        - name: runner-system-failure
          description: Job failed with runner_system_failure
          failure_reason: runner_system_failure
        - name: git-clone
          description: Git clone failure
          messages:
            - "fatal: unable to access"
            - '/error: RPC failed; curl 92 HTTP\/2 stream \d+/'

Matcher fields

Field Type Default Description
name string "" Matcher name (used in metrics and audit)
description string "" Human-readable description
failure_reason string "" Exact match on GitLab job failure_reason
job_name string "" Prefix match on job name
messages list[str] [] Literal substring or /regex/ on job trace
tail_lines int 300 Number of trace lines to scan (from the end)
retry_delays list[str] ["5m"] Escalating delays indexed by attempt ("0", "5m")
retry_limit int 3 Maximum retry count per job name
matchers list [] Nested child matchers (recursive)

Message matching

  • Literal: "fatal: unable to access" → substring match per line
  • Regex: '/error: RPC failed; curl \d+/' → re.search on joined lines (slash-delimited)

Retry safety

Before retrying a job, the executor checks:

  1. No newer job with the same name exists (prevents double retries)
  2. Retry limit not exceeded (count of jobs with the same name)

Retry timing

The retry_delays list is indexed by the job’s attempt count (0-based). When the index exceeds the list length, the last entry repeats. A delay of "0" means immediate retry; non-zero delays set pending_until to the computed not_before timestamp. The sweep notifies the item when pending_until expires, waking exactly at the earliest armed deadline.

Prometheus Metrics

The service exposes Prometheus metrics at /metrics on the same port as the API (no sidecar). Metrics follow the repo naming convention: generic operational metrics use the shared hummingbird_ prefix, service-specific metrics use hummingbird_workqueue_.

Generic operational metrics

Metric Type Labels Description
hummingbird_http_requests_total Counter method, path, status Total HTTP requests
hummingbird_http_request_duration_seconds Histogram method, path HTTP request latency
hummingbird_sqs_messages_received_total Counter — SQS messages pulled
hummingbird_sqs_messages_processed_total Counter status SQS messages processed

Service-specific metrics

Metric Type Labels Description
hummingbird_workqueue_work_items_created_total Counter item_type Work items submitted via API
hummingbird_workqueue_work_items_completed_total Counter item_type, result Action executions completed (success/failed)
hummingbird_workqueue_work_items_retried_total Counter item_type Work items retried via API
hummingbird_workqueue_action_duration_seconds Histogram action Executor action duration
hummingbird_workqueue_active_processing_threads Gauge — Running work item threads
hummingbird_workqueue_processing_lock_skipped_total Counter item_type Loops that lost the per-item session lock
hummingbird_workqueue_pending_events_inserted_total Counter — Pending events inserted into DB
hummingbird_workqueue_ci_jobs_retried_total Counter project, matcher_name CI jobs retried by the herder
hummingbird_workqueue_ci_retry_skipped_total Counter project, matcher_name, reason CI job retries skipped
hummingbird_workqueue_merge_rechecks_total Counter merge_status Fast-family merge recheck cycles by status

UUID path segments in HTTP metrics are normalized to /{id} to limit label cardinality. The /healthz and /metrics paths are excluded from HTTP metrics.

CLI

export WORKQUEUE_SERVICE_URL=https://mr.apps.cluster.example.com
export WORKQUEUE_SERVICE_TOKEN=$(oc whoami -t)

# Update an existing file (local_path:repo_path)
workqueue-service submit \
  --package kernel \
  --project org/group/project \
  --change-kind sync \
  --file /path/to/sources.spec:sources.spec \
  --commit-message "Sync sources from upstream" \
  --source-branch mr-service/sync-1 --title "Sync sources"

# Create a new file
workqueue-service submit \
  --package kernel \
  --project org/group/project \
  --change-kind sync \
  --create /path/to/new-file.txt:new-file.txt \
  --commit-message "Add new file" \
  --source-branch mr-service/add-1 --title "Add file"

# Check status
workqueue-service status --id <uuid>

# List active MR items
workqueue-service status --phase active

# Show identity and permissions
workqueue-service whoami

# Raw JSON output
workqueue-service whoami --json

--url overrides WORKQUEUE_SERVICE_URL. --no-auth skips all credential loading and sends unauthenticated requests (like aws --no-sign-request). Useful with whoami to verify anonymous access.

CLI authentication

The CLI resolves credentials in this order:

  1. WORKQUEUE_SERVICE_TOKEN env var — used directly as a Bearer token. Recommended for CI (OIDC/projected SA tokens).
  2. Cached token — if a valid token exists in the per-server cache, it is reused automatically.
  3. OAuth/Kerberos login — when neither env var nor cache is available, the CLI performs a synchronous headless SPNEGO login against the oauth-proxy, exchanges the session cookie for an OCP Bearer token via POST /oauth/token, and caches it for subsequent calls. Requires requests-gssapi and a valid Kerberos ticket (kinit).

Tokens are cached per-server under $XDG_CACHE_HOME/workqueue-service/tokens/ (defaults to ~/.cache/workqueue-service/tokens/) with 0600 permissions and a 20-hour TTL. If a cached token is rejected (HTTP 401), the CLI invalidates the cache and retries once with a fresh OAuth login.

Configuration

Config file

The service reads its configuration from a YAML file at startup. The path defaults to /etc/workqueue/config.yml and can be overridden with WORKQUEUE_SERVICE_CONFIG. The service refuses to start if the file is missing or invalid (fail-closed).

authorization:
  rules:
    # Any in-cluster SA with the right audience can read
    - issuer: k8s
      claims:
        aud: workqueue-service
      permissions: [read]

    # Human users -- OCP group membership
    - groups:
        - konflux-hummingbird-admin-access
      permissions: [read, submit, retry]

    # Specific K8s SA for CI automation
    - issuer: k8s
      claims:
        aud: workqueue-service
        sub: system:serviceaccount:hummingbird--internal:rpms-ci
      permissions: [submit, retry]
      item_types: [mr]

    # GitLab CI OIDC
    - issuer: https://gitlab.com
      claims:
        aud: workqueue-service
        project_path: redhat/hummingbird/rpms
      permissions: [read, submit, retry]
      item_types: [mr]

handled_types:
  mr:
    gitlab_url: https://gitlab.com
    projects:
      redhat/hummingbird/rpms:
        token_env: GITLAB_TOKEN_RPMS
      redhat/hummingbird/containers:
        token_env: GITLAB_TOKEN_CONTAINERS
      redhat/rhel/src:
        token_env: GITLAB_TOKEN_RHEL

Authorization rules

Each rule has either groups (OCP group membership from users/~ response) or issuer+claims (JWT bearer), not both.

  • issuer determines which JWKS keys validate the token signature. Use k8s as shorthand for the auto-detected cluster OIDC issuer.
  • claims matches JWT payload fields. All conditions must match (AND); list values match any (OR). Values are compared case-insensitively with string coercion (handles numeric project_id, boolean ref_protected).
  • permissions grants read, retry, and/or submit.
  • item_types optionally restricts the rule to specific work item types. Omit to apply to all types.
  • Every issuer rule must include aud in claims to prevent cross-service token reuse.

Handled types

handled_types declares which work item types this instance serves. Only listed types get their routers and executors registered. The processing sweep only picks up items matching these types. Empty or omitted = handle nothing.

Each type can have per-type operational config:

Type Field Description
mr gitlab_url GitLab instance URL (default: https://gitlab.com)
mr dashboard_base_url Dashboard URL for note creation (when unset, notes are not created)
mr job_retry CI job retry matcher config (when unset, retries are disabled)
mr projects Per-project/group token and URL configuration

Per-project token configuration

The projects map under handled_types.mr controls which GitLab API token the executor uses for each target project. Keys are GitLab project paths or group prefixes. Every project submitted to the service must match an entry – there is no default fallback token.

Matching order:

  1. Exact match on the full project path
  2. Longest group-prefix match on a / boundary (e.g. redhat/rhel/src matches redhat/rhel/src/kernel)

Each entry requires token_env (the name of the environment variable holding the GitLab API token) and optionally overrides gitlab_url:

handled_types:
  mr:
    gitlab_url: https://gitlab.com
    projects:
      redhat/hummingbird/rpms:
        token_env: GITLAB_TOKEN_RPMS
      redhat/hummingbird/containers:
        token_env: GITLAB_TOKEN_CONTAINERS
        gitlab_url: https://gitlab.example.com
      redhat/rhel/src:
        token_env: GITLAB_TOKEN_RHEL

At startup, the service validates that all referenced token_env variables exist in the process environment and refuses to start if any are missing.

Adding a new issuer

Add a rule with the issuer’s OIDC discovery URL. The service discovers the JWKS endpoint lazily on the first token from that issuer:

- issuer: https://token.actions.githubusercontent.com
  claims:
    aud: workqueue-service
    repository: org/repo
  permissions: [submit]
  types: [mr]

Environment variables

Variable Default Description
DATABASE_URL — PostgreSQL connection URL (required)
SENTRY_DSN — Sentry DSN for error reporting
WORKQUEUE_SERVICE_CONFIG /etc/workqueue/config.yml Config file path
WORKQUEUE_SERVICE_LOCAL_AUTH_USER — Dev-only auth bypass (warns at startup)
WORKQUEUE_SERVICE_URL — CLI: service URL
WORKQUEUE_SERVICE_TOKEN — CLI: Bearer token
(per-project token_env) — GitLab API token(s) for MR executor (set in config projects)
WORKQUEUE_SERVICE_EVENTS_QUEUE_URL — SQS queue URL for events (when unset, event consumer is disabled)
KONFLUX_KUBECONFIG_PATH — Path to Konflux kubeconfig for PLR fallback fetch
KUBEARCHIVE_URL — Kubearchive API URL for historical PLR lookups
WORKQUEUE_SERVICE_AUDIT_S3_BUCKET — S3 bucket for audit log (production)
WORKQUEUE_SERVICE_AUDIT_S3_PREFIX workqueue-service-audit Key prefix within the audit S3 bucket
AWS_ACCESS_KEY_ID — AWS credentials for S3 audit archival
AWS_SECRET_ACCESS_KEY — AWS credentials for S3 audit archival
AWS_DEFAULT_REGION — AWS region for S3 audit archival

Development

Local Development

cd workqueue-service
./dev.sh db-start    # Start local PostgreSQL
./dev.sh db-migrate  # Run Alembic migrations
./dev.sh start       # Start the service
./dev.sh db-shell    # PostgreSQL shell
./dev.sh db-stop     # Stop PostgreSQL
./dev.sh db-reset    # Stop and delete volume

Running Tests

cd workqueue-service
pip install -e ".[dev]"
python -m unittest discover tests

Tests use testcontainers[postgres] locally (auto-detects Podman) or a CI-provided PostgreSQL sidecar via DATABASE_URL.

License

This project is licensed under the GNU General Public License v3.0 or later — see the LICENSE file for details.

1 - Work Queue Service Design

Architectural design document for workqueue-service. Covers the reasoning behind every major design choice so that future changes can be made safely. For operational usage, see Work Queue Service.

1. Design Philosophy

Four principles shaped the service’s architecture:

Security boundary. The service sits between untrusted callers (AI agents, CI jobs, automation scripts) and trusted write operations (GitLab pushes, MR creation). All changes are inspectable structured data, not opaque blobs. Path blocklists and action validation run before any GitLab write.

Type-agnostic queue, type-specific behaviour. The processing loop and per-item locking (a loop-lifetime session advisory lock) are generic. Type-specific logic (MR creation, CVE remediation) lives in pluggable subpackages that register deciders and executors at import time. Adding a new work item type requires no changes to the generic layer.

Single pending action. Only one action per work item at a time. After each action, the decider re-evaluates with current state. This avoids queuing actions that assume a future state which may not hold.

Phase + dimensions, not a linear state machine. The coarse lifecycle (active/completed/failed/cancelled) is separate from independent readiness signals (CI, approval, Konflux build). Dimensions can regress (force-push resets CI), which a linear state machine cannot express.

2. Architecture

Callers submit work items via REST API. The ProcessingManager drains pending actions in per-item threads, dispatching to type-specific executors. Executors create and update GitLab MRs via the Commits API.

flowchart LR
    callers["Callers\n(CI, CLI, agents)"] --> API["REST API\n+ Auth"]
    API --> DB[("PostgreSQL")]
    Processing["ProcessingManager"] --> DB
    Processing --> GL["GitLab\nCommits API"]
    API -.-> Processing

3. Design Decisions Record

See Work Queue Service Design Decisions for all DDR entries covering data model, auth, processing, event pipeline, and MR lifecycle decisions.

4. Future Work

Event-driven dimension updates

The dimension mechanism is implemented with all MR dimensions following the uniform needs_* model (DDR-48). MR state events drive phase transitions (merge → completed, close → cancelled). Konflux PipelineRun events merge per-component state into the needs_plr dimension’s JSONB detail and re-arm needs_dashboard to trigger dashboard note updates. A 30-minute fallback executor (ensure_plr_status) fetches current PLR state from K8s/Kubearchive to catch missed events.

Non-managed MR tracking

Design how the service handles MRs it did not create (MRs without the managed-by::workqueue label). Requires subscribing to note events so the service can discover dashboard notes posted by the Lambda and adopt them as needs_dashboard dimensions. Prerequisite for retiring the Lambda.

Status rendering

The ensure_dashboard_note executor now renders a per-component Konflux build status table (using needs_plr dimension JSONB detail and the expected components from needs_expected_plrs) and updates the existing note in place via note_id.

CVE work item type

Add cve/ subpackage with CveWorkItem, CVE-specific decider (analyze → label → create advisory MR → check VEX → close), and CVE-specific executors. No changes to the generic layer.

Content validation and size limits

Add content-level inspection to the pre-commit checks: maximum file size, binary detection, and content pattern matching. Currently only path blocklists and action validation are enforced.

Reconciliation sweep

Note: The reconcile_mr executor (DDR-46) now provides per-item reconciliation. A full sweep (querying GitLab for all active items in bulk) remains a future optimization for catching items that fall through individual reconciliation.

Service rename

The service was renamed from hummingbird-mr-service to workqueue-service to reflect its generalized purpose. A further rename to hummingbird-work-queue or similar is deferred until a second work item type is implemented.

2 - Work Queue Service Design Decisions

Design Decision Registry (DDR) for workqueue-service — every major design choice, the alternatives considered, and why the chosen approach won. For the design overview, see Work Queue Service Design. For operational usage, see Work Queue Service.

DDR-1: Phase + dimensions vs linear state machine

Decision: Replace the 12-state linear state machine with a phase + dimensions model.

Context: The initial design used a single state column: pending → in_progress → created → mr_ci → approved → merged → post_merge_ci → building → releasing → completed. This broke because post-creation concerns (CI, approval, Konflux build) are independent — CI can pass before approval, approval can be revoked, force-push resets CI but not approval.

Alternatives: (a) Composite states (ci_passed_approval_pending) — exponential explosion. (b) Multiple boolean columns — rigid, schema migration for each new signal.

Rationale: Phase tracks coarse lifecycle; independent dimensions in a separate table track readiness signals with variant support for multi-instance cases (per-component Konflux builds). Dimensions can regress naturally. New dimension types need no schema migration.

DDR-2: Four generic phases (active/completed/failed/cancelled)

Decision: Narrow from 6 MR-specific phases to 4 generic phases.

Context: The intermediate design had pending, active, merged, completed, failed, closed — mixing generic concepts with MR-specific milestones.

Alternatives: Keep MR-specific phases and add more for each type.

Rationale: Phase answers “is this item done, and how?” Type-specific milestones (MR merged, CI passed) are tracked via dimensions and detail-table columns. Items start as active — there is no pending phase because the decider immediately assigns an action at creation time.

DDR-3: Dimensions in a separate table with variant support

Decision: Store dimensions in work_item_statuses table, not as columns on work_items.

Context: Initially considered a column per dimension (ci_status, approval_status, etc.).

Alternatives: (a) Column per dimension — schema migration for each new one. (b) JSONB column — flexible but loses CHECK constraints and complicates queries. (c) Separate table with composite PK (work_item_id, dimension, variant).

Rationale: Option (c). New dimensions need no migration. Variants handle multi-instance signals (multiple Konflux components). Each work item only has rows for dimensions that apply to it. The “are all dimensions satisfied?” query is a natural GROUP BY/HAVING.

DDR-4: Decider as pure function, not state machine transitions

Decision: The decider is a pure function decide(item) → action | None registered per type. It only determines the next action to execute.

Context: The original design had a TRANSITIONS dict mapping state → frozenset[state] with validate_transition().

Alternatives: (a) Transition table — rigid, can’t express “depends on which dimensions passed”. (b) Decider function — flexible, testable.

Rationale: The decider is trivially testable (given state, assert action). State transitions come from executors (they apply phase changes directly) and events (they update dimensions). The decider just picks the next action based on current state.

DDR-5: Single pending action, not a queue

Decision: One pending action per work item at a time.

Context: Considered queuing multiple actions (“merge, then trigger Konflux build”).

Alternatives: Action queue table per work item.

Rationale: Queued actions assume future state that may not hold. If “merge” fails, the queued “trigger Konflux build” is nonsense. After each action, the decider re-evaluates with current state. The attempts counter tracks consecutive failures, not total actions.

DDR-6: Claiming via leased_until (reusable)

Superseded by DDR-51. Work items are owned by a loop-lifetime session advisory lock, and the contract migration drops the leased_until column. See DDR-51 for the design and its motivation.

DDR-7: Generic + detail tables (joined table inheritance)

Decision: Split work_items into generic queue table + per-type detail tables (mr_details, future cve_details).

Context: The initial design had all MR-specific columns on the work_items table.

Alternatives: (a) Single table with nullable type-specific columns — sparse, doesn’t enforce type constraints. (b) JSONB payload column — flexible but loses type safety. (c) Joined table inheritance.

Rationale: Option (c). SQLAlchemy’s polymorphic_load="selectin" automatically loads the correct subclass. New types add their own detail table — no changes to the generic layer. Type-specific columns have proper types and constraints.

DDR-8: Type-dispatched decider/executor registries

Decision: Registry pattern for deciders and executors. Each type registers at import time via decider.register("mr", _mr_decide).

Context: Needed a way to dispatch to type-specific logic without the generic layer knowing about types.

Alternatives: (a) if/elif chains in the processing loop. (b) Class-based dispatch (strategy pattern). (c) Registry functions.

Rationale: Option (c). Simple, explicit, no class hierarchy needed. The mr/__init__.py import triggers registration. New types just add their own register() calls.

DDR-9: Structured file actions vs git patch_data

Decision: Replace opaque patch_data (BYTEA, git format-patch output) with structured file_actions (JSONB) and commit_message.

Context: The MR service sits on the security boundary — it writes to GitLab repos on behalf of untrusted callers. HUM-851 and the AI Investigation design rules require pre-commit inspection (path blocklists, content validation).

Alternatives: (a) Keep patch_data, clone repo, apply patch, then inspect diff — inspection happens after creating a working tree with credentials. (b) Parse patches with unidiff library — gives diffs not full content, still needs original files. (c) Structured file actions.

Rationale: Option (c). Inspection is a pure function over data the service already has — no clone, no git binary, no temp dirs. Path blocklists and action validation run before any API call. The file_actions JSONB column is also a queryable audit record.

DDR-10: GitLab Commits API vs git subprocess

Decision: Use the GitLab Commits API for branch creation and file commits. Delete git_ops.py.

Context: The original executor shelled out to git (clone, am, push), requiring: git binary in the container, subprocess timeout handling, GIT_ASKPASS credential management, temp directory lifecycle.

Alternatives: (a) Keep git subprocess with GIT_ASKPASS and timeout. (b) GitLab Commits API.

Rationale: Option (b). gitlab_sync.py in the same repo already proves the pattern. The Commits API’s start_branch parameter creates the branch automatically. No subprocess, no credentials in URLs, no temp dirs. Callers have full file contents (not patches), so git am semantics aren’t needed.

DDR-11: OIDC issuer from discovery, not hardcoded

Decision: Read the OIDC issuer from the /.well-known/openid-configuration discovery response. Rewrite the JWKS URI to use kubernetes.default.svc.

Context: The initial implementation hardcoded issuer = "https://kubernetes.default.svc". This failed on EaaS clusters where --service-account-issuer is https://oidc.op1.openshiftapps.com/.... The JWKS URI from discovery pointed to an IP:6443 that was unreachable from pods.

Alternatives: (a) Environment variable override (KUBE_OIDC_ISSUER). (b) Read from discovery.

Rationale: Option (b). No configuration needed — works on any cluster automatically. The JWKS fetch uses the path from discovery but rewrites the host to kubernetes.default.svc (always reachable on port 443). SA bearer token authenticates both requests (OpenShift blocks anonymous JWKS access).

DDR-12: item_type not type

Decision: Name the discriminator column item_type everywhere (DB, ORM, API, schemas).

Context: type is a Python builtin. Using it as a function parameter triggers ruff A002.

Alternatives: (a) Use type with alias in different contexts. (b) Rename to item_type everywhere.

Rationale: Option (b). Clean rename, no aliases, no special casing. Consistent at every layer.

DDR-13: source field removed

Decision: Remove the source column from mr_details. The caller_identity field (set automatically from authentication) serves the same purpose.

Context: source was a user-declared label (“rpms-ci”, “dependency-updater”) while caller_identity is the authenticated identity. Having both was redundant.

Alternatives: Keep both for cases where a shared SA submits on behalf of different logical sources.

Rationale: Removed until needed. caller_identity captures who submitted. If per-source filtering is needed later, it can be re-introduced as a field on MRWorkItemCreate.

DDR-14: attempts counts consecutive failures, not total actions

Decision: attempts increments only on executor failure and resets to 0 on success.

Context: The initial implementation incremented attempts before every action execution. A work item chaining 4 actions (ensure_mr → merge → trigger_build → complete) would exhaust MAX_ATTEMPTS on the third action.

Alternatives: (a) Separate total_actions counter. (b) Only count failures.

Rationale: Option (b). attempts answers “how many consecutive times has this action failed?” — the signal for escalation. Total actions executed is an audit concern, not a processing concern.

DDR-15: Executor applies state directly; decider only picks next action

Decision: Executors directly modify item.phase, item.mr_iid, etc. The decider only returns the next action to execute.

Context: Initially the processing loop set the phase based on ExecutorResult.success. This mixed two concerns.

Alternatives: (a) Processing loop sets phase. (b) Decider sets phase. (c) Executor sets phase.

Rationale: Option (c). The executor knows what happened (“I created an MR, so phase stays active”). Events that update dimensions also directly set state. The pattern is the same: something happens → state is updated → decider evaluates for the next action. Symmetry with event-driven dimension updates.

DDR-16: Items stay active after MR creation

Decision: After the ensure_mr executor succeeds, the item remains in active phase with no further actions pending.

Context: After MR creation, the decider returns None (no action). The item sits in active indefinitely.

Alternatives: (a) Set phase to completed after MR creation. (b) Keep active for future event-driven lifecycle tracking.

Rationale: Option (b). The active phase covers the entire lifecycle — from submission through CI, approval, merge, and post-merge tracking. The decider returns no actions until event-driven dimension updates (future SQS consumer) trigger the next step.

DDR-17: Audit log to S3, not database

Decision: Service-level audit (state transitions, decider decisions, actions taken) goes to S3 as immutable objects.

Context: Raw webhook events are already archived by sns-s3-archiver. The service needs its own audit trail for state transitions and decider evaluations.

Alternatives: (a) Database table (work_item_events). (b) S3 immutable objects. (c) Both.

Rationale: Option (b). S3 is immutable and cheap. The operational DB stays lean. Raw events are already in S3 via sns-s3-archiver; service-level audit captures the service’s reaction to those events.

DDR-18: Agent direction — dumber, not smarter

Decision: The hummingbird-agent should become a pure compute engine. The work queue service is the MR lifecycle orchestrator. The agent never calls the work queue service; the work queue service calls the agent.

Context: HUM-857 proposed adding create_merge_request as a new action in the agent’s actions.py. This was the wrong direction.

Alternatives: (a) Agent gains more actions (push, merge, MR management). (b) Agent becomes stateless compute; work queue service orchestrates.

Rationale: Option (b). The agent keeps LLM loop, tool registry, and sandbox — nothing else. Event routing, rate limiting, placeholder notes, result posting, session management, and all GitLab writes move to the work queue service. HUM-857 closed as Obsolete.

DDR-19: Shared database

Decision: The work queue service uses the same events PostgreSQL database as hummingbird-status and hummingbird-dashboard.

Context: Cross-service joins (e.g. work_items JOIN gitlab_merge_requests JOIN pipelineruns) are useful for the dashboard.

Alternatives: Separate database per service.

Rationale: Shared DB with service-specific Alembic migration chains (alembic_version_mr_service). Each service manages its own tables independently. The migration chain is isolated so one service can evolve its schema without affecting others.

DDR-20: Pydantic config over cki-lib/jsonschema

Decision: Use Pydantic for config validation. Load config from a YAML file at startup.

Context: Needed structured config for authorization rules and per-type operational settings. Alternatives: cki-lib config helpers, jsonschema, or plain dicts.

Alternatives: (a) cki-lib — adds a dependency, the service stays self-contained without it. (b) jsonschema — separate schema file, no type-safe access. (c) Pydantic — already a dependency (API schemas).

Rationale: Option (c). Pydantic provides type-safe attribute access, defaults, cross-field validators (e.g. mutual exclusivity of groups vs issuer+claims), and optional JSON Schema export. No new dependencies.

DDR-21: Claim-based rules (Vault bound_claims pattern)

Decision: Authorization rules match on arbitrary JWT claims, not just sub. All claim conditions AND; list values OR.

Context: Studied JWT trust/authz config from AWS API Gateway, HashiCorp Vault, Kubernetes Structured Auth, Istio, Traefik Hub, and oauth2-proxy. All support matching on arbitrary claims.

Alternatives: (a) Subject allowlist — rigid, requires exact sub values. (b) Claim-based matching — flexible, supports project_path, ref, groups, etc.

Rationale: Option (b). Vault’s bound_claims pattern is proven and simple. Values are coerced to lowercase strings before comparison to handle int/string mismatches (GitLab’s numeric project_id) and bool/string mismatches (ref_protected: true).

DDR-22: Issuer per rule, audience as claim

Decision: issuer is the only special field on a token rule — it determines which JWKS keys to use. All other JWT conditions (including aud) are standard claim matches in the claims dict.

Context: Considered a separate jwt.issuers section and/or treating aud as a first-class field alongside issuer.

Alternatives: (a) Global jwt section with trusted issuers + separate rules section. (b) aud as a dedicated field on each rule. (c) Issuer per rule, audience as a claim.

Rationale: Option (c). Keeps each rule self-contained and self-documenting. The Pydantic validator enforces that every issuer rule includes aud in claims to prevent cross-service token reuse.

DDR-23: Unified rule model for groups and tokens

Decision: A single rules list where every rule maps a remote identity (groups or issuer+claims) to local permissions.

Context: The original design had admin_groups/retry_groups (local permission → remote groups) alongside rules (remote identity → local permissions) — opposite mapping directions.

Alternatives: Keep separate group and token rule formats.

Rationale: Unified model. Every rule follows the same structure: remote identity → local permissions. Pydantic cross-field validators enforce mutual exclusivity.

DDR-24: Bearer-only auth with users/~ self-authentication

Decision: All API auth uses Authorization: Bearer tokens. Opaque OCP tokens (sha256~...) are validated by calling the OCP users/~ API with the user’s own token (self-authenticating).

Context: The oauth-proxy cannot simultaneously forward cookie-session tokens AND pass through non-OCP Bearer tokens on the same path. The TokenReview API would be cleaner but requires system:auth-delegator ClusterRoleBinding — a cluster-scoped permission not available on managed OCP clusters.

Alternatives: (a) Trust X-Forwarded-User from the proxy on all paths — injection risk on bypassed paths. (b) TokenReview API — requires cluster RBAC. (c) users/~ self-authentication.

Rationale: Option (c). No special RBAC needed. The users/~ endpoint returns username and groups. Results are cached with a thread-safe TTL cache (5 minutes). The /me endpoint bridges OAuth cookie sessions to Bearer tokens.

DDR-25: handled_types as explicit opt-in

Decision: handled_types declares which work item types this instance serves. Empty means handle nothing.

Context: Different deployments may serve different types from the same container image. Types carry per-type config (e.g. gitlab_url for MR).

Alternatives: (a) Handle all types by default. (b) Explicit opt-in.

Rationale: Option (b). Prevents accidental catch-all deployments. The sweep returns early when handled_types is empty — no items stolen from other instances in a shared-database deployment.

DDR-26: k8s shorthand skipped when not in-cluster

Decision: Rules with issuer: k8s are silently skipped when the in-cluster CA bundle is absent.

Context: The K8s OIDC issuer URL varies by cluster and is auto-detected at runtime. Local development doesn’t have a cluster.

Alternatives: (a) Error on startup if k8s issuer can’t be resolved. (b) Skip silently.

Rationale: Option (b). Allows using a production config file for local development and testing without startup errors.

Decision: The oauth-proxy sidecar handles only the OAuth login flow. API endpoints bypass the proxy. A proxy-protected /me endpoint bridges cookie sessions to Bearer tokens. The proxy must be configured with --cookie-refresh=1h to prevent stale-token failures.

Context: The OCP oauth-proxy cannot simultaneously forward cookie-session tokens AND pass through non-OCP Bearer tokens on the same path. Protected paths validate Bearer tokens via TokenReview (requires cluster RBAC we cannot obtain) or reject non-OCP tokens. Bypassed paths pass all Bearer tokens through but ignore cookies. Additionally, the OCP access token embedded in the proxy’s session cookie has its own expiry (default 24h), independent of the cookie lifetime. Without --cookie-refresh, the cookie outlives the token, causing the proxy to forward expired credentials while still reporting the user as authenticated — the “logged in but not authorized” failure observed in the hummingbird-dashboard deployment.

Alternatives: (a) All paths protected — blocks non-OCP Bearer tokens. (b) --openshift-delegate-urls — requires system:auth-delegator ClusterRoleBinding. (c) App reads proxy cookies directly — tight coupling to proxy internals. (d) Login-only proxy with /me bridge.

Rationale: Option (d). The /me endpoint is the only proxy-protected API path. It reads X-Forwarded-Access-Token (set by the proxy from the cookie session) and returns the OCP access token to the caller. Subsequent API calls use Authorization: Bearer sha256~... on bypassed paths, where the app validates via the OCP users/~ API. --cookie-refresh=1h keeps the embedded token fresh. TTL-bounded caching (5 minutes) on users/~ results ensures expired-token failures are never permanent (unlike @functools.cache which caches failures indefinitely).

DDR-28: Audit log with logging + optional S3 archival

Decision: Always log audit entries via Python logging at INFO level, and optionally archive to S3 as gzipped JSON objects when WORKQUEUE_SERVICE_AUDIT_S3_BUCKET is set.

Context: Needed an immutable record of state transitions, action outcomes, and auth decisions for compliance and debugging.

Alternatives: (a) Database table — bloats the operational DB with a different concern. (b) S3-only — no visibility without querying Athena; loses events if S3 is unreachable.

Why logging + S3: Logging to stdout is zero-config and integrates with existing container log collection for real-time visibility. S3 provides the durable, queryable archive (Athena) for compliance.

Key design rules:

  • Audit failures never propagate — write_audit_entry() catches all exceptions and logs them, but never raises.
  • S3 keys use YYYY/MM/DD/HH/MM/ prefix for Athena partitioning.
  • ContentEncoding: gzip set so S3 API consumers get transparent decompression.

Field conventions: See Audit Log Field Conventions in the operational documentation.

DDR-29: Events processed in work-item threads, not consumer threads

Decision: The SQS consumer thread stores events in the pending_events table. Existing per-work-item threads drain and process them.

Context: The ProcessingManager already runs one thread per work item with per-item ownership (DDR-51). Processing events in the same thread means events and actions are naturally serialized per work item.

Alternatives: (a) Consumer threads handle events directly — concurrency conflicts with action threads. (b) FIFO queue + in-memory dispatch — receipt handle management, visibility timeout extensions, message loss on crash.

Rationale: Option (c). Events join the existing thread structure. The decider runs after each event, determining follow-up actions. The single-thread-per-item invariant is preserved.

DDR-30: pending_events table as durable event store

Decision: Use a PostgreSQL table (pending_events) instead of FIFO queues or in-memory queues for event delivery to work-item threads.

Context: Since work-item threads do the processing, the SQS consumer just needs to deliver events to them durably.

Alternatives: (a) FIFO queue + in-memory dispatch — complex EventBroker, receipt handle management, message loss on crash. (b) FIFO queue + direct handling — concurrency conflicts.

Rationale: The DB provides durability (crash recovery), serialization (single thread per work item), and ordering (last_event_at guard). No FIFO queue, router thread, MessageGroupId computation, in-memory queues, or visibility timeout management needed.

DDR-31: SQS delete after INSERT, not after full processing

Decision: Delete the SQS message after INSERT+COMMIT into pending_events, not after full event processing.

Context: The hummingbird-agent deletes after handling (no DB intermediary). The workqueue-service can delete after INSERT+COMMIT because the event is crash-recoverable in PostgreSQL.

Alternatives: Delete after full processing — requires visibility timeout extensions, receipt handle passing, heartbeat management.

Rationale: After INSERT+COMMIT the event is durable in PostgreSQL. Duplicate SQS delivery may INSERT the same event twice; the last_event_at guard makes processing idempotent.

DDR-32: Resolver filters by phase, consumer stays read-only

Decision: The resolver filters by phase = 'active' on the MRWorkItem query. SQLAlchemy’s joined table inheritance handles the work_items table access implicitly. The consumer does SELECT + INSERT only, no UPDATE, no row lock contention.

Context: Only active items accept event-driven transitions (see DDR-37). Filtering at resolve time avoids unnecessary INSERTs into pending_events, thread spawns, and no-op processing for terminal items.

Alternatives: (a) Resolve without phase filter, guard only at processing time — wastes work for terminal items. (b) Consumer checks phase and applies transitions directly — contention with work-item threads.

Rationale: The phase filter adds negligible cost (PK lookup on work_items, typically one row). The consumer remains read-only. Business logic (phase transitions, decider, audit) is applied by the generic handle_event layer in the work-item thread.

DDR-33: Raw events stored, interpretation separated from application

Decision: The pending_events table stores the raw event JSON body and event type. Type-specific handlers interpret events (read-only on the work item), the generic layer applies the result.

Context: Event processing has three concerns: interpretation (what does this event mean?), application (phase transition, decider, audit), and routing (which work items?). Mixing them makes handlers hard to test and inconsistent across types.

Alternatives: (a) Handlers apply all changes directly — mixes interpretation with mutation, each handler reimplements decider/audit boilerplate. (b) Pre-process events in the consumer — consumer needs type-specific knowledge.

Rationale: Handlers return an EventResult dataclass (phase, event_ts, trigger metadata). The generic handle_event applies the phase transition via update_work_item_from_event, runs the decider, and emits audit entries. Handlers have read-only access to the work item and session (e.g. to check existing dimensions or detail fields) but EventResult is their only output — they never mutate the work item directly. This ensures decider/audit logic is consistent across all item types. See DDR-38 for the dimension delivery extension.

DDR-34: Per-work-item advisory lock for insert/drain coordination

Superseded by DDR-51. The insert/exit handshake was replaced by a SELECT ... FOR UPDATE row lock on work_items around the exit decision, which serializes every enqueue writer. See DDR-51 for the design and its motivation.

DDR-35: Sweep for startup recovery

Decision: Extend the periodic sweep to also check for work items with unprocessed pending_events rows.

Context: Events that arrive while the service is stopped, or that survive a mid-processing crash, must be processed on restart.

Alternatives: Separate startup-only scan.

Rationale: Unifying startup recovery with the periodic sweep keeps a single code path. The sweep runs immediately on manager.start(), catching events from downtime. It also serves as a safety net behind the advisory lock.

DDR-36: Consumer lifecycle and shutdown ordering

Decision: The SQS consumer is a single daemon thread started from the FastAPI lifespan. Shutdown stops the consumer before work-item threads.

Context: If work-item threads stop while the consumer is still INSERTing, events could be orphaned. The consumer must stop first.

Alternatives: Non-daemon thread with explicit join.

Rationale: Single thread is sufficient for low event volume (MR state changes). Daemon thread ensures process exit even if the thread hangs. Shutdown ordering: (1) set shutdown_event, (2) join consumer thread (timeout=30s), (3) call manager.stop(). No consumer when WORKQUEUE_SERVICE_EVENTS_QUEUE_URL is unset.

DDR-37: Terminal phase semantics — only active items accept events

Decision: Only items in phase = 'active' accept event-driven phase transitions. completed, failed, and cancelled are terminal phases — recovery requires explicit action via the retry API.

Context: Initially only failed was guarded (phase != 'failed'). completed and cancelled items could be modified by events: a closed event could transition a completed item to cancelled, and an opened event could reactivate a cancelled item.

Alternatives: (a) Guard only failed — inconsistent, completed items are accidentally mutable. (b) Growing exclusion list (NOT IN ('failed', 'completed')) — fragile if new phases are added. (c) Positive match phase = 'active'.

Rationale: Option (c). A single positive match is clearer and future-proof. If a work item was cancelled (MR closed) or completed (MR merged), only deliberate human action (retry API) should reactivate it — not an automated event. The retry API accepts both failed and cancelled items. The resolver also filters by phase = 'active' at query time (DDR-32) to avoid unnecessary INSERT/processing for terminal items.

DDR-38: Dimension updates via EventResult

Decision: Event handlers return dimension upserts as part of EventResult. The generic handle_event layer applies them.

Context: DDR-33 established that handlers return structured results and the generic layer applies them. Dimensions (DDR-1, DDR-3) need the same pattern — handlers identify which dimensions changed, the generic layer upserts them and emits audit entries.

Alternatives: (a) Handlers call crud.upsert_dimension() directly — breaks the read-only handler contract from DDR-33. (b) Separate dimension-specific consumer — duplicates the resolver/handler infrastructure.

Rationale: EventResult.dimensions is a list of DimensionUpdate dataclasses. After phase transition logic, handle_event iterates over dimensions and calls crud.upsert_dimension() for each, emitting an AuditEntry with event_type="dimension_change" using the existing dimensions field. Dimension upserts run even when the phase transition is stale (the early return is refactored to skip only the phase/decider block). This ensures dimension-only events (e.g. note events with no phase change) are still applied.

DDR-39: Server-derived idempotency keys with phase-aware uniqueness

Decision: Replace the client-supplied idempotency_key field with a server-derived key mr:{project}:{source_branch}:{generation}. Enforce uniqueness only among non-terminal phases via a partial unique index.

Context: The original API required callers to supply a unique idempotency_key per submission. Recurring callers (dist-git sync, upstream sync) had to invent unique keys for each invocation and track them across runs. The global unique constraint also blocked new submissions for a branch that had a previous completed or cancelled item with the same key.

Alternatives: (a) Client-supplied key with a prefix convention — callers still need to generate and track keys. (b) Server-derived key from the natural deduplication identity (project, source_branch) with a generation counter for lifecycle restarts.

Rationale: Option (b). The idempotency key is a DB-level safety net, not a caller concern. The natural identity is (project, source_branch); the generation counter increments when a new item is created after the previous one reached a terminal phase. The partial unique index (WHERE phase NOT IN ('completed', 'cancelled')) ensures at most one active/failed item per idempotency key while allowing completed items to coexist with their successors.

DDR-40: Upsert semantics for recurring MR management

Decision: Replace one-shot insert_mr_work_item with upsert_mr_work_item that looks up an existing active/failed item by (project, source_branch) and determines the correct action from a content hash comparison.

Context: Recurring callers submit MRs for the same branch repeatedly. Under the old insert model, each submission created a new work item — even when the previous one was still active, leading to orphaned branches and duplicate MRs.

Alternatives: (a) Client manages deduplication — check before submit, handle conflicts. (b) Server-side upsert — callers always submit current desired state, the service deduplicates transparently.

Rationale: Option (b). The upsert does a (project, source_branch) lookup among non-terminal items and branches on the result:

  • No match → create a new item (actionable = true)
  • Active, same content → no-op (return existing, "unchanged")
  • Active, different content → update fields, set actionable = true, reset needs_mr = "true"
  • Failed → reset to active, update content, clear error ("retried"), set actionable = true
  • Terminal (completed/cancelled) → create a new generation (DDR-39)

Concurrent insert races are handled via DDR-39’s partial unique index — the loser catches the IntegrityError, rolls back, and retries.

DDR-41: ThreadPoolExecutor with bounded workers

Decision: Replace unbounded per-item threading.Thread spawning with a fixed-size ThreadPoolExecutor (10 workers).

Context: The periodic tick mechanism (DDR-42) can wake many idle items simultaneously. Unbounded thread creation would cause resource spikes (~109 threads in the worst case).

Alternatives: (a) Keep per-item threads with a semaphore. (b) Bounded thread pool.

Rationale: Option (b). ThreadPoolExecutor provides built-in queuing when all workers are busy, deduplication via _submitted set, and clean shutdown via pool.shutdown(wait=True). The 10-worker limit caps concurrent GitLab API calls.

DDR-42: Hash-bucketed tick events for periodic re-evaluation

Superseded by DDR-48. The synthetic tick_30m and tick_5m events were replaced by the pending_until column on work_item_statuses. See DDR-48 for the design and its motivation.

DDR-43: Empty diff detection via repository_compare

Decision: Detect and handle empty MR diffs at two call sites: ensure_mr (prevention on create and force-push) and check_empty_diff (periodic via sweep).

Context: The detection-execution race causes empty MRs when the target branch moves between change detection and MR creation. Also, existing MRs can become empty after creation.

Alternatives: (a) Compare file content hashes before commit. (b) GitLab repository_compare API after commit/push.

Rationale: Option (b). The repository_compare endpoint returns the diff between two branches. Empty diffs list means the branches are identical. This is authoritative and handles edge cases that hash comparison might miss (e.g. whitespace normalization). Cost: one API call per check (~4 calls/minute across all items for periodic checks).

DDR-44: Per-item processing deadline

Decision: Exit the work item loop after 60 seconds of wall-clock time.

Context: The tick mechanism (DDR-42) and check_stale_mr action could cause extended processing loops for items that repeatedly generate new actions. A deadline prevents any single item from holding a pool worker indefinitely.

Alternatives: (a) Limit loop iterations. (b) Wall-clock deadline.

Rationale: Option (b). Wall-clock time is predictable regardless of action duration. The deadline is between loop iterations, not mid-executor, so executors have their own HTTP timeouts. The item is re-picked by the next sweep if work remains.

DDR-45: Dimension-driven decider with needs_* flags

Decision: The decider scans needs_* dimensions in priority order instead of encoding per-concern conditionals.

Context: The original decider checked item.mr_iid is None for create, dashboard_link dimension absence for dashboard notes, and would need more checks for auto-merge, stale detection, metadata sync. Each new concern added another conditional.

Alternatives: (a) Per-concern conditionals in decider. (b) Dimension-driven: events/executors set needs_* flags, decider scans them.

Rationale: Option (b). The decider becomes a simple priority scan. Executors own their trigger-to-next-step logic by setting the appropriate needs_* dimension. Events set needs_reconcile to request re-evaluation. Adding a new concern = new dimension + handler in reconciler, no decider changes needed.

Update (DDR-48): The original three-category model (has_*, needs_*, observed state) was unified into a single needs_* trigger model with "true"/"false" status. See DDR-48.

DDR-46: reconcile_mr pattern

Updated by DDR-52. The “merge latency to seconds” trade-off below did not hold on busy target branches: the branch churn (largely the workqueue’s own fleet) resets mergeability to unchecked faster than the 30-minute cadence re-samples it, starving managed MRs for days. DDR-52 adds the deadline-driven sweep and an adaptive recheck backoff that remove this failure mode; the direct-merge decision itself stands.

Decision: Single reconcile_mr executor handles all MR state convergence, with cheap (always) and expensive (flagged) tiers. Uses direct mr.merge() instead of GitLab auto-merge.

Context: Separate executors per concern (check_stale_mr, enable_auto_merge, sync_metadata) each needed their own registration, decider rule, and dimension tracking. GitLab auto-merge (auto_merge=True) introduced state-management complexity: merge_when_pipeline_succeeds tracking, stale SHA detection after force-push, and recurring 422 noise for non-mergeable MRs.

Rationale: One mergerequests.get() call serves all cheap checks. The reconciler checks detailed_merge_status == "mergeable" and calls mr.merge() directly — no GitLab auto-merge state to track, no skip-lists to maintain, no stale SHA detection. Pipeline completion and approval webhooks trigger reconcile via the needs_reconcile dimension, keeping merge latency to seconds. Expensive operations (empty diff via repository_compare) only run when needs_empty_check is actionable (periodic via pending_until, DDR-48). The reconciler is idempotent — all corrections are conditional on actual vs desired state.

DDR-47: CI job retry herder

Updated by DDR-48. The needs_ci_retry="pending_tick_5m" status and tick_5m promotion mechanism have been replaced by pending_until on the dimension row.

Problem: Hummingbird CI pipelines frequently fail due to transient infrastructure errors (runner system failures, git clone timeouts, OOM kills). Manual retries waste developer time and delay merges.

Options considered:

  • A. Executor action in workqueue-service (selected): Shares existing GitLab tokens, project config, DB-persisted state, and audit trail.
  • B. Standalone herder service: Maximum CKI fidelity but requires a new deployment, duplicates token management, and has no MR lifecycle integration.
  • C. Reconciliation polling: No new webhooks, but 30-minute detection latency and wastes API calls polling healthy pipelines.

Decision: Option A. The matcher engine is ported from the CKI pipeline-herder with a reduced field set. Build events are received via the existing SNS→SQS pipeline and resolved to MR work items by matching pipeline_id against head_pipeline_id on mr_details.

DDR-48: Uniform needs_* dimension model with pending_until

Decision: Unify all dimensions under a single needs_* trigger model with "true"/"false" status and an optional pending_until column for deferred re-evaluation. Drop synthetic tick events.

Context: The dimension system had grown three distinct categories (DDR-45): has_* positive completion markers, needs_* action triggers, and observed state dimensions (approval, pipeline, merge_status, has_conflicts). The has_* and needs_* split created asymmetric decider logic. Observed state dimensions were write-only — the reconciler set them but no code read them. The tick_5m/tick_30m synthetic event mechanism (DDR-42) required hash bucketing, NOT EXISTS guards, and special "pending_tick_5m" status values.

Alternatives: (a) Keep the three-category model and add pending_until alongside tick events — increases complexity with two deferral mechanisms. (b) Unify to needs_* with pending_until.

Rationale: Option (b).

  • Uniform semantics: every dimension is a needs_* trigger. Status is "true" (action needed) or "false" (handled). The decider uses a single check pattern: != "false" means actionable.
  • Absent = actionable: missing dimension rows are treated as needing action, so new items don’t require explicit initialization of every dimension.
  • pending_until replaces ticks: executors set pending_until to schedule future re-evaluation. The sweep finds expired pending_until and notifies — no synthetic events, no hash bucketing, no "pending_tick_5m" status values.
  • Dropped dimensions: approval, merge_status, has_conflicts were write-only. The pipeline dimension carried a single integer and is better modeled as a typed column on mr_details.
  • Deferred dimensions: the decider pre-filters dimensions where pending_until is in the future, treating them as "false".

DDR-49: Replace pending_action TEXT with actionable BOOLEAN

Decision: Drop the pending_action TEXT column. Replace it with actionable BOOLEAN DEFAULT FALSE. The decider is called in exactly one place: the processing loop’s derive-and-dispatch step.

Context: pending_action served three roles: (1) sweep discoverability (WHERE pending_action IS NOT NULL), (2) processing loop continue/exit signal, (3) executor dispatch key. Roles 2 and 3 made the column authoritative for which action to run, creating a sync burden with the dimension-driven decider.

Alternatives:

  • Remove entirely — rejected: the decider treats absent dimensions as actionable, so a SQL query over dimension rows cannot find items whose next action derives from a missing row. A wake-up signal persisted to the DB is required.
  • next_action TEXT (renamed cache) — rejected: requires decider.decide() calls in the CRUD layer. The authoritative-looking name invites future dispatch reads.
  • actionable BOOLEAN — selected.

Rationale: A boolean is sufficient to bridge the absent-dimension discoverability gap. Stale True is harmless (false-positive sweep → loop starts, decider returns None, exits immediately). actionable is written in exactly two places: True when work is enqueued (creation, update, retry, event insertion), False only on the loop’s exit branch (committed atomically when both decider.decide() returns None and no pending events exist). This invariant ensures the item stays sweep-visible throughout processing — a crash at any point leaves actionable = True, preventing items from becoming permanently stuck.

DDR-50: Event drain tolerates concurrent deletion

Decision: The event drain (_process_pending_events) reads the (id, event_type, event_body) of all listed pending events before processing the first one, and deletes rows by id. A row deleted by another pod mid-drain makes the delete a no-op instead of an error.

Context: The lease is acquired once per loop run and renewed only while executor actions run; it is neither renewed nor re-checked during the drain, and the per-item deadline is only checked between drains (DDR-44). A slow drain can therefore outlive the lease — most notably during rolling deployments, where the terminating pod’s in-flight drain overlaps the new pod’s startup sweep. Committing per event expires the remaining ORM instances, so the next iteration’s attribute access lazily refreshes against the database; when the other pod already deleted that row, SQLAlchemy raises ObjectDeletedError and the whole work-item loop dies (observed as Sentry issue 7829339838, 35 seconds after a rollout).

Alternatives:

  • Hold the per-item advisory lock across the drain — rejected: the lock is transaction-scoped (DDR-34) and the drain commits per event for crash isolation; spanning all events would require a single transaction and lose per-event progress.
  • Renew or re-check the lease inside the drain — rejected: shrinks but does not close the takeover window (the lease can still expire mid-iteration), and does not fix the stale-instance crash itself.
  • Re-query each event before processing it — rejected: adds a query per event, and the row can still vanish between the check and the delete.

Rationale: Reading the attributes while the instances are fresh — before the first commit — means no lazy refresh can fire mid-drain, and delete-by-id is naturally idempotent. An event may then be applied by both pods, but application is already at-least-once for crash recovery (DDR-31) and idempotent (dimension upserts, last_event_at guard). The drain becomes robust no matter which actor deletes a row.

DDR-51: Per-item processing ownership and exit safety

Decision: Loop ownership is a loop-lifetime session advisory lock — pg_try_advisory_lock(0x5751, work_item_id.int & 0x7FFFFFFF) taken on a pinned connection at loop entry — and the loop’s exit decision runs under a SELECT ... FOR UPDATE row lock on work_items. The leased_until lease with its heartbeat (DDR-6) and the insert/exit advisory-lock handshake (DDR-34) are removed; the column is dropped by the contract migration after the redesigned version has fully rolled out.

Context: DDR-29/DDR-30 require at most one processing loop per work item across pods, but no mechanism delivered that invariant. The lease was acquired once per loop run, renewed only while an action ran, and had no owner, so it excluded only the next action, not the loop. The exit wrote actionable = false blindly, so an API enqueue committing between the loop’s state read and that write was overwritten — permanently stalling items whose only wake-up signal was a PROCESS_IMMEDIATELY dimension (no pending_events row, invisible to every sweep leg). The transaction-scoped advisory lock covered only the consumer’s INSERT and the exit, not the API writers. The 2026-11-04 ObjectDeletedError (Sentry 7829339838) was one symptom: a new pod’s startup sweep took an item whose lease the old loop had cleared between actions. !1494 (DDR-50) fixed the crash, not the ownership model.

The two-32-bit-key form (classid=0x5751, objsubid=2 in pg_locks) deliberately does not overlap the legacy single-64-bit-key space (objsubid=1), so during a mixed-version rolling update an old pod’s handshake and a new pod’s session lock cannot block each other.

The key is the item UUID’s low 31 bits — the positive range of the int4 second argument. Two unrelated items whose UUIDs share those bits collide: the loser returns immediately (counted in hummingbird_workqueue_processing_lock_skipped_total) and the sweep re-spawns it once the winner’s loop finishes. A collision delays one item; it never allows two loops for the same item and never loses work.

Every enqueue path already updates the work_items row in the same transaction as its durable write (dimension upsert, pending_events insert, retry mark). The row lock serializes the exit against all of them without any new protocol; this contract is now explicit and must hold for future enqueue paths.

Alternatives:

  • Compare-and-set on a new revision column — correct and non-blocking, but adds a migration and a permanent column, and onupdate is ORM-level, so old-pod writers would not bump it during the rollout.
  • CAS on updated_at or xmin — no migration, but updated_at is transaction-start time (uniqueness only by microsecond) and xmin is an obscure system column with freeze/wrap edge cases.
  • Durable enqueue token (a wake-up row per enqueue, consumed only on a clean exit, enumerated by the sweep) — the structural fix that removes the shared-boolean race by construction; deferred as a larger change (a table, an insert on every enqueue path, a drain lifecycle, and a sweep leg).
  • Extend the advisory lock to API writers — couples synchronous HTTP handlers to the processing lock for a whole loop and reintroduces the bug whenever a writer forgets.
  • Fix the timestamp lease in place (owner token, renewal for the whole loop, re-assert before each action) — time-based expiry still cannot exclude a paused or partitioned worker holding the lock past its TTL, and keeps the heartbeat machinery.
  • SERIALIZABLE exit transaction — PostgreSQL SSI only guarantees serialization among serializable transactions; mixing isolation levels and retrying under contention is more machinery than the row lock.
  • Leader election / single-writer pod — removes rollout overlap but defeats replica throughput and distinct-item parallelism.

Rationale: The session lock makes “one loop per item” true by construction for the whole loop lifetime, not per action: a loser returns immediately without holding a worker slot, crashes recover through connection death plus the sweep, and no TTL or heartbeat is needed. The row lock makes the exit decision atomic with respect to every writer without a migration, and old-pod writers block on it too. A spawn request that arrives while the loop is still exiting is captured by a per-item rerun flag, and after a clean exit the loop unlocks, re-checks for work, and re-acquires if a writer slipped in.

Each running loop pins one pooled connection (pool sized to the worker count with overflow headroom, pool_pre_ping, libpq keepalives and tcp_user_timeout), and every loop session binds to it. A dead connection therefore fails the work — the exception propagates and the loop aborts — instead of leaving a loop running under a lock the server already released; the still-actionable item is recovered by the sweep. The API request path can block for the duration of the exit decision (a few DB-only statements), so the exit transaction must stay DB-only.

During a mixed-version rolling update, an old pod’s leased loop and a new pod’s locked loop can briefly both run the same executor action for one item: the two-key lock cannot conflict with the old lease path. Read-and-upsert actions (reconcile_mr, ensure_plr_status, ensure_expected_plrs) converge on the same state. Creation actions and retry_ci_jobs check their result field (e.g. mr_iid) before acting, so a duplicate requires both loops to pass that check before either commits — possible in the worst case, producing one duplicate external mutation (a second MR or note, or a second CI retry) for an item mid-action during the pod replacement. New pods deliberately do not skip rows with an unexpired leased_until: a lease written by a pod that died mid-rollout must not hide the item. The window is bounded by the rollout and ends when the contract migration drops the column.

DDR-52: Deadline-driven sweep and adaptive merge-recheck

Decision: The sweep wakes at deadlines instead of polling: after each pass it sleeps until the earliest armed pending_until (plus ~1 s epsilon), capped at the former 5-minute interval as a backstop — or for just the epsilon when a deferred timer is already due, letting the next pass’s expired scan pick it up. unchecked/checking/unknown merge statuses re-arm needs_reconcile at now + RetrySchedule.delay(step) on an exponential schedule (15 s, doubling, 5-minute cap) with the step persisted in the dimension detail, and a partial index on work_item_statuses (pending_until) covers the sweep’s hot path.

Context: The direct-merge design (DDR-46 lineage; !1327) merges when a reconcile GET observes detailed_merge_status == "mergeable", with a 30-minute retry cadence and the sweep (fixed 5 minutes) as the only timer. On busy target branches this starves: every push to the target branch resets stored mergeability to unchecked, GitLab recomputes it only lazily on a single-MR GET (seconds), the service samples once per 30 minutes, and GitLab emits no webhook when a recheck completes. Evidence: containers!22697 unmerged 6 days while green; 2026-11-10 logs show 15/16 reconcile GETs observing unchecked and the two mergeable observations losing the merge PUT race to concurrent fleet merges (422 “Branch cannot be merged”). The branch churn is largely the workqueue’s own lockfile fleet (59 merges in two hours on 2026-11-10), so the service invalidates its own merge candidates. The branch churn also means any fixed short interval is a compromise: real blockers need minutes, rechecks need seconds.

now is passed to the deadline query as a bound parameter rather than SQL now() because pending_until values are written with the Python clock; comparing in one clock domain avoids app-vs-DB skew shifting wakeups. The MIN query mirrors find_expired_pending’s item filters (active phase, accepted types) so completed items — which keep their last dimension rows — cannot pin the sweep to a stale deadline. The deadline lookup computes the earliest future deadline and a past-due flag in one filtered-aggregate query — one snapshot — so a timer expiring between the sweep’s expired scan and the lookup cannot be missed by both; the past-due flag excludes actionable items, mirroring the expired scan, so a running loop’s timer cannot spin the sweep. A wake event interrupts the sleep when an earlier deadline is armed in-process (set from spawn_loop()’s enqueue paths and after executor action commits; sweep-initiated submissions deliberately do not poke — they arm no timer, and poking would wake the sweep into a pass that re-submits, spinning cross-pod while another pod’s loop holds the item lock during a rollout); spurious wakes cost one indexed query and a no-op sweep. Multi-pod: each pod computes its own MIN; the arming pod wakes exactly, others see the work within the backstop cap, and duplicate spawns are absorbed by the per-item advisory lock (DDR-51).

Alternatives:

  • Shorter fixed sweep interval — wasted cycles and still granular; a +30 s timer would still round up to the interval.
  • Short dimension timers without a sweep change — the 5-minute sweep granularity silently becomes the effective interval; the timer is a lie without deadline-driven wakeups.
  • Sleeping and re-GETting inside the executor — holds a worker slot (10 max) and the pinned DB connection, brushes the 60-second item deadline (DDR-44), and violates the “reconcile_mr: cheap operations only” contract (DDR-46).
  • Fixed fast interval without backoff — a permanently-unchecked MR would poll ~2880×/day.
  • Backoff step derived from row timestamps (updated_at deltas) — fragile, implicit, breaks on any other write to the dimension row.
  • RetryAfter state merged into the crud state union with a reserved detail key — more machinery and magic than needed for one consumer; revisit on a second non-MR consumer.
  • Re-enabling GitLab server-side auto-merge — removes service ownership of the merge decision (DDR-46); the adaptive recheck achieves convergence without it.

Rationale: A mergeable MR converges within one or two recheck probes (~15–45 s) instead of winning a 30-minute lottery, and concurrent-merge race losers recover within seconds, so the fleet race self-heals — the 422 → immediate retry → unchecked → +15 s → mergeable → merged sequence replaces the 30-minute re-sample. unknown shares the fast family (GitLab documents it alongside unchecked); a persistent unknown settles at the 5-minute cap, and the mr.state != "opened" guard keeps closed/merged MRs from being polled. Permanently-unchecked MRs poll at the 5-minute cap (the same cadence as error backoff), visible via the hummingbird_workqueue_merge_rechecks_total counter labeled by merge status. Real blockers keep the 30-minute cadence and reset the backoff step (detail=None, the documented clear path); webhook re-arms preserve the step (the _PRESERVE sentinel) and consume one — each immediate reconcile observing unchecked advances the backoff, deliberately throttling poll storms on chatty MRs. The GitLab-error path (the 5-minute error re-arm without a detail write) also preserves a matured step, so the first unchecked probe after an outage waits at the cap — deliberate. The sweep runs as often as the earliest deadline (potentially every ~15 s under recheck storms); each pass is one indexed MIN query, one expired-pending query, and the actionable-or-pending-events union query, and only items with due work spawn loops — hence the partial index. The direct-merge decision stands; this record removes its starvation failure mode on busy branches. Rollout: the deadline-driven sweep and its supporting partial index land first, then the reusable RetrySchedule type, then the merge-recheck application (schedule constant, detail step, outcome enum, metric, and docs); the migration adds the partial index only.