Generic work queue service for Hummingbird. Manages work items through type-specific lifecycles with a shared processing framework. Currently handles MR creation and updates; designed to support CVE remediation, advisory tracking, and other work item types.
For design decisions and architectural rationale, see Work Queue Service Design.
Architecture
The service runs as a Kubernetes deployment sharing the same PostgreSQL database as hummingbird-status and hummingbird-dashboard (with a service-specific Alembic migration chain).
flowchart TD
clients["Internal services"] -- "POST /work-items/mr" --> svc["Work Queue Service"]
gl_events["GitLab\n(events)"] -- webhook --> gl_fwd["gitlab-event-forwarder"]
gl_fwd --> sns["SNS"]
k8s_events["Konflux K8s\n(PipelineRun events)"] -- watch --> k8s_fwd["k8s-event-forwarder"]
k8s_fwd --> sns
sns -->|"filter: source=gitlab,\nevent_type=[merge_request, build]"| sqs["SQS Queue"]
sns -->|"filter: source=kubernetes,\nkind=PipelineRun"| sqs
sqs --> svc
svc -- "Commits API\n(create + update)\nJobs API\n(trace + retry)" --> gl["GitLab\n(API)"]
svc -- "PLR fallback fetch\n(label selector on SHA)" --> k8s_api["Konflux K8s API"]
svc -- "archived PLR fetch" --> ka["Kubearchive"]
svc -- "shared DB" --> pg[("PostgreSQL")]
The SQS consumer thread receives GitLab merge_request and pipeline
events and Kubernetes PipelineRun events from the SNS topic via one
filtered SQS subscription with a two-branch $or filter policy covering
both source=gitlab and source=kubernetes. Events are stored in the
pending_events table and processed by existing work-item threads,
which map MR state changes to phase transitions, track Konflux build
status, and trigger CI job retries on pipeline failure.
Design
Generic + detail tables
The schema separates queue machinery from type-specific data:
work_items— generic: id, type, phase, actionable, attempts, error, timestamps. Shared by all work item types.mr_details— MR-specific: package, project, file_actions, commit_message, content_hash, mr_iid, etc. Joined to work_items via foreign key.work_item_statuses— independent readiness dimensions with composite key (work_item_id, dimension, variant). Includespending_untilfor deferred re-evaluation.
New work item types add their own detail table (e.g. cve_details) —
no changes to the generic layer. SQLAlchemy joined table inheritance
loads the correct subclass automatically via polymorphic_load="selectin".
Phases
Four generic phases track coarse lifecycle status:
| Phase | Description |
|---|---|
active |
Being processed — the only mutable phase |
completed |
All lifecycle steps done (terminal) |
failed |
Error occurred (terminal, retryable) |
cancelled |
Cancelled (terminal, retryable) |
Items start as active. Only active items accept event-driven phase
transitions and executor actions. The three terminal phases
(completed, failed, cancelled) are immutable to events — recovery
requires explicit action via the retry API. GitLab merge_request events
drive transitions from active: merge -> completed, close ->
cancelled.
Dimensions
Independent readiness signals stored in work_item_statuses. All
dimensions follow a uniform needs_* trigger model: status is
"true" (action needed) or "false" (handled). Absent dimensions
are treated as actionable by the decider (!= "false"). Each
dimension has an optional variant for multi-instance signals (e.g.
per-component Konflux builds) and an optional pending_until
timestamp for deferred re-evaluation (see
Deferred dimensions below).
Dimensions are set by executors or event handlers via
EventResult.dimensions; the generic handle_event layer applies
event-driven dimensions and emits audit entries.
Exception —
needs_plr: Unlike other dimensions,needs_plrrequires explicit arming to"true"to be actionable. Absent or"false"means inactive (not awaiting PLR events). The decider checksdims.get("needs_plr") == "true", so a missing row is treated as inactive, not actionable.
Dimension schema
work_item_statuses:
work_item_id: UUID (PK, FK)
dimension: Text (PK)
variant: Text (PK, default="")
status: Text (NOT NULL) -- "true" or "false"
updated_at: DateTime(tz)
detail: JSONB (nullable)
pending_until: DateTime(tz, nullable)
Unified state diagram
All dimensions share the same state machine:
stateDiagram-v2
[*] --> absent : item created
absent --> true : decider treats absent as actionable
true --> true_deferred : executor (defer for later)
true --> false : executor (done, no further checks)
true_deferred --> true : sweep notifies (pending_until expired)
true_deferred --> true : webhook/handler (clears pending_until)
false --> true : webhook/handler (re-activates)
note right of absent : no dimension row exists
note right of true : status="true", pending_until=NULL
note right of true_deferred : status="true", pending_until=future
note right of false : status="false", decider ignores
Per-dimension transitions
| Dimension | On success (→ "false") |
On error / re-trigger (→ "true") |
Deferred? |
|---|---|---|---|
needs_mr |
MR created or already exists | — | No |
needs_expected_plrs |
Expected components fetched | GitLab API error, unexpected error (malformed manifest permanently completes with error: "parse_error") |
Yes — 5 min error backoff |
needs_plr |
All PLRs terminal or fetched | PLR event or PLR fallback fetch (non-terminal) | Yes — 30 min fallback poll |
needs_dashboard |
Dashboard note posted or updated | PLR event, PLR fallback fetch, transient GitLab 5xx | No — always set “false”; re-armed by PLR handler/executor |
needs_reconcile |
Reconciler completed | Webhook (approved, pipeline, update), sweep expiry | Yes — 30 min cycle / 5 min error backoff / 15 s→5 min recheck backoff for unchecked/checking/unknown |
needs_empty_check |
Empty-diff check completed | Sweep expiry | Yes — 30 min cycle / 5 min error backoff |
needs_ci_retry |
All jobs processed | Build event (job failed), sweep expiry | Yes — per-job not_before delay |
Dimension detail schemas
The detail JSONB column carries executor-specific state. Dimensions
without detail (needs_mr, needs_empty_check) store null;
needs_reconcile stores null except during a merge-recheck streak.
Frozen dataclass definitions live in
workqueue_service/mr/dimension_types.py; typed accessor functions
(get_plr_detail, get_dashboard_detail, get_expected_plrs_detail)
deserialize the stored JSONB into the dataclass, or return None when
absent. JSON arrays round-trip as tuples — NeedsCiRetryDetail.jobs
and NeedsExpectedPlrsDetail.components are immutable.
Detail preservation. DimensionUpdate.detail defaults to a
_PRESERVE sentinel. Event handlers that return a DimensionUpdate
without explicit detail automatically keep whatever is stored in the
database (via SQL COALESCE). Pass detail=None explicitly to clear,
or a detail dataclass to replace. See Dimension update
helpers below.
Updating detail. Event handlers update the existing detail instead
of rebuilding it: read it via the typed accessor, then apply a mutation
helper — NeedsPlrDetail.with_component() adds or replaces one
component entry, NeedsCiRetryDetail.with_job() appends one job.
Both return a new frozen instance; the original is untouched. Handlers
pass the dataclass to DimensionUpdate, and executors that snapshot
authoritative state (ensure_plr_status, ensure_expected_plrs,
retry_ci_jobs) construct a fresh dataclass. crud.upsert_dimension
serializes dataclasses via dataclasses.asdict() before the DB write,
so callers never handle raw dicts.
needs_expected_plrs detail
Stored after fetching the Konflux trigger manifest for a commit SHA.
Set by ensure_expected_plrs; read by ensure_plr_status,
ensure_dashboard_note, and the PLR event handler.
| Field | Type | Required | Description |
|---|---|---|---|
sha |
str | null |
yes | Commit SHA the manifest was fetched for. null for items without a SHA. |
components |
list[str] |
yes | Expected component names (e.g. "caddy--hummingbird--default"). Empty when the repo has no trigger manifest (404). |
error |
str |
no | Set on parse failure; empty components list accompanies it. |
needs_plr detail
Tracks observed PipelineRun state per component. Updated by the PLR
SNS handler (merge per event) and ensure_plr_status (full snapshot
from K8s/Kubearchive). Read by ensure_dashboard_note.
| Field | Type | Required | Description |
|---|---|---|---|
components |
dict[str, PlrComponentInfo] |
yes | Keyed by component name. Empty dict on init. |
Each PlrComponentInfo value:
| Field | Type | Required | Description |
|---|---|---|---|
status |
str |
yes | PLR condition: "Running", "Succeeded", "Failed", "Unknown" |
plr_name |
str |
yes | Kubernetes PipelineRun resource name |
start_time |
str |
yes | ISO 8601 timestamp from status.startTime (pipeline start, not resource creation) |
needs_dashboard detail
Stored after posting or updating the MR dashboard note. Set by
ensure_dashboard_note; read on subsequent runs to decide whether
to update or post. The PLR handler and ensure_plr_status re-arm
needs_dashboard to "true" without passing detail, which preserves
this detail.
| Field | Type | Required | Description |
|---|---|---|---|
note_id |
int |
yes | GitLab note ID for in-place updates |
url |
str |
yes | Dashboard URL included in the note body |
note_body |
str |
no | Last rendered note body; compared to skip no-op GitLab API calls |
needs_reconcile detail
Optional — only present during a merge-recheck streak. When
reconcile_mr observes detailed_merge_status in the fast family
(unchecked, checking, unknown — recheck in flight), it re-arms the
dimension at now + MERGE_RECHECK_SCHEDULE.delay(retry_step) (15 s
doubling to a 5-minute cap) and bumps the step via advance_retry. The
converge outcomes (merge attempted, real blocker, idle) clear the
detail, so a fresh streak starts at 15 s again; the early exits (no MR
yet, GitLab fetch error) leave the stored step untouched. Webhook
re-arms preserve the stored step through the _PRESERVE sentinel.
| Field | Type | Required | Description |
|---|---|---|---|
retry_step |
int |
yes | Consecutive fast-family recheck count (starts at 1) |
needs_ci_retry detail
Accumulates failed CI jobs eligible for retry. Set by the build event
handler (appends per job) and retry_ci_jobs executor (removes
processed jobs). Each entry represents one failed GitLab CI job.
| Field | Type | Required | Description |
|---|---|---|---|
jobs |
list[CiRetryJobEntry] |
yes | List of job entries (see below) |
Each job entry:
| Field | Type | Required | Description |
|---|---|---|---|
build_id |
int |
yes | GitLab job ID |
build_name |
str |
yes | Job name (e.g. "test", "check") |
build_failure_reason |
str |
yes | GitLab failure reason (e.g. "script_failure") |
build_finished_at |
str |
yes | ISO 8601 timestamp when the job finished |
retries_count |
int |
yes | Number of retries already attempted |
pipeline_id |
int |
yes | GitLab pipeline ID the job belongs to |
matcher_name |
str |
no | Name of the retry matcher rule that matched this job |
not_before |
str |
no | ISO 8601 timestamp — earliest time the retry may be attempted |
Deferred dimensions
When an executor or handler sets pending_until to a future timestamp
alongside status="true", the decider treats the dimension as "false"
(not actionable) until that time. The periodic sweep finds items with
expired pending_until and notifies them — no synthetic events are
inserted. The sweep wakes exactly at the earliest armed deadline — or
after ~1 s when a deferred timer is already due, letting the next
pass’s expired scan pick it up (see
Processing loop) — so even short timers are honored
precisely rather than rounded up to a sweep tick.
status |
pending_until |
Decider sees | Meaning |
|---|---|---|---|
| (absent) | — | actionable | No row — needs action |
"true" |
NULL |
actionable | Needs action now |
"true" |
past | actionable | Sweep notified, ready to act |
"true" |
future | not actionable | Deferred — sweep will notify |
"false" |
any | not actionable | Done |
Dimension update helpers
crud.upsert_dimension provides safe-by-default mutation patterns via
its detail parameter:
- Omitted — keep existing detail (SQL
COALESCEon conflict); e.g. re-triggeringneeds_dashboardafter a PLR event without clobberingnote_id - Detail dataclass — replace with a full snapshot; e.g. initialising a dimension or completing an executor action
None— clear the stored detail explicitly
Partial detail updates do not use a merge helper — event handlers
apply the frozen dataclass mutation helpers (with_component(),
with_job()) to the existing detail instead, and executors construct a
fresh dataclass for a full snapshot. crud.upsert_dimension takes a
detail dataclass (or None) and serializes it via dataclasses.asdict()
internally; callers never handle raw dicts.
See Dimension detail schemas for the
_PRESERVE sentinel, frozen dataclass definitions, and typed accessors.
Dimension state (status plus optional deferred re-arm) is expressed via the
DimensionState enum in workqueue_service/mr/constants.py. Passing a bare status string
or timedelta to the state parameter is a type error; always use an enum member:
| Member | Status | Delay | Used for |
|---|---|---|---|
DONE |
false |
— | Dimension fulfilled, no action needed |
PROCESS_IMMEDIATELY |
true |
— | Needs action now |
PROCESS_AFTER_TIMEOUT |
true |
30 min | PLR silence fallback, reconcile/empty-check cycle |
PROCESS_AFTER_ERROR |
true |
5 min | Transient API error backoff, CI job not_before delay |
For computed absolute times (e.g. the CI retry not_before), use
DimensionState.process_at(when). It returns a ProcessAt wrapper that the CRUD layer
resolves to status="true" with pending_until set to the given time.
Decider
A pure function registered per type that derives the next action from
the work item’s current state. The decider reads the selectin-loaded
item.statuses list (in-memory, no DB call) to check dimensions,
pre-filtering deferred dimensions (where pending_until is in the
future). See
workqueue_service/mr/decider.py
for the MR decider implementation.
The decider scans dimensions in priority order
(DDR-45, DDR-48).
Absent dimensions are treated as actionable (!= "false"). Deferred
dimensions (pending_until in the future) are excluded from the
scan. Adding a new concern = new dimension + handler, no decider
changes.
The decider determines the next action to execute — it does not set phases or modify state.
MR upsert semantics
The MR submission endpoint uses upsert logic keyed on
(project, source_branch). Callers always submit the current desired
state; the service decides what to do based on existing work items:
stateDiagram-v2
[*] --> created : no active item
[*] --> unchanged : active, same hash
[*] --> updated : active, different hash
[*] --> retried : failed item
created --> active : new work item
unchanged --> active : no-op
updated --> active : refresh fields
retried --> active : reset phase
| Upsert action | Condition | HTTP status | Effect |
|---|---|---|---|
created |
No active/failed item for (project, branch) | 201 | New work item, actionable = true |
unchanged |
Active item exists, same content_hash |
200 | No-op, return existing item |
updated |
Active item exists, different content_hash |
200 | Refresh file_actions/title/description, set actionable = true |
retried |
Failed item exists for (project, branch) | 200 | Reset to active, clear error, update content, set actionable = true |
Content hash. A SHA-256 of the normalized file_actions JSON
(sorted by file_path + action, compact separators, sorted keys).
Enables cheap same-content detection without comparing the full payload.
Idempotency key. Derived server-side as
mr:{project}:{source_branch}:{generation} where generation is a
counter per (project, source_branch) pair. A partial unique index
ensures uniqueness only among non-terminal phases — completed and
cancelled items can share a key prefix without blocking new items for
the same branch.
Race handling. If two concurrent inserts race for the same
idempotency key, the loser catches the IntegrityError, rolls back,
and retries the full upsert — the winner is now visible and the
correct action (unchanged/updated) is taken.
MR action chain
The MR work item lifecycle chains executor actions. After upsert, the processing loop calls the decider to select the first action; subsequent actions are also selected by the decider after each executor completes:
ensure_mr— creates or updates the MR. Ifmr_iidis not set, creates the branch, commit, and MR via the GitLab Commits API and applies themanaged-by::workqueuescoped label. Ifmr_iidis already set, force-pushes updated content to the existing MR branch and syncs MR metadata (title, description). Detects empty diffs (before creation) or closes an empty MR after force-push. Captures the commit SHA ascurrent_sha, resets theneeds_plrdimension (clearing stale component state), and re-armsneeds_expected_plrs. Setsneeds_mrto"false"on success.ensure_expected_plrs— fetches the Konflux trigger manifest (.tekton/trigger-manifest-pull-request.json) from the target project via the GitLab repository files API, parses the component-to-path mappings, and derives the set of affected components from the MR’s changed file paths. Stores the component list in theneeds_expected_plrsdimension detail. If the manifest is missing (404), stores an empty list. Retries on transient API errors with a 5-minute backoff.ensure_plr_status— fallback executor that fetches current PLR state from the Konflux K8s API (with Kubearchive fallback) using label selectors on the commit SHA. Fires on 30-minutepending_untilre-arm when PipelineRun events have been silent. Writes the complete component snapshot into theneeds_plrdimension detail and re-armsneeds_dashboardto trigger a note update. Per component, the newest PipelineRun bystatus.startTimewins — after a retrigger, the older archived run does not overwrite the newer live one. If any component is still non-terminal, re-armsneeds_plrwith another 30-minute delay. RequiresKONFLUX_KUBECONFIG_PATHand optionallyKUBEARCHIVE_URLenvironment variables. The KubeArchive client reuses the bearer token from the kubeconfig’s auth settings.ensure_dashboard_note— posts or updates an internal note on the MR with the Hummingbird dashboard link and a per-component Konflux build status table. The note uses the:robot: **Hummingbird Status**header shared with the dashboard-mr-linker Lambda. On first run, it creates the note and storesnote_idin the dimension detail. On subsequent runs (triggered by PipelineRun events or the PLR fallback fetch), updates the existing note in place if the rendered body differs from the storednote_body. Always setsneeds_dashboardto"false"on completion — re-arming is handled by the PLR SNS handler and theensure_plr_statusexecutor, which setneeds_dashboard = "true"whenever PLR state changes.check_empty_diff— checks whether an active MR’s diff became empty (target branch moved). Triggered by theneeds_empty_checkdimension viapending_untilexpiry (~every 30 minutes per item). If the diff is empty, closes the MR with an explanatory note, deletes the branch, and cancels the item. Setsneeds_empty_checkto"false"on completion. Transient GitLab API errors are logged but do not fail the item.reconcile_mr— reconciles desired vs actual MR state. Triggered by theneeds_reconciledimension, set by webhooks (approved, unapproved, update, pipeline success/failure) andpending_untilexpiry. Merges directly whendetailed_merge_statusismergeableandauto_mergeis enabled. Syncs title and description if drifted. Re-armsneeds_reconcileaccording to the merge outcome: a failed merge PUT retries immediately;unchecked/checking/unknown(a mergeability recheck in flight, completing seconds later) retry on an exponential schedule — 15 s doubling to a 5-minute cap, with the step tracked in the dimension detail; every other status keeps the 30-minute cycle.
Draft state. The draft submission option controls initial MR
creation only. After creation, GitLab owns Ready/Draft state;
update_mr and reconcile_mr preserve the current GitLab state while
synchronizing title text. The details.draft response field reports the
initial creation setting, not the live GitLab status.
After all actions complete, the decider returns None and the item
sits in active phase waiting for event-driven lifecycle tracking
(merge → completed, close → cancelled).
Label coordination with dashboard-mr-linker. The managed-by::workqueue
label causes the dashboard-mr-linker Lambda to skip the MR, so the
workqueue-service creates the dashboard note for its MRs. When
dashboard_base_url is not configured, ensure_dashboard_note returns
immediately without posting a note.
head_pipeline_id on mr_details
The current head pipeline ID is stored as a typed integer column
(head_pipeline_id) on the mr_details table rather than as a
dimension. Build events and the reconciler update it directly. The CI
retry executor uses it to filter stale jobs from previous pipelines.
Processing loop
The ProcessingManager uses a bounded ThreadPoolExecutor (10 workers):
- One loop per item — a loop takes a session-level advisory lock
pg_try_advisory_lock(0x5751, item_id & 0x7FFFFFFF)on a pinned connection and holds it for its whole lifetime. Another loop (another pod, the sweep, or the local rerun path) returns immediately when the lock is held; duplicate submissions are cheap. Losers incrementhummingbird_workqueue_processing_lock_skipped_total, labelled by item type. The key is the item UUID’s low 31 bits, so two unrelated items can rarely collide; the skipped item is re-spawned by the sweep once the winner finishes — delayed, never lost. - Serial per item — one action at a time per work item, across pods.
- Parallel across items — independent items processed concurrently.
- Per-item deadline — processing loops exit after 60 seconds to prevent stuck items from holding pool workers.
- Executors apply state updates directly (phase, MR fields) and must
not call
session.commit()orsession.rollback(). - After execution, the loop re-derives the next action via the decider.
- Crash recovery — the lock lives on a pinned database connection;
if the pod or connection dies, PostgreSQL releases it. A death at any
loop step (inside an executor or at the post-execution state write)
aborts the loop; it never continues on a reconnected connection
without the lock. The item stays
actionable, so the periodic sweep re-spawns it. There is no lease, heartbeat, or TTL. - The periodic sweep submits actionable items and items with pending
events or due
pending_untildimensions (no synthetic events). It is deadline-driven: after each pass it sleeps until the earliest armedpending_untildeadline (plus ~1 s so the deadline is already expired on wake), or for ~1 s when a deferred timer is already due (a single query returns the earliest future deadline and a past-due flag over one snapshot, so a timer expiring between the sweep’s scans cannot be missed by both), keeping the former 5-minute interval as a backstop cap for the un-timed legs — actionable crash recovery, pending events, and deadlines armed by other pods. A wake event cuts a long wait short whenever an enqueue or an executor commit arms an earlier deadline in-process; sweep-initiated submissions do not wake it, as they arm no timer themselves.
Connection pinning. Each loop checks out one connection and binds
every session it opens to that connection, so the session lock survives
the per-action commits. The engine pool is sized to the worker count
with overflow headroom, and libpq keepalives plus tcp_user_timeout
detect a dead server instead of leaving a lock on a half-open socket.
The loop releases the lock explicitly before returning the connection
to the pool; if the release fails, it invalidates the DBAPI connection,
because a pooled close() alone would keep the session lock alive.
MR labeling
The executor always adds the managed-by::workqueue scoped label to
every MR it creates, merged with any caller-provided labels. This label
is a coordination signal: other services (e.g.
dashboard-mr-linker)
check for this label and
skip managed MRs, since the workqueue-service handles dashboard note
creation itself via the ensure_dashboard_note action.
Event processing
The SQS consumer thread receives events from the SNS topic — GitLab
webhooks (merge_request, build) and Kubernetes resource events
(PipelineRun) — and stores them in the pending_events table.
Work-item threads drain and process these events as part of their main
loop.
Consumer thread. A single daemon thread (poll_loop) long-polls SQS
with MaxNumberOfMessages=10 and WaitTimeSeconds=20. For each
message it decodes the SNS envelope (handles gzip+base64), resolves
matching work items via the resolver registry, INSERTs into
pending_events, commits, deletes the SQS message, and calls
manager.spawn_loop() for each affected work item. The consumer is started
only when WORKQUEUE_SERVICE_EVENTS_QUEUE_URL is set.
State mapping. MR state changes are mapped to phases:
| GitLab state | Phase | Notes |
|---|---|---|
merged |
completed |
Terminal |
closed |
cancelled |
Terminal (recover via retry API) |
opened |
— | Ignored (only active items accept events) |
locked |
— | Ignored (transient during merge) |
Only items in active phase accept event-driven transitions. Events
for terminal items (completed, failed, cancelled) are silently
discarded.
PipelineRun events. Kubernetes PipelineRun events (forwarded via
the event forwarder → SNS) are resolved to MR work items by matching
the commit SHA in the PipelineRun’s labels against
mr_details.current_sha. Only type=build PipelineRuns are
processed; test PipelineRuns are ignored. The handler also rejects
events whose SHA label no longer matches the item’s current_sha
(stale SNS delivery after a force-push). Each accepted event merges
the component’s status into the needs_plr dimension’s JSONB detail
and re-arms needs_dashboard to trigger a note update. Events are
attributed to a PipelineRun by metadata.name: every event for the
same PipelineRun carries the same status.startTime, so matching
names always apply the newer status (a terminal run never regresses
to a non-terminal one). For a different PipelineRun — a retrigger —
the event is accepted only when its status.startTime is strictly
newer (compared as parsed instants, so fractional-second formats
order correctly). An event without a startTime cannot overwrite a
component already recorded as Succeeded or Failed, nor an
in-progress component that already has a stored startTime.
Konflux SHA labels. Build and test PipelineRuns carry different SHA labels — using the wrong label returns zero results:
| PLR type | SHA label | Type label value |
|---|---|---|
| Build | pipelinesascode.tekton.dev/sha |
build |
| Test | pac.test.appstudio.openshift.io/sha |
test |
The event resolver checks pipelinesascode.tekton.dev/sha first
(build PLRs), falling back to pac.test.appstudio.openshift.io/sha
for edge cases. The K8s/Kubearchive fallback fetcher
(list_build_plrs_for_sha) queries only
pipelinesascode.tekton.dev/sha with type=build, matching the
label that build PLRs actually carry.
Out-of-order guard. The last_event_at column on work_items
rejects events with updated_at older than the last processed event.
Same-timestamp events pass the <= guard (processing is idempotent).
Exit coordination. The work-item exit decision runs under a
SELECT ... FOR UPDATE row lock on work_items, so an enqueue that
updates the item in the same transaction is either visible to the exit
or blocks until the exit commits and then leaves actionable = true.
The former pg_advisory_xact_lock insert/exit handshake was removed;
the row lock covers every writer, including the API enqueue paths the
handshake never covered. See
DDR-51.
After a no-work exit commits, the loop releases the session lock and
re-checks pending_events and the decider on the pinned connection.
If a writer committed in that gap, the loop re-acquires the session
lock and continues instead of returning. A writer that arrives after
the re-check is caught by the sweep; the item remains actionable.
Sweep extension. The deadline-driven periodic sweep submits
actionable items and items with unprocessed pending_events, providing
startup recovery and a safety net behind the row-lock exit and the
unlock/re-check. It wakes exactly at the earliest armed pending_until
deadline; the 5-minute backstop interval covers everything un-timed
(see DDR-52).
Concurrent drain tolerance. In steady state the per-item session
lock prevents two pods from draining the same item. During the rolling
update that introduced the session lock, an old pod still used
leased_until while a new pod used the session lock, so both could
briefly drain the same item. The drain therefore reads each listed
event’s (id, event_type, event_body) before processing the first one
and deletes by id — a row another pod already deleted is a no-op. An
event may be applied by both pods; application is idempotent (dimension
upserts, last_event_at guard), matching the at-least-once semantics
of crash recovery. See
DDR-50.
The same coexistence applied to executor actions, not just the drain:
the two-key session lock deliberately could not conflict with the old
single-key/lease path, so an old pod’s leased loop and a new pod’s
locked loop could both run the same action for one item during the
overlap. Read-and-upsert actions (reconcile_mr, ensure_plr_status,
ensure_expected_plrs) converge on the same state. Creation actions
(ensure_mr, ensure_dashboard_note) and retry_ci_jobs check their
result field (e.g. mr_iid) before acting, so a duplicate requires
both loops to pass that check before either commits — possible in the
worst case, producing one duplicate external mutation (a second MR or
note, or a second CI retry) for an item that was mid-action during the
pod replacement. New pods deliberately ignored leased_until rather
than skipping rows with an unexpired value: a lease written by a pod
that died mid-rollout must not hide the item from the sweep. That
coexistence window ended when the locking redesign finished rolling
out; the contract migration drops the leased_until
column, so the lease mechanism no longer exists. See
DDR-51.
Adding a new work item type
- Create a detail table and SQLAlchemy model (e.g.
CveWorkItem) - Register a decider function:
decider.register("cve", _cve_decide) - Register executor functions for each action
- Add a submission endpoint:
POST /api/work-items/cve
No changes needed to the processing loop, claiming, or generic API.
API
Endpoints
| Method | Path | Description |
|---|---|---|
POST |
/api/work-items/mr |
Upsert an MR work item (create, update, or skip) |
GET |
/api/work-items/mr |
List MR items (source, project, etc.) |
GET |
/api/work-items |
List all items (type, phase filters) |
GET |
/api/work-items/{id} |
Get item with nested details |
POST |
/api/work-items/{id}/retry |
Retry a failed item |
GET |
/api/whoami |
Introspect caller identity and permissions |
GET |
/oauth/start |
Initiate OAuth login (handled by oauth-proxy) |
GET |
/oauth/callback |
OAuth provider callback (handled by oauth-proxy) |
POST |
/oauth/token |
Exchange OAuth session cookie for Bearer token |
GET |
/healthz |
Health check |
GET |
/metrics |
Prometheus metrics |
Responses include a nested details dict with type-specific fields:
id: "..."
type: mr
phase: active
details:
source: rpms-ci
mr_iid: 42
mr_url: https://gitlab.com/org/group/project/-/merge_requests/42
Authentication
All callers authenticate via Authorization: Bearer tokens. The
service validates each token in one of two ways:
| Caller | Token type | Validation |
|---|---|---|
| K8s SA | JWT (projected) | JWKS signature verification |
| GitLab CI | JWT (OIDC) | JWKS signature verification |
| Human (web/CLI) | sha256~... (OCP) |
OCP users/~ API |
| Local dev | none | WORKQUEUE_SERVICE_LOCAL_AUTH_USER env var |
X-Forwarded-* headers are never trusted on API paths. The proxy
protects only the OAuth login flow (/oauth/start, /oauth/callback)
and the POST /oauth/token endpoint, which bridges cookie sessions to Bearer
tokens for browser and CLI users. The /oauth/token endpoint uses POST (not
GET) to prevent CSRF — browser-initiated GETs via <img> or
<script> tags cannot trigger it.
OAuth proxy deployment requirements
The oauth-proxy sidecar handles only the OAuth login flow. All /api/
paths bypass the proxy — the app handles auth itself. The proxy must be
configured with these flags:
| Flag | Purpose |
|---|---|
--bypass-auth-except-for=^/oauth/ |
Proxy only protects the OAuth flow (login + token exchange) |
--pass-access-token |
Sets X-Forwarded-Access-Token on cookie-authenticated requests (required for /oauth/token). Cookie secret must be exactly 16, 24, or 32 bytes (AES key size) |
--pass-user-headers |
Sets X-Forwarded-User on protected paths |
--cookie-refresh=1h |
Refreshes the OCP token inside the cookie before it expires (see below) |
--cookie-expire=24h |
Hard session limit, forces re-login |
--cookie-samesite=lax |
CSRF defense-in-depth for /oauth/token (POST already blocks the main vectors) |
Why --cookie-refresh is critical
The proxy manages two independent lifetimes that are easy to confuse:
- Session cookie — controlled by the proxy (
--cookie-expire, defaults to browser session). Contains the encrypted OCP access token. - OCP access token — controlled by OCP (default 24h). The actual
credential the app uses to call
users/~.
The proxy does not know when the OCP token inside its cookie expires.
It only checks whether the cookie is valid (signature, expiry).
Without --cookie-refresh, the cookie outlives the token:
| Time | What happens |
|---|---|
| T+0h | User logs in. Proxy creates cookie with fresh OCP token (expires T+24h) |
| T+24h | OCP token expires. Cookie still valid. |
| T+25h | User makes request. Proxy sets X-Forwarded-User (valid cookie) and X-Forwarded-Access-Token (expired token). App calls users/~ with expired token → 401 → anonymous → “not authorized”. |
--cookie-refresh=1h tells the proxy to silently refresh the OCP token
inside the cookie every hour, keeping it well within the 24h expiry.
The workqueue-service also uses a TTL-bounded cache (5 minutes) for
users/~ results, so expired-token failures are never permanent.
Authorization
Access rules are defined in the config file. Each rule maps a remote
identity to permissions (read, retry, submit), optionally scoped
to work item types. Rules use either groups (OCP group membership) or
issuer+claims (JWT bearer). Claim matching follows Vault’s
bound_claims pattern: all conditions AND, list values OR.
All permissions (including read) must be granted by a matching rule.
Authenticated callers with no matching rules get 403. Multiple matching
rules are merged (union of permissions).
See Configuration for the config file format.
Whoami introspection
GET /api/whoami returns the caller’s identity and effective
permissions. Example response:
authentication:
tier: ocp-user
identity: admin-user
groups:
- konflux-hummingbird-admin-access
authorization:
effective_permissions:
mr:
- read
- retry
- submit
matching_rules:
- groups:
- konflux-hummingbird-admin-access
permissions:
- read
- submit
- retry
item_types: []
API Documentation
Interactive API documentation available without authentication:
/docs (Swagger UI), /redoc (ReDoc), /openapi.json.
Audit Log
State-changing operations (processing, API calls, auth decisions)
emit audit entries. Every entry is logged at INFO level. When
WORKQUEUE_SERVICE_AUDIT_S3_BUCKET is set, entries are additionally
archived to S3 as gzipped JSON objects for long-term querying via
Athena.
Audited events
| Event type | Source | Description |
|---|---|---|
action_started |
Worker | Action execution began |
action_completed |
Worker | Action finished (success or failure) |
action_decided |
Decider | Decider assigned next action |
action_no_op |
Decider | Decider returned no next action |
phase_change |
Worker / API / Events | Phase transition (max attempts, submit, event) |
item_updated |
API | Work item content updated via upsert |
dimension_change |
Events / Executor | Dimension upsert (needs_dashboard posted, etc.) |
retry_requested |
API | Retry endpoint reset a failed item |
auth_failed |
Auth | Anonymous request rejected (401) |
auth_denied |
Auth | Authenticated caller lacks permission (403) |
S3 key format
When S3 archival is enabled, entries are stored as gzipped JSON with an Athena-partitionable key layout:
{prefix}/YYYY/MM/DD/HH/MM/{timestamp}#{work_item_id}::{event_type}.json.gz
The ContentEncoding: gzip header is set so S3 API consumers get
transparent decompression.
Field conventions
Follow these conventions when adding new audit calls:
actor: caller identity for API/auth events; thread name for worker events;"system"for decider/system-triggered transitions.old_state: state before the change. Includephaseandactionwhen both are relevant. Omit on creation (no prior state) and on auth events (no work item state).new_state: state after the change. Same keys asold_state. Omit fields that areNone(filtered byto_dict()).trigger: what caused this event. Always includesource("api","worker", or"decider"). Addactionfor deliberate operations,reasonfor error/system transitions,errorfor failure details.metadata: context orthogonal to the state transition — request info on auth events, type-specific data on submit. Not for duplicating state already inold_state/new_state.
CI Job Retry
When a CI pipeline on a managed MR fails, the service can automatically retry jobs that match known transient failure patterns. This is ported from the CKI pipeline-herder’s proven regex-based job log matching engine.
How it works
- GitLab sends a
build(job) webhook when a CI job finishes - The event handler appends the failed job to the
needs_ci_retrydimension’s jobs list and sets status to"true" - The decider selects the
retry_ci_jobsexecutor - The executor processes all jobs in the list, matching each against the config tree and retrying matching jobs via the GitLab Jobs API
- Jobs with non-zero delay set
pending_untilto the earliest per-jobnot_beforetimestamp — the sweep notifies when it expires - Multiple failed jobs in the same pipeline accumulate in the list
needs_ci_retry dimension
The needs_ci_retry dimension follows the unified state machine (see
Unified state diagram). Jobs with non-zero
delay use the deferred path: status="true" with pending_until set
to the earliest per-job not_before timestamp. The sweep notifies
when any job’s delay elapses.
The true → true self-loop fires when another job in the same
pipeline fails — the build event handler appends the new job to the
existing list (clearing pending_until so the decider picks it up
immediately).
The detail JSONB stores a jobs list. Each entry carries build_id,
build_name, build_failure_reason, build_finished_at,
retries_count, and pipeline_id — all from the build webhook
payload. The executor deduplicates by build_id and skips jobs whose
pipeline_id does not match the MR’s current head_pipeline_id.
When a job has a non-zero delay, the executor stamps a per-job
not_before ISO timestamp based on build_finished_at + retry_delays[retries_count]. The dimension’s pending_until is set
to the earliest per-job not_before so the sweep fires as soon as
any job is ready.
Matcher configuration
CI job retry is configured per type under handled_types.mr.job_retry.
The config uses a recursive matcher tree where parent fields are
inherited by children:
handled_types:
mr:
job_retry:
retry_delays: ["0", "5m", "10m"]
retry_limit: 5
matchers:
- name: runner-system-failure
description: Job failed with runner_system_failure
failure_reason: runner_system_failure
- name: git-clone
description: Git clone failure
messages:
- "fatal: unable to access"
- '/error: RPC failed; curl 92 HTTP\/2 stream \d+/'
Matcher fields
| Field | Type | Default | Description |
|---|---|---|---|
name |
string | "" |
Matcher name (used in metrics and audit) |
description |
string | "" |
Human-readable description |
failure_reason |
string | "" |
Exact match on GitLab job failure_reason |
job_name |
string | "" |
Prefix match on job name |
messages |
list[str] | [] |
Literal substring or /regex/ on job trace |
tail_lines |
int | 300 |
Number of trace lines to scan (from the end) |
retry_delays |
list[str] | ["5m"] |
Escalating delays indexed by attempt ("0", "5m") |
retry_limit |
int | 3 |
Maximum retry count per job name |
matchers |
list | [] |
Nested child matchers (recursive) |
Message matching
- Literal:
"fatal: unable to access"→ substring match per line - Regex:
'/error: RPC failed; curl \d+/'→re.searchon joined lines (slash-delimited)
Retry safety
Before retrying a job, the executor checks:
- No newer job with the same name exists (prevents double retries)
- Retry limit not exceeded (count of jobs with the same name)
Retry timing
The retry_delays list is indexed by the job’s attempt count
(0-based). When the index exceeds the list length, the last entry
repeats. A delay of "0" means immediate retry; non-zero delays set
pending_until to the computed not_before timestamp. The sweep
notifies the item when pending_until expires, waking exactly at the
earliest armed deadline.
Prometheus Metrics
The service exposes Prometheus metrics at /metrics on the same port
as the API (no sidecar). Metrics follow the repo naming convention:
generic operational metrics use the shared hummingbird_ prefix,
service-specific metrics use hummingbird_workqueue_.
Generic operational metrics
| Metric | Type | Labels | Description |
|---|---|---|---|
hummingbird_http_requests_total |
Counter | method, path, status | Total HTTP requests |
hummingbird_http_request_duration_seconds |
Histogram | method, path | HTTP request latency |
hummingbird_sqs_messages_received_total |
Counter | — | SQS messages pulled |
hummingbird_sqs_messages_processed_total |
Counter | status | SQS messages processed |
Service-specific metrics
| Metric | Type | Labels | Description |
|---|---|---|---|
hummingbird_workqueue_work_items_created_total |
Counter | item_type | Work items submitted via API |
hummingbird_workqueue_work_items_completed_total |
Counter | item_type, result | Action executions completed (success/failed) |
hummingbird_workqueue_work_items_retried_total |
Counter | item_type | Work items retried via API |
hummingbird_workqueue_action_duration_seconds |
Histogram | action | Executor action duration |
hummingbird_workqueue_active_processing_threads |
Gauge | — | Running work item threads |
hummingbird_workqueue_processing_lock_skipped_total |
Counter | item_type | Loops that lost the per-item session lock |
hummingbird_workqueue_pending_events_inserted_total |
Counter | — | Pending events inserted into DB |
hummingbird_workqueue_ci_jobs_retried_total |
Counter | project, matcher_name | CI jobs retried by the herder |
hummingbird_workqueue_ci_retry_skipped_total |
Counter | project, matcher_name, reason | CI job retries skipped |
hummingbird_workqueue_merge_rechecks_total |
Counter | merge_status | Fast-family merge recheck cycles by status |
UUID path segments in HTTP metrics are normalized to /{id} to limit
label cardinality. The /healthz and /metrics paths are excluded
from HTTP metrics.
CLI
export WORKQUEUE_SERVICE_URL=https://mr.apps.cluster.example.com
export WORKQUEUE_SERVICE_TOKEN=$(oc whoami -t)
# Update an existing file (local_path:repo_path)
workqueue-service submit \
--package kernel \
--project org/group/project \
--change-kind sync \
--file /path/to/sources.spec:sources.spec \
--commit-message "Sync sources from upstream" \
--source-branch mr-service/sync-1 --title "Sync sources"
# Create a new file
workqueue-service submit \
--package kernel \
--project org/group/project \
--change-kind sync \
--create /path/to/new-file.txt:new-file.txt \
--commit-message "Add new file" \
--source-branch mr-service/add-1 --title "Add file"
# Check status
workqueue-service status --id <uuid>
# List active MR items
workqueue-service status --phase active
# Show identity and permissions
workqueue-service whoami
# Raw JSON output
workqueue-service whoami --json
--url overrides WORKQUEUE_SERVICE_URL. --no-auth skips all
credential loading and sends unauthenticated requests (like
aws --no-sign-request). Useful with whoami to verify anonymous
access.
CLI authentication
The CLI resolves credentials in this order:
WORKQUEUE_SERVICE_TOKENenv var — used directly as a Bearer token. Recommended for CI (OIDC/projected SA tokens).- Cached token — if a valid token exists in the per-server cache, it is reused automatically.
- OAuth/Kerberos login — when neither env var nor cache is
available, the CLI performs a synchronous headless SPNEGO login
against the oauth-proxy, exchanges the session cookie for an OCP
Bearer token via
POST /oauth/token, and caches it for subsequent calls. Requiresrequests-gssapiand a valid Kerberos ticket (kinit).
Tokens are cached per-server under
$XDG_CACHE_HOME/workqueue-service/tokens/ (defaults to
~/.cache/workqueue-service/tokens/) with 0600 permissions and a
20-hour TTL. If a cached token is rejected (HTTP 401), the CLI
invalidates the cache and retries once with a fresh OAuth login.
Configuration
Config file
The service reads its configuration from a YAML file at startup. The
path defaults to /etc/workqueue/config.yml and can be overridden with
WORKQUEUE_SERVICE_CONFIG. The service refuses to start if the file is missing
or invalid (fail-closed).
authorization:
rules:
# Any in-cluster SA with the right audience can read
- issuer: k8s
claims:
aud: workqueue-service
permissions: [read]
# Human users -- OCP group membership
- groups:
- konflux-hummingbird-admin-access
permissions: [read, submit, retry]
# Specific K8s SA for CI automation
- issuer: k8s
claims:
aud: workqueue-service
sub: system:serviceaccount:hummingbird--internal:rpms-ci
permissions: [submit, retry]
item_types: [mr]
# GitLab CI OIDC
- issuer: https://gitlab.com
claims:
aud: workqueue-service
project_path: redhat/hummingbird/rpms
permissions: [read, submit, retry]
item_types: [mr]
handled_types:
mr:
gitlab_url: https://gitlab.com
projects:
redhat/hummingbird/rpms:
token_env: GITLAB_TOKEN_RPMS
redhat/hummingbird/containers:
token_env: GITLAB_TOKEN_CONTAINERS
redhat/rhel/src:
token_env: GITLAB_TOKEN_RHEL
Authorization rules
Each rule has either groups (OCP group membership from users/~
response) or issuer+claims (JWT bearer), not both.
issuerdetermines which JWKS keys validate the token signature. Usek8sas shorthand for the auto-detected cluster OIDC issuer.claimsmatches JWT payload fields. All conditions must match (AND); list values match any (OR). Values are compared case-insensitively with string coercion (handles numericproject_id, booleanref_protected).permissionsgrantsread,retry, and/orsubmit.item_typesoptionally restricts the rule to specific work item types. Omit to apply to all types.- Every issuer rule must include
audinclaimsto prevent cross-service token reuse.
Handled types
handled_types declares which work item types this instance serves.
Only listed types get their routers and executors registered. The
processing sweep only picks up items matching these types. Empty or
omitted = handle nothing.
Each type can have per-type operational config:
| Type | Field | Description |
|---|---|---|
mr |
gitlab_url |
GitLab instance URL (default: https://gitlab.com) |
mr |
dashboard_base_url |
Dashboard URL for note creation (when unset, notes are not created) |
mr |
job_retry |
CI job retry matcher config (when unset, retries are disabled) |
mr |
projects |
Per-project/group token and URL configuration |
Per-project token configuration
The projects map under handled_types.mr controls which GitLab API
token the executor uses for each target project. Keys are GitLab
project paths or group prefixes. Every project submitted to the service
must match an entry – there is no default fallback token.
Matching order:
- Exact match on the full project path
- Longest group-prefix match on a
/boundary (e.g.redhat/rhel/srcmatchesredhat/rhel/src/kernel)
Each entry requires token_env (the name of the environment variable
holding the GitLab API token) and optionally overrides gitlab_url:
handled_types:
mr:
gitlab_url: https://gitlab.com
projects:
redhat/hummingbird/rpms:
token_env: GITLAB_TOKEN_RPMS
redhat/hummingbird/containers:
token_env: GITLAB_TOKEN_CONTAINERS
gitlab_url: https://gitlab.example.com
redhat/rhel/src:
token_env: GITLAB_TOKEN_RHEL
At startup, the service validates that all referenced token_env
variables exist in the process environment and refuses to start if any
are missing.
Adding a new issuer
Add a rule with the issuer’s OIDC discovery URL. The service discovers the JWKS endpoint lazily on the first token from that issuer:
- issuer: https://token.actions.githubusercontent.com
claims:
aud: workqueue-service
repository: org/repo
permissions: [submit]
types: [mr]
Environment variables
| Variable | Default | Description |
|---|---|---|
DATABASE_URL |
— | PostgreSQL connection URL (required) |
SENTRY_DSN |
— | Sentry DSN for error reporting |
WORKQUEUE_SERVICE_CONFIG |
/etc/workqueue/config.yml |
Config file path |
WORKQUEUE_SERVICE_LOCAL_AUTH_USER |
— | Dev-only auth bypass (warns at startup) |
WORKQUEUE_SERVICE_URL |
— | CLI: service URL |
WORKQUEUE_SERVICE_TOKEN |
— | CLI: Bearer token |
(per-project token_env) |
— | GitLab API token(s) for MR executor (set in config projects) |
WORKQUEUE_SERVICE_EVENTS_QUEUE_URL |
— | SQS queue URL for events (when unset, event consumer is disabled) |
KONFLUX_KUBECONFIG_PATH |
— | Path to Konflux kubeconfig for PLR fallback fetch |
KUBEARCHIVE_URL |
— | Kubearchive API URL for historical PLR lookups |
WORKQUEUE_SERVICE_AUDIT_S3_BUCKET |
— | S3 bucket for audit log (production) |
WORKQUEUE_SERVICE_AUDIT_S3_PREFIX |
workqueue-service-audit |
Key prefix within the audit S3 bucket |
AWS_ACCESS_KEY_ID |
— | AWS credentials for S3 audit archival |
AWS_SECRET_ACCESS_KEY |
— | AWS credentials for S3 audit archival |
AWS_DEFAULT_REGION |
— | AWS region for S3 audit archival |
Development
Local Development
cd workqueue-service
./dev.sh db-start # Start local PostgreSQL
./dev.sh db-migrate # Run Alembic migrations
./dev.sh start # Start the service
./dev.sh db-shell # PostgreSQL shell
./dev.sh db-stop # Stop PostgreSQL
./dev.sh db-reset # Stop and delete volume
Running Tests
cd workqueue-service
pip install -e ".[dev]"
python -m unittest discover tests
Tests use testcontainers[postgres] locally (auto-detects Podman) or a
CI-provided PostgreSQL sidecar via DATABASE_URL.
License
This project is licensed under the GNU General Public License v3.0 or later — see the LICENSE file for details.