Architecture Decision Records

• 1: ADR-0001: Unified Observability Stack
• 2: ADR-0002: AI-Assisted SDLC Workqueue
• 3: ADR-0003: Metadata and Versioning Specification

1 - ADR-0001: Unified Observability Stack

• Status: Implemented
• Date: 2026-07-19 (updated 2026-08-19)
• Author: Robert Sturla
• Jira: HUM-4790 (Epic), HUM-4791 (Design task)

Context

Hummingbird had no single place to view the health of builds, infrastructure, and services. Multiple disconnected monitoring and alerting implementations existed — CloudWatch dashboards, a custom cloudwatch-log-forwarder Lambda, an SNS AlertsTopic for consumer CloudWatch alarms, and per-service custom dashboards. Workloads span multiple clusters with no cross-cluster visibility. This created blind spots, duplicated effort, and inconsistent alerting behaviour.

One standardised observability stack is needed. No duplication. One toolchain, used everywhere, federated across clusters from day one.

Operational runbooks live in documentation/monitoring.md in the infrastructure repository. This ADR records the architecture decision.

Cluster Constraints

Hummingbird has no cluster-admin access on any cluster. All resources must be deployed within tenant-provisioned namespaces. Operator CRDs and RBAC grants are controlled by the cluster platform team — what is available differs between clusters.

This constraint drives a hub-spoke split architecture: the hub (MPP) runs the backends and Grafana, while spoke clusters run lightweight collectors that forward all telemetry to the hub.

Decision

Stack

Hub (MPP, hummingbird--monitoring-hub):

Spoke (every cluster with hummingbird--monitoring):

Spoke clusters have no local backends. All storage, alerting, and visualisation is centralised on the hub. Each spoke forwards to both the staging and production hubs.

Distributed tracing (Tempo) is not deployed. See Future Work.

Alerting Architecture

Mimir evaluates PromQL alerting rules (ruler) and delivers notifications through its built-in Alertmanager. Grafana is not in the evaluation path.

Mimir ruler (PromQL) → Mimir Alertmanager → Slack / email

• Rules are Git-provisioned in the infrastructure repository and mounted into Mimir as a ConfigMap.
• All alerting runs on the hub. Spoke clusters do not run Alertmanager. Spoke alerting depends on network connectivity to the hub — if a spoke loses connectivity, alerts for that cluster are delayed until the connection is restored. This is an accepted tradeoff given the operator constraints on spoke clusters (see Cluster Constraints).
• Staging Alertmanager receivers are empty (a blackhole) so staging does not page Slack or email.

Alert Routing

Production only; staging is a blackhole.

Deployment Model

The hub runs the backends, Grafana, and Alloy Hub. Spoke clusters run Alloy and kube-state-metrics.

Hub (MPP):

• Namespace: dedicated hummingbird--monitoring-hub namespace (tenant-provisioned)
• Tenant-scoped: all resources deployed in namespaces owned by Hummingbird, no cluster-admin dependency
• Mimir (monolithic): accepts Prometheus remote-write from spoke Alloy and Alloy Hub, stores TSDB blocks in S3, serves PromQL, and runs the ruler and Alertmanager. Grafana queries Mimir in-cluster (no nginx basic auth on the Service port).
• Loki (monolithic): receives logs from spoke Alloy via authenticated Routes. Grafana queries Loki in-cluster.
• Alloy Hub: a dedicated Alloy instance in the hub namespace. CloudWatch metrics are ingested with prometheus.exporter.cloudwatch (YACE), filtered to Hummingbird resources (app-code=RPRM-001). Credential-expiry metrics are ingested with the textfile collector from a CI-generated ConfigMap. Alloy Hub remote-writes to in-cluster Mimir (Service DNS; no Route or CA bundle).
• Grafana: exposed via an internal Route. Authentication is Generic OAuth against gitlab.com (redhat/hummingbird group). GitLab Owner and Maintainer map to Grafana Admin, Developer to Editor, everyone else in the allowed group to Viewer. Datasources and dashboards live in grafana_data/ and are synced with grafana_data/deploy.sh. Grafana also has a CloudWatch datasource for ad-hoc browsing of AWS logs and metrics; that path is query-only and is not used for alerting or pod-log forwarding.
• External access: Loki and Mimir Routes sit behind an nginx sidecar with HTTP basic auth. In-cluster Service access (Grafana, Alloy Hub) does not require auth.

Spoke (every cluster):

• Namespace: hummingbird--monitoring
• Alloy: collects container logs via loki.source.kubernetes and scrapes prometheus.io/scrape Services, including kube-state-metrics. Forwards logs to hub Loki and metrics to hub Mimir. Dual-writes to staging and production hubs.
• kube-state-metrics: Kubernetes object metrics (deployments, pods, jobs, PVCs, resource quotas) with a curated allowlist.
• No local Mimir, Loki, Grafana, or Alertmanager.

AWS Lambda Services

Some Hummingbird services (e.g. container-catalog) run as AWS Lambda functions. Observability for those services uses the existing AWS signals rather than an OpenTelemetry Lambda extension:

• Metrics: Alloy Hub YACE scrapes curated AWS/Lambda metrics (Invocations, Errors, Duration, Throttles) into Mimir. Mimir alerting rules cover error rate and throttling.
• Logs: Grafana queries CloudWatch Logs directly via the CloudWatch datasource. Pod logs are not involved; Lambda logs stay in CloudWatch.
• Low-traffic utility Lambdas are covered by the same YACE allowlist when tagged app-code=RPRM-001.

Log Collection

Container stdout/stderr is tailed by spoke Alloy using loki.source.kubernetes (Kubernetes API; no DaemonSet, no sidecar, no ClusterLogForwarder). Logs are labeled with cluster, namespace, pod, container, and app, then dual-written to hub Loki.

This requires no application code changes. ClusterLogForwarder is available on some clusters but is not used.

High Availability

Hub:

Spoke:

Spoke clusters have no stateful observability components. HA concerns are concentrated on the hub. Single-replica monolithic Mimir and Loki are an accepted tradeoff against the proposed Prometheus + Thanos HA pair: there is no existing Prometheus fleet to retrofit, and Alloy already remote-writes.

Retention

S3 is provisioned with ObjectBucketClaim. There is no separate warm/cold tier or multi-year archive.

Operator Availability

CRD availability was audited on 2026-08-09. Hummingbird has no cluster-admin access — only operator CRDs explicitly granted to tenant namespaces are usable. That audit is why this stack does not use Cluster Observability Operator (MonitoringStack / Thanos), LokiStack, TempoStack, ClusterLogForwarder, or an OpenTelemetry Collector as the live path:

• LokiStack and TempoStack are not available to tenant namespaces on any cluster. Loki is a standalone Deployment. Tempo is deferred.
• COO MonitoringStack is available on MPP but not on spoke, and would have required Prometheus + Thanos for a fleet we do not have. Mimir accepts remote-write directly from Alloy.
• ClusterLogForwarder is available, but Alloy loki.source.kubernetes collects the same pod logs without a second operator.
• OpenTelemetry Collector was deployed earlier and removed once Alloy wrote directly to Loki and Mimir.

Relationship to Existing Dashboard

The existing hummingbird-dashboard app serves two distinct roles:

1. Data visualisation — CVE status charts, failed release counts, build metrics
2. Operational workflows — click to rerun a failed release, interactive triage, actionable controls

This observability stack takes over data collection, storage, and visualisation (role 1). Grafana is the primary place for charts, time-series dashboards, and alert-driven visualisations. The dashboard app retains operational workflow features (role 2) — interactive actions that Grafana cannot provide.

The two systems complement each other:

• The dashboard app can embed Grafana panels for richer visualisation without reimplementing charting
• The dashboard app can query the same backends (Mimir, Loki) directly for data it needs to drive workflows
• Over time, the dashboard app consumes data from the unified backends instead of maintaining parallel collection

Alerting (e.g. failed release Slack notifications) moves to Alertmanager. The dashboard app no longer needs its own alerting path.

What This Replaces

The dashboard app itself is not decommissioned — its operational workflow features (release reruns, interactive triage) remain. Only its data collection and visualisation responsibilities shift to the unified stack.

hummingbird-events-topic is the live event bus. It was not an alerting channel and is not replaced by this stack.

Consumer CloudWatch alarms and the log-forwarder were removed after the Mimir/Loki path was in place. Remaining CloudWatch use is intentional:

• Grafana CloudWatch datasource for ad-hoc AWS log and metric browsing
• CVE error-budget SLO dashboards and hummingbird-slo-alerts (no Grafana port yet)

Operations

Day-to-day endpoints, datasources, query examples, and onboarding a new cluster are documented in documentation/monitoring.md in the infrastructure repository.

Future Work

• Distributed tracing: Tempo (or equivalent) for request and pipeline traces. Not deployed; traces are not a current signal.
• CVE lifecycle tracing: instrument CVE analysis, RPM build, and container image pipelines with spans. Trace a CVE from detection → RPM fix → image rebuild → publish. Requires a trace backend and a schema (separate story).
• Spoke local alerting: if operator access on spoke clusters changes, a local evaluation path could remove the hub dependency for spoke alerts. Not required today.

Alternatives Considered

Alerting: Grafana-only vs Alertmanager-only vs hybrid

• Grafana-only: less mature routing, dedup, and silencing.
• Hybrid Grafana + Alertmanager (original proposal): Grafana would evaluate rules across metrics, logs, and traces, then forward to Alertmanager. That is not what shipped: there is no Tempo, and log alerting is not required yet.
• Mimir ruler + Alertmanager (chosen): PromQL rules and delivery in one component. Matches the metrics-first rollout. Applications can still POST to the Alertmanager API later if needed.

Metrics backend: Prometheus + Thanos vs Mimir

Prometheus + Thanos (MonitoringStack) was the original hub design. It fits retrofitting an existing Prometheus fleet. Hummingbird had no such fleet — Alloy already remote-writes. Mimir is one component for receive, storage, PromQL, ruler, and Alertmanager, instead of Prometheus + Thanos Sidecar + Store Gateway + Querier + a separate Alertmanager.

Logs: Sumo Logic vs Loki

Red Hat has a company-wide Sumo Logic license. However, the Sumo Logic Grafana plugin requires Grafana Enterprise or Grafana Cloud — it is not available in open-source Grafana. Loki integrates natively with open-source Grafana and keeps logs on-cluster.

Log collection: ClusterLogForwarder vs Alloy

ClusterLogForwarder is available on hub and spoke. Alloy loki.source.kubernetes collects the same pod logs and already runs on every spoke for metrics, so a second operator was not justified.

Spoke architecture: full stack vs collector-only

• Full stack per cluster: each cluster runs its own backends and Grafana. Local alerting survives network partitions. Requires operator CRDs Hummingbird does not have on spoke.
• Collector-only on spoke (chosen): Alloy + kube-state-metrics forwarding to the hub. Simpler spoke footprint. Tradeoff: spoke alerting depends on hub connectivity.

Lambda: OTel extension vs CloudWatch ingest

An OpenTelemetry Collector Lambda extension would export OTLP to the hub. That needs a network path from Lambda to MPP and a collector that no longer exists. YACE metrics into Mimir plus Grafana’s CloudWatch datasource for logs reuse AWS signals already produced by the functions.

HA: all components vs selective vs single replica

Full HA doubles cost with little benefit for view-only tools. The original proposal kept HA only on Prometheus and Alertmanager. The as-built stack uses single-replica monolithic Mimir and Loki; Alertmanager is in-process with Mimir. Brief downtime is accepted.

Consequences

• Team must learn Grafana dashboarding and PromQL/LogQL basics
• New hub services should expose Prometheus metrics (scrape annotation) and may add a Grafana dashboard under grafana_data/
• New spoke services must be scraped by Alloy (scrape annotation) and emit logs to stdout/stderr
• Spoke alerting depends on hub connectivity — no local alerting on spoke
• Deploying a new hub cluster requires Mimir, Loki, Grafana, and Alloy Hub, plus ObjectBucketClaims
• Deploying a new spoke cluster requires a hummingbird--monitoring namespace, Alloy, and kube-state-metrics pointing at both hubs
• CloudWatch remains for Lambda logs (Grafana datasource) and CVE SLO error-budget dashboards; it is not used for pod-log forwarding or consumer alarms

2 - ADR-0002: AI-Assisted SDLC Workqueue

• Status: Proposed
• Date: 2026-08-24
• Author: Michael Hofmann
• Jira: HUM-6342 (spike)

Context

The containers and rpms repositories produce over 1,000 automated commits per week. The automation that drives this throughput is spread across multiple independent mechanisms: GitLab CI scheduled jobs running shell scripts, Renovate CronJobs in the infrastructure repo, AWS Lambda functions for MR approval, and manual developer workflows run from dev machines. Each mechanism creates and manages MRs with its own lifecycle, credentials, and error handling.

There is no unified system that tracks an MR from creation through approval to merge. The Hummingbird Agent already runs AI-powered failure analysis and code review on every MR, but cannot create or modify MRs. The security foundation for agent-authored MRs is designed in HUM-851 but not yet implemented.

This ADR documents the current state of MR automation across all repositories and entrypoints as discovery for a spike to design a unified workqueue-based system.

Scope

This document inventories what exists today. It does not prescribe the target architecture — that is the output of the spike. Design questions and patterns are included to inform the spike investigation.

Current State

Repositories

rpms (redhat/hummingbird/rpms): RPM package definitions. Each package lives in its own directory with a .spec file and metadata/<package>.json for upstream repo URL, CVE product mapping, and modification status. Fedora dist-git imports and upstream version bumps are the primary automated flows.

containers (redhat/hummingbird/containers): Container image definitions. Each image lives in images/<name>/ with Containerfiles and RPM lockfiles (rpms.lock.yaml). Lockfile refresh and Renovate dependency updates are the primary automated flows.

tools (redhat/hummingbird/tools): Lambdas, CLI tools, and the Hummingbird Agent framework. Contains mr-auto-approver, gitlab-event-forwarder, hummingbird-agent, and the CVE analysis pipeline.

infrastructure (redhat/hummingbird/infrastructure): Kubernetes manifests, AWS SAM templates, and deployment configuration. Hosts Renovate CronJob definitions and agent deployment manifests.

MR Creation Entrypoints

MR Creation Mechanism: Git Push Options

All shell-script-based MR creation uses git push options, not the GitLab Merge Requests REST API. The push options control MR creation, title, description, labels, draft status, and merge-when-pipeline-succeeds (MWPS):

git push \
  --push-option merge_request.create \
  --push-option "merge_request.title=chore(rpms): Update foo to 1.2.3" \
  --push-option merge_request.remove_source_branch \
  --push-option merge_request.merge_when_pipeline_succeeds \
  origin HEAD:refs/heads/chore/upstream-update-foo

Approval is always a separate step — never part of the creation scripts.

MR Lifecycle and Approval Flows

GitLab Project Settings (containers and rpms)

Bot Identities

Approval Paths

Three parallel approval mechanisms exist, with overlapping scope:

1. CI approval jobs — delayed jobs in each repo’s .gitlab-ci.yml:

• rpms chore_mr_approval: gates on check_konflux_statuses.py (polls every 60s up to 2 hours for Konflux statuses to appear and succeed), then approves with CHORE_MR_APPROVAL_GITLAB_TOKEN. Triggered for author chore-mr or pipelines-as-code on chore/* branches.

• containers chore_mr_approval: same author/branch gate, approves with CHORE_MR_APPROVAL_GITLAB_TOKEN. No Konflux polling. allow_failure: true.

• containers lockfile_update_approval: triggered for pipelines-as-code on lock-file-maintenance(-vulnerability)? branches. Real when: delayed / start_in: 10 minutes. No Konflux polling.

2. mr-auto-approver Lambda — event-driven via GitLab webhooks:

GitLab webhook → gitlab-event-forwarder → SNS → mr-auto-approver Lambda

Approves only (does not set MWPS). Rejects forks. Checks Konflux statuses where configured.

The Lambda is intended to replace CI approval jobs; both currently run in parallel for chore/lockfile paths.

3. Renovate automerge — Renovate sets automerge: true in its MR configuration, but GitLab still requires a non-author approval before merge. For rpms Renovate MRs, neither CI approval jobs nor the Lambda cover renovate/.* branches — these MRs are stuck until a human approves.

Konflux Status Gate

External commit statuses prefixed Konflux kflux-prd-rh03 / are set by Konflux PipelineRuns triggered via Pipelines-as-Code. The check_konflux_statuses.py script (rpms) polls these statuses:

• Missing → wait (poll continues)
• Failed → do not approve
• Invoked/running/pending/success → approve (MWPS then waits for final green, including Testing Farm via IntegrationTestScenarios)

Lifecycle by MR Type

RPMs dist-git clean (chore/dist-git-update-<pkg>): schedule creates → MWPS at creation → GitLab CI + Konflux builds + Testing Farm → CI approval job (poll Konflux, up to 2h) + Lambda → merge. Typical: tens of minutes to hours.

RPMs dist-git conflict: same branch, but MR is created as draft with no-test label and no MWPS. Requires human conflict resolution (or claude_resolve_conflict.sh), then remove draft/no-test and re-trigger.

RPMs upstream (chore/upstream-update-<pkg>): same as dist-git clean. Uses OIDC AWS lookaside upload for tarballs.

RPMs rebuild (chore/rebuild-<pkg>): same approval path. Typically operator-triggered, not scheduled.

RPMs Renovate (renovate/…): Renovate sets automerge, but neither CI approval job nor Lambda covers renovate/.* branches on rpms → stuck until human approves. The renovate-babysit agent workflow posts a triage note but does not approve or merge.

Containers Renovate lockfile (lock-file-maintenance): lockfile_update_approval (10 min delay) + Lambda (renovate/.* with Konflux check) → Renovate automerge / MWPS. Typical: 30–90+ minutes.

Containers other Renovate: Lambda covers renovate/.*. No lockfile_update_approval unless branch matches lockfile pattern.

Containers metrics report (chore/metrics-report): daily schedule → MWPS → chore_mr_approval + Lambda.

Containers CI lockfile (rpm-lockfile-updates/<image>): schedule is currently active: false. Approval path is incomplete — CI rule matches lock-file-maintenance, Lambda does not match rpm-lockfile-updates/.*.

Shell Script Automation Detail

rpms create_mr.sh (shared helper)

Used by all rpms multi-MR scripts. Expects commits already on HEAD.

Inputs: --branch NAME (required), --title TEXT (required), optional --description, --auto-merge, --draft, --label NAME (repeatable).

Algorithm:

1. Count commits ahead of target branch; if 0 → exit 0.
2. Dedup: git ls-remote --heads for branch name. If present → exit 2 (“MR already exists”).
3. Create branch, push with MR push options.
4. Parse push stdout for MR URL; if missing → exit 1.

Exit code contract: 0 = MR created, 2 = remote branch exists (skip), 1 = failure.

Limitation: stale remote branch from a closed MR still triggers exit 2. Does not update existing MR/branch content. No --force-with-lease.

rpms dist_git_update_multi_mr.sh

Creates one MR per package needing Fedora dist-git sync.

Modes: production (cwd = repo, always create MRs) vs clone dry-run vs clone + --create-mrs.

Algorithm:

1. Select metadata/*.json files; apply --clean-only / --modified-only filter; skip native packages.
2. Record START_COMMIT on target branch.
3. For each package: run ./ci/dist_git.py update PACKAGE. Exit 0 + new commit = clean update. Exit 2 + new commit = conflict update.
4. Collect all new commits, hard-reset target to START_COMMIT.
5. Per commit: parse subject (^(Update|Sync) ([^ ]+) from ... to ...), create branch chore/dist-git-update-<pkg>, cherry-pick. Conflict → draft + no-test, no auto-merge. Clean → --auto-merge.
6. Call create_mr.sh.

Branch convention: chore/dist-git-update-<pkg>. Title: chore(rpms): <commit subject> or CONFLICT: chore(rpms): <subject>.

rpms upstream_update_multi_mr.sh

Same multi-MR / cherry-pick pattern as dist-git, for upstream version bumps.

Algorithm: runs ./ci/check_upstream_versions.py check --update --sign-off, collects commits, splits into per-package MRs. Always --auto-merge. Branch: chore/upstream-update-<pkg>.

rpms rebuild_multi_mr.sh

Does not generate rebuilds. Expects local commits from dist_git.py rebuild or rebuild-rev-deps, then splits them into per-package MRs.

Unique feature: --force flag deletes existing remote branch before create, working around stale branches from closed MRs. Branch: chore/rebuild-<pkg>.

containers create_lockfile_update_mrs.sh

Creates or updates one MR per image group when RPM lockfiles changed. Prerequisite: lockfiles already refreshed (make all-host FORCE_REFRESH=true).

Algorithm (process_single_image):

1. Branch: rpm-lockfile-updates/<image>.
2. For each distro/variant lockfile: compare to HEAD after stripping .arches[].packages[].url and .arches[].source[].url via yq (URL churn ignored).
3. If no meaningful changes → skip.
4. If push-mode ≠ force and origin/<branch> exists: compare to remote; if identical → skip.
5. git switch --force-create branch, commit lockfiles.
6. Push --force-with-lease + MR push options.

Key difference from rpms: uses --force-with-lease to update existing branches in place. Dedup compares lockfile content, not just branch existence.

containers create_mr.sh (simpler helper)

Used by metrics report. Creates from working tree (not existing commits). git add --all, commit, --force-with-lease push. No dedup — overwrites existing branch.

Deduplication Summary

Hummingbird Agent Framework

The Hummingbird Agent is an event-driven LLM agent deployed on OpenShift that processes GitLab events via SQS.

Architecture

• Event-driven: consumes SQS events (pipeline failures, MR events, slash commands). No REST API — triggered only by SQS or CLI.
• Workflow-driven: investigation logic defined in markdown files as LLM system prompts. New behaviors do not require code changes.
• Sandboxed execution: all LLM commands run in isolated containers (Podman/K8s/KubeVirt) with no network access and no credentials.
• Multi-model: supports Gemini and Claude via Vertex AI.
• Token separation: model tokens (read-only, Reporter-level) are distinct from orchestrator tokens (write-capable, used only by deterministic code outside the sandbox).
• Single output action: post_gitlab_note is the only way the agent affects the outside world. It cannot create, modify, or merge MRs.

Current Workflows

analyze-failures: triggered on pipeline failure. Bulk-fetches GitLab CI job logs, Konflux PipelineRun/TaskRun metadata and logs, and Testing Farm results. Groups failures by root cause. Posts a structured MR note with collapsible forensic evidence and links.

• Tools: gitlab_get_mr_details, gitlab_get_mr_diff, gitlab_get_commit_statuses, konflux_list_pipelineruns, konflux_get_pod_log, tf_get_results, tf_get_test_log, sandbox_exec
• Pattern: bulk metadata fetch → selective deep log pulls → grouped analysis → structured note
• Flexibility: medium-low (tightly coupled to Konflux/Tekton/TF pipelines)

code-review: triggered on MR open/update. Reads the unified diff, prior review discussions, and per-repo rules. Posts review findings with severity, category, and concrete code fix examples.

• Tools: gitlab_get_mr_details, gitlab_get_mr_unified_diff, gitlab_get_mr_discussions, gitlab_get_file_at_ref, sandbox_exec
• Pattern: diff-first analysis, discussion-aware follow-up (avoids re-raising resolved issues), per-project rules overlay
• Per-repo rules: workflows/repo-rules/<group>-<repo>.md files provide project-specific review guidance (e.g., hummingbird-rpms.md documents package metadata conventions)
• Flexibility: high (generic for any GitLab MR; domain knowledge plugs in via repo-rules/)

renovate-babysit: triggered on successful Renovate MR pipeline. Classifies the update as safe or risky based on diff scope, upstream changelogs (via web search), and local API usage. Posts a triage verdict.

• Tools: gitlab_get_mr_details, gitlab_get_mr_unified_diff, web_search, gitlab_get_repo_archive, sandbox_exec
• Pattern: diff scope classification, embedded changelog first then web search, archive + grep for call-site impact
• Flexibility: medium-high (ecosystem-agnostic risk matrix; does not approve/merge — triage note only)

What the Agent Has That Maps to a Workqueue

• SQS FIFO queue with message grouping (serializes per session)
• Session persistence (S3 context.json)
• Config hot-reload in serve mode
• Per-project token separation (model / action / orchestrator)
• Rate limiting (per-workflow per-MR)
• Data source abstraction (GitLab, Konflux, Testing Farm)
• Auto-resolve on push/success

What the Agent Lacks for MR Lifecycle Management

• No MR creation capability (no git write operations, no push access)
• Read-only GitLab tokens for the model
• No workqueue for MR lifecycle state (events are fire-and-forget)
• Single output action (post_gitlab_note only)
• No safe-outputs pattern (no validation of proposed writes)
• No deterministic fallback paths for well-understood operations
• No request/response API (SQS consumption and CLI only; external services cannot invoke a workflow and retrieve structured results)

Event Bus

The existing event pipeline could serve as the backbone for a workqueue:

GitLab MR / note / pipeline webhooks
  → gitlab-event-forwarder (API Gateway → Lambda)
  → SNS hummingbird-events-topic
  → mr-auto-approver Lambda (approve safe bot MRs)
  → hummingbird-agent SQS (analyze / review / babysit)
  → hummingbird-status SQS (pipeline tracking)

mr-auto-approver is a pure rule-based approver — no LLM involvement. hummingbird-mr-human-tracker detects human fixes on Renovate/bot MRs and tracks them in Jira epics.

Security Foundation: HUM-851

HUM-851 (“Enable Secure Agent MR Creation”) is an existing epic (status: New) that designs the security posture for agents creating MRs.

Core Principle

Instead of protecting CI secrets from agent-authored MR pipelines, eliminate all secrets from containers/rpms/tools CI entirely. Move all secret-dependent jobs to the infrastructure repo’s CI. With zero secrets, a compromised agent MR pipeline can only access CI_JOB_TOKEN (single-project-scoped for bot users) and the runner’s network.

What Moves to Infrastructure Repo CI

This migration directly enables centralized MR creation.

Target Security Posture

• Zero CI secrets in target projects
• Restricted runners with egress filtered to package registries only
• Agent never holds write tokens — writes happen in deterministic orchestrator code
• Diff safety validation — blocklist prevents changes to CI config, .tekton/, dependency files
• Human review required for every agent MR, auto-labeled agent-generated
• Rate limiting per-project and global; auto-close stale MRs
• Monitoring: every agent MR logged and Slack-notified

Task Breakdown

Renovate Deployment

Deployed as Kubernetes CronJobs in the infrastructure repo (kubernetes/renovate/20-cronjobs.yml.j2):

• 5 repos: containers, infrastructure, k8s-test-pipeline, rpms, tools
• Each gets its own CronJob running hourly with concurrencyPolicy: Replace
• Uses MintMaker image (MINTMAKER_IMAGE)
• containers/rpms use bash wrappers and custom config files
• Custom env variables for RPM lockfile cache, Rust/Cargo paths

CVE Analysis Pipeline

The CVE analysis pipeline is a complex, mature system. The relevant interface points for a workqueue are:

1. hummingbird-cve-analysis CronJob (K8s, every 30 min prod) — runs cve_analysis --resolve which triages Jira Security tickets, transitions statuses, creates advisory MRs, manages labels.
2. collect_cve_dashboard — runs after analysis, posts lifecycle data to the dashboard API.
3. /cve skill — reads staged CVE analysis data from Jira tickets and creates fix MRs in the rpms repo.

The workqueue integration point: when /cve or CVE analysis determines an MR should be created, that request becomes a work item.

RPM-to-Container Pipeline Flow

A single RPM spec change triggers a cascade that spans both repos and multiple build systems. The full flow is documented in the RPM Pipeline and Image Pipeline docs.

RPM spec change (rpms repo)
  → MR validation (GitLab CI + Konflux build + Testing Farm)
  → Merge to main
  → Konflux RPM build (Tekton PipelineRun per package, mock hermetic)
  → RPM signing (Kerberos-based)
  → Publish to Pulp (packages.redhat.com)
  → Lockfile update detects new RPM versions (containers repo)
  → Lockfile MR created (Renovate or create_lockfile_update_mrs.sh)
  → Container build (Konflux, multi-arch)
  → Container testing (Testing Farm + K8s tests)
  → Enterprise Contract validation (Conforma)
  → Release to Quay.io

Key observations for the workqueue:

• The cascade is implicit — there is no explicit trigger from RPM publish to container lockfile update. Lockfile updates discover new RPMs by polling Pulp (via make all-host FORCE_REFRESH=true or Renovate’s RPM lockfile maintenance).
• A single RPM change can fan out to many container images (any image that includes that package in its lockfile).
• The RPM pipeline has 5 stages (spec change, MR validation, build, signing, publishing). The container pipeline has 6 stages (source templates, generation, build, testing, Enterprise Contract, release).
• Testing Farm runs integration tests on merge requests only, not on main branch builds. Container tests include reverse dependency testing (rebuilding dependent images locally in the test environment).
• Enterprise Contract (Conforma) validates supply chain security, hermetic builds, and policy compliance before release.

Decision

To be determined by the spike investigation. This section will be updated with the chosen architecture.

The spike should investigate a unified workqueue-based system where:

• Queue feeders replace or wrap the current scheduled scripts and Renovate outputs
• Queue processors handle MR creation, approval, rebasing, and merge through deterministic code
• Agentic capabilities (review, patch generation, triage) are invoked as composable services with structured output
• The security model follows HUM-851’s design: agents reason read-only, orchestrators execute writes

Design Patterns and Prior Art

GitHub Agentic Workflows (gh-aw)

gh-aw provides a reference architecture for AI-assisted repository automation with security guardrails:

• Read-only agent execution: agents run with no write permissions, no secrets
• Safe Outputs: agents buffer intended write operations as structured JSON artifacts; separate permission-controlled jobs validate and execute them
• Threat detection: a separate AI agent scans agent outputs before write execution
• Plan-level trust: trust is embedded in the workflow definition (what the agent is allowed to do), not in the agent’s runtime decisions

Key pattern: separate reasoning (agentic, sandboxed, read-only) from execution (deterministic, permission-controlled, auditable).

Existing Hummingbird Patterns

• SQS FIFO queue with message grouping: already serializes work per session; can be extended to serialize per-MR
• Workflow-as-prompt: markdown files define behavior; new behaviors do not need code changes
• Data source abstraction: GitLab API access is modular
• Token tier separation: model read tokens, action write tokens, orchestrator tokens — maps directly to the safe-outputs pattern
• Sandbox isolation: reusable for agentic review/analysis tasks

Agent as Composable AI Service

One approach to bridge the agent framework and a workqueue is to evolve the agent from a self-contained event processor into a building block that non-AI services can call:

• Request/response API: HTTP endpoint accepts workflow requests, returns a UUID; caller polls for structured JSON results. The caller (e.g., a workqueue herder) applies deterministic logic to decide what to do with the output.
• Structured output: agent produces typed JSON (not just a GitLab note), enabling callers to process results programmatically.
• Safe-output actions: new action types beyond post_gitlab_note (e.g., create_mr, update_mr) with per-workflow path allowlists, diff size limits, and branch naming constraints. The orchestrator validates each action against the workflow’s configured constraints before execution.

Open Questions

1. Queue technology: extend SQS FIFO (already in use by the agent) or use a different queue (K8s-native, Redis, PostgreSQL)?
2. Processor deployment: K8s Deployments polling SQS, GitLab CI jobs triggered by webhooks, Lambda functions, or a combination?
3. Renovate strategy: keep Renovate as a separate MR creator and add a herder, or replace Renovate’s MR creation with the workqueue?
4. Write credentials: per-repo project access tokens (current model) or a single bot account? How does HUM-851’s “eliminate CI secrets” model interact with workqueue processors that need write access?
5. Safe outputs validation: simple schema validation, deterministic allowlist checks, or full threat detection (as in gh-aw)?
6. Review scope: which MRs need AI review before merging vs which can be auto-merged with deterministic checks only?
7. State tracking: queue, GitLab labels, a database, or the existing hummingbird-status PostgreSQL?
8. Agent evolution: request/response API, safe-output actions, or hybrid? How does this relate to HUM-857?
9. HUM-851 relationship: is the workqueue spike a superset of HUM-851, or a separate epic that depends on HUM-851 Phase 1?

Consequences

To be updated after the spike investigation determines the architecture.

Known consequences of any workqueue approach:

• Centralizing MR creation provides a single point for audit, rate limiting, and monitoring
• Moving secret-dependent jobs to the infrastructure repo (per HUM-851) is a prerequisite regardless of workqueue design
• The existing CI approval jobs, Lambda approver, and Renovate automerge have overlapping scope that must be reconciled
• RPMs Renovate MRs currently have no automated approval path — any design must address this gap
• The agent’s current workflows (failure analysis, code review, Renovate babysit) complement the workqueue and should not be disrupted

File and System Inventory

Infrastructure repo

Tools repo

RPMs repo

Containers repo

3 - ADR-0003: Metadata and Versioning Specification

• Status: Proposed
• Date: 2026-08-31
• Author: Brent Baude
• Jira: HUM-6212

Context

The metadata file at metadata/<package>.json and the package directory at rpms/<package>/ form a pair, one per package. The package directory holds the build input: the spec file, patches, and the sources checksum manifest that mock consumes to produce RPMs. This specification does not change anything about that directory.

The metadata file is the machine-readable record of a package’s relationship to Fedora and upstream. Automation reads it to decide what it may do: whether dist_git.py update may overwrite the package directory, which upstream project check_upstream_versions.py tracks, and how CVE analysis maps the package to its upstream identity.

The metadata file does not duplicate build inputs, and it does not record the release currently shipped: the spec’s Release: line is the sole authoritative value for what ships, and is what NVR computation uses (see Computing the built NVR). For packages on Fedora’s version, fedora.release also appears in metadata — but only as the base value the spec’s Release: is built from, never as a substitute for it.

The two locations are updated independently; neither is derived from the other. They must stay consistent by convention, not by construction — modification_status is correct only if it accurately reflects whether the package directory actually diverges from Fedora’s import.

This document normatively defines the target structure of metadata/<package>.json and how release and versioning behave as a function of modification_status. It describes the target metadata shape, not what is present on disk in metadata/*.json today. Bringing existing metadata in line with this specification is separate, tracked work.

Decision

Package states

Every package occupies exactly one of three states, describing its relationship to Fedora at a given moment. A state is not itself a metadata field — it is what modification_status means in combination with which version the package currently ships. The rest of this specification refers to these states by name, so they are defined here first.

A package’s state follows from its modification_status, which is itself defined by the package’s relationship to a Fedora origin. modification_status is one of three values:

• clean — the package has a Fedora origin, and currently matches that Fedora import.
• modified — the package has a Fedora origin, but currently diverges from it — either through local changes while still on Fedora’s version, or by having moved ahead to a newer upstream version.
• independent — the package has no Fedora origin at all.

For a modified package, the state further depends on whether it currently ships Fedora’s exact version or has moved ahead to a newer upstream version. The fedora object’s shape — complete or partial — records which of those is the case; it does not itself define the state.

The table below names all three states by the version they currently ship, not just by modification_status, since modified alone is ambiguous between the two Fedora-origin cases:

Field reference

This is the complete field-by-field reference for metadata/<package>.json — every field consumed by tooling in this repository, not only the fields that determine modification_status or the Fedora relationship. Fields are listed in four groups, in order: package identity and state (version through modification_reason), the Fedora-origin group (fedora and its sub-fields), fields that configure upstream-version tracking, and fields unrelated to either Fedora or upstream tracking (cve_product, version_transform, fix_status). The “Presence” column states the condition under which a field appears in the JSON; fields are omitted entirely rather than set to null or an empty value when their condition does not hold.

Computing the built NVR

The built RPM is identified by its NVR: Name, Version, and Release. Name is the spec/package name and is unaffected by any of this — only Version: and Release: vary with state.

Metadata version and the spec Version: line always match — metadata stores it separately not because the two can diverge, but because reading Version: reliably from every spec (which may build it from macros or indirection) isn’t practical, so metadata is the field automation trusts instead. Release: works the other way: the spec Release: line is the sole authoritative value for what ships, in every state. fedora.release, when present, is only the base value Release: is built from — never a substitute for it.

The table below shows, for each state, where the spec’s Version: and Release: lines come from. The prose beneath it gives the precise rebuild-increment rules that don’t compress into a table cell.

On Fedora version (clean or modified): fedora.release holds Fedora’s confirmed release number, dist tag stripped. The spec Release: is that value plus %{?dist}, with an optional trailing .N if the package has been locally rebuilt without a source change since the last import or sync. dist_git.py rebuild computes the next .N by comparing the spec’s current Release: against fedora.release: if they match exactly, this is the first local rebuild since import or sync, and .1 is appended; if the spec already carries a trailing .N ahead of fedora.release, that trailing number is incremented.

On upstream version (always modified): No fedora.release, and no other metadata field records a base value. The spec uses Release: 0.1%{?dist} the first time a package moves ahead of Fedora. Every subsequent local rebuild increments the trailing dot-number directly from the spec’s own current Release: value (0.1 → 0.2 → 0.3, and so on) — never by re-deriving it from a stored metadata value, because none exists. This is what makes the bug class behind HUM-5182 structurally impossible: that bug occurred because a stored placeholder value could disagree with the spec’s actual current Release:; with nothing stored to disagree with, the mis-increment cannot happen.

Independent: The same rule as the on-upstream-version case applies — no metadata field records a base value, and the spec Release: is authoritative. Rebuilds increment its trailing .N directly. The only difference is the very first value chosen when the package is added (typically 1 or 0.1) is a one-time human decision, not derived from anything.

Validation rules

Every rule below is a single, unconditional statement: given the field values present, exactly one outcome is correct. None of them carry a qualifier like “should,” “generally,” or “in most cases,” and none fold an exception into the wording — a rule that needs an exception is really two rules, stated separately. The rationale for why each rule holds lives elsewhere in this document (Package states, Field reference, Computing the built NVR); this section exists so a validator can implement each line as a direct check with nothing left to interpret.

1. version is present.
2. modification_status is present.
3. upstream_repo is present.
4. modification_status is clean, modified, or independent.
5. version_source, when present, is gitlab_tags.
6. source_availability_check, when present, is a name registered in SOURCE_AVAILABILITY_CHECKERS.
7. The top-level field release does not appear.
8. If modification_status is clean, fedora is present.
9. If modification_status is modified, fedora is present.
10. If modification_status is independent, fedora is absent.
11. If fedora is present, it includes git_url, branch, and sha.
12. If modification_status is clean, fedora.release is present.
13. If modification_status is modified, fedora.release is present if and only if version equals the version Fedora ships at fedora.sha.
14. fedora.release, when present, does not include a dist-tag suffix (e.g. .fc42, .el9).
15. If modification_status is modified, modification_reason is present.
16. If modification_status is clean or independent, modification_reason is absent.
17. If track_upstream is present and version_source is absent, release_monitoring_project_id is present.
18. If release_monitoring_project_id is present, track_upstream is present.
19. If version_source is gitlab_tags, release_monitoring_project_id is absent.
20. If tag_strip_prefix is present, version_source is gitlab_tags.
21. The spec’s Version: line, resolved through macro expansion, equals metadata version.
22. If version_from_ref is present, upstream_repo is present.
23. version_from_ref.type, when present, is commit-date.

Examples

Metadata for a package on Fedora version, clean:

{
  "version": "5.4.3.0",
  "modification_status": "clean",
  "fedora": {
    "release": "2",
    "git_url": "https://src.fedoraproject.org/rpms/dnf5.git",
    "branch": "rawhide",
    "sha": "f174d0fcedc78a34d74b800697862b93683ad5e0"
  },
  "upstream_repo": "https://github.com/rpm-software-management/dnf5"
}

Metadata for a package on Fedora version, modified:

{
  "version": "5.3.15",
  "modification_status": "modified",
  "modification_reason": "Added gorget source-pipeline.yaml (HUM-5841; pattern from HUM-4622): a fresh fetch of Source0's bash-5.3.tar.gz no longer matches the recorded checksum -- upstream re-published the base tarball (translation file + build stamp only, GPG signature re-verified against the already-trusted key) -- so Fedora's cache no longer serves bytes matching the corrected checksum",
  "fedora": {
    "release": "2",
    "git_url": "https://src.fedoraproject.org/rpms/bash.git",
    "branch": "rawhide",
    "sha": "2768211b5135c7169f513965d79e9f89e0ca6124"
  },
  "upstream_repo": "https://git.savannah.gnu.org/git/bash.git"
}

Metadata for a package on an upstream version ahead of Fedora, modified:

{
  "version": "0.9~rc4",
  "modification_status": "modified",
  "modification_reason": "update to 0.9rc4",
  "fedora": {
    "git_url": "https://src.fedoraproject.org/rpms/avahi.git",
    "branch": "rawhide",
    "sha": "195919c9f9a8dff0e996921afd8736df8f12715b"
  },
  "upstream_repo": "https://github.com/avahi/avahi"
}

Note the absence of fedora.release and of any top-level release field — this package is ahead of whatever Fedora currently ships, so there is no confirmed Fedora release to record.

Metadata for an independent package:

{
  "version": "1.3.3",
  "modification_status": "independent",
  "upstream_repo": "https://github.com/oras-project/oras",
  "track_upstream": "latest",
  "release_monitoring_project_id": 205787
}

No fedora object and no release-related field — the release currently shipped lives only in that package’s spec Release: line.

Consequences

• metadata/*.json across the rpms repository must be brought in line with this specification; that migration is separate, tracked work and is not complete as of this ADR.
• dist_git.py, check_upstream_versions.py, and CVE analysis tooling can implement the Validation rules directly, since each rule is a single unconditional check with no interpretation left to the implementer.
• The spec’s Release: line remains the sole authoritative value for what ships; no metadata field may be treated as a substitute for it, including fedora.release.
• Removing a stored base value for the on-upstream-version and independent states (see Computing the built NVR) makes the bug class behind HUM-5182 structurally impossible, since there is no longer a stored value that can disagree with the spec’s actual current Release:.