Hummingbird Tools

Database backup and restore tools for the Hummingbird status database. Creates compressed PostgreSQL dumps, uploads to S3 with rotation, and supports restore from the latest backup.

The image also contains Tekton Tasks for MR artifact promotion: they find the tested image and create a push-style Component Snapshot after merge.

Features

  • Streaming Backup: Streams pg_dump through gzip to temp file, then S3 multipart upload - handles arbitrarily large databases
  • Streaming Restore: Streams S3 through gzip and psql with bounded memory
  • Automatic Rotation: Keeps configurable number of backups per group (daily/weekly/monthly)
  • Separate IAM Users: Write user for production backups, read-only user for staging restore
  • MR Artifact Lookup: Derive the MR head from a merge commit and verify its tested Snapshot against the linked build PipelineRun
  • Promotion Component Snapshot: Create a single-Component Snapshot using the tested revision and digest, allowing Integration Service to release it

Prerequisites

  • AWS CLI configured with appropriate credentials
  • PostgreSQL client tools (pg_dump, psql)
  • Python 3.11 or later
  • Access to the hummingbird-status PostgreSQL database

The MR artifact lookup Task needs a Git checkout Trusted Artifact at the merge commit, a digest-pinned Hummingbird tools image, and a ServiceAccount that can list and get Snapshots and PipelineRuns in the Konflux namespace. KubeArchive must be reachable by HTTPS; it authorizes reads using the Task’s projected ServiceAccount token and the same namespace permissions.

The promotion Component Snapshot Task also needs permission to list IntegrationTestScenarios and create Snapshots in that namespace. Creating a Component Snapshot for an Application with automatic ReleasePlans can start a release.

Both Tasks unpack the source Trusted Artifact into /var/workdir/source and run their Python step from /var/workdir. Tekton creates a step’s working directory before any step in the Task runs; using the extraction path as the working directory makes it root-owned and prevents unpacking under the restricted SCC.

MR Artifact Lookup

The Task manifest is hummingbird-tools/tekton/find-tested-mr-artifact.yaml. Pass the merge commit SHA, target repository URL, application, Component, and namespace. Pass the clone Task’s SOURCE_ARTIFACT result as sourceArtifact. The clone must include full Git history. Supply the Hummingbird tools image by digest through toolImage. Set archiveUrl to the cluster’s KubeArchive API URL, for example https://kubearchive.apps.cluster.example.com.

The Task reads the merge commit’s second Git parent, then selects the newest passed MR Snapshot from the live API and KubeArchive. A passed /test or /retest build is eligible when its Snapshot and build carry the same PaC test or retest event. Multiple passed Snapshots for one commit are normal — a /test run creates another — and while RPM builds are not reproducible their image digests may differ (HUM-9785); the newest eligible Snapshot is selected. It reads a linked build PipelineRun from KubeArchive when the live object has been removed, then checks the source URL and revision, target branch, and image digest. The Task fails when KubeArchive cannot be queried. Its results are SNAPSHOT, BUILD_PIPELINERUN, IMAGE, SOURCE_SHA, and SOURCE_URL. It does not create a Snapshot or Release.

Promotion Component Snapshot

The Task manifest is hummingbird-tools/tekton/create-promotion-component.yaml. Pass the same merge commit, target repository, Application, Component, and namespace as the lookup Task. Pass the same sourceArtifact and the lookup Task’s SNAPSHOT, BUILD_PIPELINERUN, IMAGE, SOURCE_SHA, and SOURCE_URL results as sourceSnapshot, buildPipelineRun, image, sourceSha, and sourceUrl. Supply the tools image by digest through toolImage. Pass the same archiveUrl used by the lookup Task and the current PipelineRun name as promotionPipelineRun.

The Task rechecks Git parent B, the MR Snapshot, its linked build PipelineRun, and the selected digest. It reads removed MR evidence from KubeArchive while checking current IntegrationTestScenarios and creating the Component Snapshot through the live Kubernetes API. It fails if an IntegrationTestScenario for the Application could run on the push-style component Snapshot. It then creates a Snapshot named deterministically from the full Component name and merge SHA, containing only the selected Component entry. The Snapshot has Application and Component labels, type=component, and a push event label. Its annotations link the merge commit, MR Snapshot, build PipelineRun, and promotion PipelineRun. The test.appstudio.openshift.io/pipelinerunstarttime annotation is the promotion PipelineRun’s status.startTime in epoch milliseconds. The COMPONENT_SNAPSHOT result contains its name.

Before creation, the Task checks the target branch with Git. For RPM builds, it compares the build’s monorepo-subdir with the current branch when other merges have advanced it. An unrelated package merge does not block promotion; a newer change to the same package does. Builds without a monorepo-subdir require the merge commit to remain the branch head. A retry reuses a live Component Snapshot only when its Application, Component, source revision, image digest, provenance, and original promotion start time match; otherwise it reports the conflicting fields. It never reuses an archived Snapshot. Git branch access is required for this check.

The Task creates no Release or build attestation. Integration Service handles the automatic Release. See MR artifact promotion for the complete flow and Snapshot ordering. If a retry finds a matching Snapshot, the Task returns that name without creating another one.

Promotion failure and recovery

If a stale run stops, let promotion for the current target branch proceed. If the Component Snapshot conflicts, inspect it and its source MR evidence; do not delete or replace it to force another Release. A failed automatic Release needs an explicit operator-owned retry through the release workflow. If the tested MR Snapshot or build evidence is unavailable, recover through an explicit new build and required tests before release. Check Git access for branch lookup failures, ServiceAccount permissions for forbidden writes, and tenant quota for quota failures. A normal Task retry does not repair a failed Release.

Deployment

The SAM template creates:

  • S3 bucket for database backups
  • IAM user with write permissions (for production backup CronJobs)
  • IAM user with read-only permissions (for staging restore CronJob)

Build and deploy using containerized AWS SAM CLI:

cd hummingbird-tools
make build     # Build SAM application
make deploy    # First deployment (interactive/guided)
make redeploy  # Subsequent deployments (non-interactive)

Note: Only deployed in production - staging restore reads from the production backup bucket.

Parameters

Parameter Description Default
ResourcePrefix Prefix for all resource names myapp-prod

Resource naming:

  • S3 bucket: {ResourcePrefix}
  • Write IAM user: {ResourcePrefix}-write-user
  • Read IAM user: {ResourcePrefix}-read-user

Usage

Backup

Run a backup with rotation:

python3 -m hummingbird_tools.backup <group> <rotation>

Arguments:

  • group: Backup group name (e.g., daily, weekly, monthly)
  • rotation: Number of backups to keep for this group

Environment variables:

Variable Description
S3_BUCKET S3 bucket name
S3_PREFIX Optional prefix for S3 keys
POSTGRESQL_HOST Database hostname
POSTGRESQL_USER Database username
POSTGRESQL_PASSWORD Database password
POSTGRESQL_DATABASE Database name

Example:

export S3_BUCKET=myapp-prod-db-backups
export S3_PREFIX=mydb/
export POSTGRESQL_HOST=myapp-postgres
export POSTGRESQL_USER=postgres
export POSTGRESQL_PASSWORD=secret
export POSTGRESQL_DATABASE=mydb

python3 -m hummingbird_tools.backup daily 7

This creates a backup like mydb/2026-01-16-02-30.daily.sql.gz and removes any daily backups beyond 7.

Restore

Restore from the latest backup:

python3 -m hummingbird_tools.restore

Environment variables are the same as for backup.

The restore process:

  1. Lists all backups matching *.{daily,weekly,monthly}.sql.gz
  2. Selects the latest by filename sort (most recent timestamp)
  3. Opens the backup stream and starts a database transaction
  4. Removes the rpm_tracker schema and existing public tables
  5. Streams the decompressed SQL through psql with ON_ERROR_STOP=1
  6. Commits only after the complete gzip stream validates and all SQL succeeds

SQL errors and damaged backups fail the restore and roll back both cleanup and import. This replaces tracker data rather than merging it. An older backup without tracker tables removes the target tracker schema rather than retaining stale data. Other schemas are not removed automatically. If the dump conflicts with an existing schema outside this cleanup scope, the restore fails and rolls back instead of merging it.

Pause all database writers and wait for active Jobs to finish before restoring. SCALE_DEPLOYMENT pauses only its configured Deployment, not tracker CronJobs or other services. Ensure database roles referenced by the dump already exist. After restoring, run service initializers to apply migrations and refresh environment specific permissions before resuming workloads.

CronJob Schedule

Deployed via Kubernetes CronJobs in kubernetes/hummingbird-status/:

CronJob Schedule Environment Command
daily 30 2 * * * production python3 -m hummingbird_tools.backup daily 7
weekly 30 6 * * 0 production python3 -m hummingbird_tools.backup weekly 4
monthly 30 10 1 * * production python3 -m hummingbird_tools.backup monthly 12
restore (suspended) staging python3 -m hummingbird_tools.restore

To manually trigger a staging restore:

kubectl create job --from=cronjob/hummingbird-status-restore manual-restore-$(date +%s)

S3 Key Structure

Backups are stored with a flat key structure per database:

{database}/{timestamp}.{group}.sql.gz

Examples:

mydb/2026-01-16-02-30.daily.sql.gz
mydb/2026-01-12-06-30.weekly.sql.gz
mydb/2026-01-01-10-30.monthly.sql.gz

Multiple databases can share the same bucket using different prefixes.

Development

See the main README for development workflows.

make hummingbird-tools/setup  # Install dependencies
make check                     # Lint code (ruff)
make fmt                       # Format code
make test                      # Run unit tests
make coverage                  # Run tests with coverage

Restore integration tests require PostgreSQL client tools and an empty disposable database whose name starts with test_restore_. They refuse an existing populated database and use fake S3 input, without contacting AWS or Kubernetes:

cd hummingbird-tools
export PGHOST=localhost PGPORT=5432 PGUSER=postgres
export RESTORE_TEST_DATABASE=test_restore_local
python3 -m unittest tests.test_restore_db

Create the empty test database first. Supply PGPASSWORD through your local test environment if required. Never point these tests at a deployed database.

Security

  • S3 bucket blocks all public access
  • S3 objects encrypted at rest (AES256)
  • Separate IAM users with least privilege:
    • Write user: s3:PutObject, s3:DeleteObject, s3:ListBucket
    • Read user: s3:GetObject, s3:ListBucket
  • Database passwords passed via environment variables (Kubernetes secrets)

License

This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.