Hummingbird Tools
Database backup and restore tools for the Hummingbird status database. Creates compressed PostgreSQL dumps, uploads to S3 with rotation, and supports restore from the latest backup.
The image also contains Tekton Tasks for MR artifact promotion: they find the tested image and create a push-style Component Snapshot after merge.
Features
- Streaming Backup: Streams pg_dump through gzip to temp file, then S3 multipart upload - handles arbitrarily large databases
- Streaming Restore: Streams S3 through gzip and psql with bounded memory
- Automatic Rotation: Keeps configurable number of backups per group (daily/weekly/monthly)
- Separate IAM Users: Write user for production backups, read-only user for staging restore
- MR Artifact Lookup: Derive the MR head from a merge commit and verify its tested Snapshot against the linked build PipelineRun
- Promotion Component Snapshot: Create a single-Component Snapshot using the tested revision and digest, allowing Integration Service to release it
Prerequisites
- AWS CLI configured with appropriate credentials
- PostgreSQL client tools (pg_dump, psql)
- Python 3.11 or later
- Access to the hummingbird-status PostgreSQL database
The MR artifact lookup Task needs a Git checkout Trusted Artifact at the merge commit, a digest-pinned Hummingbird tools image, and a ServiceAccount that can list and get Snapshots and PipelineRuns in the Konflux namespace. KubeArchive must be reachable by HTTPS; it authorizes reads using the Task’s projected ServiceAccount token and the same namespace permissions.
The promotion Component Snapshot Task also needs permission to list IntegrationTestScenarios and create Snapshots in that namespace. Creating a Component Snapshot for an Application with automatic ReleasePlans can start a release.
Both Tasks unpack the source Trusted Artifact into /var/workdir/source and run their Python
step from /var/workdir. Tekton creates a step’s working directory before any step in the Task runs;
using the extraction path as the working directory makes it root-owned and prevents unpacking
under the restricted SCC.
MR Artifact Lookup
The Task manifest is hummingbird-tools/tekton/find-tested-mr-artifact.yaml.
Pass the merge commit SHA, target repository URL, application, Component, and
namespace. Pass the clone Task’s SOURCE_ARTIFACT result as sourceArtifact. The
clone must include full Git history. Supply the
Hummingbird tools image by digest through toolImage. Set archiveUrl to the
cluster’s KubeArchive API URL, for example
https://kubearchive.apps.cluster.example.com.
The Task reads the merge commit’s second Git parent, then selects the newest
passed MR Snapshot from the live API and KubeArchive. A passed /test or
/retest build is eligible when its Snapshot and build carry the same PaC
test or retest event. Multiple passed Snapshots for one commit are normal — a
/test run creates another — and while RPM builds are not reproducible their
image digests may differ (HUM-9785); the newest eligible Snapshot is selected.
It reads a linked build PipelineRun from KubeArchive when the live object has
been removed, then checks the source URL and revision, target branch, and image
digest. The Task fails when KubeArchive cannot be queried. Its results are
SNAPSHOT, BUILD_PIPELINERUN, IMAGE, SOURCE_SHA, and SOURCE_URL. It
does not create a Snapshot or Release.
Promotion Component Snapshot
The Task manifest is hummingbird-tools/tekton/create-promotion-component.yaml.
Pass the same merge commit, target repository, Application, Component, and
namespace as the lookup Task. Pass the same sourceArtifact and the lookup
Task’s SNAPSHOT, BUILD_PIPELINERUN, IMAGE, SOURCE_SHA, and SOURCE_URL
results as sourceSnapshot,
buildPipelineRun, image, sourceSha, and sourceUrl. Supply the tools image
by digest through toolImage. Pass the same archiveUrl used by the lookup
Task and the current PipelineRun name as promotionPipelineRun.
The Task rechecks Git parent B, the MR Snapshot, its linked build PipelineRun,
and the selected digest. It reads removed MR evidence from KubeArchive while
checking current IntegrationTestScenarios and creating the Component Snapshot
through the live Kubernetes API. It fails if an IntegrationTestScenario for the
Application could run on the push-style component Snapshot. It then creates a Snapshot
named deterministically from the full Component name and merge SHA, containing
only the selected Component entry. The Snapshot has Application and Component
labels, type=component, and a push event label. Its annotations link the merge
commit, MR Snapshot, build PipelineRun, and promotion PipelineRun. The
test.appstudio.openshift.io/pipelinerunstarttime annotation is the promotion
PipelineRun’s status.startTime in epoch milliseconds. The COMPONENT_SNAPSHOT
result contains its name.
Before creation, the Task checks the target branch with Git. For RPM builds,
it compares the build’s monorepo-subdir with the current branch when other
merges have advanced it. An unrelated package merge does not block promotion;
a newer change to the same package does. Builds without a monorepo-subdir
require the merge commit to remain the branch head. A retry reuses a live
Component Snapshot only when its Application, Component, source revision, image
digest, provenance, and original promotion start time match; otherwise it reports
the conflicting fields. It never reuses an archived Snapshot. Git branch access
is required for this check.
The Task creates no Release or build attestation. Integration Service handles the automatic Release. See MR artifact promotion for the complete flow and Snapshot ordering. If a retry finds a matching Snapshot, the Task returns that name without creating another one.
Promotion failure and recovery
If a stale run stops, let promotion for the current target branch proceed. If the Component Snapshot conflicts, inspect it and its source MR evidence; do not delete or replace it to force another Release. A failed automatic Release needs an explicit operator-owned retry through the release workflow. If the tested MR Snapshot or build evidence is unavailable, recover through an explicit new build and required tests before release. Check Git access for branch lookup failures, ServiceAccount permissions for forbidden writes, and tenant quota for quota failures. A normal Task retry does not repair a failed Release.
Deployment
The SAM template creates:
- S3 bucket for database backups
- IAM user with write permissions (for production backup CronJobs)
- IAM user with read-only permissions (for staging restore CronJob)
Build and deploy using containerized AWS SAM CLI:
cd hummingbird-tools
make build # Build SAM application
make deploy # First deployment (interactive/guided)
make redeploy # Subsequent deployments (non-interactive)
Note: Only deployed in production - staging restore reads from the production backup bucket.
Parameters
| Parameter | Description | Default |
|---|---|---|
ResourcePrefix |
Prefix for all resource names | myapp-prod |
Resource naming:
- S3 bucket:
{ResourcePrefix} - Write IAM user:
{ResourcePrefix}-write-user - Read IAM user:
{ResourcePrefix}-read-user
Usage
Backup
Run a backup with rotation:
python3 -m hummingbird_tools.backup <group> <rotation>
Arguments:
group: Backup group name (e.g.,daily,weekly,monthly)rotation: Number of backups to keep for this group
Environment variables:
| Variable | Description |
|---|---|
S3_BUCKET |
S3 bucket name |
S3_PREFIX |
Optional prefix for S3 keys |
POSTGRESQL_HOST |
Database hostname |
POSTGRESQL_USER |
Database username |
POSTGRESQL_PASSWORD |
Database password |
POSTGRESQL_DATABASE |
Database name |
Example:
export S3_BUCKET=myapp-prod-db-backups
export S3_PREFIX=mydb/
export POSTGRESQL_HOST=myapp-postgres
export POSTGRESQL_USER=postgres
export POSTGRESQL_PASSWORD=secret
export POSTGRESQL_DATABASE=mydb
python3 -m hummingbird_tools.backup daily 7
This creates a backup like mydb/2026-01-16-02-30.daily.sql.gz and removes
any daily backups beyond 7.
Restore
Restore from the latest backup:
python3 -m hummingbird_tools.restore
Environment variables are the same as for backup.
The restore process:
- Lists all backups matching
*.{daily,weekly,monthly}.sql.gz - Selects the latest by filename sort (most recent timestamp)
- Opens the backup stream and starts a database transaction
- Removes the
rpm_trackerschema and existingpublictables - Streams the decompressed SQL through psql with
ON_ERROR_STOP=1 - Commits only after the complete gzip stream validates and all SQL succeeds
SQL errors and damaged backups fail the restore and roll back both cleanup and import. This replaces tracker data rather than merging it. An older backup without tracker tables removes the target tracker schema rather than retaining stale data. Other schemas are not removed automatically. If the dump conflicts with an existing schema outside this cleanup scope, the restore fails and rolls back instead of merging it.
Pause all database writers and wait for active Jobs to finish before restoring.
SCALE_DEPLOYMENT pauses only its configured Deployment, not tracker CronJobs or other
services. Ensure database roles referenced by the dump already exist. After restoring,
run service initializers to apply migrations and refresh environment specific permissions
before resuming workloads.
CronJob Schedule
Deployed via Kubernetes CronJobs in kubernetes/hummingbird-status/:
| CronJob | Schedule | Environment | Command |
|---|---|---|---|
| daily | 30 2 * * * |
production | python3 -m hummingbird_tools.backup daily 7 |
| weekly | 30 6 * * 0 |
production | python3 -m hummingbird_tools.backup weekly 4 |
| monthly | 30 10 1 * * |
production | python3 -m hummingbird_tools.backup monthly 12 |
| restore | (suspended) | staging | python3 -m hummingbird_tools.restore |
To manually trigger a staging restore:
kubectl create job --from=cronjob/hummingbird-status-restore manual-restore-$(date +%s)
S3 Key Structure
Backups are stored with a flat key structure per database:
{database}/{timestamp}.{group}.sql.gz
Examples:
mydb/2026-01-16-02-30.daily.sql.gz
mydb/2026-01-12-06-30.weekly.sql.gz
mydb/2026-01-01-10-30.monthly.sql.gz
Multiple databases can share the same bucket using different prefixes.
Development
See the main README for development workflows.
make hummingbird-tools/setup # Install dependencies
make check # Lint code (ruff)
make fmt # Format code
make test # Run unit tests
make coverage # Run tests with coverage
Restore integration tests require PostgreSQL client tools and an empty disposable database
whose name starts with test_restore_. They refuse an existing populated database and use
fake S3 input, without contacting AWS or Kubernetes:
cd hummingbird-tools
export PGHOST=localhost PGPORT=5432 PGUSER=postgres
export RESTORE_TEST_DATABASE=test_restore_local
python3 -m unittest tests.test_restore_db
Create the empty test database first. Supply PGPASSWORD through your local test environment
if required. Never point these tests at a deployed database.
Security
- S3 bucket blocks all public access
- S3 objects encrypted at rest (AES256)
- Separate IAM users with least privilege:
- Write user:
s3:PutObject,s3:DeleteObject,s3:ListBucket - Read user:
s3:GetObject,s3:ListBucket
- Write user:
- Database passwords passed via environment variables (Kubernetes secrets)
License
This project is licensed under the GNU General Public License v3.0 or later - see the LICENSE file for details.