Upgrade & Maintenance Procedures

Upgrading the Keeptrusts platform involves coordinating changes across the gateway, API server, database migrations, and console. This guide covers rolling update strategies, migration sequencing, rollback plans, and maintenance windows.

Use this page when

You are upgrading Keeptrusts components (gateway, API, console) to a new version.
You need to understand migration sequencing and component dependency order.
You are planning maintenance windows and rollback strategies for production upgrades.
You want post-upgrade verification steps to confirm all services are healthy.

Primary audience

Primary: Technical Engineers
Secondary: AI Agents, Technical Leaders

Upgrade Architecture

Component Dependencies

                    ┌──────────┐
                    │ Database │ ← migrate first
                    │ (schema) │
                    └────┬─────┘
                         │
                    ┌────▼─────┐
                    │   API    │ ← upgrade second
                    │  Server  │
                    └────┬─────┘
                    ┌──────────┼──────────┐
                    ▼          ▼
                  ┌────────┐ ┌────────┐
                  │Console │ │Gateway │ ← upgrade last
                  └────────┘ └────────┘

Upgrade order: Database migrations → API server → Console → Gateway instances.

Version Compatibility

The API maintains backward compatibility within a major version:

Gateway N-1 works with API N (one version behind)
Console N requires API N (same version)
Database migrations are forward-only and auto-applied at API startup

Pre-Upgrade Checklist

#!/usr/bin/env bash
# pre-upgrade-check.sh
set -euo pipefail

echo "=== Pre-Upgrade Checklist ==="

# 1. Backup database
echo "[1/6] Creating database backup..."
pg_dump -h db.internal -U keeptrusts -d keeptrusts \
  --format=custom --compress=9 \
  --file="/var/backups/keeptrusts/pre-upgrade-$(date +%Y%m%d_%H%M%S).dump"

# 2. Record current versions
echo "[2/6] Current versions:"
curl -s http://api.internal:8080/health | jq '.version'
kt --version

# 3. Check disk space for migrations
echo "[3/6] Disk space:"
df -h /var/lib/postgresql | tail -1

# 4. Verify backup
echo "[4/6] Verifying backup..."
pg_restore --list /var/backups/keeptrusts/pre-upgrade-*.dump > /dev/null

# 5. Check active connections
echo "[5/6] Active API connections:"
curl -s http://api.internal:8080/health | jq '.connections'

# 6. Check pending export jobs
echo "[6/6] Pending exports:"
curl -s http://api.internal:8080/v1/exports?status=pending \
  -H "Authorization: Bearer $API_TOKEN" | jq '.total'

echo "=== Pre-upgrade checks complete ==="

Rolling Update (Zero-Downtime)

Docker Compose Rolling Update

# Pull new images
docker compose pull

# Update API first (migrations auto-apply)
docker compose up -d --no-deps keeptrusts-api
docker compose exec keeptrusts-api curl -s localhost:8080/health

# Wait for API to be healthy
until curl -sf http://localhost:8080/health > /dev/null; do
  echo "Waiting for API..."
  sleep 5
done

# Update the console frontend
docker compose up -d --no-deps keeptrusts-console

# Update gateway instances one at a time
docker compose up -d --no-deps --scale keeptrusts-gateway=2 keeptrusts-gateway
sleep 30  # Allow in-flight requests to drain
docker compose up -d --no-deps keeptrusts-gateway

Kubernetes Rolling Update

apiVersion: apps/v1
kind: Deployment
metadata:
  name: keeptrusts-api
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  template:
    spec:
      containers:
        - name: api
          image: keeptrusts/api:2.5.0
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 10

# Update API deployment
kubectl set image deployment/keeptrusts-api api=keeptrusts/api:2.5.0
kubectl rollout status deployment/keeptrusts-api

# Update gateway deployment
kubectl set image deployment/keeptrusts-gateway gateway=keeptrusts/gateway:2.5.0
kubectl rollout status deployment/keeptrusts-gateway

Database Migration Strategy

How Migrations Work

Keeptrusts API auto-applies pending migrations at startup:

API binary starts
  → Connect to PostgreSQL
  → Check _sqlx_migrations table
  → Apply any pending migrations in order
  → Start serving requests

Safe Migration Practices

# Check pending migrations before upgrade
ls api/migrations/*.sql | tail -10

# Migrations are forward-only — never edit shipped files
# Each migration is a new sequential file: 0145_feature.sql, 0146_fix.sql

# Manual migration (if needed)
cd api && sqlx migrate run

Long-Running Migrations

For migrations that modify large tables, plan a maintenance window:

# Check estimated migration time
psql -h db.internal -U keeptrusts -d keeptrusts -c \
  "SELECT pg_size_pretty(pg_total_relation_size('events'));"

# If events table > 10 GB, schedule downtime for ALTER TABLE migrations

Rollback Procedures

Rollback API Server

# Docker Compose — revert to previous image
docker compose stop keeptrusts-api
docker compose run --rm keeptrusts-api-previous  # using previous image tag

# Or pin the previous version
docker compose up -d --no-deps keeptrusts-api
# (edit docker-compose.yml to reference the previous image tag first)

Rollback Database Migration

Database rollbacks require a pre-upgrade backup because migrations are forward-only:

# 1. Stop the API
docker compose stop keeptrusts-api

# 2. Restore from pre-upgrade backup
pg_restore -h db.internal -U keeptrusts -d keeptrusts \
  --clean --if-exists \
  /var/backups/keeptrusts/pre-upgrade-20250420_020000.dump

# 3. Start previous version of API
docker compose up -d keeptrusts-api  # with previous image tag

Kubernetes Rollback

# View rollout history
kubectl rollout history deployment/keeptrusts-api

# Rollback to previous revision
kubectl rollout undo deployment/keeptrusts-api

# Rollback to specific revision
kubectl rollout undo deployment/keeptrusts-api --to-revision=3

Maintenance Windows

Planned Downtime Procedure

#!/usr/bin/env bash
# maintenance-start.sh
set -euo pipefail

# 1. Notify users (via your notification system)
echo "Starting maintenance window at $(date)"

# 2. Enable maintenance mode on load balancer
# Return 503 to new requests, allow in-flight to complete
nginx -s reload  # after updating config to maintenance mode

# 3. Wait for in-flight requests to drain (2 minutes)
sleep 120

# 4. Stop gateway instances
docker compose stop keeptrusts-gateway

# 5. Backup database
pg_dump -h db.internal -U keeptrusts -d keeptrusts \
  --format=custom --compress=9 \
  --file="/var/backups/keeptrusts/maintenance-$(date +%Y%m%d_%H%M%S).dump"

# 6. Perform upgrade
docker compose pull
docker compose up -d keeptrusts-api
# Wait for migrations and health check
until curl -sf http://localhost:8080/health > /dev/null; do sleep 5; done

# 7. Start remaining services
docker compose up -d

# 8. Verify health
curl -s http://localhost:8080/health | jq .
curl -s http://localhost:41002/health | jq .

# 9. Disable maintenance mode
nginx -s reload  # revert to normal config

echo "Maintenance window closed at $(date)"

Version Compatibility Matrix

API Version	Min Gateway	Min Console	Min CLI	Notes
2.5.x	2.4.x	2.5.x	2.4.x	Gateway tolerates one minor behind
2.4.x	2.3.x	2.4.x	2.3.x
2.3.x	2.2.x	2.3.x	2.2.x

Post-Upgrade Verification

#!/usr/bin/env bash
# post-upgrade-verify.sh
set -euo pipefail

echo "=== Post-Upgrade Verification ==="

# Check component health
echo "[1/5] API health:"
curl -s http://api.internal:8080/health | jq '{version, status}'

echo "[2/5] Gateway health:"
curl -s http://gateway.internal:41002/health | jq '{version, status}'

echo "[3/5] Console accessible:"
curl -s -o /dev/null -w "%{http_code}" http://console.internal:3000/

echo "[4/5] Database migration status:"
psql -h db.internal -U keeptrusts -d keeptrusts -c \
  "SELECT version, description FROM _sqlx_migrations ORDER BY version DESC LIMIT 5;"

echo "[5/5] Smoke test — policy evaluation:"
curl -s -o /dev/null -w "%{http_code}" \
  http://gateway.internal:41002/v1/chat/completions \
  -H "Authorization: Bearer kt_gk_test" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"test"}]}'

echo "=== Verification complete ==="

Next steps

Backup & Recovery — always backup before upgrading
Docker Deployment — container-specific upgrade patterns
Monitoring Infrastructure — watch metrics during upgrades

For AI systems

Canonical terms: Keeptrusts upgrade procedures, database migration, rolling update, maintenance window, version compatibility, container update, health verification.
Key config/commands: docker compose pull && docker compose up -d for container updates; migrations auto-applied at API startup; psql -c "SELECT version, description FROM _sqlx_migrations ORDER BY version DESC LIMIT 5;" for migration status; post-upgrade smoke test via gateway health and policy evaluation.
Best next pages: Backup & Recovery, Docker Deployment, Monitoring Infrastructure.

For engineers

Prerequisites: Full database backup before any upgrade (see Backup & Recovery); monitoring active during upgrade window.
Always backup before upgrading; migrations are auto-applied at API startup and cannot be rolled back.
Validate with: check migration status (SELECT version FROM _sqlx_migrations ORDER BY version DESC LIMIT 5); run health checks on all services; execute policy evaluation smoke test through gateway.
For zero-downtime upgrades: use rolling updates with load balancer draining; upgrade gateway instances one at a time.

For leaders

Database migrations are forward-only — rollback requires restoring from backup, making pre-upgrade backups critical.
Plan maintenance windows for major version upgrades; minor versions support rolling updates with no downtime.
Monitor metrics during and after upgrades to detect performance regressions or new error patterns.
Upgrade cadence should balance security patches (urgent) against stability requirements (validated in staging first).

Use this page when​

Primary audience​

Upgrade Architecture​

Component Dependencies​

Version Compatibility​

Pre-Upgrade Checklist​

Rolling Update (Zero-Downtime)​

Docker Compose Rolling Update​

Kubernetes Rolling Update​

Database Migration Strategy​

How Migrations Work​

Safe Migration Practices​

Long-Running Migrations​

Rollback Procedures​

Rollback API Server​

Rollback Database Migration​

Kubernetes Rollback​

Maintenance Windows​

Planned Downtime Procedure​

Version Compatibility Matrix​

Post-Upgrade Verification​

Next steps​

For AI systems​

For engineers​

For leaders​