Database Migrations

Production migration has one invariant: one release-scoped job completes successfully before any new API or worker replica starts. Do not rely on every replica racing through the image's default migration bootstrap.

Supported command

The generic migration command uses Prisma's checked-in migration history and retries only database connectivity failures:

python -m src.prisma_bootstrap \
  --schema ./prisma/schema.prisma \
  --max-attempts 30 \
  --sleep-seconds 2 \
  --timeout-seconds 300

It runs prisma migrate deploy; it does not run prisma db push. A non-connectivity error is fatal and requires investigation rather than an automatic destructive repair.

The packaged image uses the bundled native Prisma CLI. It does not load the generated Python database models to run this command. Keep this image path for migration Jobs; the Python CLI wrapper can exceed the declared memory limit.

Some releases require a named coordinator, backfill, or compatibility cutover in addition to the generic command. Read every rollout page associated with the target release. For example, Organization deletion and the router-state schema cutover have stricter sequencing requirements.

Kubernetes job example

Create a new job name for every release and use the same immutable image that will run the application. This example assumes deltallm-runtime-secrets contains DATABASE_URL:

apiVersion: batch/v1
kind: Job
metadata:
  name: deltallm-migrate-<release-id>
  namespace: deltallm
  labels:
    app.kubernetes.io/name: deltallm
    app.kubernetes.io/component: migration
spec:
  backoffLimit: 0
  activeDeadlineSeconds: 330
  template:
    metadata:
      labels:
        app.kubernetes.io/name: deltallm
        app.kubernetes.io/component: migration
    spec:
      restartPolicy: Never
      automountServiceAccountToken: false
      containers:
        - name: migrate
          image: deltallm/deltallm@sha256:<release-image-digest>
          command: ["python", "-m", "src.prisma_bootstrap"]
          args:
            - --schema
            - ./prisma/schema.prisma
            - --max-attempts
            - "30"
            - --sleep-seconds
            - "2"
            - --timeout-seconds
            - "300"
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: deltallm-runtime-secrets
                  key: DATABASE_URL
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]

Apply the reviewed manifest, wait for it, and retain the logs with the release evidence:

kubectl apply -f deltallm-migration-<release-id>.yaml
kubectl wait --for=condition=complete --timeout=15m \
  job/deltallm-migrate-<release-id> -n deltallm
kubectl logs job/deltallm-migrate-<release-id> -n deltallm

Do not reuse a completed Job name for a different image or command.

Rollout after migration

The production chart runs its migration hook before either Deployment changes. The database Secret must already exist; the hook has no application ConfigMap, Redis or application-key dependency. Keep the default managed image command.

For a separately orchestrated migration stage, use this configuration only after the exact-image job and any named coordinator complete:

migrationJob:
  enabled: false
  external: true
config:
  general_settings:
    migration_mode: external

Each API and worker verifies required migration names/checksums and rejects missing, failed or unfinished history before serving. This is read-only and uses an existing startup allocation. Completed additional migrations permit a schema-compatible application rollback. See Process lifecycle for hook identity, timeouts, non-root execution, retained failure logs and rollout commands.

Migration design and failure handling

• Prefer expand/backfill/contract changes: add compatible structures, deploy compatible code, backfill with bounded work, then remove old structures in a later release.
• Take and verify a recoverable backup before risky or contract migrations.
• Monitor locks, long transactions, connection saturation, replication lag, and disk during the job.
• If a migration fails, stop the application rollout. Preserve job logs and database state; do not edit _prisma_migrations, mark a migration applied, or use db push without a reviewed recovery plan.
• DeltaLLM v0.1.48 has one narrowly scoped recovery command for the known missing-pgcrypto failure in 20260927150000_model_api_identity. Follow Recover the v0.1.48 model identity migration; do not use that command for any other failed migration.
• Rolling application code back does not roll schema back. A safe rollback requires the old binary to remain compatible with the migrated schema.

Continue with Upgrades and rollbacks.