Deployment, rollback and recovery

Pre-release deployment gate

Coordinate a PostgreSQL snapshot and Artifact Store recovery point, record the release and contract versions, and stop all but one migration job. From the gamex-panel directory run:

npm ci
npm test
npm run typecheck
npm run lint
npm run format:check
npm run build
npm run db:status
npm run db:migrate
npm run db:check

db:check verifies that the current schema is fully applied. Before the first stable release, Panel and Registry support clean installation only: Setup rejects every nonempty database, and Daemon enrollment accepts only the current v1 contract. Do not repair a failed check with direct database edits.

Start one canary Panel, require readiness, Panel-only Chat, task SSE replay, one harmless Workspace read and one Daemon reconnect, then roll replicas one at a time.

Rollback decision

Pre-release application rollback requires restoring the matching database and Artifact Store recovery point. Database rollback is separate and never automatic:

1. Stop every Panel writer and keep Daemons disconnected.
2. Take a new coordinated backup.
3. Restore a copy in isolation and run the down migration there.
4. Run npm run db:check and verify preserved task counts/content.
5. Only after the rehearsal succeeds, run npm run db:rollback against the intended stopped environment.

After restoring a recovery point, review every Automation owner, Workspace assignment and fixed action before enabling schedules.

Redis loss and multiple Panel replicas

PostgreSQL remains authoritative for Agent Turns, schedules, commands, events, worker leases and outbox state. Redis loss makes readiness fail and removes replicas from new load-balanced traffic. Database-fenced workers keep durable state safe, while authentication refresh, shared rate limits, singleton maintenance and wake-up hints remain degraded. Do not switch replicas to independent memory caches. Restore Redis, wait for readiness, and verify task SSE cursors and worker backlogs before returning traffic.

Every replica of one logical Panel uses the same deployment UUID and a unique Gateway instance ID. Copying a database under a different marker fails closed. Use PANEL_IDENTITY_RESEED=true only to create a deliberately independent environment; it creates a new private-data namespace and is not a restart fix.

Daemon offline and reconnect

Do not duplicate a command when a Daemon goes offline. The replacement Gateway owner takes a higher connection epoch; the Daemon journal returns an already recorded result rather than repeating the side effect. Check locally:

sudo gamex doctor
sudo curl --unix-socket /run/gamex-daemon/admin.sock http://localhost/status
sudo systemctl status gamex-daemon@gamex.service

After Panel or Daemon restart, allow the old default lease up to about 45 seconds to expire. Treat failure to reach controlConnected: true within 90 seconds as an incident. Rotate client identity only with an administrator-issued single-use link and sudo gamex rotate.

Backup and restore

Back up PostgreSQL and immutable Artifact objects as one recovery point. For a single-Panel SQLite deployment, stop Panel and copy the database together with its WAL and shared-memory files, or use SQLite's online backup command. Back up the logical Panel identity marker and Gateway trust material separately with restricted access. Stop Daemon before copying its journal database, WAL and shared-memory files; this journal is not a Workspace-volume backup.

Restore into isolation in this order: PostgreSQL/SQLite, Artifact Store, logical Panel identity, fresh Redis, then Gateway material. Run npm run db:check, sample Artifact digest checks and audit/event continuity checks before starting one Panel. Reconnect one canary Daemon and verify snapshot/event reconciliation before widening traffic. Never restore a database with a newly generated Panel identity unless the environment is intentionally being reseeded.

Product-limit tuning

agent-product-limits/v1 defines the compiled defaults and hard maxima for tasks, items, context, SSE, search, Plan/Input, Memory and Review export. An operator may supply a JSON object in AGENT_PRODUCT_LIMIT_OVERRIDES; every key must exist and every positive safe integer must be no greater than its compiled default. Overrides only narrow limits and require a restart.

AGENT_PRODUCT_LIMIT_OVERRIDES='{"sse.connections_per_user":4,"search.results":25}'

Reject an override if startup validation reports an unknown key, non-integer or increase. Before a production change, soak representative task and search loads, test multi-tab reconnect and confirm that reset/snapshot recovery works at the narrowed boundary.