Cold-restore rehearsal

2026-10-20 23:18:14 -04:00
parent 5b21074462
commit 0ac2073038
4 changed files with 30 additions and 13 deletions
@@ -63,7 +63,7 @@ An operator must choose recovery point/retention/RPO/RTO, practice off-host reco

 The prior file-only `node:backup` still refuses PostgreSQL-bound node directories. A separate explicit [paired backup helper](../compose.custody-backup.yaml) and [utility](../server/coordinated-backup.ts) now create a small-sandbox rehearsal package containing the SQLite ledger/market pair and a custom-format PostgreSQL database dump. Creation holds the exact node-directory lock, then a dedicated PostgreSQL session advisory writer lock, checks the local `custody-postgres.json` binding against the database custodian identity, verifies any retained SQLite cutover source is frozen and its digest matches PostgreSQL's `import_source`, audits PostgreSQL liabilities under the lock, snapshots the active node files and runs `pg_dump`. A frozen SQLite source is deliberately omitted from the active backup pair; PostgreSQL is authoritative after cutover. The pair manifest is written last and binds network/chain/profile/custodian ID, the node manifest digest, PostgreSQL database name and dump size/hash. No restore/reset operation is included.

-The PostgreSQL dump uses a **separate read-only `custody_backup` login**, not the app runtime or schema owner. The local Compose provisioner now creates its credential volume, the one-shot bootstrap grants only database connect, schema usage, table SELECT and sequence SELECT, and the database HBA permits only TLS/SCRAM for this role. Existing local v1 secret volumes get an additive backup-role secret upgrade without rotating database credentials. This changes the next explicit persistent-stack provision/migrate cycle; no running stack was restarted here. The helper image has `pg_dump` and is a one-shot maintenance container; those tools and backup credentials are not added to the long-running app image. The disposable integration test checks that the role can dump but cannot update balances, delete entries or advance journal sequences.
+The PostgreSQL dump uses a **separate read-only `custody_backup` login**, not the app runtime or schema owner. The local Compose provisioner now creates its credential volume, the one-shot bootstrap grants only database connect, schema usage, table SELECT and sequence SELECT, and the database HBA permits only TLS/SCRAM for this role. Existing local v1 secret volumes get an additive backup-role secret upgrade without rotating database credentials. This changes the next explicit persistent-stack provision/migrate cycle; no running stack was restarted here. The helper image has `pg_dump` and is a one-shot maintenance container; those tools and backup credentials are not added to the long-running app image. The disposable integration test checks that the role can dump but cannot update balances, delete entries or advance journal sequences. A cold test restore also replays the paired node SQLite database through the native RandomX verifier and checks tip/NFT ownership; the test-only fake-PoW fixtures are not used for that proof-revalidation assertion.

 ### Local sandbox invocation

@@ -83,6 +83,8 @@ Automated tests cover default proxy distrust/spoofed-header rejection, explicit

 The native HTTP integration now cleanly stops its disposable nodes, backs up their **real three-block RandomX chain and market journal**, restores to a fresh disposable directory, opens/revalidates copied proofs with the actual reference engine, and checks canonical ownership, market instance, order buyer and nonce receipt history. It then restarts the original isolated fixture. This is a local automated drill, not an off-host operational recovery or crash/power-loss certification.

+**Coordinated custody recovery evidence (2026-10-20):** the latest build passed 172 offline regressions and 31 isolated PostgreSQL integration tests. The new paired-backup test mines a valid native-RandomX source block, writes fictitious PostgreSQL custody state, freezes/binds the simulated SQLite cutover source, creates and verifies the pair, restores PostgreSQL into a scratch database, and restores/replays the node database into a new directory with the reference RandomX engine. It also verifies backup-login restrictions, mismatched identity/cutover detection, stale node-lock refusal, failed `pg_dump` cleanup/writer-lock release, output overwrite refusal, incomplete package handling and dump tamper detection. Tests use tmpfs/internal trust-auth databases and synthetic local files; this does not demonstrate protected persistent-volume provisioning, host-backed destination ACLs, off-host encryption or production operational recovery. The persistent sandbox was not restarted or migrated for this work.
+
 Final Node 24 TypeScript/Vite and production runtime builds passed. The suite passed **98 tests, 0 failed, 0 skipped**. A separate packaged-runtime smoke drill ran with no external network, read-only root, dropped capabilities and a disposable 64 MiB tmpfs: monitoring correctly returned nonzero for isolation, empty chain, not-yet-reconciled market and low disk; clean stopped-node backup, verification and new-directory restore each exited zero. That empty-chain CLI drill reported proof revalidation as false, as designed; nonempty real-proof replay is covered by the integration drill above. No host volume or production wallet was used.

 SEC-07/SEC-09 are only partially addressed: actual edge/source-IP/TLS checks, a production monitoring agent and verified alert delivery, off-host backup storage, long-history restore timing, filesystem/power-loss testing, independent peer/miner topology and named operators remain unverified. See [operator launch register](operator-launch.md) and [security review](security-review.md). No startup or payment approval is granted by these tools.