Cold-restore rehearsal

docs/operations-recovery.md

@@ -63,7 +63,7 @@ An operator must choose recovery point/retention/RPO/RTO, practice off-host reco
 
 The prior file-only `node:backup` still refuses PostgreSQL-bound node directories. A separate explicit [paired backup helper](../compose.custody-backup.yaml) and [utility](../server/coordinated-backup.ts) now create a small-sandbox rehearsal package containing the SQLite ledger/market pair and a custom-format PostgreSQL database dump. Creation holds the exact node-directory lock, then a dedicated PostgreSQL session advisory writer lock, checks the local `custody-postgres.json` binding against the database custodian identity, verifies any retained SQLite cutover source is frozen and its digest matches PostgreSQL's `import_source`, audits PostgreSQL liabilities under the lock, snapshots the active node files and runs `pg_dump`. A frozen SQLite source is deliberately omitted from the active backup pair; PostgreSQL is authoritative after cutover. The pair manifest is written last and binds network/chain/profile/custodian ID, the node manifest digest, PostgreSQL database name and dump size/hash. No restore/reset operation is included.
 
-The PostgreSQL dump uses a **separate read-only `custody_backup` login**, not the app runtime or schema owner. The local Compose provisioner now creates its credential volume, the one-shot bootstrap grants only database connect, schema usage, table SELECT and sequence SELECT, and the database HBA permits only TLS/SCRAM for this role. Existing local v1 secret volumes get an additive backup-role secret upgrade without rotating database credentials. This changes the next explicit persistent-stack provision/migrate cycle; no running stack was restarted here. The helper image has `pg_dump` and is a one-shot maintenance container; those tools and backup credentials are not added to the long-running app image. The disposable integration test checks that the role can dump but cannot update balances, delete entries or advance journal sequences.
+The PostgreSQL dump uses a **separate read-only `custody_backup` login**, not the app runtime or schema owner. The local Compose provisioner now creates its credential volume, the one-shot bootstrap grants only database connect, schema usage, table SELECT and sequence SELECT, and the database HBA permits only TLS/SCRAM for this role. Existing local v1 secret volumes get an additive backup-role secret upgrade without rotating database credentials. This changes the next explicit persistent-stack provision/migrate cycle; no running stack was restarted here. The helper image has `pg_dump` and is a one-shot maintenance container; those tools and backup credentials are not added to the long-running app image. The disposable integration test checks that the role can dump but cannot update balances, delete entries or advance journal sequences. A cold test restore also replays the paired node SQLite database through the native RandomX verifier and checks tip/NFT ownership; the test-only fake-PoW fixtures are not used for that proof-revalidation assertion.
 
 ### Local sandbox invocation
 
@@ -83,6 +83,8 @@ Automated tests cover default proxy distrust/spoofed-header rejection, explicit
 
 The native HTTP integration now cleanly stops its disposable nodes, backs up their **real three-block RandomX chain and market journal**, restores to a fresh disposable directory, opens/revalidates copied proofs with the actual reference engine, and checks canonical ownership, market instance, order buyer and nonce receipt history. It then restarts the original isolated fixture. This is a local automated drill, not an off-host operational recovery or crash/power-loss certification.
 
+**Coordinated custody recovery evidence (2026-10-20):** the latest build passed 172 offline regressions and 31 isolated PostgreSQL integration tests. The new paired-backup test mines a valid native-RandomX source block, writes fictitious PostgreSQL custody state, freezes/binds the simulated SQLite cutover source, creates and verifies the pair, restores PostgreSQL into a scratch database, and restores/replays the node database into a new directory with the reference RandomX engine. It also verifies backup-login restrictions, mismatched identity/cutover detection, stale node-lock refusal, failed `pg_dump` cleanup/writer-lock release, output overwrite refusal, incomplete package handling and dump tamper detection. Tests use tmpfs/internal trust-auth databases and synthetic local files; this does not demonstrate protected persistent-volume provisioning, host-backed destination ACLs, off-host encryption or production operational recovery. The persistent sandbox was not restarted or migrated for this work.
+
 Final Node 24 TypeScript/Vite and production runtime builds passed. The suite passed **98 tests, 0 failed, 0 skipped**. A separate packaged-runtime smoke drill ran with no external network, read-only root, dropped capabilities and a disposable 64 MiB tmpfs: monitoring correctly returned nonzero for isolation, empty chain, not-yet-reconciled market and low disk; clean stopped-node backup, verification and new-directory restore each exited zero. That empty-chain CLI drill reported proof revalidation as false, as designed; nonempty real-proof replay is covered by the integration drill above. No host volume or production wallet was used.
 
 SEC-07/SEC-09 are only partially addressed: actual edge/source-IP/TLS checks, a production monitoring agent and verified alert delivery, off-host backup storage, long-history restore timing, filesystem/power-loss testing, independent peer/miner topology and named operators remain unverified. See [operator launch register](operator-launch.md) and [security review](security-review.md). No startup or payment approval is granted by these tools.

docs/persistent-custody-sandbox.md

@@ -41,7 +41,7 @@ The **Verify persistent custody after recreation** task runs [the deployment che
 
 The optional **Check persistent custody TLS and credit** task adds a single 1-XMR **fictitious** test credit under an ephemeral test identity before auditing. It does not create Monero outputs, touch existing wallets or transfer NFTs. Repeating that task creates another fictitious customer credit; use the read-only verification task for routine checks. Test identities are not recoverable users and should never be used with valuable funds.
 
-The **Stop persistent custody without deleting data** task uses Compose `down` without volume removal. Restarting with the same project/file recreates containers and reuses database, credentials, TLS and node state. **Never add `--volumes`/`-v` to normal shutdown:** it destroys custody history and credentials for this stack. Unclean node shutdown may leave the existing exclusive node lock; verify the old process is gone and follow the stale-lock review procedure—do not clear it automatically.
+The **Stop persistent custody without deleting data** task uses Compose `down` without volume removal. Restarting with the same project/file recreates containers and reuses database, credentials, TLS and node state. **Never add `--volumes`/`-v` to normal shutdown:** it destroys custody history and credentials for this stack. Unclean node shutdown may leave the existing exclusive node lock; verify the old process is gone and follow the stale-lock review procedure—do not clear it automatically. The provisioner preserves existing v1 secret volumes, then additively creates the backup-only role credential volume and upgrades its completion marker; role SQL grants are applied by the explicit one-shot migrator. Do not run this provisioning/migration upgrade during unreviewed production use.
 
 Existing SQLite-only node backup intentionally refuses PostgreSQL-bound nodes. An explicit [coordinated backup rehearsal](operations-recovery.md#coordinated-postgresql--node-custody-backup-rehearsal) is now available as a separate one-shot compose file; it does not run continuously and is not invoked on ordinary app startup. After stopping the app, it takes a node+PostgreSQL snapshot under both writer fences and a full custody audit. This small-sandbox helper is not an encrypted/off-host backup system, WAL/PITR, restore orchestrator or RPO/RTO claim. Named-volume persistence is **not a backup**, and Docker Desktop resets, disk loss or volume removal can still lose data. A v1 secret-volume upgrade adds the backup-only credential without rotating existing secrets, but should first be rehearsed on a copy of the intended deployment volumes.
 

docs/production-plan.md

@@ -90,9 +90,7 @@ Blocked on P0 ADR; use P1 amount types and P2 durable events. Consensus work is
 
 **Canonical/per-order reorg rehearsal increment, 2026-10-20:** the gated disposable PostgreSQL observer stores bounded pages from an in-process `Ledger` adapter, checks stream/anchor/hash/tip/ancestor continuity, supports retries, and audits retained history. Tests cover a block mined and independently verified with native RandomX flowing through the local ledger into PostgreSQL, plus a fake-PoW equal-work branch/reorg. A separate synthetic-event test links the deterministic candidate terminal transaction ID to its posted order; disconnecting that ID (even followed by same-page re-inclusion) commits the event history and durable order review hold atomically, without undoing or repeating the payout. Generic incident resolution cannot clear a sale-reorg record. The server now has an opt-in polling lifecycle, restricted to PostgreSQL `custody_test`, both disposable flags and non-production mode; it drains on shutdown, resumes the persistent cursor after restart and stops on review/failure. Final build, 172 offline regressions and 30 PostgreSQL tests pass. The worker remains disabled in all regular deployments, active v6 cannot emit candidate IDs, and synthetic linkage is not canonical sale evidence. Stream hashes are not tamper-proof against the schema owner; 4,096-event cap/full scan are non-production. M3 remains blocked on reviewed candidate activation, actual candidate IDs from an activated local ledger, production observer/checkpoint design and production readiness.
 
-**Coordinated backup rehearsal increment, 2026-10-20:** a separate stopped-node maintenance utility now holds the node lock and custody advisory writer lock, checks the local/PostgreSQL custodian ID and runs a complete accounting audit before pairing SQLite ledger/market snapshots with a custom-format `pg_dump`. It uses a dedicated SELECT-only backup login (the local persistent stack provisions it additively, without rotating the existing credentials); the helper is separate from the normal app image. A strict pair manifest is written last. Disposable integration tests restore the dump into an isolated database and verify identity/balances; lock, mismatch, failed dump, incomplete output, overwrite and tamper paths fail closed. Final build, 172 offline regressions and 31 PostgreSQL tests pass. This package is not encrypted or off-host, has no restore orchestrator, PITR/retention/scheduler/RPO/RTO, and has not been run against the persistent stack. Existing services/data were not restarted or changed. Production coordinated recovery remains open.
-
-**Coordinated-backup rehearsal increment, 2026-10-20:** a stopped-node maintenance utility now holds the node startup lock and PostgreSQL custody advisory writer lock, confirms local/database custodian identity, performs a liability audit, snapshots node SQLite databases, writes a full custom-format PostgreSQL dump through a dedicated read-only backup login, and writes a pair manifest last. A disposable integration verifies package hashes and restores the dump into an isolated database with balances/identity preserved; failure/lock/tamper behavior is tested. The persistent local sandbox provisioner adds the backup secret/role without rotating existing credentials. This is not encrypted, off-host, signed, scheduled, PITR, long-history performance or production recovery evidence. No persistent deployment was restarted. Production backups and measurable RPO/RTO remain open.
+**Coordinated-backup rehearsal increment, 2026-10-20:** the stopped-node maintenance utility holds the node startup lock and PostgreSQL custody advisory writer lock, checks local/database custodian identity and frozen cutover digest, performs a full liability audit, snapshots active node SQLite state and writes a custom-format PostgreSQL dump through a dedicated SELECT-only backup login. The strict pair manifest is written last. Its integration mines and independently validates a genuine native-RandomX block, creates the archive, restores the node files into a new directory and reopens them through the reference validator (checking tip and NFT owner); it also restores PostgreSQL into a separate database and compares custodian ID, import digest, balances, audit and receipt state. A post-cutover frozen SQLite source is checked against PostgreSQL `import_source` and omitted as historical rather than misrepresented as current custody. Lock, identity mismatch, failed dump, incomplete output, overwrite, least-privilege and tamper cases are covered. Final build, 172 offline regressions and 31 PostgreSQL tests pass. The persistent sandbox received only an additive backup-role implementation; it was not restarted or migrated. The package is not encrypted/off-host/signed/scheduled and provides no PITR, retention, measured long-history timing, RPO/RTO or production recovery certification. Coordinated recovery remains open.
 
 - If a reviewed trustless design is selected: implement its actual cross-chain validity/evidence rules, compatible cryptographic primitives, both-party failure/refund paths and commission semantics. Run maintained reference vectors, negative tests, fuzz/property tests and specialist audit before claiming atomicity.
 - If a custodial/attested fallback is explicitly approved: define chain-enforced NFT lock, order identity, authorized releaser, cancellation/refund/expiry and stale-signature rules. Separate host control keys from seller wallet and processor payout keys. Reserve/refund procedures must not permit both buyer refund and later seller delivery/release. Describe who can steal, censor or lock assets and how recovery is authorized.

tests/postgres/custody.test.ts

@@ -18,7 +18,7 @@ import { migrateCustody, grantCustodyRuntime } from '../../server/market/postgre
 import { importSqliteCustody } from '../../server/market/custody-import.ts';
 import { postgresConfig, custodyStorage } from '../../server/market/postgres-config.ts';
 import { openCustodyStorage } from '../../server/market/custody-storage.ts';
-import { createBackup } from '../../server/recovery.ts';
+import { createBackup, restoreBackup } from '../../server/recovery.ts';
 import { createCoordinatedCustodyBackup, verifyCoordinatedCustodyBackup } from '../../server/coordinated-backup.ts';
 import { CUSTODY_POLICY, custodyMessage, type CustodyAction, type CustodyCommand, type CustodyActivity,
   type CustodyListing, type CustodyOrder, type CustodyWithdrawal } from '../../shared/custody.ts';
@@ -374,13 +374,22 @@ test('coordinated node/PostgreSQL backup holds both writer fences and verifies m
   t.after(() => rm(dir, { recursive: true, force: true }));
   const data = join(dir, 'node-data'), networkDir = join(data, 'testnet');
   await mkdir(networkDir, { recursive: true });
-  const ledgerDb = new DatabaseSync(join(networkDir, 'ledger.sqlite'));
-  try {
-    ledgerDb.exec(`CREATE TABLE metadata(key TEXT PRIMARY KEY,value TEXT NOT NULL);
-      INSERT INTO metadata VALUES('schema_version','1'),('network','${NETWORK}'),('chain_id','${CHAIN_ID}');
-      CREATE TABLE blocks(sequence INTEGER PRIMARY KEY AUTOINCREMENT,hash TEXT NOT NULL,previous_hash TEXT NOT NULL,block_json TEXT NOT NULL);
-      CREATE TABLE pending_transfers(position INTEGER PRIMARY KEY,id TEXT NOT NULL,transaction_json TEXT NOT NULL);`);
-  } finally { ledgerDb.close(); }
+  const nativeEngine = new RandomX({ binary: process.env.RANDOMX_BINARY });
+  const sourceLedger = await Ledger.open(data, nativeEngine);
+  const blockTemplate = makeTemplate([], {}, seller.actor, seller.publicKey, [], fixtureArticle(56000), fixtureStats(), GENESIS_TIME + 30);
+  const header = structuredClone(blockTemplate.header);
+  let mined: Awaited<ReturnType<typeof nativeEngine.mine>> = null;
+  for (let batch = 0; batch < 16 && !mined; batch++) {
+    mined = await nativeEngine.mine(header.key, miningBlob(header), header.target, 32, 64);
+    if (!mined) header.nonce += 64;
+  }
+  assert.ok(mined, 'Native RandomX cold-backup fixture mining exceeded bounded attempts');
+  header.nonce = mined.nonce;
+  const blockHash = sha256(miningBlob(header));
+  const sourceBlock: TestnetBlock = { ...header, hash: blockHash, tokenId: mintId(header), powHash: mined.hash,
+    signature: seller.sign(blockClaimMessage(blockHash)), transactions: [] };
+  await sourceLedger.submit(sourceBlock);
+  sourceLedger.close(); nativeEngine.close();
   const marketDb = new DatabaseSync(join(networkDir, 'market.sqlite'));
   try {
     marketDb.exec('CREATE TABLE market_meta(key TEXT PRIMARY KEY,value TEXT NOT NULL)');
@@ -410,6 +419,14 @@ test('coordinated node/PostgreSQL backup holds both writer fences and verifies m
   assert.equal(manifest.sqliteCutoverSourceDigest, cutoverDigest);
   assert.equal((await verifyCoordinatedCustodyBackup(destination)).nodeManifestSha256, manifest.nodeManifestSha256);
   await assert.rejects(lstat(join(destination, 'node', 'custody-sandbox.sqlite')), { code: 'ENOENT' });
+  const restoredNodeDir = join(dir, 'restored-node');
+  await restoreBackup(join(destination, 'node'), restoredNodeDir);
+  const verifierEngine = new RandomX({ binary: process.env.RANDOMX_BINARY });
+  const verifierLedger = await Ledger.open(restoredNodeDir, verifierEngine);
+  try {
+    assert.equal(verifierLedger.summary().tip, sourceBlock.hash);
+    assert.equal(verifierLedger.nft(sourceBlock.tokenId)?.owner, seller.actor);
+  } finally { verifierLedger.close(); verifierEngine.close(); }
   const restoreDatabase = `pair_restore_${randomUUID().replaceAll('-', '')}`, run = promisify(execFile);
   await f.admin.query(`CREATE DATABASE "${restoreDatabase}"`);
   try {