Separate one-shot utility

docs/custody-sale-rehearsal.md

@@ -45,7 +45,7 @@ The local-ledger adapter is **not** linked to these terminal candidate IDs: acti
 
 ## Validation and remaining gates
 
-TypeScript/Vite Docker build passed; **172 offline regressions and 30 PostgreSQL integration tests passed, zero failures/skips**. [Candidate tests](../tests/sale-candidate.test.ts) cover signature compatibility/recipient tampering and candidate transition rules. [PostgreSQL tests](../tests/postgres/custody.test.ts) add:
+TypeScript/Vite Docker build passed; **172 offline regressions and 31 PostgreSQL integration tests passed, zero failures/skips**. [Candidate tests](../tests/sale-candidate.test.ts) cover signature compatibility/recipient tampering and candidate transition rules. [PostgreSQL tests](../tests/postgres/custody.test.ts) add:
 
 - explicit instance/database consent and exact signed reservation matching;
 - seed-binding changes, malformed/cyclic/oversized collections, and denial of ordinary cancellation after binding;

docs/market-host-configuration.md

@@ -32,4 +32,4 @@ Existing signature bytes remain valid: omission retains the original message; ex
 
 The custody SQLite fixture records newly settled simulated host revenue under its snapshotted recipient. PostgreSQL production settlement/host withdrawals are still unimplemented: configuring an address does not create automatic transfers. Historical unallocated host revenue remains historical; it is not silently reassigned. The offline sale-lock candidate now preserves the hosting recipient in a tagged signed-terms extension. A [disposable PostgreSQL accounting rehearsal](custody-sale-rehearsal.md) verifies that the signed snapshot receives simulated revenue even after current host policy changes; this is not active-chain delivery or a live host withdrawal service.
 
-Validation: current TypeScript/Vite build, 172 offline regressions and 28 PostgreSQL integration tests passed. Tests cover strict flag/address validation, paused/disabled behavior, continuing existing-order delivery, preserved balances/cancellations, signed-recipient tampering and recipient snapshots across config changes, including the isolated accounting rehearsal. Real XMR remains disabled.
+Validation: current TypeScript/Vite build, 172 offline regressions and 31 PostgreSQL integration tests passed. Tests cover strict flag/address validation, paused/disabled behavior, continuing existing-order delivery, preserved balances/cancellations, signed-recipient tampering and recipient snapshots across config changes, including the isolated accounting rehearsal. Real XMR remains disabled.

docs/operations-recovery.md

@@ -59,6 +59,24 @@ Action `verify` needs `--source <backup-directory>` only. It reads a bounded str
 
 An operator must choose recovery point/retention/RPO/RTO, practice off-host recovery and approve any production restore explicitly. Never remove a lock or replace a running volume just to make a recovery test pass.
 
+## Coordinated PostgreSQL + node custody backup rehearsal
+
+The prior file-only `node:backup` still refuses PostgreSQL-bound node directories. A separate explicit [paired backup helper](../compose.custody-backup.yaml) and [utility](../server/coordinated-backup.ts) now create a small-sandbox rehearsal package containing the SQLite ledger/market pair and a custom-format PostgreSQL database dump. Creation holds the exact node-directory lock, then a dedicated PostgreSQL session advisory writer lock, checks the local `custody-postgres.json` binding against the database custodian identity, verifies any retained SQLite cutover source is frozen and its digest matches PostgreSQL's `import_source`, audits PostgreSQL liabilities under the lock, snapshots the active node files and runs `pg_dump`. A frozen SQLite source is deliberately omitted from the active backup pair; PostgreSQL is authoritative after cutover. The pair manifest is written last and binds network/chain/profile/custodian ID, the node manifest digest, PostgreSQL database name and dump size/hash. No restore/reset operation is included.
+
+The PostgreSQL dump uses a **separate read-only `custody_backup` login**, not the app runtime or schema owner. The local Compose provisioner now creates its credential volume, the one-shot bootstrap grants only database connect, schema usage, table SELECT and sequence SELECT, and the database HBA permits only TLS/SCRAM for this role. Existing local v1 secret volumes get an additive backup-role secret upgrade without rotating database credentials. This changes the next explicit persistent-stack provision/migrate cycle; no running stack was restarted here. The helper image has `pg_dump` and is a one-shot maintenance container; those tools and backup credentials are not added to the long-running app image. The disposable integration test checks that the role can dump but cannot update balances, delete entries or advance journal sequences.
+
+### Local sandbox invocation
+
+This procedure is **not yet a production backup runbook**. Use only after reviewing the exact Compose project/volume names and destination permissions:
+
+1. Stop only the app service and verify no other node/application process writes the same data directory. Leave the PostgreSQL service running. Do not stop/restart or remove any user deployment as part of development tests.
+2. With the matching stopped-node directory, run the one-shot provision and migration services under an approved maintenance window so older v1 local secret volumes receive the additive read-only backup credential and database role. Verify existing passwords, CA and custodian ID were preserved; do not regenerate volumes.
+3. Set `WIKIDECK_BACKUP_DIR` to an **existing, restricted** host directory on encrypted storage and `WIKIDECK_BACKUP_NAME` to a new leaf name. Run `docker compose -p wikideck-custody-local -f compose.custody-backup.yaml run --rm backup`. The helper requires an existing destination parent and refuses overwrite. The node lock blocks startup during the entire paired snapshot. A PostgreSQL session lock fences all compliant Wikideck custody writers; privileged/manual DB writers must also be operationally fenced.
+4. Run the helper with `verify --source /backups/<name>` (overriding the service command) to check the strict pair manifest, standalone node databases, hashes, and exact package contents. Verify pg_dump compatibility/restorability in a separately isolated PostgreSQL database; verification here does not by itself parse/restore the archive.
+5. Store an encrypted off-host copy with separately protected authenticity metadata. This package is **not encrypted** by the utility, and colocating it with the source host does not satisfy disaster recovery.
+
+Creation holds locks and runs a full financial audit; a large journal may exceed configured query/audit budgets and fail closed, so this needs measured long-history tests before production use. A database superuser/schema owner can bypass application advisory-lock discipline or rewrite records; external writer fencing remains mandatory. The output hash/manifest detects accidental damage, not a malicious coherent replacement. The PostgreSQL dump is not atomic with arbitrary unmanaged ledger or schema-owner activity. There is no WAL/PITR, remote scheduler, retention, restore orchestration, RPO/RTO claim, encryption, signature or production failover process in this increment. The local Compose backup service mounts the external host destination read-write into a container with a stopped-node source lock; review Docker volume access and the helper image before using it around any valuable data.
+
 ## Evidence and remaining hold points
 
 Automated tests cover default proxy distrust/spoofed-header rejection, explicit loopback proxy recognition, invalid trust/port settings, private listener and threshold checks, paired backup/restore identity preservation, refusal to overwrite/ignore locks/back up within data, and rejection of tampering, wrong network, incomplete packages and unexpected sidecars.

docs/persistent-custody-sandbox.md

@@ -21,7 +21,7 @@ Completed one-shot services are expected to show **Exited (0)**. PostgreSQL and
 
 ## Persistent volumes and isolation
 
-Named volumes separately retain PostgreSQL data, node data, admin credentials, schema-owner credentials, runtime credentials and database TLS material. Generated secrets are not written into source, command arguments, Compose environment or browser variables. The app receives no admin/owner credential or server private key. Generated files are scoped by volume mounts; the admin password file is readable inside the trusted database/provisioning containers for the PostgreSQL entrypoint, not mounted in the app.
+Named volumes separately retain PostgreSQL data, node data, admin credentials, schema-owner credentials, runtime credentials, a read-only `custody_backup` login, and database TLS material. Generated secrets are not written into source, command arguments, Compose environment or browser variables. The app receives no admin/owner/backup credential or server private key. The separate one-shot migration provisions backup role grants limited to SELECT on custody tables/sequences; the role cannot advance sequence values or update/delete financial records. The admin password file is readable inside the trusted database/provisioning containers for the PostgreSQL entrypoint, not mounted in the app.
 
 This is a **new independent market**, not a migration of any existing node or wallet balance. Do not attach an existing SQLite custody volume and expect automatic import. Follow the stopped-node, backup and source-freezing procedure in [ADR 003](postgres-financial-storage.md) for that case. Do not point multiple independent market identities at this one custody schema.
 
@@ -43,7 +43,7 @@ The optional **Check persistent custody TLS and credit** task adds a single 1-XM
 
 The **Stop persistent custody without deleting data** task uses Compose `down` without volume removal. Restarting with the same project/file recreates containers and reuses database, credentials, TLS and node state. **Never add `--volumes`/`-v` to normal shutdown:** it destroys custody history and credentials for this stack. Unclean node shutdown may leave the existing exclusive node lock; verify the old process is gone and follow the stale-lock review procedure—do not clear it automatically.
 
-Existing SQLite-only node backup intentionally refuses PostgreSQL-bound nodes. This stack does not yet supply a coordinated backup job: full PostgreSQL backups, protected role/secret material and matching node state must be covered together. Named-volume persistence is **not a backup**, and Docker Desktop resets, disk loss or volume removal can still lose data. WAL/PITR, off-host encrypted copies, restore/failover exercises and RPO/RTO remain production gates.
+Existing SQLite-only node backup intentionally refuses PostgreSQL-bound nodes. An explicit [coordinated backup rehearsal](operations-recovery.md#coordinated-postgresql--node-custody-backup-rehearsal) is now available as a separate one-shot compose file; it does not run continuously and is not invoked on ordinary app startup. After stopping the app, it takes a node+PostgreSQL snapshot under both writer fences and a full custody audit. This small-sandbox helper is not an encrypted/off-host backup system, WAL/PITR, restore orchestrator or RPO/RTO claim. Named-volume persistence is **not a backup**, and Docker Desktop resets, disk loss or volume removal can still lose data. A v1 secret-volume upgrade adds the backup-only credential without rotating existing secrets, but should first be rehearsed on a copy of the intended deployment volumes.
 
 ## Evidence and limits
 

docs/postgres-financial-storage.md

@@ -79,7 +79,7 @@ There is no live dual-write, automatic migration or automatic rollback. An old p
 
 ## Backup, restore and failure handling
 
-Existing node backup packages cover local SQLite databases only. They now **refuse** a PostgreSQL-bound data directory or frozen cutover source instead of claiming to capture all customer liabilities. No automatic cross-database backup coordinator is implemented.
+Existing node backup packages cover local SQLite databases only. They now **refuse** a PostgreSQL-bound data directory or frozen cutover source instead of claiming to capture all customer liabilities. A separate [coordinated stopped-node rehearsal](operations-recovery.md#coordinated-postgresql--node-custody-backup-rehearsal) can now package node SQLite plus a PostgreSQL custom dump while holding the node lock, the database advisory writer lock and an audit snapshot. It uses a separate read-only backup role in the local persistent sandbox. This is not an automatic production coordinator, and uncooperative privileged writers still require external fencing.
 
 Recommended operations before any production release:
 
@@ -89,7 +89,7 @@ Recommended operations before any production release:
 - Reconcile Monero external history before resuming real spending once that integration exists. No local backup can reverse an external payment. Database timeouts/connection loss leave requests failed closed; retry the original signed command, not a new withdrawal.
 - Monitor application and database errors without logging statements/parameters containing private customer data in production. Fixture logs deliberately show failed constraint probes on fictitious data; do not adopt their logging/auth configuration.
 
-The integration suite performs a real full `pg_dump`/isolated `pg_restore` rehearsal preserving ID, exact liabilities and receipt replay. It does **not** certify managed backup schedules, PITR, encrypted off-host custody, power-loss/fsync behavior or production failover.
+The integration suite performs both a PostgreSQL `pg_dump`/isolated `pg_restore` rehearsal preserving ID, exact liabilities and receipt replay, and a paired node/PostgreSQL archive-creation/verification/restore test using the actual runtime adapter plus a read-only backup role. This does **not** certify managed backup schedules, PITR, encrypted off-host custody, signature/authenticity, power-loss/fsync behavior, long-history timing or production failover.
 
 ## Evidence and remaining gates
 

docs/production-plan.md

@@ -90,6 +90,10 @@ Blocked on P0 ADR; use P1 amount types and P2 durable events. Consensus work is
 
 **Canonical/per-order reorg rehearsal increment, 2026-10-20:** the gated disposable PostgreSQL observer stores bounded pages from an in-process `Ledger` adapter, checks stream/anchor/hash/tip/ancestor continuity, supports retries, and audits retained history. Tests cover a block mined and independently verified with native RandomX flowing through the local ledger into PostgreSQL, plus a fake-PoW equal-work branch/reorg. A separate synthetic-event test links the deterministic candidate terminal transaction ID to its posted order; disconnecting that ID (even followed by same-page re-inclusion) commits the event history and durable order review hold atomically, without undoing or repeating the payout. Generic incident resolution cannot clear a sale-reorg record. The server now has an opt-in polling lifecycle, restricted to PostgreSQL `custody_test`, both disposable flags and non-production mode; it drains on shutdown, resumes the persistent cursor after restart and stops on review/failure. Final build, 172 offline regressions and 30 PostgreSQL tests pass. The worker remains disabled in all regular deployments, active v6 cannot emit candidate IDs, and synthetic linkage is not canonical sale evidence. Stream hashes are not tamper-proof against the schema owner; 4,096-event cap/full scan are non-production. M3 remains blocked on reviewed candidate activation, actual candidate IDs from an activated local ledger, production observer/checkpoint design and production readiness.
 
+**Coordinated backup rehearsal increment, 2026-10-20:** a separate stopped-node maintenance utility now holds the node lock and custody advisory writer lock, checks the local/PostgreSQL custodian ID and runs a complete accounting audit before pairing SQLite ledger/market snapshots with a custom-format `pg_dump`. It uses a dedicated SELECT-only backup login (the local persistent stack provisions it additively, without rotating the existing credentials); the helper is separate from the normal app image. A strict pair manifest is written last. Disposable integration tests restore the dump into an isolated database and verify identity/balances; lock, mismatch, failed dump, incomplete output, overwrite and tamper paths fail closed. Final build, 172 offline regressions and 31 PostgreSQL tests pass. This package is not encrypted or off-host, has no restore orchestrator, PITR/retention/scheduler/RPO/RTO, and has not been run against the persistent stack. Existing services/data were not restarted or changed. Production coordinated recovery remains open.
+
+**Coordinated-backup rehearsal increment, 2026-10-20:** a stopped-node maintenance utility now holds the node startup lock and PostgreSQL custody advisory writer lock, confirms local/database custodian identity, performs a liability audit, snapshots node SQLite databases, writes a full custom-format PostgreSQL dump through a dedicated read-only backup login, and writes a pair manifest last. A disposable integration verifies package hashes and restores the dump into an isolated database with balances/identity preserved; failure/lock/tamper behavior is tested. The persistent local sandbox provisioner adds the backup secret/role without rotating existing credentials. This is not encrypted, off-host, signed, scheduled, PITR, long-history performance or production recovery evidence. No persistent deployment was restarted. Production backups and measurable RPO/RTO remain open.
+
 - If a reviewed trustless design is selected: implement its actual cross-chain validity/evidence rules, compatible cryptographic primitives, both-party failure/refund paths and commission semantics. Run maintained reference vectors, negative tests, fuzz/property tests and specialist audit before claiming atomicity.
 - If a custodial/attested fallback is explicitly approved: define chain-enforced NFT lock, order identity, authorized releaser, cancellation/refund/expiry and stale-signature rules. Separate host control keys from seller wallet and processor payout keys. Reserve/refund procedures must not permit both buyer refund and later seller delivery/release. Describe who can steal, censor or lock assets and how recovery is authorized.
 - Completed-sale inclusion must require the approved evidence that payment is sufficiently confirmed. A peer must not insert a completed sale merely because its local database says paid. Define threshold/policy version, proof/attestation authorization and deterministic validation; no external HTTP calls in consensus validation.

docs/sale-lock-candidate.md

@@ -66,7 +66,7 @@ An already mature terminal record remains usable evidence if the NFT is later gi
 
 ## Evidence and next release gates
 
-Initial candidate evidence: 162 offline tests passed, including 13 candidate tests. Current TypeScript/Vite Docker build and **172 offline tests plus 28 PostgreSQL integration tests passed, zero failures/skips**, including 14 candidate tests in [tests/sale-candidate.test.ts](../tests/sale-candidate.test.ts). Coverage includes current-consensus rejection, field/signature and hosting-recipient tampering, exact height boundaries, sequence exhaustion, native transfer/trade lock bypass, competing-market/reused-order locks, atomic conflicting-outcome rejection, reorg financial decisions, deterministic snapshot restore, state commitments, actual custody-order drafting and enumerated adversarial histories through expiry. The [accounting rehearsal](custody-sale-rehearsal.md) records actual PostgreSQL/fictitious-funds postings, a local-ledger event adapter test with fake PoW, and synthetic candidate-terminal disconnect linkage. No protocol guarantee is inferred from a test count.
+Initial candidate evidence: 162 offline tests passed, including 13 candidate tests. Current TypeScript/Vite Docker build and **172 offline tests plus 31 PostgreSQL integration tests passed, zero failures/skips**, including 14 candidate tests in [tests/sale-candidate.test.ts](../tests/sale-candidate.test.ts). Coverage includes current-consensus rejection, field/signature and hosting-recipient tampering, exact height boundaries, sequence exhaustion, native transfer/trade lock bypass, competing-market/reused-order locks, atomic conflicting-outcome rejection, reorg financial decisions, deterministic snapshot restore, state commitments, actual custody-order drafting and enumerated adversarial histories through expiry. The [accounting rehearsal](custody-sale-rehearsal.md) records actual PostgreSQL/fictitious-funds postings, a local-ledger event adapter with native RandomX validation, fake-PoW branch/reorg fixtures, and synthetic candidate-terminal disconnect linkage. Coordinated backup tests use a separate read-only fixture role and isolated restore. No protocol or production-recovery guarantee is inferred from a test count.
 
 **This does not close M3.** Required next steps: