wikideck/docs/postgres-financial-storage.md

# ADR 003 — PostgreSQL financial storage

Status: **selected and integrated for the custody sandbox; real-money release not authorized**. Evidence date: 2026-10-20. Related: [custody design](custodial-market-design.md), [production plan](production-plan.md).

## Decision and scope

Use **PostgreSQL** for financial state. Exact arithmetic, transactional constraints, concurrent coherent reads, cross-process locks, established backup/restore tooling and future WAL/PITR operations make it preferable to extending an embedded financial database indefinitely. PostgreSQL 17.11 was exercised in the disposable fixture; pin and review the intended deployed release/image separately.

This is an integrated backend, not a second disconnected balance demo: [custody storage selection](../server/market/custody-storage.ts) supplies [the existing HTTP custody routes](../server/testnet/http.ts) with [PostgreSQL custody](../server/market/postgres-custody.ts). Signed command domains, user identities, amounts and UI stay unchanged. SQLite remains the default disposable/local sandbox for compatibility; choose PostgreSQL explicitly. Chain DAG/canonical indexes and legacy mock marketplace remain in their original SQLite databases. Moving those independent concerns is not needed to use PostgreSQL for customer liabilities.

**All balances still represent fictitious sandbox XMR.** Monero observer and subaddress allocation groundwork are implemented separately in additive custody migrations v2-v4, but remain unwired and unavailable to public custody routes. Schema v5 adds an unexposed reviewed pending-deposit coordinator with its own role and immutable credit evidence; v6/v7 add append-only, reviewer-authorized release and reorg-resolution records. A staged credit remains non-spendable until a separate signed review moves it to `available`; detected reorg holds block new spending for that beneficiary until the output is freshly re-observed as present, unlocked, sufficiently confirmed and separately reviewed. There is no API/startup integration, automatic scan/reorg worker, production treasury backing, live signer, real-address withdrawal, NFT sale witness or payout. The role is provisioned explicitly using `grant-monero-credit-runtime ROLE`; no credentials or reviewer keys are created by the CLI. No mainnet funds are activated.

## Storage and transaction guarantees

- Dedicated `wk_custody` schema with version/digest-checked **explicit migration**, not DDL at HTTP startup. A runtime connection with administrative privileges or schema-owner membership is rejected.
- Money uses unconstrained-precision PostgreSQL `NUMERIC` plus checks for integral, nonnegative, bounded atomic values. Using `NUMERIC(30,0)` alone could silently round a fractional input before checking; this implementation rejects fractions instead. Account balances and individual entry legs remain below $10^{30}$ atomic units. The driver returns monetary values as strings; no global floating-point parser override.
- Journals/entries and replay receipts are immutable to the application. Deferred constraint triggers reject empty/unbalanced journals at commit, including entry inserts. Committed journals reject appended entries. An isolated `SECURITY DEFINER` entry trigger with a fixed schema search path locks accounts and updates balance projections; the runtime cannot directly update balances.
- Reservations, listing/order state, withdrawal holds, journal entries, blocked outbox and receipts commit in the **same database transaction**. Exactly the same signed nonce returns its original receipt; changed terms fail. No receipt pruning or reinterpretation of mock balances as live deposits.
- One writer per financial domain initially: transaction-scoped advisory lock shared across pools/processes, plus row locks and constraints. Reads use a repeatable-read snapshot. This improves separation/concurrency of readers and durable server operations, **not a claim of unlimited concurrent writers**. Sharding/partitioning or finer write locking requires measurements and another reviewed change.
- Pool maximum four connections per service; per-instance active admission sixteen; finite connect/statement/lock/idle transaction timeouts; synchronous commit explicitly on. Scale the total pool budget across instances deliberately. No wallet/network side effect is performed in a database transaction.
- No automatic retry after an uncertain commit. A client refreshes or resends the original signed envelope, letting durable receipts establish whether it committed. PostgreSQL failure never causes SQLite fallback. Startup audits compare every account projection with exact journal sums and check backing totals; these are internal accounting checks, not proof that real Monero assets exist.

**Reconciliation hardening:** [financial business audits](financial-reconciliation.md) additionally check order/withdrawal encodings, exact allocations and held amounts, lifecycle journal lines, task payloads and orphan/duplicate records. They run at startup/import, on a nonoverlapping periodic schedule and on affected mutation boundaries. Persistent integrity incidents block writers and survive restart; only schema-owner acknowledgement with reviewed evidence and successful full audit can resolve them. External database/process loss still requires operational writer fencing. These checks do not prove real Monero backing.

These controls do not make a compromised application harmless: it still has posting privileges, and an administrative database owner can alter schema/history. Restricted roles prevent accidental/direct mutation, not fraudulent balanced postings by a fully compromised service. Independent monitoring, statement review, external backups and authorization/treasury isolation remain required.

## Configuration and operator workflow

The following are configuration values, not credentials:

| Setting | Behavior |
| --- | --- |
| `WIKIDECK_CUSTODY_MODE=sandbox` | Enables the existing testnet-only sandbox. Live/mainnet custody stays disabled. |
| `WIKIDECK_CUSTODY_STORAGE=postgres` | Selects PostgreSQL; default `sqlite` is retained for existing sandbox users. |
| `WIKIDECK_CUSTODY_PG_URL_FILE` | Absolute path to a private secret file containing a PostgreSQL URL. Never a Vite/browser variable. |
| `WIKIDECK_CUSTODY_PG_CA_FILE` | Optional private file with trusted PEM CA certificate(s); otherwise use system trust. TLS hostname/certificate verification remains enabled. |
| `WIKIDECK_CUSTODY_PG_TRANSPORT` | Default `tls`: verified PostgreSQL TLS. Optional `infrastructure`: the driver uses plaintext and relies on an independently enforced encrypted/authenticated infrastructure path; see conditions below. |
| `WIKIDECK_CUSTODY_PG_INFRASTRUCTURE_ACK` | Required **only** in infrastructure mode, exactly `encrypted-authenticated-path-enforced`. An operator acknowledgement, NOT automatic verification of infrastructure security. |
| `WIKIDECK_CUSTODY_PG_LOCAL_TEST=1` | **Disposable loopback tests only:** disables TLS solely for literal loopback/localhost. Never use for production. |

URL query parameters/fragments are rejected, preventing `sslmode` options from overriding transport policy. Missing/malformed configuration fails with a generic error, not a URL/password dump. Secret files must be regular non-symlink bounded files; mount them read-only with restricted filesystem ACLs. Production network firewall, verified TLS, password/secret rotation and external database logs need an operator review. The automated fixture tests configuration policy, not an actual production certificate renewal/handshake drill.

### Kubernetes/infrastructure-managed transport

Prefer the default verified PostgreSQL TLS, with certificate issuance/renewal provided by the database operator, managed service or cluster infrastructure. **Wikideck does not need a bespoke certificate lifecycle service.** The local Docker certificate generator is a sandbox convenience, not the Kubernetes production architecture.

If an enforced service-mesh mTLS path or equivalent infrastructure authenticates the intended database peer and encrypts the complete inter-workload path, deployments may explicitly select `infrastructure` plus the exact acknowledgement above. This sets the PostgreSQL driver's `ssl` to false: application-to-proxy and proxy-to-database plaintext segments must be local/isolated within the reviewed boundary. A firewall, ordinary Kubernetes networking or NetworkPolicy alone does not satisfy that condition. Ensure sidecar interception cannot be bypassed, plaintext inter-pod access is denied, destination identities are restricted, and policies remain enforced after restarts/failover. Protect migration, audit and recovery clients too—not just the HTTP application.

Infrastructure mode cannot be combined with a CA file or the loopback-test switch; a stray acknowledgement in TLS mode is rejected. URL TLS overrides remain forbidden and file-based password credentials remain required. No insecure “accept any certificate” TLS option is added. Wikideck cannot verify external mesh enforcement; the acknowledgement is not proof and does not authorize live funds. Cluster evidence/review remains a release gate.

Validation for this transport increment: TypeScript/Vite build, **171 offline tests and 16 PostgreSQL integration tests passed**, zero failures/skips. Configuration tests cover secure defaults, exact opt-in, missing acknowledgement/password, invalid mode, conflicting CA/test settings and URL override rejection. No Kubernetes cluster or actual service-mesh enforcement was tested or certified.

Existing Compose stacks remain configured for verified native PostgreSQL TLS. Their TLS-only database HBA and deployment smoke probe intentionally reject infrastructure/plaintext mode: do not change just the client flag on that stack. This option is for a separately configured infrastructure deployment, not automatic TLS fallback. No Kubernetes resources or running database/container settings were changed. Custom certificate renewal implementation is deferred to infrastructure; the local sandbox's documented short-lived certificates still need controlled operational replacement if retained.

The operator CLI is [scripts/custody-db.ts](../scripts/custody-db.ts), available through `npm run custody:db -- ACTION`. Credentials are provided only by the secret-file environment setting, never positional arguments or browser UI:

1. Provision a dedicated database, a migration/schema-owner identity, and a separate existing login role without superuser/create-role/create-db/replication/bypass-RLS privileges. Create passwords privately outside this tool; no credentials are generated or displayed by the CLI.
2. With migration credentials, run action `migrate`. It checks the pinned version/digest or creates a fresh schema transactionally. Unknown versions are refused; no reset/drop/rebuild mode exists.
3. With migration credentials, run action `grant-runtime ROLE`. It validates the literal role identifier and grants schema usage, reads, account/journal column-specific inserts, entry/receipt inserts and operational record updates. It never grants direct balance updates, history deletion, schema ownership or migrations. Use a **new dedicated role**; the helper grants required privileges but is not a scrubber for inherited privileges from an existing unrelated role.
4. If the data directory has existing SQLite custody, stop the node, preserve a stopped-node backup, and run `import-sqlite DATA_DIR` against a migrated **empty** target using migration credentials. See cutover behavior below.
5. Switch the server to runtime credentials and explicit PostgreSQL selection. Startup verifies schema, restricted identity, custodian binding and accounting before serving. A new directory can initialize a fresh sandbox identity; it does not inherit another host's credit.
6. Action `audit`, using runtime credentials, prints exact aggregate sandbox totals only. It does not print customer histories or database credentials. Running it on a previously uninitialized schema initializes a sandbox custodian identity just like startup, so do not use it as a supposedly read-only probe of an empty migration target.

The CLI is included in the runtime image. No existing deployment, database secret or fund wallet was changed automatically by this implementation.

**Persistent local deployment now available:** [deployment guide](persistent-custody-sandbox.md) and [standalone Compose stack](../compose.custody.yaml) provision isolated named volumes, scoped generated credentials, verified PostgreSQL TLS/SCRAM, separate roles and a one-shot migration service. Recreated containers retained a signed fictitious credit and custodian identity in rehearsal. This explicitly started a new sandbox, not a cutover of existing node data or a live-custody deployment; production backup, certificate rotation and activation gates remain open.

## Explicit SQLite cutover and rollback limits

[Importer](../server/market/custody-import.ts) is bounded to the existing sandbox source size (128 MiB), not a general live-money migration utility. The CLI takes the same exclusive stopped-node directory lock as node startup. Calling the underlying library assumes the caller has already stopped all source writers.

1. Acquire SQLite immediate transaction and take a consistent snapshot. Verify source identity/integrity and canonical amount encoding. Hash identity and all logical rows in deterministic insertion order.
2. **Freeze source first:** commit its `postgres_cutover` digest before contacting the target. SQLite startup refuses this marker. This closes the crash window where PostgreSQL might commit while SQLite remains writable.
3. Lock PostgreSQL financial domain. Require an empty migrated target. Import accounts initially at zero and replay original journal entries through the database posting trigger; compare rebuilt balances to source projections. Preserve custodian identity, all receipts/digests, fictitious-credit claims, listing/order/withdrawal order, prepared attempt/transaction identities and blocked outbox state.
4. Validate accounting, then commit the imported source digest. Repeating the **same** frozen-source import returns success without duplicating balances. A changed source/different imported target is rejected. Failure before commit rolls target rows back; the source stays frozen. Correct the privately diagnosed issue and resume the same import—do not delete markers to restart SQLite.
5. PostgreSQL startup records a private fsynced local binding of custodian ID. A directory with that binding refuses SQLite even if no source database exists. An imported source must match the target's stored source digest.

There is no live dual-write, automatic migration or automatic rollback. An old pre-cutover backup cannot know about a later cutover: restoring it and running SQLite would fork accounting operationally. Fence old hosts/credentials and archive old backups as **historical, non-spending**. This cannot be solved by a local marker alone; coordinated restore and authority controls are release gates.

## Backup, restore and failure handling

Existing node backup packages cover local SQLite databases only. They now **refuse** a PostgreSQL-bound data directory or frozen cutover source instead of claiming to capture all customer liabilities. A separate [coordinated stopped-node rehearsal](operations-recovery.md#coordinated-postgresql--node-custody-backup-rehearsal) can now package node SQLite plus a PostgreSQL custom dump while holding the node lock, the database advisory writer lock and an audit snapshot. It uses a separate read-only backup role in the local persistent sandbox. This is not an automatic production coordinator, and uncooperative privileged writers still require external fencing.

Recommended operations before any production release:

- Managed PostgreSQL storage with encrypted backups, WAL archival/PITR, monitored replica/backup lag, checked restore retention and selected RPO/RTO. Define synchronous-replica requirements based on acceptable loss/failover risk; `synchronous_commit=on` alone does not configure a synchronous replica.
- Fence writers/treasury services during backup/restore cutover and record matching node ledger/market checkpoints, custodian identity, database dump/backup identity and external withdrawal state. Do not restore a financial snapshot in isolation and start spending.
- Use full-database dumps that include schema/functions/triggers/ACLs, plus separately protected role configuration; a schema-filtered dump may omit necessary trigger dependencies. Restore into an isolated database first, using administrative credentials; connect only the restricted runtime for audit/replay checks after restore.
- Reconcile Monero external history before resuming real spending once that integration exists. No local backup can reverse an external payment. Database timeouts/connection loss leave requests failed closed; retry the original signed command, not a new withdrawal.
- Monitor application and database errors without logging statements/parameters containing private customer data in production. Fixture logs deliberately show failed constraint probes on fictitious data; do not adopt their logging/auth configuration.

The integration suite performs both a PostgreSQL `pg_dump`/isolated `pg_restore` rehearsal preserving ID, exact liabilities and receipt replay, and a paired node/PostgreSQL archive-creation/verification/restore test using the actual runtime adapter plus a read-only backup role. Pair verification records PostgreSQL server/dump/restore versions, checks exact package entries and checksums, and parses the custom archive using `pg_restore --list` while requiring core custody tables and a compatible restore client. The dump subprocess is isolated from inherited libpq endpoint/TLS overrides. This does **not** certify managed backup schedules, PITR, encrypted off-host custody, signature/authenticity, power-loss/fsync behavior, long-history timing or production failover.

## Evidence and remaining gates

[Disposable test compose](../compose.postgres-test.yaml) runs PostgreSQL on tmpfs in an isolated internal network, no published ports; test runner shares the database network namespace and uses loopback. Trust authentication is **fixture-only**, not a deployment template. [Tests](../tests/postgres/custody.test.ts) are a separate explicit suite and refuse to run without `WIKIDECK_PG_DISPOSABLE_TEST=1`.

Verified: restricted runtime authorization; deferred balance constraints; fractional/negative/oversized rejection; committed journal append rejection; independent-pool simultaneous nonce/listing/withdrawal races; insufficient-balance rollback; injected receipt failure atomicity; exact amounts beyond JavaScript safe integer; reconnect/replay; import of prepared withdrawals and pending orders; failed-import freeze/rollback/resume; real HTTP backend integration; configuration policy; fallback/backup refusal; full dump/restore audit.

Final evidence: TypeScript/Vite test-image build passed; **149 existing regression tests and 10 isolated PostgreSQL integration tests passed**, zero failures/skips. PostgreSQL fixture resources were removed after testing. Existing user data directories and deployed node database settings were not switched. This verifies the current sandbox implementation, not production acceptance.

**Not closed:** live Monero and NFT settlement; production dataset/SLOs and pool contention benchmarks; audit scheduling and very-large-history startup cost; normalized durable real-deposit/withdrawal/compensation states; account recovery/second factor; external security/accounting review; coordinated node/financial/treasury backup; deployment TLS drills, PITR and failover rehearsal; funded exposure/fee/loss policies; legal/operations sign-off. The scalable database choice does not turn the existing sandbox into a production custodian.

### Reviewed Monero pending-credit operations (sandbox only)

`npm run custody:db -- ACTION` now supports these explicit, non-HTTP actions for the pending-credit ledger stage:

- `deposit-prepare-credit WALLET_FINGERPRINT OBSERVATION_ID` emits the exact review subject for an eligible, currently observed output.
- `deposit-stage-credit APPROVAL_JSON_FILE` verifies the allow-listed reviewer signature and records the amount in a non-spendable pending liability.
- `deposit-prepare-release CREDIT_UUID` emits the current output and unresolved-hold snapshot.
- `deposit-release APPROVAL_JSON_FILE` moves the matching liability to `available` after a distinct reviewer signature.
- `deposit-reconcile-holds WALLET_FINGERPRINT` records unseen/changed/reappeared observations as append-only holds.
- `deposit-prepare-resolve-hold CREDIT_UUID HOLD_UUID` and `deposit-resolve-hold APPROVAL_JSON_FILE` resolve a hold only after a fresh eligible observation and a new reviewer signature.

The operator must separately provision the coordinator role with action `grant-monero-credit-runtime ROLE`; database migration remains explicit. The review policy comes from `WIKIDECK_MONERO_DEPOSIT_REVIEW_POLICY_FILE`, an owner-only JSON file containing only approved reviewer owner public-key hashes, a confirmation floor (at least 10 in this sandbox policy), and maximum observer age. Approval JSON files are bounded, absolute-path, regular files opened without following symlinks, and owner-only on POSIX. **These review-file commands currently refuse Windows** because the implementation cannot verify Windows ACLs; run them only in a secured POSIX environment. No reviewer private key is loaded by the application: snapshots are emitted for external signing and signed approval files are then submitted back to the CLI. Two-argument commands parse both identifiers explicitly. These CLI actions do not enable scanning, an HTTP endpoint, an unattended worker, or live deposits.

The added `tests/custody-db-args.test.ts` verifies two-argument IDs are preserved in the proper positions and rejects missing, empty, unknown, or extra arguments. `tests/monero-deposit-review-config.test.ts` checks strict policy parsing, approval-file bounds, POSIX owner-only permissions, no-follow/symlink rejection, and fail-closed Windows behavior. Full offline validation passed **194/194**; PostgreSQL integration passed **36/36** after the operator workflow changes. These tests verify offline command/config boundaries only; no operator keys or live RPC were used.