diff --git a/docs/financial-reconciliation.md b/docs/financial-reconciliation.md new file mode 100755 index 0000000..f23f412 --- /dev/null +++ b/docs/financial-reconciliation.md @@ -0,0 +1,48 @@ +# Custodial business-liability reconciliation + +Status: implemented PostgreSQL **sandbox** integrity checks; not external proof of Monero backing or production settlement acceptance. Related: [production plan](production-plan.md), [financial storage](postgres-financial-storage.md). Updated 2026-10-20. + +## Why totals are insufficient + +A balanced journal and matching account projections can coexist with a corrupted order: its buyer, amount, status or pending task may disagree with the funds actually reserved. Aggregate assets equalling liabilities alone cannot detect a substituted recipient or orphan task. + +[Business integrity checks](../server/market/custody-integrity.ts) now validate both directions: + +- Strict persisted listing/order/withdrawal encodings, indexed identity/status columns and JSON record agreement. +- Order card, seller, buyer, sequence, price and historical hosting recipient against the original listing; exact independently rounded seller/author/host allocation and fixed author policy. Current operator fee configuration never rewrites history. +- Each awaiting-lock order has its exact credit-normal purchase reservation; cancelled/fixture-settled orders have zero reservation. The actual reservation and terminal journal lines must match the expected lifecycle, accounts and amounts, not just net to zero. +- Each withdrawal hold equals recipient amount plus maximum approved fee. Prepared/complete fixture records require an attempt ID, unique transaction identity and fee within cap. Cancelled/complete holds are zero; terminal postings and returned unused fee allowance match exactly. +- Blocked/cancelled/prepared/completed outbox state and payload correspond to the business record. Imported SQLite fixture terminal evidence is recognized explicitly; no real NFT/Monero settlement is inferred. +- Reverse checks reject orphan reservation accounts, pending work and lifecycle postings, duplicate listing orders, and reused prepared attempt/transaction identities. + +Reads use the caller's coherent transaction. Record batches use UUID keyset pagination, maximum 128 per batch, with no growing process-wide customer cache. A 15-second business-scan budget is checked between batches/queries; the existing per-query timeout can extend the final boundary. This bounds application memory and failure behavior, **not a production capacity claim**. Very large histories may require indexed normalized schemas, incremental auditing and a dedicated database connection/worker budget before release. + +The existing exact journal/projection/backing audit remains, and now invokes these checks at startup, explicit audit, SQLite import validation and deployment checks. Targeted pre/post checks run inside order/withdrawal mutation transactions. No database schema version or old financial record is rewritten by this increment. + +## Runtime scheduling and failure behavior + +The PostgreSQL backend schedules one nonoverlapping audit every 60 seconds. Mutations are held if the latest audit failed or the last success is over 120 seconds old. A transient availability failure can recover after a successful later audit; it does not automatically classify customer behavior as fraud. + +A confirmed integrity mismatch latches the detecting process and stores a redacted `integrity_incident_v1` record in custody metadata. Subsequent writers check that marker under the existing cross-process advisory lock. Startup refuses it. Private activity and original signed receipt replay remain available; no new posting, refund, cancellation or withdrawal reservation is allowed while integrity is uncertain. This is deliberately stricter than an ordinary market-disabled flag, which still permits normal resolution. + +For a targeted write-path mismatch, a savepoint rolls back business work while retaining the writer lock; the incident is committed before another cooperating writer can proceed. A background/read-only audit or startup failure records its incident separately. **Process crash or database unavailability can prevent that marker from committing.** A redacted error explicitly requests fencing all writers if marker persistence fails. A periodic audit is not an atomic fraud-prevention barrier for every unrelated write, nor protection from a privileged operator or fully compromised application. + +Private loopback operations now expose only backend, mutation/incident hold, audit timestamps, freshness and a stable failure code. No customer ID, order ID, address, SQL statement, balance or connection secret is attached. [Operations checks](../server/operations-check.ts) emit `custody-financial-review` and `custody-audit-stale` warnings. No new public admin or financial audit endpoint was added. + +## Controlled resolution — never reset journals + +1. Stop/fence all custody writers and preserve matching database/node evidence and backups. Diagnose privately; a public-facing error intentionally does not disclose the affected customer. +2. Have the responsible operator/reviewer establish the correct state from immutable receipts/journals, trusted backup evidence and any external effects. This increment supplies **no automatic repair, compensation, destructive reset or payout retry**. Repair/restore authorization is separate from acknowledging an incident. +3. Using the schema-owner credential, run the existing database CLI action `resolve-integrity EVIDENCE_UUID`. The UUID refers to the externally maintained reviewed incident case; the tool does not certify the truth of that case or supply a second human approval. +4. [Resolution code](../server/market/custody-incident.ts) takes the financial writer lock, verifies schema-owner membership, tentatively removes the unresolved marker and runs the full accounting/business audit in the same transaction. Failure rolls everything back, preserving the hold. Success appends a resolution record with evidence UUID, timestamp and incident digest; it changes no journal/account/business rows. +5. Restart previously latched instances only after controlled resolution, then verify successful startup/audit and operations state. A detecting instance stays locally held even after a successful later audit. The restricted runtime role cannot acknowledge incidents or delete the marker. + +This process is intentionally not a “clear error” button. Manual SQL balance edits, deleting a marker outside the reviewed procedure, restoring stale isolated financial state or inventing journal resets can create duplicate payouts or loss of liabilities. An external database owner retains administrative power; independent evidence retention remains necessary. + +## Evidence and limitations + +Tests cover business-record corruptions that leave all journal totals balanced; bad allocations/fee metadata; missing/mismatched outbox; orphan reserves/tasks; targeted mutation rollback; durable cross-instance/startup holds; private reads/original replay during a hold; restricted-role resolution denial; failed-resolution rollback; successful reviewed acknowledgement plus restart; traversal beyond 128 records; and import of historical fixture completions/cancellations. No actual fund transfer, incident on deployed user data or consensus change was performed. + +Final validation: TypeScript/Vite Docker build passed; **168 offline regression tests and 16 real-PostgreSQL integration tests passed**, zero failures/skips. The first corruption-test run exposed an unguarded chained amount refinement on malformed stored decimals; the parser was corrected to emit a redacted integrity incident and all suites were rerun. Deployed sandbox services were not recreated or subjected to deliberate corruption; new code takes effect only after the selected runtime image is rebuilt/recreated. + +SQLite remains a disposable compatibility model with its original journal audit; this richer runtime checking is implemented in the PostgreSQL backend, including imported SQLite histories. Source-freeze/import procedures are unchanged. Production live-deposit attribution, treasury asset reconciliation, normalized financial states, compensation, observer/signing services, independent review and measured load/SLOs remain open. A successful internal audit is **not** proof of external XMR reserves, a guarantee of solvency, or launch permission. diff --git a/docs/postgres-financial-storage.md b/docs/postgres-financial-storage.md index a89880a..9043864 100755 --- a/docs/postgres-financial-storage.md +++ b/docs/postgres-financial-storage.md @@ -20,6 +20,8 @@ This is an integrated backend, not a second disconnected balance demo: [custody - Pool maximum four connections per service; per-instance active admission sixteen; finite connect/statement/lock/idle transaction timeouts; synchronous commit explicitly on. Scale the total pool budget across instances deliberately. No wallet/network side effect is performed in a database transaction. - No automatic retry after an uncertain commit. A client refreshes or resends the original signed envelope, letting durable receipts establish whether it committed. PostgreSQL failure never causes SQLite fallback. Startup audits compare every account projection with exact journal sums and check backing totals; these are internal accounting checks, not proof that real Monero assets exist. +**Reconciliation hardening:** [financial business audits](financial-reconciliation.md) additionally check order/withdrawal encodings, exact allocations and held amounts, lifecycle journal lines, task payloads and orphan/duplicate records. They run at startup/import, on a nonoverlapping periodic schedule and on affected mutation boundaries. Persistent integrity incidents block writers and survive restart; only schema-owner acknowledgement with reviewed evidence and successful full audit can resolve them. External database/process loss still requires operational writer fencing. These checks do not prove real Monero backing. + These controls do not make a compromised application harmless: it still has posting privileges, and an administrative database owner can alter schema/history. Restricted roles prevent accidental/direct mutation, not fraudulent balanced postings by a fully compromised service. Independent monitoring, statement review, external backups and authorization/treasury isolation remain required. ## Configuration and operator workflow diff --git a/docs/production-plan.md b/docs/production-plan.md index 98286f7..ad87708 100755 --- a/docs/production-plan.md +++ b/docs/production-plan.md @@ -67,6 +67,8 @@ Can run alongside P0/P1. Addresses SEC-01/SEC-06 rather than adding more fixed c **Next financial-storage increment:** [ADR 003 — PostgreSQL](postgres-financial-storage.md) selects and integrates the custody SQL backend with existing HTTP callers, exact constrained accounting, explicit migration/runtime roles, source-freezing SQLite import and dump/restore rehearsal. The adapter remains on the fictitious sandbox domain; it is not a production deposit/withdrawal worker. SQLite node-only backup deliberately refuses PostgreSQL-bound financial state. Production hardware/SLO tests, coordinated backup/PITR/failover, real payment schema/worker controls and independent review remain open, so this does not close M2/M4 or authorize funds. +**Financial reconciliation increment:** [business-liability checks](financial-reconciliation.md) now tie each order/withdrawal to its exact reservation, journal lifecycle and pending work, including reverse orphan/duplicate checks. PostgreSQL startup/import/explicit/periodic audits and targeted mutations use these checks; redacted durable incident holds fence new postings, while private reads and original receipt replay remain available. Private operations reports audit freshness/holds, and schema-owner incident acknowledgement requires a successful locked full audit plus evidence reference. No automatic repair, real payment, schema reset or consensus activation is enabled. Production capacity, coordinated recovery and external review remain open. + **Deployment increment:** [persistent local custody stack](persistent-custody-sandbox.md) now starts PostgreSQL and the actual app together, with private TLS/SCRAM networking, separate owner/runtime roles, explicit migrations and scoped persistent credential/data volumes. A signed fictitious credit and custodian identity survived container/network removal and recreation; the server rejected plaintext authentication. Existing deployments and SQLite custody were not switched. Production certificate rotation, coordinated backup/restore and external operating review remain open; all XMR remains simulated. - Indexed canonical ownership, NFT/transfer lookup and paged wallet/explorer APIs: browser signing must not require loading all history. Tip-pinned queries detect reorgs and preserve complete ownership semantics. diff --git a/index.html b/index.html index 1cd184d..fb7bb16 100644 --- a/index.html +++ b/index.html @@ -6,7 +6,7 @@ Wikideck — Mine knowledge. - +
diff --git a/public/original-logo.svg b/public/original-logo.svg new file mode 100755 index 0000000..9ac970f --- /dev/null +++ b/public/original-logo.svg @@ -0,0 +1 @@ + diff --git a/scripts/custody-db.ts b/scripts/custody-db.ts index b73612d..d179348 100755 --- a/scripts/custody-db.ts +++ b/scripts/custody-db.ts @@ -6,11 +6,12 @@ import { migrateCustody, grantCustodyRuntime } from '../server/market/postgres-s import { importSqliteCustody } from '../server/market/custody-import.ts'; import { PostgresCustody } from '../server/market/postgres-custody.ts'; import { IS_MAINNET } from '../shared/network.ts'; +import { resolveCustodyIncident } from '../server/market/custody-incident.ts'; // Explicit operator task, never invoked by node startup. No passwords or database URLs printed. if (IS_MAINNET) throw new Error('Only sandbox financial schema operations are implemented'); const [action, argument] = process.argv.slice(2); -if (!['migrate', 'grant-runtime', 'import-sqlite', 'audit'].includes(action)) throw new Error('Expected migrate, grant-runtime ROLE, import-sqlite DATA_DIR, or audit'); +if (!['migrate', 'grant-runtime', 'import-sqlite', 'audit', 'resolve-integrity'].includes(action)) throw new Error('Expected migrate, grant-runtime ROLE, import-sqlite DATA_DIR, audit, or resolve-integrity EVIDENCE_UUID'); try { const config = await postgresConfig(); if (action === 'audit') { @@ -21,6 +22,7 @@ try { try { if (action === 'migrate') { await migrateCustody(pool); console.log('Custody PostgreSQL schema verified/migrated. No funds activated.'); } if (action === 'grant-runtime') { await grantCustodyRuntime(pool, argument ?? ''); console.log('Runtime grants applied; no credentials printed.'); } + if (action === 'resolve-integrity') console.log(JSON.stringify(await resolveCustodyIncident(pool, argument ?? ''))); if (action === 'import-sqlite') { if (!argument) throw new Error('Stopped source node DATA_DIR required'); const directory = resolve(argument); diff --git a/server/market/custody-incident.ts b/server/market/custody-incident.ts new file mode 100755 index 0000000..fc5dbe3 --- /dev/null +++ b/server/market/custody-incident.ts @@ -0,0 +1,29 @@ +import { randomUUID, createHash } from 'node:crypto'; +import type { Pool } from 'pg'; +import { z } from 'zod'; +import { CUSTODY_WRITE_LOCK } from './postgres-schema.ts'; +import { auditPostgres } from './postgres-custody.ts'; + +/** Operator-only acknowledgement AFTER externally reviewed repair/restore; never repairs or resets a journal. */ +export async function resolveCustodyIncident(pool: Pool, evidenceId: string): Promise<{ resolutionId: string }> { + z.string().uuid().parse(evidenceId); + const c = await pool.connect(); + try { + await c.query('BEGIN'); + await c.query("SET LOCAL statement_timeout='10s'; SET LOCAL lock_timeout='3s'; SET LOCAL idle_in_transaction_session_timeout='10s'; SET LOCAL synchronous_commit=on"); + await c.query('SELECT pg_advisory_xact_lock($1::bigint)', [CUSTODY_WRITE_LOCK]); + const allowed = await c.query("SELECT pg_has_role(current_user,nspowner,'MEMBER') AS owns FROM pg_namespace WHERE nspname='wk_custody'"); + if (allowed.rows[0]?.owns !== true) throw new Error('Schema-owner approval required for integrity resolution'); + const incident = await c.query("SELECT value FROM wk_custody.custody_meta WHERE key='integrity_incident_v1' FOR UPDATE"); + if (!incident.rowCount) throw new Error('No unresolved custody integrity incident'); + // Delete is UNCOMMITTED and exclusively locked. Any audit failure rolls it back, leaving the hold intact. + await c.query("DELETE FROM wk_custody.custody_meta WHERE key='integrity_incident_v1'"); + await auditPostgres(c); + const resolutionId = randomUUID(); + await c.query('INSERT INTO wk_custody.custody_meta VALUES($1,$2)', [`integrity_resolution:${resolutionId}`, + JSON.stringify({ version: 1, evidenceId, resolvedAt: Date.now(), + incidentDigest: createHash('sha256').update(incident.rows[0].value).digest('hex') })]); + await c.query('COMMIT'); return { resolutionId }; + } catch (error) { await c.query('ROLLBACK').catch(() => {}); throw error; } + finally { c.release(); } +} diff --git a/server/market/custody-integrity.ts b/server/market/custody-integrity.ts new file mode 100755 index 0000000..a2a2c57 --- /dev/null +++ b/server/market/custody-integrity.ts @@ -0,0 +1,199 @@ +import { z } from 'zod'; +import type { PoolClient, QueryResult } from 'pg'; +import { ADDITIONAL_MARKET_FEE } from '../../shared/market.ts'; +import { parseAtomicXmr } from '../../shared/xmr-amount.ts'; +import { allocateXmrSale } from '../../shared/xmr-pricing.ts'; +import { validateXmrAddress } from './xmr.ts'; + +/** Codes only: never attach customer IDs, payloads, SQL errors or addresses to operational diagnostics. */ +export class CustodyIntegrityError extends Error { + readonly code: string; + constructor(code: string) { super(`Custody integrity check failed: ${code}`); this.code = code; } +} +function fail(code: string): never { throw new CustodyIntegrityError(code); } +const uuid = z.string().uuid(), hex = z.string().regex(/^[a-f0-9]{64}$/); +const integer = z.number().int().nonnegative().safe(); +const atomic = z.string().max(22).refine(s => { try { parseAtomicXmr(s); return true; } catch { return false; } }); +const positive = atomic.refine(s => { try { return parseAtomicXmr(s) > 0n; } catch { return false; } }); +const wallet = z.string().regex(/^[1-9A-HJ-NP-Za-km-z]{95}$/); +const listingSchema = z.object({ id: uuid, seller: hex, cardHash: hex, sequence: integer, + title: z.string().min(1).max(200), priceAtomicXmr: positive, feeBps: z.number().int().min(0).max(5000), + hostFeeAddress: wallet.optional(), status: z.enum(['open', 'reserved', 'cancelled', 'sold']), createdAt: integer }).strict(); +const orderSchema = z.object({ id: uuid, listingId: uuid, seller: hex, buyer: hex, cardHash: hex, sequence: integer, + grossAtomicXmr: positive, sellerAtomicXmr: atomic, authorAtomicXmr: atomic, hostAtomicXmr: atomic, + authorAddress: wallet, hostFeeAddress: wallet.optional(), status: z.enum(['awaiting-lock', 'cancelled', 'settled']), createdAt: integer }).strict(); +const withdrawalSchema = z.object({ id: uuid, actor: hex, destination: z.literal('mock-external'), amountAtomicXmr: positive, + maxFeeAtomicXmr: atomic, status: z.enum(['reserved', 'prepared', 'complete', 'cancelled']), + attemptId: uuid.optional(), transactionId: z.string().regex(/^fixture:[a-f0-9]{64}$/).optional(), + feeAtomicXmr: atomic.optional(), createdAt: integer }).strict(); +type Entry = { account: string; debit: string; credit: string }; +type Journal = { id: string; kind: string; lines: Entry[] }; +function parse(schema: z.ZodType, data: unknown, code: string): T { + const result = schema.safeParse(data); return result.success ? result.data : fail(code); +} +function same(a: unknown, b: unknown): boolean { + // Compare parsed records without depending on JSONB object key ordering. + if (a === b) return true; + if (!a || !b || typeof a !== 'object' || typeof b !== 'object') return false; + const x = a as Record, y = b as Record; + const keys = Object.keys(x).filter(k => x[k] !== undefined).sort(); + return keys.join('\0') === Object.keys(y).filter(k => y[k] !== undefined).sort().join('\0') && keys.every(k => same(x[k], y[k])); +} +function json(text: string | null): unknown { try { return JSON.parse(text ?? ''); } catch { return fail('outbox-encoding'); } } +const d = (account: string, amount: string): Entry => ({ account, debit: amount, credit: '0' }); +const c = (account: string, amount: string): Entry => ({ account, debit: '0', credit: amount }); +async function journals(client: PoolClient, ids: string[]): Promise> { + const result = await client.query(`SELECT j.id,j.kind,jsonb_agg(jsonb_build_object('account',e.account, + 'debit',e.debit::text,'credit',e.credit::text) ORDER BY e.position) FILTER(WHERE e.journal IS NOT NULL) AS lines + FROM wk_custody.journals j LEFT JOIN wk_custody.entries e ON e.journal=j.id + WHERE j.id=ANY($1::text[]) GROUP BY j.id`, [ids]); + return new Map(result.rows.map(r => [r.id, r as Journal])); +} +function expectJournal(all: Map, id: string, kind: string, expected: Entry[] | null) { + const got = all.get(id); + if (expected === null) { if (got) fail('unexpected-terminal-posting'); return; } + if (!got || got.kind !== kind || !Array.isArray(got.lines)) fail('missing-lifecycle-posting'); + const normalized = (lines: Entry[]) => lines.filter(l => BigInt(l.debit) !== 0n || BigInt(l.credit) !== 0n) + .map(l => JSON.stringify([l.account, l.debit, l.credit])).sort(); + if (!same(normalized(got.lines), normalized(expected))) fail('lifecycle-posting-mismatch'); +} +async function validateWallets(values: (string | undefined)[]) { + // Bounded per-batch cache, not a growing process-wide map of customer data. + for (const address of new Set(values.filter((v): v is string => !!v))) { + try { await validateXmrAddress(address); } catch { fail('fee-recipient-invalid'); } + } +} + +type OrderRow = { id: string; buyer: string; seller: string; status: string; data: unknown; + listing_data: unknown; listing_status: string; listing_card: string; balance: string | null; + normal: string | null; box_kind: string | null; box_status: string | null; box_data: string | null }; +async function checkOrders(client: PoolClient, rows: OrderRow[]) { + const posting = await journals(client, rows.flatMap(r => [`reserve:${r.id}`, `cancel:${r.id}`, `settle:${r.id}`])); + const wallets: string[] = []; + for (const row of rows) { + const o = parse(orderSchema, row.data, 'order-record-invalid'), l = parse(listingSchema, row.listing_data, 'order-listing-invalid'); + if (o.id !== row.id || o.buyer !== row.buyer || o.seller !== row.seller || o.status !== row.status || + l.status !== row.listing_status || l.cardHash !== row.listing_card || o.listingId !== l.id || o.seller !== l.seller || + o.cardHash !== l.cardHash || o.sequence !== l.sequence || o.grossAtomicXmr !== l.priceAtomicXmr || o.seller === o.buyer || + (o.hostFeeAddress ?? ADDITIONAL_MARKET_FEE.address) !== (l.hostFeeAddress ?? ADDITIONAL_MARKET_FEE.address)) fail('order-terms-mismatch'); + const split = allocateXmrSale(BigInt(l.priceAtomicXmr), l.feeBps); + if (o.sellerAtomicXmr !== split.sellerBeforeNetworkCosts.toString() || o.hostAtomicXmr !== split.hosting.toString() || + o.authorAtomicXmr !== split.author.toString() || o.authorAddress !== ADDITIONAL_MARKET_FEE.address) fail('order-allocation-mismatch'); + wallets.push(o.authorAddress, o.hostFeeAddress ?? ADDITIONAL_MARKET_FEE.address); + const account = `customer:${o.buyer}:purchase:${o.id}`, available = `customer:${o.buyer}:available`; + if (row.normal !== 'credit' || row.balance !== (o.status === 'awaiting-lock' ? o.grossAtomicXmr : '0')) fail('order-reservation-mismatch'); + const expectedStatus = o.status === 'awaiting-lock' ? 'reserved' : o.status === 'settled' ? 'sold' : 'cancelled'; + if (l.status !== expectedStatus || row.box_kind !== 'nft-lock') fail('order-state-mismatch'); + if (o.status === 'settled') { + if (row.box_status !== 'fixture-complete' || !/^fixture:[a-f0-9]{64}$/.test(row.box_data ?? '')) fail('settlement-evidence-mismatch'); + } else { + const original = parse(orderSchema, json(row.box_data), 'order-outbox-invalid'); + if (row.box_status !== (o.status === 'cancelled' ? 'cancelled' : 'blocked') || + !same(original, { ...o, status: 'awaiting-lock' })) fail('order-outbox-mismatch'); + } + expectJournal(posting, `reserve:${o.id}`, 'purchase-reservation', [d(available, o.grossAtomicXmr), c(account, o.grossAtomicXmr)]); + expectJournal(posting, `cancel:${o.id}`, 'purchase-cancellation', o.status === 'cancelled' ? [d(account, o.grossAtomicXmr), c(available, o.grossAtomicXmr)] : null); + expectJournal(posting, `settle:${o.id}`, 'mock-sale-settlement', o.status === 'settled' ? [d(account, o.grossAtomicXmr), + c(`customer:${o.seller}:available`, o.sellerAtomicXmr), c(`author:${o.authorAddress}`, o.authorAtomicXmr), + c(o.hostFeeAddress ? `revenue:host:${o.hostFeeAddress}` : 'revenue:host', o.hostAtomicXmr)] : null); + } + await validateWallets(wallets); +} +const orderSelect = `SELECT o.id::text,o.buyer,o.seller,o.status,o.data,l.data AS listing_data,l.status AS listing_status, + l.card AS listing_card,a.balance::text,a.normal,b.kind AS box_kind,b.status AS box_status,b.data AS box_data + FROM wk_custody.orders o LEFT JOIN wk_custody.listings l ON l.id::text=o.data->>'listingId' + LEFT JOIN wk_custody.accounts a ON a.id='customer:'||o.buyer||':purchase:'||o.id::text + LEFT JOIN wk_custody.outbox b ON b.id=o.id`; + +type WithdrawalRow = { id: string; actor: string; status: string; data: unknown; balance: string | null; + normal: string | null; box_kind: string | null; box_status: string | null; box_data: string | null }; +async function checkWithdrawals(client: PoolClient, rows: WithdrawalRow[]) { + const posting = await journals(client, rows.flatMap(r => [`withdraw:${r.id}`, `withdraw-cancel:${r.id}`, `withdraw-confirm:${r.id}`])); + for (const row of rows) { + const w = parse(withdrawalSchema, row.data, 'withdrawal-record-invalid'); + if (w.id !== row.id || w.actor !== row.actor || w.status !== row.status) fail('withdrawal-identity-mismatch'); + const total = (BigInt(w.amountAtomicXmr) + BigInt(w.maxFeeAtomicXmr)).toString(); + const active = w.status === 'reserved' || w.status === 'prepared'; + if (row.normal !== 'credit' || row.balance !== (active ? total : '0')) fail('withdrawal-reservation-mismatch'); + const prepared = w.status === 'prepared' || w.status === 'complete'; + if (prepared ? !w.attemptId || !w.transactionId || w.feeAtomicXmr === undefined || BigInt(w.feeAtomicXmr) > BigInt(w.maxFeeAtomicXmr) + : w.attemptId !== undefined || w.transactionId !== undefined || w.feeAtomicXmr !== undefined) fail('withdrawal-attempt-invalid'); + const status = w.status === 'reserved' ? 'blocked' : w.status === 'cancelled' ? 'cancelled' + : w.status === 'prepared' ? 'fixture-prepared' : 'fixture-complete'; + const box = parse(withdrawalSchema, json(row.box_data), 'withdrawal-outbox-invalid'); + if (row.box_kind !== 'withdrawal' || row.box_status !== status || !same(box, { ...w, status: prepared ? 'prepared' : 'reserved' })) fail('withdrawal-outbox-mismatch'); + const account = `customer:${w.actor}:withdrawal:${w.id}`, available = `customer:${w.actor}:available`; + expectJournal(posting, `withdraw:${w.id}`, 'withdrawal-reservation', [d(available, total), c(account, total)]); + expectJournal(posting, `withdraw-cancel:${w.id}`, 'withdrawal-cancellation', w.status === 'cancelled' ? [d(account, total), c(available, total)] : null); + expectJournal(posting, `withdraw-confirm:${w.id}`, 'mock-withdrawal', w.status === 'complete' ? [d(account, total), + c('treasury:mock', (BigInt(w.amountAtomicXmr) + BigInt(w.feeAtomicXmr!)).toString()), + c(available, (BigInt(w.maxFeeAtomicXmr) - BigInt(w.feeAtomicXmr!)).toString())] : null); + } +} +const withdrawalSelect = `SELECT w.id::text,w.actor,w.status,w.data,a.balance::text,a.normal, + b.kind AS box_kind,b.status AS box_status,b.data AS box_data FROM wk_custody.withdrawals w + LEFT JOIN wk_custody.accounts a ON a.id='customer:'||w.actor||':withdrawal:'||w.id::text + LEFT JOIN wk_custody.outbox b ON b.id=w.id`; + +/** Targeted mutation checks share the same transaction as postings/receipts. No external side effects. */ +export async function auditOrder(client: PoolClient, id: string) { + const result = await client.query(`${orderSelect} WHERE o.id=$1`, [id]); + if (result.rowCount !== 1) fail('order-record-missing'); await checkOrders(client, result.rows); +} +export async function auditWithdrawal(client: PoolClient, id: string) { + const result = await client.query(`${withdrawalSelect} WHERE w.id=$1`, [id]); + if (result.rowCount !== 1) fail('withdrawal-record-missing'); await checkWithdrawals(client, result.rows); +} +/** Caller supplies a coherent transaction. Keyset pages bound application memory, not total audit duration. */ +export async function auditBusinessLiabilities(client: PoolClient): Promise { + const deadline = performance.now() + 15000; + const budget = () => { if (performance.now() > deadline) throw new Error('Custody business audit budget exceeded'); }; + let after: string | null = null; + for (;;) { + budget(); + const result: QueryResult = await client.query(`${orderSelect} WHERE ($1::uuid IS NULL OR o.id>$1::uuid) ORDER BY o.id LIMIT 128`, [after]); + await checkOrders(client, result.rows); if (result.rows.length < 128) break; after = result.rows.at(-1)!.id; + } + after = null; + for (;;) { + budget(); + const result: QueryResult = await client.query(`${withdrawalSelect} WHERE ($1::uuid IS NULL OR w.id>$1::uuid) ORDER BY w.id LIMIT 128`, [after]); + await checkWithdrawals(client, result.rows); if (result.rows.length < 128) break; after = result.rows.at(-1)!.id; + } + after = null; + for (;;) { + budget(); + const result: QueryResult<{ id: string; card: string; status: string; data: unknown }> = await client.query('SELECT id::text,card,status,data FROM wk_custody.listings WHERE ($1::uuid IS NULL OR id>$1::uuid) ORDER BY id LIMIT 128', [after]); + const wallets: string[] = []; + for (const row of result.rows) { + const l = parse(listingSchema, row.data, 'listing-record-invalid'); + if (row.id !== l.id || row.card !== l.cardHash || row.status !== l.status) fail('listing-identity-mismatch'); + wallets.push(l.hostFeeAddress ?? ADDITIONAL_MARKET_FEE.address); + } + await validateWallets(wallets); + if (result.rows.length < 128) break; after = result.rows.at(-1)!.id; + } + // Check both directions: a correct order cannot excuse orphan reserves, tasks or terminal postings. + const checks: [string, string][] = [ + ['orphan-purchase-reserve', `SELECT 1 FROM wk_custody.accounts a WHERE a.id LIKE 'customer:%:purchase:%' + AND NOT EXISTS(SELECT 1 FROM wk_custody.orders o WHERE a.id='customer:'||o.buyer||':purchase:'||o.id::text) LIMIT 1`], + ['orphan-withdrawal-reserve', `SELECT 1 FROM wk_custody.accounts a WHERE a.id LIKE 'customer:%:withdrawal:%' + AND NOT EXISTS(SELECT 1 FROM wk_custody.withdrawals w WHERE a.id='customer:'||w.actor||':withdrawal:'||w.id::text) LIMIT 1`], + ['orphan-outbox', `SELECT 1 FROM wk_custody.outbox b WHERE NOT EXISTS(SELECT 1 FROM wk_custody.orders o WHERE o.id=b.id AND b.kind='nft-lock') + AND NOT EXISTS(SELECT 1 FROM wk_custody.withdrawals w WHERE w.id=b.id AND b.kind='withdrawal') LIMIT 1`], + ['listing-order-mismatch', `SELECT 1 FROM wk_custody.listings l LEFT JOIN wk_custody.orders o ON o.data->>'listingId'=l.id::text + WHERE (l.status IN ('reserved','sold') AND o.id IS NULL) OR l.id::text IS DISTINCT FROM l.data->>'id' + OR l.status IS DISTINCT FROM l.data->>'status' OR l.card IS DISTINCT FROM l.data->>'cardHash' LIMIT 1`], + ['duplicate-listing-orders', `SELECT 1 FROM wk_custody.orders GROUP BY data->>'listingId' HAVING count(*)<>1 LIMIT 1`], + ['orphan-lifecycle-posting', `SELECT 1 FROM wk_custody.journals j WHERE + (j.kind IN ('purchase-reservation','purchase-cancellation','mock-sale-settlement') AND NOT EXISTS + (SELECT 1 FROM wk_custody.orders o WHERE j.id IN ('reserve:'||o.id::text,'cancel:'||o.id::text,'settle:'||o.id::text))) OR + (j.kind IN ('withdrawal-reservation','withdrawal-cancellation','mock-withdrawal') AND NOT EXISTS + (SELECT 1 FROM wk_custody.withdrawals w WHERE j.id IN ('withdraw:'||w.id::text,'withdraw-cancel:'||w.id::text,'withdraw-confirm:'||w.id::text))) LIMIT 1`], + ['duplicate-prepared-attempt', `SELECT 1 FROM wk_custody.withdrawals WHERE status IN ('prepared','complete') + GROUP BY data->>'attemptId' HAVING count(*)>1 LIMIT 1`], + ['duplicate-prepared-transaction', `SELECT 1 FROM wk_custody.withdrawals WHERE status IN ('prepared','complete') + GROUP BY data->>'transactionId' HAVING count(*)>1 LIMIT 1`], + ]; + for (const [code, sql] of checks) { budget(); if ((await client.query(sql)).rowCount) fail(code); } +} diff --git a/server/market/postgres-custody.ts b/server/market/postgres-custody.ts index 17f374a..0269faa 100755 --- a/server/market/postgres-custody.ts +++ b/server/market/postgres-custody.ts @@ -11,6 +11,7 @@ import { CUSTODY_POLICY, MOCK_CREDIT_ATOMIC, custodyCommandSchema, custodyMessag import { CustodyError } from './custody.ts'; import { CUSTODY_WRITE_LOCK, verifyCustodySchema } from './postgres-schema.ts'; import { resolvedMarketPolicy, validateMarketPolicy, permitsMarketAction, type MarketPolicy } from './policy.ts'; +import { CustodyIntegrityError, auditBusinessLiabilities, auditOrder, auditWithdrawal } from './custody-integrity.ts'; export type CustodyOptions = { feeBps: number; card(hash: string, owner: string, sequence: number): { title: string } | undefined; now?: () => number } & MarketPolicy; type Result = CustodyActivity | CustodyListing | CustodyOrder | CustodyWithdrawal; @@ -27,7 +28,18 @@ export class PostgresCustody { private readonly pool: Pool; private readonly options: CustodyOptions; private active = 0; - private constructor(pool: Pool, options: CustodyOptions, id: string) { this.pool = pool; this.options = options; this.id = id; } + private readonly auditTimer: ReturnType; + private auditInFlight: Promise | undefined; + private integrityHold = false; + private auditState: { status: 'ok' | 'failed'; lastAttemptAt: number; lastSuccessAt: number; failureCode: string | null }; + private constructor(pool: Pool, options: CustodyOptions, id: string) { + this.pool = pool; this.options = options; this.id = id; + this.auditState = { status: 'ok', lastAttemptAt: Date.now(), lastSuccessAt: Date.now(), failureCode: null }; + this.auditTimer = setInterval(() => { void this.audit().catch(() => { + console.error('Custody financial audit unavailable or inconsistent; mutations held pending operator review.'); + }); }, 60000); + this.auditTimer.unref(); + } static async open(config: PoolConfig, options: CustodyOptions): Promise { if (IS_MAINNET) throw new Error('Custody sandbox cannot run on mainnet'); await validateMarketPolicy(options); @@ -61,12 +73,23 @@ export class PostgresCustody { } await auditPostgres(c); await c.query('COMMIT'); - } catch (e) { await c.query('ROLLBACK').catch(() => {}); throw e; } + } catch (e) { + await c.query('ROLLBACK').catch(() => {}); + if (e instanceof CustodyIntegrityError) await recordIntegrityIncident(c, e.code).catch(() => { + console.error('Custody integrity incident could not be persisted; fence all custody writers and inspect database availability.'); + }); + throw e; + } finally { c.release(); } return new PostgresCustody(pool, options, id); } catch (e) { await pool.end(); throw e; } } - close(): Promise { return this.pool.end(); } + async close(): Promise { clearInterval(this.auditTimer); await this.auditInFlight?.catch(() => {}); await this.pool.end(); } + diagnostics() { + const stale = Date.now() - this.auditState.lastSuccessAt > 120000; + return { backend: 'postgres', mutationHold: this.integrityHold || this.auditState.status !== 'ok' || stale, + integrityHold: this.integrityHold, audit: { ...this.auditState, stale } }; + } info(): CustodyInfo { return { enabled: true, mode: 'sandbox', policy: CUSTODY_POLICY, network: NETWORK, chainId: CHAIN_ID, custodianId: this.id, feeBps: this.options.feeBps, ...resolvedMarketPolicy(this.options), liveDeposits: false, liveWithdrawals: false, nftSettlement: false }; @@ -75,17 +98,40 @@ export class PostgresCustody { check(this.active < 16, 'Custody database busy; retry the original command shortly'); this.active++; let c: PoolClient | undefined; + let mutationSavepoint = false; try { c = await this.pool.connect(); await c.query(write ? 'BEGIN' : 'BEGIN ISOLATION LEVEL REPEATABLE READ READ ONLY'); await c.query("SET LOCAL statement_timeout='10s'; SET LOCAL lock_timeout='3s'; SET LOCAL idle_in_transaction_session_timeout='10s'; SET LOCAL synchronous_commit=on"); // One writer per financial domain initially; other readers remain nonblocking. Lock is database-scoped, // shared by every node/process, migrations and import, not an in-process promise queue. - if (write) await c.query('SELECT pg_advisory_xact_lock($1::bigint)', [CUSTODY_WRITE_LOCK]); + if (write) { + await c.query('SELECT pg_advisory_xact_lock($1::bigint)', [CUSTODY_WRITE_LOCK]); + await c.query('SAVEPOINT custody_mutation'); mutationSavepoint = true; + } const result = await fn(c); await c.query('COMMIT'); return result; } catch (e) { + if (e instanceof CustodyIntegrityError) { + this.integrityHold = true; + this.auditState = { ...this.auditState, status: 'failed', lastAttemptAt: Date.now(), failureCode: e.code }; + // Roll back business changes but retain the cross-process writer lock until the incident is committed. + // Read-only/background audit failures require a separate marker transaction; DB/process loss can still + // prevent persistence, so external writer fencing remains an incident-response requirement. + if (c) { + try { + if (mutationSavepoint) await c.query('ROLLBACK TO SAVEPOINT custody_mutation'); + else await c.query('ROLLBACK'); + await recordIntegrityIncident(c, e.code); + if (mutationSavepoint) await c.query('COMMIT'); + } catch { + await c.query('ROLLBACK').catch(() => {}); + console.error('Custody integrity incident could not be persisted; fence all custody writers and inspect database availability.'); + } + } + throw e; + } await c?.query('ROLLBACK').catch(() => {}); // Never retry after an ambiguous COMMIT. Signed receipts resolve the SAME nonce on a client retry. if (e instanceof CustodyError) throw e; @@ -147,13 +193,21 @@ export class PostgresCustody { const key = `${cmd.actor}:${cmd.nonce}`, hash = createHash('sha256').update(message).digest('hex'); const old = await c.query('SELECT digest,data FROM wk_custody.receipts WHERE id=$1', [key]); if (old.rowCount) { check(old.rows[0].digest === hash, 'Nonce already used for different custody terms'); return old.rows[0].data as Result; } + const incident = await c.query("SELECT 1 FROM wk_custody.custody_meta WHERE key='integrity_incident_v1'"); + check(!incident.rowCount, 'Custody integrity incident is unresolved; operator review required'); + check(!this.diagnostics().mutationHold, 'Custody financial reconciliation requires operator review; balances remain readable and original receipts can be replayed'); check(permitsMarketAction(this.options, a.type), 'New market trading is disabled by this host; existing balances and resolution remain available'); // Sandbox admission budget matches SQLite. Receipt/history are NEVER pruned; cancellation stays available. if (!['cancel-listing', 'cancel-reservation', 'cancel-withdrawal'].includes(a.type)) { const count = await c.query('SELECT count(*)::integer AS n FROM (SELECT 1 FROM wk_custody.receipts LIMIT 10000) b'); check(count.rows[0].n < 10000, 'Custody sandbox command budget reached'); } + // Verify existing liabilities BEFORE resolving them, and validate the new business state before COMMIT. + if (a.type === 'cancel-reservation') await auditOrder(c, a.orderId); + if (a.type === 'cancel-withdrawal') await auditWithdrawal(c, a.withdrawalId); const result = await this.mutate(c, cmd); + if (a.type === 'reserve' || a.type === 'cancel-reservation') await auditOrder(c, (result as CustodyOrder).id); + if (a.type === 'withdraw' || a.type === 'cancel-withdrawal') await auditWithdrawal(c, (result as CustodyWithdrawal).id); await c.query('INSERT INTO wk_custody.receipts VALUES($1,$2,$3)', [key, hash, JSON.stringify(result)]); return result; }); @@ -226,7 +280,19 @@ export class PostgresCustody { } throw new CustodyError('Unsupported custody action'); } - audit(): Promise { return this.transaction(false, auditPostgres); } + audit(): Promise { + if (this.auditInFlight) return this.auditInFlight; + this.auditState.lastAttemptAt = Date.now(); + const pending = this.transaction(false, auditPostgres).then(result => { + this.auditState = { ...this.auditState, lastSuccessAt: Date.now(), + status: this.integrityHold ? 'failed' : 'ok', failureCode: this.integrityHold ? this.auditState.failureCode : null }; + return result; + }).catch(error => { + this.auditState = { ...this.auditState, status: 'failed', failureCode: error instanceof CustodyIntegrityError ? error.code : 'audit-unavailable' }; + throw error; + }).finally(() => { this.auditInFlight = undefined; }); + this.auditInFlight = pending; return pending; + } importSource(): Promise { return this.transaction(false, async c => (await c.query("SELECT value FROM wk_custody.custody_meta WHERE key='import_source'")).rows[0]?.value); } @@ -234,16 +300,24 @@ export class PostgresCustody { /** Exact NUMERIC/text aggregation in a coherent transaction; no Number conversion of money. */ export async function auditPostgres(c: PoolClient): Promise { + const incident = await c.query("SELECT 1 FROM wk_custody.custody_meta WHERE key='integrity_incident_v1'"); + if (incident.rowCount) throw new CustodyIntegrityError('unresolved-integrity-incident'); const bad = await c.query(`SELECT 1 FROM wk_custody.journals j LEFT JOIN wk_custody.entries e ON e.journal=j.id GROUP BY j.id HAVING count(e.position)=0 OR sum(e.debit-e.credit)<>0 LIMIT 1`); - if (bad.rowCount) throw new Error('Unbalanced PostgreSQL custody history'); + if (bad.rowCount) throw new CustodyIntegrityError('unbalanced-journal'); const mismatch = await c.query(`SELECT 1 FROM wk_custody.accounts a LEFT JOIN wk_custody.entries e ON e.account=a.id GROUP BY a.id HAVING a.balance<>COALESCE(sum(CASE WHEN a.normal='debit' THEN e.debit-e.credit ELSE e.credit-e.debit END),0) LIMIT 1`); - if (mismatch.rowCount) throw new Error('PostgreSQL custody balance projection mismatch'); + if (mismatch.rowCount) throw new CustodyIntegrityError('balance-projection-mismatch'); const totals = await c.query(`SELECT COALESCE(sum(balance) FILTER(WHERE normal='debit'),0)::text AS assets, COALESCE(sum(balance) FILTER(WHERE normal='credit'),0)::text AS claims FROM wk_custody.accounts`); - if (totals.rows[0].assets !== totals.rows[0].claims) throw new Error('PostgreSQL custody backing mismatch'); + if (totals.rows[0].assets !== totals.rows[0].claims) throw new CustodyIntegrityError('accounting-backing-mismatch'); + await auditBusinessLiabilities(c); const n = await c.query('SELECT count(*)::text AS n FROM wk_custody.journals'); if (BigInt(n.rows[0].n) > BigInt(Number.MAX_SAFE_INTEGER)) throw new Error('Journal count exceeds supported range'); return { journals: Number(n.rows[0].n), assetsAtomicXmr: totals.rows[0].assets, claimsAndRevenueAtomicXmr: totals.rows[0].claims }; } + +async function recordIntegrityIncident(c: PoolClient, code: string): Promise { + await c.query('INSERT INTO wk_custody.custody_meta(key,value) VALUES($1,$2) ON CONFLICT(key) DO NOTHING', + ['integrity_incident_v1', JSON.stringify({ version: 1, code, detectedAt: Date.now() })]); +} diff --git a/server/operations-check.ts b/server/operations-check.ts index 9a55adc..3df9993 100755 --- a/server/operations-check.ts +++ b/server/operations-check.ts @@ -6,7 +6,10 @@ const schema=z.object({schema:z.literal('wikideck-operations-v1'),observedAt:z.s peers:z.object({configured:nonnegative,connected:nonnegative}), eventLoop:z.object({meanMs:nonnegative}),disk:z.unknown(),diskSampledAt:nonnegative.nullable(), market:z.object({queued:nonnegative,lastReconciledAt:nonnegative.nullable(), - orderStates:z.array(z.object({status:z.string(),count:nonnegative})).max(20)})}); + orderStates:z.array(z.object({status:z.string(),count:nonnegative})).max(20)}), + custody:z.object({backend:z.string(),mutationHold:z.boolean().optional(),integrityHold:z.boolean().optional(), + audit:z.object({status:z.enum(['ok','failed']),lastAttemptAt:nonnegative,lastSuccessAt:nonnegative, + failureCode:z.string().nullable(),stale:z.boolean()}).optional()}).optional()}); export function checkOperations(raw:unknown,now=Date.now()){ const s=schema.parse(raw),warnings:string[]=[]; const observation=Date.parse(s.observedAt); @@ -18,6 +21,11 @@ export function checkOperations(raw:unknown,now=Date.now()){ if(s.eventLoop.meanMs>250)warnings.push('event-loop-delay'); if(!s.market.lastReconciledAt||now-s.market.lastReconciledAt>60000)warnings.push('market-reconciliation-stale'); if(s.market.orderStates.some(r=>(r.status==='review'||r.status==='refund-required')&&r.count>0))warnings.push('market-operator-review'); + if(s.custody?.backend==='postgres') { + const audit=s.custody.audit; + if(s.custody.mutationHold||s.custody.integrityHold||audit?.status==='failed')warnings.push('custody-financial-review'); + if(!audit||audit.stale||now-audit.lastSuccessAt>120000||audit.lastSuccessAt>now+60000)warnings.push('custody-audit-stale'); + } const disk=z.object({availableBytes:nonnegative}).safeParse(s.disk); if(!disk.success||!s.diskSampledAt||now-s.diskSampledAt>90000)warnings.push('disk-sample-unavailable-or-stale'); else if(disk.data.availableBytes<512*1024*1024)warnings.push('disk-low'); diff --git a/server/testnet/http.ts b/server/testnet/http.ts index 455ec10..bda9dbf 100644 --- a/server/testnet/http.ts +++ b/server/testnet/http.ts @@ -225,7 +225,8 @@ const server=app.listen(port,host,()=>{ const operations=startOperations(opsPort,()=>({network:NETWORK,paymentMode:'mock', ledger:ledger.diagnostics(),native:engine.diagnostics(), peers:{configured:peers.status().length,connected:peers.status().filter(p=>p.connected).length}, - rpc:{api:apiBudget.status(),p2p:peerBudget.status()},market:market.diagnostics()}),resolve(dataDir,PROFILE)); + rpc:{api:apiBudget.status(),p2p:peerBudget.status()},market:market.diagnostics(), + custody: custody && 'diagnostics' in custody ? custody.diagnostics() : { backend: custody ? 'sqlite-sandbox' : 'off' }}),resolve(dataDir,PROFILE)); server.maxConnections=256; server.headersTimeout=10000; server.requestTimeout=15000; diff --git a/src/App.tsx b/src/App.tsx index 685be4b..6673125 100644 --- a/src/App.tsx +++ b/src/App.tsx @@ -160,7 +160,7 @@ export default function App() { return