Market-journal reconciliation benchmark

docs/canonical-indexes.md

@@ -46,6 +46,8 @@ Observed in the development Docker environment:
 
 These numbers are a reproducible **baseline**, not acceptance SLOs: host CPU/storage and production concurrency were not selected, outputs include synthetic article payloads, warm caches apply, one transaction amortizes insertion fsync, and real validation/startup/HTTP/market workloads are excluded. They do not establish a supported production envelope.
 
+The synthetic market-journal harness [benchmark](../scripts/benchmark-market-reconciliation.ts), available as `npm run benchmark:market -- 10000`, separately measures bounded reconciliation of terminal order/listing records in its own temporary SQLite directory. On the same development Docker environment, 10,000 orders plus 10,000 listings completed in 157 reconciliation passes in 772 ms (about 25,903 records/s), with pass-latency p50/p95/max of 4.57/9.13/20.04 ms and a 7,180,288-byte SQLite file. This dataset intentionally avoids active-order delivery and listing validation branches; it measures local keyset/SQLite overhead only. It is not a target-hardware result or throughput/SLO claim and excludes concurrent HTTP traffic, event-loop interference from other work, PostgreSQL, chain validation and adversarial workloads.
+
 ## Remaining gates
 
 M2 still needs target-hardware/data/SLO decisions, populated long-journal/large-event benchmarks, bounded worker/event-loop latency under load, production financial database/schema and measured recovery/retention/storage growth. The new scheduler bounds work per pass, but does not yet provide queue isolation, multi-worker coordination, or a demonstrated production throughput envelope. Ledger PoW replay and DAG state memory still scale with retained history; indexed queries do not solve those costs. Startup validation checks undo-record counts but not every possible future undo mutation against a second reconstructed undo model; comprehensive corruption/fuzz coverage and independent review remain required.

docs/production-plan.md

@@ -63,7 +63,7 @@ Acceptance: exact-amount property/boundary tests, versioned signature tests, zer
 
 Can run alongside P0/P1. Addresses SEC-01/SEC-06 rather than adding more fixed caps.
 
-**Implemented M2 increments, 2026-10-20–21:** [canonical indexes and durable market observation](canonical-indexes.md). Canonical ownership/transaction SQL indexes now serve NFT, deck, custody availability and mock-market queries; per-block undo and anchored connect/disconnect events commit with ledger updates. Market checkpoint/reorg holds persist atomically and bounded catch-up blocks new mutations. Order/listing reconciliation now advances in durable, stable-ID keyset batches of at most 128 records per table, resumes after restart, re-fences when the chain tip/revision changes, and runs from an HTTP lifecycle poller rather than relying only on page loads. Final TypeScript/Vite build and **174 offline tests passed**; focused market tests passed 23/23 and the disposable PostgreSQL integration suite passed 31/31. A disposable 10,000-NFT SQL baseline is recorded, **not** a target-hardware SLO or complete M2 acceptance. Queue isolation, target-scale throughput/latency, production financial storage/recovery, retained-DAG capacity and operator dataset/SLO targets remain open.
+**Implemented M2 increments, 2026-10-20–21:** [canonical indexes and durable market observation](canonical-indexes.md). Canonical ownership/transaction SQL indexes now serve NFT, deck, custody availability and mock-market queries; per-block undo and anchored connect/disconnect events commit with ledger updates. Market checkpoint/reorg holds persist atomically and bounded catch-up blocks new mutations. Order/listing reconciliation now advances in durable, stable-ID keyset batches of at most 128 records per table, resumes after restart, re-fences when the chain tip/revision changes, and runs from an HTTP lifecycle poller rather than relying only on page loads. Final TypeScript/Vite build and **174 offline tests passed**; focused market tests passed 23/23 and the disposable PostgreSQL integration suite passed 31/31. A new synthetic local benchmark processed 10,000 terminal orders plus 10,000 listings in 772 ms on the development Docker host; this is a reproducible harness result, **not** a target-hardware SLO or complete M2 acceptance. Queue isolation, target-scale/concurrent throughput and latency, production financial storage/recovery, retained-DAG capacity and operator dataset/SLO targets remain open.
 
 **Next financial-storage increment:** [ADR 003 — PostgreSQL](postgres-financial-storage.md) selects and integrates the custody SQL backend with existing HTTP callers, exact constrained accounting, explicit migration/runtime roles, source-freezing SQLite import and dump/restore rehearsal. The adapter remains on the fictitious sandbox domain; it is not a production deposit/withdrawal worker. SQLite node-only backup deliberately refuses PostgreSQL-bound financial state. Production hardware/SLO tests, coordinated backup/PITR/failover, real payment schema/worker controls and independent review remain open, so this does not close M2/M4 or authorize funds.
 

package.json

@@ -12,6 +12,7 @@
     "analyze:consensus": "node --experimental-strip-types scripts/consensus-analysis.ts",
     "benchmark:randomx": "node --experimental-strip-types scripts/benchmark-randomx.ts",
     "benchmark:index": "node --experimental-strip-types scripts/benchmark-canonical-index.ts",
+    "benchmark:market": "node --experimental-strip-types scripts/benchmark-market-reconciliation.ts",
     "custody:db": "node --experimental-strip-types scripts/custody-db.ts",
     "custody:backup": "node --experimental-strip-types scripts/custody-pair-backup.ts",
     "node:backup": "node --experimental-strip-types scripts/node-backup.ts",

scripts/benchmark-market-reconciliation.ts

@@ -0,0 +1,74 @@
+import { DatabaseSync } from 'node:sqlite';
+import { mkdtemp, rm, stat } from 'node:fs/promises';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { performance } from 'node:perf_hooks';
+import { randomUUID } from 'node:crypto';
+import { Market } from '../server/market/store.ts';
+import { GENESIS_HASH } from '../server/testnet/consensus.ts';
+import { NETWORK } from '../shared/testnet.ts';
+
+// Synthetic terminal-order journal only. It is isolated in a temporary SQLite database and
+// never connects to a Ledger, peer, payment adapter, wallet, or production data directory.
+const text = process.argv[2] ?? '10000';
+if (!/^[1-9]\d{0,5}$/.test(text) || Number(text) > 100000) throw new Error('Fixture size must be 1..100000 orders');
+const count = Number(text);
+const directory = await mkdtemp(join(tmpdir(), 'wikideck-market-reconciliation-'));
+const streamId = randomUUID(), tip = 'a'.repeat(64);
+let market: Market | undefined;
+try {
+  // Indexed-only adapter: the benchmark must not fall back to materializing chain snapshots.
+  const chain = {
+    snapshot: () => { throw new Error('Benchmark must not request a full chain snapshot'); },
+    summary: () => ({ tip, revision: 0, height: 1 }),
+    nft: () => undefined,
+    canonicalTransaction: () => undefined,
+    pendingTransactions: () => [],
+    canonicalHead: () => 0,
+    canonicalEvents: (after: number) => ({ streamId, previousAnchor: GENESIS_HASH, anchor: GENESIS_HASH,
+      events: [], cursor: after, head: 0, caughtUp: true }),
+    submitTransfer: async () => { throw new Error('Benchmark never submits transfers'); },
+  };
+  market = await Market.open(directory, chain, { feeBps: 250, confirmations: 2 });
+  const db = new DatabaseSync(join(directory, NETWORK.includes('mainnet') ? 'mainnet' : 'testnet', 'market.sqlite'));
+  try {
+    const insertListing = db.prepare('INSERT INTO listings(id,card,status,data) VALUES(?,?,?,?)');
+    const insertOrder = db.prepare('INSERT INTO orders(id,data) VALUES(?,?)');
+    const seller = 'synthetic-seller';
+    db.exec('BEGIN IMMEDIATE');
+    try {
+      for (let i = 0; i < count; i++) {
+        const id = i.toString().padStart(12, '0');
+        const listingId = `listing-${id}`, orderId = `order-${id}`, cardHash = i.toString(16).padStart(64, '0');
+        const listing = { id: listingId, cardHash, seller, sequence: 0, kind: 'fixed', status: 'cancelled' };
+        const order = { id: orderId, listingId, seller, buyer: 'synthetic-buyer', cardHash, sequence: 0, status: 'refunded' };
+        insertListing.run(listingId, cardHash, 'cancelled', JSON.stringify(listing));
+        insertOrder.run(orderId, JSON.stringify(order));
+      }
+      db.exec('COMMIT');
+    } catch (error) { db.exec('ROLLBACK'); throw error; }
+  } finally { db.close(); }
+
+  const latencies: number[] = [];
+  const maxPasses = Math.ceil(count / 128) * 2 + 4;
+  const start = performance.now();
+  for (let pass = 0; pass < maxPasses && (pass === 0 || market.hasReconciliationBacklog()); pass++) {
+    const before = performance.now();
+    market.browse();
+    latencies.push(performance.now() - before);
+  }
+  const elapsedMs = performance.now() - start;
+  if (market.hasReconciliationBacklog()) throw new Error(`Reconciliation did not finish within ${maxPasses} bounded passes`);
+  market.close(); market = undefined;
+  const bytes = (await stat(join(directory, NETWORK.includes('mainnet') ? 'mainnet' : 'testnet', 'market.sqlite'))).size;
+  latencies.sort((a, b) => a - b);
+  const percentile = (p: number) => latencies[Math.min(latencies.length - 1, Math.ceil(latencies.length * p) - 1)];
+  console.log(JSON.stringify({ fixture: 'synthetic-terminal-orders-and-listings', orders: count, listings: count,
+    batchSizePerTable: 128, reconciliationPasses: latencies.length, elapsedMs,
+    rowsPerSecond: Math.round((count * 2 * 1000) / elapsedMs), passLatencyP50Ms: percentile(0.50),
+    passLatencyP95Ms: percentile(0.95), passLatencyMaxMs: latencies.at(-1), sqliteBytes: bytes,
+    warning: 'Synthetic terminal records on this development host only; excludes live chain validation, active-order logic, concurrent HTTP load, PostgreSQL, and production SLO certification.' }, null, 2));
+} finally {
+  market?.close();
+  await rm(directory, { recursive: true, force: true });
+}