diff --git a/docs/canonical-indexes.md b/docs/canonical-indexes.md index 48b3360..cf880c7 100755 --- a/docs/canonical-indexes.md +++ b/docs/canonical-indexes.md @@ -46,7 +46,7 @@ Observed in the development Docker environment: These numbers are a reproducible **baseline**, not acceptance SLOs: host CPU/storage and production concurrency were not selected, outputs include synthetic article payloads, warm caches apply, one transaction amortizes insertion fsync, and real validation/startup/HTTP/market workloads are excluded. They do not establish a supported production envelope. -The synthetic market-journal harness [benchmark](../scripts/benchmark-market-reconciliation.ts), available as `npm run benchmark:market -- 10000`, separately measures bounded reconciliation of terminal order/listing records in its own temporary SQLite directory. On the same development Docker environment, 10,000 orders plus 10,000 listings completed in 157 reconciliation passes in 772 ms (about 25,903 records/s), with pass-latency p50/p95/max of 4.57/9.13/20.04 ms and a 7,180,288-byte SQLite file. This dataset intentionally avoids active-order delivery and listing validation branches; it measures local keyset/SQLite overhead only. It is not a target-hardware result or throughput/SLO claim and excludes concurrent HTTP traffic, event-loop interference from other work, PostgreSQL, chain validation and adversarial workloads. +The synthetic market-journal harness [benchmark](../scripts/benchmark-market-reconciliation.ts), available as `npm run benchmark:market -- 10000 [terminal|confirmed-delivery]`, measures two bounded reconciliation paths in its own temporary SQLite directory. On the same development Docker environment, 10,000 terminal orders plus 10,000 listings completed in 776 ms (about 25,770 records/s), with pass-latency p50/p95/max of 4.56/8.03/17.73 ms, sampled event-loop-delay p95/max of 16.70/19.55 ms at 10ms resolution, and an 11,927,552-byte SQLite file. The synthetic confirmed-delivery scenario transitioned 10,000 paid orders and reserved listings to complete/sold in 2,031 ms (about 9,846 records/s), with pass-latency p50/p95/max of 12.27/24.75/35.59 ms, sampled event-loop-delay p95/max of 28.77/48.46 ms, and a 12,832,768-byte SQLite file. Its canonical transaction IDs come from an in-memory adapter fixture, not validated Ledger evidence. The event-loop samples demonstrate synchronous batch stalls locally; yielding between benchmark passes allows the monitor to sample but does not offload production work. These are not target-hardware throughput/SLO claims and exclude concurrent HTTP traffic, other server work, PostgreSQL, actual chain validation and adversarial workloads. ## Remaining gates diff --git a/docs/production-plan.md b/docs/production-plan.md index fd705c6..4a074a8 100755 --- a/docs/production-plan.md +++ b/docs/production-plan.md @@ -63,7 +63,7 @@ Acceptance: exact-amount property/boundary tests, versioned signature tests, zer Can run alongside P0/P1. Addresses SEC-01/SEC-06 rather than adding more fixed caps. -**Implemented M2 increments, 2026-10-20–21:** [canonical indexes and durable market observation](canonical-indexes.md). Canonical ownership/transaction SQL indexes now serve NFT, deck, custody availability and mock-market queries; per-block undo and anchored connect/disconnect events commit with ledger updates. Market checkpoint/reorg holds persist atomically and bounded catch-up blocks new mutations. Order/listing reconciliation now advances in durable, stable-ID keyset batches of at most 128 records per table, resumes after restart, re-fences when the chain tip/revision changes, and runs from an HTTP lifecycle poller rather than relying only on page loads. Final TypeScript/Vite build and **174 offline tests passed**; focused market tests passed 23/23 and the disposable PostgreSQL integration suite passed 31/31. A new synthetic local benchmark processed 10,000 terminal orders plus 10,000 listings in 772 ms on the development Docker host; this is a reproducible harness result, **not** a target-hardware SLO or complete M2 acceptance. Queue isolation, target-scale/concurrent throughput and latency, production financial storage/recovery, retained-DAG capacity and operator dataset/SLO targets remain open. +**Implemented M2 increments, 2026-10-20–21:** [canonical indexes and durable market observation](canonical-indexes.md). Canonical ownership/transaction SQL indexes now serve NFT, deck, custody availability and mock-market queries; per-block undo and anchored connect/disconnect events commit with ledger updates. Market checkpoint/reorg holds persist atomically and bounded catch-up blocks new mutations. Order/listing reconciliation now advances in durable, stable-ID keyset batches of at most 128 records per table, resumes after restart, re-fences when the chain tip/revision changes, and runs from an HTTP lifecycle poller rather than relying only on page loads. Final TypeScript/Vite build and **174 offline tests passed**; focused market tests passed 23/23 and the disposable PostgreSQL integration suite passed 31/31. Synthetic 10,000-order/listing runs processed terminal records in 776 ms (event-loop-delay max 19.55 ms) and paid orders with confirmed-delivery adapter fixtures in 2,031 ms (max 48.46 ms); these measurements show synchronous batches can block the local loop, but are not target-hardware SLOs or complete M2 acceptance. Queue isolation, target-scale/concurrent throughput and latency, production financial storage/recovery, retained-DAG capacity and operator dataset/SLO targets remain open. **Next financial-storage increment:** [ADR 003 — PostgreSQL](postgres-financial-storage.md) selects and integrates the custody SQL backend with existing HTTP callers, exact constrained accounting, explicit migration/runtime roles, source-freezing SQLite import and dump/restore rehearsal. The adapter remains on the fictitious sandbox domain; it is not a production deposit/withdrawal worker. SQLite node-only backup deliberately refuses PostgreSQL-bound financial state. Production hardware/SLO tests, coordinated backup/PITR/failover, real payment schema/worker controls and independent review remain open, so this does not close M2/M4 or authorize funds. diff --git a/scripts/benchmark-market-reconciliation.ts b/scripts/benchmark-market-reconciliation.ts index 4cde78b..79e1c7a 100755 --- a/scripts/benchmark-market-reconciliation.ts +++ b/scripts/benchmark-market-reconciliation.ts @@ -2,27 +2,31 @@ import { DatabaseSync } from 'node:sqlite'; import { mkdtemp, rm, stat } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; -import { performance } from 'node:perf_hooks'; +import { monitorEventLoopDelay, performance } from 'node:perf_hooks'; import { randomUUID } from 'node:crypto'; +import { setImmediate as yieldToEventLoop } from 'node:timers/promises'; import { Market } from '../server/market/store.ts'; -import { GENESIS_HASH } from '../server/testnet/consensus.ts'; +import { CHAIN_ID, GENESIS_HASH, transactionId } from '../server/testnet/consensus.ts'; import { NETWORK } from '../shared/testnet.ts'; -// Synthetic terminal-order journal only. It is isolated in a temporary SQLite database and -// never connects to a Ledger, peer, payment adapter, wallet, or production data directory. +// Synthetic market journal only. It is isolated in a temporary SQLite database and never +// connects to a Ledger, peer, payment adapter, wallet, or production data directory. const text = process.argv[2] ?? '10000'; if (!/^[1-9]\d{0,5}$/.test(text) || Number(text) > 100000) throw new Error('Fixture size must be 1..100000 orders'); -const count = Number(text); +const count = Number(text), scenario = process.argv[3] ?? 'terminal'; +if (!['terminal', 'confirmed-delivery'].includes(scenario)) throw new Error('Scenario must be terminal or confirmed-delivery'); const directory = await mkdtemp(join(tmpdir(), 'wikideck-market-reconciliation-')); const streamId = randomUUID(), tip = 'a'.repeat(64); +const deliveryRows = new Map(); +const seller = '1'.repeat(64), buyer = '2'.repeat(64); let market: Market | undefined; try { // Indexed-only adapter: the benchmark must not fall back to materializing chain snapshots. const chain = { snapshot: () => { throw new Error('Benchmark must not request a full chain snapshot'); }, - summary: () => ({ tip, revision: 0, height: 1 }), + summary: () => ({ tip, revision: 0, height: 2 }), nft: () => undefined, - canonicalTransaction: () => undefined, + canonicalTransaction: (id: string) => deliveryRows.get(id), pendingTransactions: () => [], canonicalHead: () => 0, canonicalEvents: (after: number) => ({ streamId, previousAnchor: GENESIS_HASH, anchor: GENESIS_HASH, @@ -34,15 +38,23 @@ try { try { const insertListing = db.prepare('INSERT INTO listings(id,card,status,data) VALUES(?,?,?,?)'); const insertOrder = db.prepare('INSERT INTO orders(id,data) VALUES(?,?)'); - const seller = 'synthetic-seller'; db.exec('BEGIN IMMEDIATE'); try { for (let i = 0; i < count; i++) { const id = i.toString().padStart(12, '0'); const listingId = `listing-${id}`, orderId = `order-${id}`, cardHash = i.toString(16).padStart(64, '0'); - const listing = { id: listingId, cardHash, seller, sequence: 0, kind: 'fixed', status: 'cancelled' }; - const order = { id: orderId, listingId, seller, buyer: 'synthetic-buyer', cardHash, sequence: 0, status: 'refunded' }; - insertListing.run(listingId, cardHash, 'cancelled', JSON.stringify(listing)); + const terminal = scenario === 'terminal'; + const listingStatus = terminal ? 'cancelled' : 'reserved'; + const listing = { id: listingId, cardHash, seller, sequence: 0, kind: 'fixed', status: listingStatus }; + const order = { id: orderId, listingId, seller, buyer, cardHash, sequence: 0, + status: terminal ? 'refunded' : 'paid', payout: null, deliveryConfirmations: 0, + requiredConfirmations: 2, createdAt: 1, payBy: 2, deliverBy: 3 }; + if (!terminal) { + const delivery = { network: NETWORK, chainId: CHAIN_ID, cardHash, from: seller, to: buyer, + sequence: 0, publicKey: '', signature: '' }; + deliveryRows.set(transactionId(delivery), { height: 1, blockHash: tip }); + } + insertListing.run(listingId, cardHash, listingStatus, JSON.stringify(listing)); insertOrder.run(orderId, JSON.stringify(order)); } db.exec('COMMIT'); @@ -51,23 +63,30 @@ try { const latencies: number[] = []; const maxPasses = Math.ceil(count / 128) * 2 + 4; + const eventLoopDelay = monitorEventLoopDelay({ resolution: 10 }); + eventLoopDelay.enable(); const start = performance.now(); - for (let pass = 0; pass < maxPasses && (pass === 0 || market.hasReconciliationBacklog()); pass++) { - const before = performance.now(); - market.browse(); - latencies.push(performance.now() - before); - } + try { + for (let pass = 0; pass < maxPasses && (pass === 0 || market.hasReconciliationBacklog()); pass++) { + const before = performance.now(); + market.browse(); + latencies.push(performance.now() - before); + await yieldToEventLoop(); // Permit the delay monitor to observe synchronous batch stalls. + } + } finally { eventLoopDelay.disable(); } const elapsedMs = performance.now() - start; if (market.hasReconciliationBacklog()) throw new Error(`Reconciliation did not finish within ${maxPasses} bounded passes`); market.close(); market = undefined; const bytes = (await stat(join(directory, NETWORK.includes('mainnet') ? 'mainnet' : 'testnet', 'market.sqlite'))).size; latencies.sort((a, b) => a - b); const percentile = (p: number) => latencies[Math.min(latencies.length - 1, Math.ceil(latencies.length * p) - 1)]; - console.log(JSON.stringify({ fixture: 'synthetic-terminal-orders-and-listings', orders: count, listings: count, + console.log(JSON.stringify({ fixture: `synthetic-${scenario}-orders-and-listings`, orders: count, listings: count, batchSizePerTable: 128, reconciliationPasses: latencies.length, elapsedMs, rowsPerSecond: Math.round((count * 2 * 1000) / elapsedMs), passLatencyP50Ms: percentile(0.50), passLatencyP95Ms: percentile(0.95), passLatencyMaxMs: latencies.at(-1), sqliteBytes: bytes, - warning: 'Synthetic terminal records on this development host only; excludes live chain validation, active-order logic, concurrent HTTP load, PostgreSQL, and production SLO certification.' }, null, 2)); + eventLoopDelayResolutionMs: 10, eventLoopDelayP95Ms: eventLoopDelay.percentile(95) / 1e6, + eventLoopDelayMaxMs: eventLoopDelay.max / 1e6, + warning: 'Synthetic fixture on this development host only; canonical deliveries are adapter fixtures, not Ledger evidence. Excludes concurrent HTTP load, PostgreSQL, and production SLO certification.' }, null, 2)); } finally { market?.close(); await rm(directory, { recursive: true, force: true });