Reconciliation uses 32-row SQLite slices

docs/canonical-indexes.md

@@ -9,10 +9,9 @@ Status: implemented local infrastructure, **not full M2 acceptance or production
 - Events contain sequence, connect/disconnect type, block identity/height/parent, transaction IDs and a hash chained to the previous event. An independently generated local stream ID binds the history; it is **not consensus data or a public proof of honest observation**. Pages contain at most 64 events and no article payloads. Financial/chain history is not silently pruned.
 - On first open, original blocks are replayed and independently proof/state validated before additive local index backfill. Later opens compare indexed block identity, ownership, transaction inclusion and event continuity/anchors to that validated chain. Inconsistent state fails closed; existing consensus version, genesis, signatures and free-transfer/trade rules are unchanged.
 - [Market reconciliation](../server/market/store.ts) uses indexed current delivery/ownership queries and a persisted `(stream ID, sequence, anchor)` checkpoint. Every observed disconnect of a previously completed order causes a durable review hold, even if delivery reappeared before polling. Review is never automatically repaid. Hold updates and checkpoint advancement commit together in the market database.
-- Order and listing review now uses stable-ID keyset batches capped at 128 rows each. A strict, durable job record stores stage, continuation ID, observed time, chain tip/revision, and the bounded canonical event page. Each batch commits row effects and its continuation atomically; the canonical event cursor advances only after both tables finish. A chain tip/revision change during a multi-batch scan rewinds the job and captures a fresh event page, preventing a mixed-view scan from being marked caught up. Completion checks the current event head so events appended during the scan remain behind the write fence.
-- While any job/event backlog exists, ordinary market mutations and time-dependent payout/expiry decisions stop; diagnostic/read access remains available. Subsequent requests advance bounded work, and the HTTP lifecycle has an unref'd 250ms poller that drives catch-up independently of user page loads. The poller is not a separately isolated worker: SQLite work is synchronous on the server event loop. There is no claim of instant catch-up, a throughput SLO, or horizontal-worker safety.
+- Order and listing review now uses stable-ID keyset batches capped at 32 rows each. A strict, durable job record stores stage, continuation ID, observed time, chain tip/revision, and the bounded canonical event page. Each batch commits row effects and its continuation atomically; the canonical event cursor advances only after both tables finish. A chain tip/revision change during a multi-batch scan rewinds the job and captures a fresh event page, preventing a mixed-view scan from being marked caught up. Completion checks the current event head so events appended during the scan remain behind the write fence.
+- While any job/event backlog exists, ordinary market mutations and time-dependent payout/expiry decisions stop; diagnostic/read access remains available. Requests advance one bounded slice, and an unref'd HTTP-lifecycle pump drains remaining work through `setImmediate` turns without depending on page loads. This is cooperative yielding, **not a worker thread**: one slice still runs synchronous SQLite work on the server event loop. There is no claim of instant catch-up, a throughput SLO, or horizontal-worker safety.
 - Reconciliation restart, mid-job event append/reorg fencing, disconnect review, and injected SQLite batch/cursor failures are tested. A failed batch leaves both order effects and durable keyset progress unchanged; restart resumes committed work. Restore mismatch (different stream, different history anchor or a cursor ahead of the restored ledger) is refused rather than resetting the cursor. Restore the matching stopped-node ledger/market backup; any alternative recovery requires reviewed evidence, not deleting a checkpoint.
-- Restore mismatch (different stream, different history anchor or a cursor ahead of the restored ledger) is refused rather than resetting the cursor. Restore the matching stopped-node ledger/market backup; any alternative recovery requires reviewed evidence, not deleting a checkpoint.
 - [HTTP NFT lookup and custody card checks](../server/testnet/http.ts) use the same index. Wallet deck pagination accepts an optional `tip` query; stale tips return 409. Existing callers without `tip` remain compatible. There is no new public event/admin endpoint, no payment observer and no NFT-lock signer.
 
 ## Tests and measurements
@@ -25,7 +24,7 @@ The final Docker test-target build passed TypeScript/Vite compilation. All **174
 - additive migration of a prior-schema copy, refusing corrupt ownership and event records;
 - production market adapter exercised with full snapshots forbidden;
 - multi-page catch-up and persisted restart checkpoint;
-- 128-row order/listing keyset progress, restart continuation, and a new canonical event arriving during an in-flight job;
+- 32-row order/listing keyset progress, restart continuation, and a new canonical event arriving during an in-flight job;
 - chain revision change fencing that resets continuation and refreshes the event page before the scan continues;
 - injected batch-write failure rolling back order effects and keyset progress, plus cursor-write failure rolling back checkpoint advancement while preserving committed review holds;
 - numeric cursor reuse with a different historical hash rejected;
@@ -46,7 +45,7 @@ Observed in the development Docker environment:
 
 These numbers are a reproducible **baseline**, not acceptance SLOs: host CPU/storage and production concurrency were not selected, outputs include synthetic article payloads, warm caches apply, one transaction amortizes insertion fsync, and real validation/startup/HTTP/market workloads are excluded. They do not establish a supported production envelope.
 
-The synthetic market-journal harness [benchmark](../scripts/benchmark-market-reconciliation.ts), available as `npm run benchmark:market -- 10000 [terminal|confirmed-delivery]`, measures two bounded reconciliation paths in its own temporary SQLite directory. On the same development Docker environment, 10,000 terminal orders plus 10,000 listings completed in 776 ms (about 25,770 records/s), with pass-latency p50/p95/max of 4.56/8.03/17.73 ms, sampled event-loop-delay p95/max of 16.70/19.55 ms at 10ms resolution, and an 11,927,552-byte SQLite file. The synthetic confirmed-delivery scenario transitioned 10,000 paid orders and reserved listings to complete/sold in 2,031 ms (about 9,846 records/s), with pass-latency p50/p95/max of 12.27/24.75/35.59 ms, sampled event-loop-delay p95/max of 28.77/48.46 ms, and a 12,832,768-byte SQLite file. Its canonical transaction IDs come from an in-memory adapter fixture, not validated Ledger evidence. The event-loop samples demonstrate synchronous batch stalls locally; yielding between benchmark passes allows the monitor to sample but does not offload production work. These are not target-hardware throughput/SLO claims and exclude concurrent HTTP traffic, other server work, PostgreSQL, actual chain validation and adversarial workloads.
+The synthetic market-journal harness [benchmark](../scripts/benchmark-market-reconciliation.ts), available as `npm run benchmark:market -- 10000 [terminal|confirmed-delivery]`, measures two bounded reconciliation paths in its own temporary SQLite directory. With 32-row slices, the development Docker host processed 10,000 terminal orders plus 10,000 listings in 2,549 ms (about 7,845 records/s), with pass-latency p50/p95/max of 3.44/6.48/42.13 ms, sampled event-loop-delay p95/max of 14.97/48.99 ms at 10ms resolution, and an 11,927,552-byte SQLite file. The confirmed-delivery fixture transitioned 10,000 paid orders and reserved listings to complete/sold in 4,667 ms (about 4,286 records/s), with pass-latency p50/p95/max of 6.88/14.30/62.12 ms, sampled event-loop-delay p95/max of 20.63/62.23 ms, and a 12,832,768-byte SQLite file. The 32-row size increases pass count (625 vs. 157 at 128 rows) and reduces typical delay, but these runs still show long-tail event-loop stalls; a smaller batch is not equivalent to worker isolation. The canonical transaction IDs are in-memory adapter fixtures, not validated Ledger evidence. These are not target-hardware throughput/SLO claims and exclude concurrent HTTP traffic, other server work, PostgreSQL, actual chain validation and adversarial workloads.
 
 ## Remaining gates
 

docs/production-plan.md

@@ -63,7 +63,7 @@ Acceptance: exact-amount property/boundary tests, versioned signature tests, zer
 
 Can run alongside P0/P1. Addresses SEC-01/SEC-06 rather than adding more fixed caps.
 
-**Implemented M2 increments, 2026-10-20–21:** [canonical indexes and durable market observation](canonical-indexes.md). Canonical ownership/transaction SQL indexes now serve NFT, deck, custody availability and mock-market queries; per-block undo and anchored connect/disconnect events commit with ledger updates. Market checkpoint/reorg holds persist atomically and bounded catch-up blocks new mutations. Order/listing reconciliation now advances in durable, stable-ID keyset batches of at most 128 records per table, resumes after restart, re-fences when the chain tip/revision changes, and runs from an HTTP lifecycle poller rather than relying only on page loads. Final TypeScript/Vite build and **174 offline tests passed**; focused market tests passed 23/23 and the disposable PostgreSQL integration suite passed 31/31. Synthetic 10,000-order/listing runs processed terminal records in 776 ms (event-loop-delay max 19.55 ms) and paid orders with confirmed-delivery adapter fixtures in 2,031 ms (max 48.46 ms); these measurements show synchronous batches can block the local loop, but are not target-hardware SLOs or complete M2 acceptance. Queue isolation, target-scale/concurrent throughput and latency, production financial storage/recovery, retained-DAG capacity and operator dataset/SLO targets remain open.
+**Implemented M2 increments, 2026-10-20–21:** [canonical indexes and durable market observation](canonical-indexes.md). Canonical ownership/transaction SQL indexes now serve NFT, deck, custody availability and mock-market queries; per-block undo and anchored connect/disconnect events commit with ledger updates. Market checkpoint/reorg holds persist atomically and bounded catch-up blocks new mutations. Order/listing reconciliation advances in durable, stable-ID keyset batches of at most 32 records per table, resumes after restart, re-fences when the chain tip/revision changes, and drains through a yielding HTTP-lifecycle pump rather than relying only on page loads. Final TypeScript/Vite build and **174 offline tests passed**; focused market tests passed 23/23 and the disposable PostgreSQL integration suite passed 31/31. Synthetic 10,000-order/listing runs processed terminal records in 2,549 ms (event-loop-delay p95/max 14.97/48.99 ms) and paid orders with confirmed-delivery adapter fixtures in 4,667 ms (20.63/62.23 ms). Slicing reduces typical per-pass delay but increases total work and still has long-tail stalls; it is not worker-thread isolation or target-hardware evidence. Queue isolation, target-scale/concurrent throughput and latency, production financial storage/recovery, retained-DAG capacity and operator dataset/SLO targets remain open.
 
 **Next financial-storage increment:** [ADR 003 — PostgreSQL](postgres-financial-storage.md) selects and integrates the custody SQL backend with existing HTTP callers, exact constrained accounting, explicit migration/runtime roles, source-freezing SQLite import and dump/restore rehearsal. The adapter remains on the fictitious sandbox domain; it is not a production deposit/withdrawal worker. SQLite node-only backup deliberately refuses PostgreSQL-bound financial state. Production hardware/SLO tests, coordinated backup/PITR/failover, real payment schema/worker controls and independent review remain open, so this does not close M2/M4 or authorize funds.
 
@@ -72,7 +72,7 @@ Can run alongside P0/P1. Addresses SEC-01/SEC-06 rather than adding more fixed c
 **Deployment increment:** [persistent local custody stack](persistent-custody-sandbox.md) now starts PostgreSQL and the actual app together, with private TLS/SCRAM networking, separate owner/runtime roles, explicit migrations and scoped persistent credential/data volumes. A signed fictitious credit and custodian identity survived container/network removal and recreation; the server rejected plaintext authentication. Existing deployments and SQLite custody were not switched. Production certificate rotation, coordinated backup/restore and external operating review remain open; all XMR remains simulated.
 
 - Indexed canonical ownership, NFT/transfer lookup and paged wallet/explorer APIs: browser signing must not require loading all history. Tip-pinned queries detect reorgs and preserve complete ownership semantics.
-- Incremental market reconciliation from durable canonical/reorg events. **Local bounded implementation delivered:** rollback-safe canonical event pages, durable 128-row order/listing keyset jobs, tip/revision re-fencing, restart continuation, mutation fencing, and background catch-up polling; regression tests cover appends/reorgs and injected SQLite failures. **Still open:** move synchronous SQLite batches off the HTTP event loop, coordinate a single worker across processes, measure lag/throughput and request latency at declared order/event volumes, and demonstrate overload/recovery limits before production claims.
+- Incremental market reconciliation from durable canonical/reorg events. **Local bounded implementation delivered:** rollback-safe canonical event pages, durable 32-row order/listing keyset jobs, tip/revision re-fencing, restart continuation, mutation fencing, and a cooperative background pump that yields between batches; regression tests cover appends/reorgs and injected SQLite failures. **Still open:** true worker-thread isolation (the pump still runs synchronous SQLite on the main event loop), coordination of a single worker across processes, and lag/throughput/request-latency plus overload/recovery measurements at declared order/event volumes.
 - A versioned production market/settlement schema with append-only financial events and balanced accounting entries for buyer liabilities, host revenue, author liabilities, seller liabilities, paid/refunded amounts and actual network fees. Derived views are rebuildable; financial history and idempotency keys are not discarded to reclaim space.
 - Durable inbox/outbox, processor attempts and reconciliation states; no network operation inside a long DB transaction. Store intent before external side effects; unknown POST outcomes stay quarantined until evidenced resolution. Per-market isolation, actor-scoped paged reads, queue/backpressure and tenant/global exposure budgets.
 - Database architecture ADR: retain a measured single-writer SQLite deployment if it meets declared limits, or choose a production DB/worker topology and supported migration. Do not claim horizontal safety merely by launching multiple workers against local state. Start with one authoritative writer per market.

scripts/benchmark-market-reconciliation.ts

@@ -5,7 +5,7 @@ import { join } from 'node:path';
 import { monitorEventLoopDelay, performance } from 'node:perf_hooks';
 import { randomUUID } from 'node:crypto';
 import { setImmediate as yieldToEventLoop } from 'node:timers/promises';
-import { Market } from '../server/market/store.ts';
+import { MARKET_RECONCILIATION_BATCH_SIZE, Market } from '../server/market/store.ts';
 import { CHAIN_ID, GENESIS_HASH, transactionId } from '../server/testnet/consensus.ts';
 import { NETWORK } from '../shared/testnet.ts';
 
@@ -62,7 +62,7 @@ try {
   } finally { db.close(); }
 
   const latencies: number[] = [];
-  const maxPasses = Math.ceil(count / 128) * 2 + 4;
+  const maxPasses = Math.ceil(count / MARKET_RECONCILIATION_BATCH_SIZE) * 2 + 4;
   const eventLoopDelay = monitorEventLoopDelay({ resolution: 10 });
   eventLoopDelay.enable();
   const start = performance.now();
@@ -81,7 +81,7 @@ try {
   latencies.sort((a, b) => a - b);
   const percentile = (p: number) => latencies[Math.min(latencies.length - 1, Math.ceil(latencies.length * p) - 1)];
   console.log(JSON.stringify({ fixture: `synthetic-${scenario}-orders-and-listings`, orders: count, listings: count,
-    batchSizePerTable: 128, reconciliationPasses: latencies.length, elapsedMs,
+    batchSizePerTable: MARKET_RECONCILIATION_BATCH_SIZE, reconciliationPasses: latencies.length, elapsedMs,
     rowsPerSecond: Math.round((count * 2 * 1000) / elapsedMs), passLatencyP50Ms: percentile(0.50),
     passLatencyP95Ms: percentile(0.95), passLatencyMaxMs: latencies.at(-1), sqliteBytes: bytes,
     eventLoopDelayResolutionMs: 10, eventLoopDelayP95Ms: eventLoopDelay.percentile(95) / 1e6,

server/market/store.ts

@@ -52,7 +52,8 @@ const reconciliationJobSchema = z.object({ version: z.literal(1), stage: z.enum(
   afterId: z.string().nullable(), observedAt: z.number().int().nonnegative().safe(),
   tip: z.string().regex(/^[a-f0-9]{64}$/).nullable(), revision: z.number().int().nonnegative().safe().nullable(),
   eventPage: canonicalEventPageSchema.nullable() }).strict();
-const RECONCILE_BATCH_SIZE = 128;
+// Keep each synchronous SQLite slice small enough to limit request-loop stalls.
+export const MARKET_RECONCILIATION_BATCH_SIZE = 32;
 const RECONCILIATION_JOB_KEY = 'canonical_reconciliation_job';
 
 /** Node-local marketplace, NOT consensus escrow. Its SQLite journal is backed up with the ledger. */
@@ -128,7 +129,7 @@ export class Market {
   }
   private nextReconciliationRows(table: 'listings' | 'orders', afterId: string | null): { id: string; data: string }[] {
     return this.db.prepare(`SELECT id,data FROM ${table} WHERE (? IS NULL OR id>?) ORDER BY id LIMIT ?`)
-      .all(afterId, afterId, RECONCILE_BATCH_SIZE) as { id: string; data: string }[];
+      .all(afterId, afterId, MARKET_RECONCILIATION_BATCH_SIZE) as { id: string; data: string }[];
   }
   private listing(id: string): MarketListingRecord {
     const row = this.db.prepare('SELECT data FROM listings WHERE id=?').get(id) as { data: string } | undefined;
@@ -270,10 +271,10 @@ export class Market {
       }
 
       if (job.stage === 'orders') {
-        if (rows.length < RECONCILE_BATCH_SIZE) nextJob = { ...job, stage: 'listings', afterId: null };
+        if (rows.length < MARKET_RECONCILIATION_BATCH_SIZE) nextJob = { ...job, stage: 'listings', afterId: null };
         else nextJob = { ...job, afterId: rows.at(-1)!.id };
       } else {
-        if (rows.length < RECONCILE_BATCH_SIZE) {
+        if (rows.length < MARKET_RECONCILIATION_BATCH_SIZE) {
           this.db.prepare('DELETE FROM receipts WHERE expires < ?').run(job.observedAt - 86400_000);
           if (eventPage) {
             this.db.prepare('INSERT INTO market_meta(key,value) VALUES(?,?) ON CONFLICT(key) DO UPDATE SET value=excluded.value')
@@ -305,7 +306,7 @@ export class Market {
     }
     // Empty/short order pages can advance to the listing stage in the same bounded call. This
     // avoids rejecting the first command on an empty market while still capping work at 2×128 rows.
-    if (!completed && job.stage === 'orders' && rows.length < RECONCILE_BATCH_SIZE) this.reconcile();
+    if (!completed && job.stage === 'orders' && rows.length < MARKET_RECONCILIATION_BATCH_SIZE) this.reconcile();
   }
   browse(offset = 0): { listings: MarketListingRecord[]; next: number | null; info: MarketInfo } {
     if (!Number.isSafeInteger(offset) || offset < 0 || offset > 10000) throw new Error('Invalid marketplace page');

server/testnet/http.ts

@@ -251,17 +251,29 @@ server.setTimeout(30000,socket=>socket.destroy());
 // Advance simulated payout/expiry state even when no browser is open. No real funds are sent.
 const marketTimer=setInterval(()=>{try {market.browse();}catch{console.error('Marketplace reconciliation failed; operator review required.');}},15000);
 marketTimer.unref();
-// When durable reconciliation is behind, advance bounded keyset pages without requiring a browser request.
+let stopping=false;
+let catchupPaused=false;
+let catchupImmediate:NodeJS.Immediate|undefined;
+// Drain durable catch-up without page-load dependence. Each pass is a smaller synchronous
+// SQLite slice, then setImmediate yields so HTTP and peer work can run between passes.
+const advanceMarketCatchup=()=>{
+  catchupImmediate=undefined;
+  if(stopping||catchupPaused||!market.hasReconciliationBacklog())return;
+  try {market.browse();}
+  catch {catchupPaused=true;console.error('Marketplace canonical catch-up paused; operator review required.');return;}
+  if(market.hasReconciliationBacklog()){catchupImmediate=setImmediate(advanceMarketCatchup);catchupImmediate.unref();}
+};
 const marketCatchupTimer=setInterval(()=>{
-  if(!market.hasReconciliationBacklog())return;
-  try {market.browse();}catch{console.error('Marketplace canonical catch-up failed; operator review required.');}
+  if(!stopping&&!catchupPaused&&!catchupImmediate&&market.hasReconciliationBacklog()){
+    catchupImmediate=setImmediate(advanceMarketCatchup);catchupImmediate.unref();
+  }
 },250);
 marketCatchupTimer.unref();
-let stopping=false;
 for(const signal of ['SIGTERM','SIGINT']as const)process.once(signal,()=>{
   if(stopping)return;stopping=true;peers.close();operations.close();clearInterval(marketTimer);clearInterval(marketCatchupTimer);
+  if(catchupImmediate)clearImmediate(catchupImmediate);
   const observerDrain=custodyObserver?.close()??Promise.resolve();
   const timeout=setTimeout(()=>process.exit(1),25000);timeout.unref();
   server.close(async()=>{await observerDrain;await custody?.close();market.close();ledger.close();engine.close();await lock.close();await unlink(lockPath).catch(()=>{});clearTimeout(timeout);process.exit(0);});
 });
-server.on('error',async()=>{clearInterval(marketTimer);clearInterval(marketCatchupTimer);operations.close();peers.close();await custodyObserver?.close();await custody?.close();market.close();ledger.close();engine.close();await lock.close();await unlink(lockPath).catch(()=>{});process.exit(1);});
+server.on('error',async()=>{stopping=true;clearInterval(marketTimer);clearInterval(marketCatchupTimer);if(catchupImmediate)clearImmediate(catchupImmediate);operations.close();peers.close();await custodyObserver?.close();await custody?.close();market.close();ledger.close();engine.close();await lock.close();await unlink(lockPath).catch(()=>{});process.exit(1);});

tests/market.test.ts

@@ -171,18 +171,23 @@ test('market reconciliation keyset progress is bounded, durable and resumes afte
     try { return JSON.parse((conn.prepare('SELECT value FROM market_meta WHERE key=?').get('canonical_reconciliation_job') as { value: string }).value); }
     finally { conn.close(); }
   };
-  assert.equal(progress().stage, 'orders'); assert.equal(progress().afterId, ids[127]);
+  assert.equal(progress().stage, 'orders'); assert.equal(progress().afterId, ids[31]);
   assert.equal(f.market.hasReconciliationBacklog(), true);
-  await assert.rejects(f.list(), /catching up/); // second 128-order batch; no new listing is admitted.
-  assert.equal(progress().stage, 'orders'); assert.equal(progress().afterId, ids[255]);
+  await assert.rejects(f.list(), /catching up/); // Second 32-order batch; no new listing is admitted.
+  assert.equal(progress().stage, 'orders'); assert.equal(progress().afterId, ids[63]);
   // The ledger advances while this durable multi-batch job is in flight.
   f.events.push({ sequence: 2, type: 'connect', height: 2, hash: 'f'.repeat(64), previousHash: 'd'.repeat(64),
     transactionIds: [], eventHash: '2'.repeat(64) });
   f.market.browse(); // Tip/revision changed: rewind and bind the updated event page.
   assert.equal(progress().eventPage.cursor, 2);
-  assert.equal(progress().afterId, ids[127]);
-  f.market.browse(); // Second order batch of the refreshed scan.
-  f.market.browse(); // Remaining four orders, then at most one listing page.
+  assert.equal(progress().afterId, ids[31]);
+  const tailStatus = () => {
+    const conn = new DatabaseSync(join(f.dir, 'testnet', 'market.sqlite'));
+    try { return JSON.parse((conn.prepare('SELECT data FROM orders WHERE id=?').get(ids[259]) as { data: string }).data).status; }
+    finally { conn.close(); }
+  };
+  for (let i = 0; i < 12 && tailStatus() !== 'review'; i++) f.market.browse();
+  assert.equal(tailStatus(), 'review'); // Disconnect reached the tail order before checkpoint advancement.
   assert.equal(f.market.diagnostics().canonicalEventCursor, 0);
   assert.equal(f.market.hasReconciliationBacklog(), true, 'new events appended during job must stay behind the work fence');
   const reviewDb = new DatabaseSync(join(f.dir, 'testnet', 'market.sqlite'));
@@ -194,7 +199,7 @@ test('market reconciliation keyset progress is bounded, durable and resumes afte
 
   // Simulate process death after committed batch progress, then resume using the SQLite job cursor.
   await f.restart(); assert.equal(f.market.diagnostics().eventBacklog, true);
-  for (let i = 0; i < 12 && f.market.hasReconciliationBacklog(); i++) f.market.browse();
+  for (let i = 0; i < 24 && f.market.hasReconciliationBacklog(); i++) f.market.browse();
   assert.equal(f.market.hasReconciliationBacklog(), false);
   const conn = new DatabaseSync(join(f.dir, 'testnet', 'market.sqlite'));
   try { assert.equal(conn.prepare('SELECT 1 FROM market_meta WHERE key=?').get('canonical_reconciliation_job'), undefined); }