Moved canonical transaction, NFT, and mempool reads into a separate read-only Ledger SQLite connection

docs/canonical-indexes.md

@@ -16,7 +16,7 @@ Status: implemented local infrastructure, **not full M2 acceptance or production
 
 ## Tests and measurements
 
-The final Docker test-target build passed TypeScript/Vite compilation. All **177 offline tests passed, zero failures/skips**, including native RandomX multi-process HTTP/peer/restart and backup rehearsal. The focused market suite passed **26 tests**; the disposable PostgreSQL suite passed **31 tests** in the preceding integration run. New failure evidence includes:
+The final Docker test-target build passed TypeScript/Vite compilation. All **178 offline tests passed, zero failures/skips**, including native RandomX multi-process HTTP/peer/restart and backup rehearsal. The focused market suite passed **27 tests**, and the disposable PostgreSQL suite passed **31/31**.
 
 - indexed ownership/sequence versus full replay for transfers, multi-card trades and forks;
 - append/disconnect/reconnect ordering, side-branch/mempool exclusion, bounded defensive-copy pages and durable restart;
@@ -30,6 +30,7 @@ The final Docker test-target build passed TypeScript/Vite compilation. All **177
 - worker-owned row evaluation/batch/finalize transaction with injected failure, mutation fence and restart continuation;
 - canonical event arrival during an in-flight worker batch forces a fresh bounded event page before the cursor reaches the new head;
 - separate worker read-only Ledger connection supplies canonical NFT and delivery facts and re-fences when an event arrives during that snapshot;
+- main-thread timer fires while a large worker reconciliation backlog remains in flight;
 - numeric cursor reuse with a different historical hash rejected;
 - real HTTP NFT results and stale-tip deck pagination.
 

docs/production-plan.md

@@ -63,7 +63,7 @@ Acceptance: exact-amount property/boundary tests, versioned signature tests, zer
 
 Can run alongside P0/P1. Addresses SEC-01/SEC-06 rather than adding more fixed caps.
 
-**Implemented M2 increments, 2026-10-20–21:** [canonical indexes and durable market observation](canonical-indexes.md). Canonical ownership/transaction SQL indexes now serve NFT, deck, custody availability and mock-market queries; per-block undo and anchored connect/disconnect events commit with ledger updates. Market checkpoint/reorg holds persist atomically and bounded catch-up blocks new mutations. Order/listing reconciliation advances in durable, stable-ID keyset batches of at most 32 records per table, resumes after restart, and re-fences when the chain tip/revision changes. The HTTP process now uses a single worker thread to select/evaluate bounded row batches, read canonical facts using its own read-only Ledger connection, and atomically persist changes with cursor progress. Final TypeScript/Vite build and **177 offline tests passed**; focused market tests passed 26/26; the disposable PostgreSQL suite passed 31/31. In a same-fixture synthetic 10,000 confirmed-delivery comparison, worker processing took 4,088 ms with sampled event-loop-delay max 12.48 ms versus 5,853 ms and 55.67 ms synchronously on the development host. This is local evidence, not an SLO. Main-thread event-page acquisition/post-batch fencing, concurrent HTTP load, multi-process fencing, target hardware/data, production financial storage/recovery, retained-DAG capacity and operator SLO decisions remain open.
+**Implemented M2 increments, 2026-10-20–21:** [canonical indexes and durable market observation](canonical-indexes.md). Canonical ownership/transaction SQL indexes now serve NFT, deck, custody availability and mock-market queries; per-block undo and anchored connect/disconnect events commit with ledger updates. Market checkpoint/reorg holds persist atomically and bounded catch-up blocks new mutations. Order/listing reconciliation advances in durable, stable-ID keyset batches of at most 32 records per table, resumes after restart, and re-fences when the chain tip/revision changes. The HTTP process now uses a single worker thread to select/evaluate bounded row batches, read canonical facts using its own read-only Ledger connection, and atomically persist changes with cursor progress. Final TypeScript/Vite build and **178 offline tests passed**; focused market tests passed 27/27; the disposable PostgreSQL suite passed 31/31. A main-loop timer regression confirms a timer can run while a large worker backlog is still processing. In a same-fixture synthetic 10,000 confirmed-delivery comparison, worker processing took 4,088 ms with sampled event-loop-delay max 12.48 ms versus 5,853 ms and 55.67 ms synchronously on the development host. This is local evidence, not an SLO. Main-thread event-page acquisition/post-batch fencing, concurrent HTTP load, multi-process fencing, target hardware/data, production financial storage/recovery, retained-DAG capacity and operator SLO decisions remain open.
 
 **Next financial-storage increment:** [ADR 003 — PostgreSQL](postgres-financial-storage.md) selects and integrates the custody SQL backend with existing HTTP callers, exact constrained accounting, explicit migration/runtime roles, source-freezing SQLite import and dump/restore rehearsal. The adapter remains on the fictitious sandbox domain; it is not a production deposit/withdrawal worker. SQLite node-only backup deliberately refuses PostgreSQL-bound financial state. Production hardware/SLO tests, coordinated backup/PITR/failover, real payment schema/worker controls and independent review remain open, so this does not close M2/M4 or authorize funds.
 

tests/market.test.ts

@@ -418,6 +418,34 @@ test('worker reads canonical facts from a separate read-only Ledger snapshot and
   } finally { verifyDb.close(); }
 });
 
+test('worker reconciliation yields to event-loop timers during a large durable backlog', async t => {
+  const f = await fixture(t, true, true, true), db = new DatabaseSync(join(f.dir, 'testnet', 'market.sqlite'));
+  try {
+    const insertListing = db.prepare('INSERT INTO listings VALUES(?,?,?,?)');
+    const insertOrder = db.prepare('INSERT INTO orders VALUES(?,?)');
+    db.exec('BEGIN IMMEDIATE');
+    try {
+      for (let i = 0; i < 3000; i++) {
+        const id = `60000000-0000-4000-8000-${(i + 1).toString(16).padStart(12, '0')}`;
+        const listingId = `70000000-0000-4000-8000-${(i + 1).toString(16).padStart(12, '0')}`;
+        const cardHash = (i + 1).toString(16).padStart(64, '0');
+        const listing = { id: listingId, cardHash, seller: alice.actor, sequence: 0, kind: 'fixed', status: 'cancelled' };
+        insertListing.run(listingId, cardHash, 'cancelled', JSON.stringify(listing));
+        insertOrder.run(id, JSON.stringify({ id, listingId, seller: alice.actor, buyer: bob.actor,
+          cardHash, sequence: 0, status: 'refunded' }));
+      }
+      db.exec('COMMIT');
+    } catch (error) { db.exec('ROLLBACK'); throw error; }
+  } finally { db.close(); }
+  f.market.browse();
+  assert.equal(f.market.isReconciliationWorkerBusy(), true);
+  let timerRan = false;
+  await new Promise<void>(resolve => setTimeout(() => { timerRan = true; resolve(); }, 0));
+  assert.equal(timerRan, true);
+  assert.equal(f.market.isReconciliationWorkerBusy(), true,
+    'large reconciliation should still be in flight while the main-loop timer has run');
+});
+
 test('USD and mock XMR accounting use bounded integer amounts and exact fee splitting', () => {
   assert.equal(parseUsd('15.01'), 1501); assert.equal(parseUsd('0.01'), 1);
   for (const invalid of ['0', '-1', '1e3', '1.001', ' 1', 'NaN', '1000001', '01.00']) assert.throws(() => parseUsd(invalid));