From 74f269923cb2f7521677f3228663a74bd32a6394 Mon Sep 17 00:00:00 2001 From: Eric Meehan Date: Wed, 21 Oct 2026 12:24:53 -0400 Subject: [PATCH] Moved canonical transaction, NFT, and mempool reads into a separate read-only Ledger SQLite connection --- docs/canonical-indexes.md | 3 ++- docs/production-plan.md | 2 +- tests/market.test.ts | 28 ++++++++++++++++++++++++++++ 3 files changed, 31 insertions(+), 2 deletions(-) diff --git a/docs/canonical-indexes.md b/docs/canonical-indexes.md index 803e272..187267a 100755 --- a/docs/canonical-indexes.md +++ b/docs/canonical-indexes.md @@ -16,7 +16,7 @@ Status: implemented local infrastructure, **not full M2 acceptance or production ## Tests and measurements -The final Docker test-target build passed TypeScript/Vite compilation. All **177 offline tests passed, zero failures/skips**, including native RandomX multi-process HTTP/peer/restart and backup rehearsal. The focused market suite passed **26 tests**; the disposable PostgreSQL suite passed **31 tests** in the preceding integration run. New failure evidence includes: +The final Docker test-target build passed TypeScript/Vite compilation. All **178 offline tests passed, zero failures/skips**, including native RandomX multi-process HTTP/peer/restart and backup rehearsal. The focused market suite passed **27 tests**, and the disposable PostgreSQL suite passed **31/31**. - indexed ownership/sequence versus full replay for transfers, multi-card trades and forks; - append/disconnect/reconnect ordering, side-branch/mempool exclusion, bounded defensive-copy pages and durable restart; @@ -30,6 +30,7 @@ The final Docker test-target build passed TypeScript/Vite compilation. All **177 - worker-owned row evaluation/batch/finalize transaction with injected failure, mutation fence and restart continuation; - canonical event arrival during an in-flight worker batch forces a fresh bounded event page before the cursor reaches the new head; - separate worker read-only Ledger connection supplies canonical NFT and delivery facts and re-fences when an event arrives during that snapshot; +- main-thread timer fires while a large worker reconciliation backlog remains in flight; - numeric cursor reuse with a different historical hash rejected; - real HTTP NFT results and stale-tip deck pagination. diff --git a/docs/production-plan.md b/docs/production-plan.md index 5dabb23..95a5c69 100755 --- a/docs/production-plan.md +++ b/docs/production-plan.md @@ -63,7 +63,7 @@ Acceptance: exact-amount property/boundary tests, versioned signature tests, zer Can run alongside P0/P1. Addresses SEC-01/SEC-06 rather than adding more fixed caps. -**Implemented M2 increments, 2026-10-20–21:** [canonical indexes and durable market observation](canonical-indexes.md). Canonical ownership/transaction SQL indexes now serve NFT, deck, custody availability and mock-market queries; per-block undo and anchored connect/disconnect events commit with ledger updates. Market checkpoint/reorg holds persist atomically and bounded catch-up blocks new mutations. Order/listing reconciliation advances in durable, stable-ID keyset batches of at most 32 records per table, resumes after restart, and re-fences when the chain tip/revision changes. The HTTP process now uses a single worker thread to select/evaluate bounded row batches, read canonical facts using its own read-only Ledger connection, and atomically persist changes with cursor progress. Final TypeScript/Vite build and **177 offline tests passed**; focused market tests passed 26/26; the disposable PostgreSQL suite passed 31/31. In a same-fixture synthetic 10,000 confirmed-delivery comparison, worker processing took 4,088 ms with sampled event-loop-delay max 12.48 ms versus 5,853 ms and 55.67 ms synchronously on the development host. This is local evidence, not an SLO. Main-thread event-page acquisition/post-batch fencing, concurrent HTTP load, multi-process fencing, target hardware/data, production financial storage/recovery, retained-DAG capacity and operator SLO decisions remain open. +**Implemented M2 increments, 2026-10-20–21:** [canonical indexes and durable market observation](canonical-indexes.md). Canonical ownership/transaction SQL indexes now serve NFT, deck, custody availability and mock-market queries; per-block undo and anchored connect/disconnect events commit with ledger updates. Market checkpoint/reorg holds persist atomically and bounded catch-up blocks new mutations. Order/listing reconciliation advances in durable, stable-ID keyset batches of at most 32 records per table, resumes after restart, and re-fences when the chain tip/revision changes. The HTTP process now uses a single worker thread to select/evaluate bounded row batches, read canonical facts using its own read-only Ledger connection, and atomically persist changes with cursor progress. Final TypeScript/Vite build and **178 offline tests passed**; focused market tests passed 27/27; the disposable PostgreSQL suite passed 31/31. A main-loop timer regression confirms a timer can run while a large worker backlog is still processing. In a same-fixture synthetic 10,000 confirmed-delivery comparison, worker processing took 4,088 ms with sampled event-loop-delay max 12.48 ms versus 5,853 ms and 55.67 ms synchronously on the development host. This is local evidence, not an SLO. Main-thread event-page acquisition/post-batch fencing, concurrent HTTP load, multi-process fencing, target hardware/data, production financial storage/recovery, retained-DAG capacity and operator SLO decisions remain open. **Next financial-storage increment:** [ADR 003 — PostgreSQL](postgres-financial-storage.md) selects and integrates the custody SQL backend with existing HTTP callers, exact constrained accounting, explicit migration/runtime roles, source-freezing SQLite import and dump/restore rehearsal. The adapter remains on the fictitious sandbox domain; it is not a production deposit/withdrawal worker. SQLite node-only backup deliberately refuses PostgreSQL-bound financial state. Production hardware/SLO tests, coordinated backup/PITR/failover, real payment schema/worker controls and independent review remain open, so this does not close M2/M4 or authorize funds. diff --git a/tests/market.test.ts b/tests/market.test.ts index 31fb3e2..9029bbb 100755 --- a/tests/market.test.ts +++ b/tests/market.test.ts @@ -418,6 +418,34 @@ test('worker reads canonical facts from a separate read-only Ledger snapshot and } finally { verifyDb.close(); } }); +test('worker reconciliation yields to event-loop timers during a large durable backlog', async t => { + const f = await fixture(t, true, true, true), db = new DatabaseSync(join(f.dir, 'testnet', 'market.sqlite')); + try { + const insertListing = db.prepare('INSERT INTO listings VALUES(?,?,?,?)'); + const insertOrder = db.prepare('INSERT INTO orders VALUES(?,?)'); + db.exec('BEGIN IMMEDIATE'); + try { + for (let i = 0; i < 3000; i++) { + const id = `60000000-0000-4000-8000-${(i + 1).toString(16).padStart(12, '0')}`; + const listingId = `70000000-0000-4000-8000-${(i + 1).toString(16).padStart(12, '0')}`; + const cardHash = (i + 1).toString(16).padStart(64, '0'); + const listing = { id: listingId, cardHash, seller: alice.actor, sequence: 0, kind: 'fixed', status: 'cancelled' }; + insertListing.run(listingId, cardHash, 'cancelled', JSON.stringify(listing)); + insertOrder.run(id, JSON.stringify({ id, listingId, seller: alice.actor, buyer: bob.actor, + cardHash, sequence: 0, status: 'refunded' })); + } + db.exec('COMMIT'); + } catch (error) { db.exec('ROLLBACK'); throw error; } + } finally { db.close(); } + f.market.browse(); + assert.equal(f.market.isReconciliationWorkerBusy(), true); + let timerRan = false; + await new Promise(resolve => setTimeout(() => { timerRan = true; resolve(); }, 0)); + assert.equal(timerRan, true); + assert.equal(f.market.isReconciliationWorkerBusy(), true, + 'large reconciliation should still be in flight while the main-loop timer has run'); +}); + test('USD and mock XMR accounting use bounded integer amounts and exact fee splitting', () => { assert.equal(parseUsd('15.01'), 1501); assert.equal(parseUsd('0.01'), 1); for (const invalid of ['0', '-1', '1e3', '1.001', ' 1', 'NaN', '1000001', '01.00']) assert.throws(() => parseUsd(invalid));