From a127a402933bc9c84e467d39d8d5beb4b66a8340 Mon Sep 17 00:00:00 2001 From: Eric Meehan Date: Wed, 21 Oct 2026 10:29:45 -0400 Subject: [PATCH] Reconciliation uses 32-row SQLite slices --- docs/canonical-indexes.md | 9 ++++----- docs/production-plan.md | 4 ++-- scripts/benchmark-market-reconciliation.ts | 6 +++--- server/market/store.ts | 11 ++++++----- server/testnet/http.ts | 22 +++++++++++++++++----- tests/market.test.ts | 19 ++++++++++++------- 6 files changed, 44 insertions(+), 27 deletions(-) diff --git a/docs/canonical-indexes.md b/docs/canonical-indexes.md index cf880c7..77fc000 100755 --- a/docs/canonical-indexes.md +++ b/docs/canonical-indexes.md @@ -9,10 +9,9 @@ Status: implemented local infrastructure, **not full M2 acceptance or production - Events contain sequence, connect/disconnect type, block identity/height/parent, transaction IDs and a hash chained to the previous event. An independently generated local stream ID binds the history; it is **not consensus data or a public proof of honest observation**. Pages contain at most 64 events and no article payloads. Financial/chain history is not silently pruned. - On first open, original blocks are replayed and independently proof/state validated before additive local index backfill. Later opens compare indexed block identity, ownership, transaction inclusion and event continuity/anchors to that validated chain. Inconsistent state fails closed; existing consensus version, genesis, signatures and free-transfer/trade rules are unchanged. - [Market reconciliation](../server/market/store.ts) uses indexed current delivery/ownership queries and a persisted `(stream ID, sequence, anchor)` checkpoint. Every observed disconnect of a previously completed order causes a durable review hold, even if delivery reappeared before polling. Review is never automatically repaid. Hold updates and checkpoint advancement commit together in the market database. -- Order and listing review now uses stable-ID keyset batches capped at 128 rows each. A strict, durable job record stores stage, continuation ID, observed time, chain tip/revision, and the bounded canonical event page. Each batch commits row effects and its continuation atomically; the canonical event cursor advances only after both tables finish. A chain tip/revision change during a multi-batch scan rewinds the job and captures a fresh event page, preventing a mixed-view scan from being marked caught up. Completion checks the current event head so events appended during the scan remain behind the write fence. -- While any job/event backlog exists, ordinary market mutations and time-dependent payout/expiry decisions stop; diagnostic/read access remains available. Subsequent requests advance bounded work, and the HTTP lifecycle has an unref'd 250ms poller that drives catch-up independently of user page loads. The poller is not a separately isolated worker: SQLite work is synchronous on the server event loop. There is no claim of instant catch-up, a throughput SLO, or horizontal-worker safety. +- Order and listing review now uses stable-ID keyset batches capped at 32 rows each. A strict, durable job record stores stage, continuation ID, observed time, chain tip/revision, and the bounded canonical event page. Each batch commits row effects and its continuation atomically; the canonical event cursor advances only after both tables finish. A chain tip/revision change during a multi-batch scan rewinds the job and captures a fresh event page, preventing a mixed-view scan from being marked caught up. Completion checks the current event head so events appended during the scan remain behind the write fence. +- While any job/event backlog exists, ordinary market mutations and time-dependent payout/expiry decisions stop; diagnostic/read access remains available. Requests advance one bounded slice, and an unref'd HTTP-lifecycle pump drains remaining work through `setImmediate` turns without depending on page loads. This is cooperative yielding, **not a worker thread**: one slice still runs synchronous SQLite work on the server event loop. There is no claim of instant catch-up, a throughput SLO, or horizontal-worker safety. - Reconciliation restart, mid-job event append/reorg fencing, disconnect review, and injected SQLite batch/cursor failures are tested. A failed batch leaves both order effects and durable keyset progress unchanged; restart resumes committed work. Restore mismatch (different stream, different history anchor or a cursor ahead of the restored ledger) is refused rather than resetting the cursor. Restore the matching stopped-node ledger/market backup; any alternative recovery requires reviewed evidence, not deleting a checkpoint. -- Restore mismatch (different stream, different history anchor or a cursor ahead of the restored ledger) is refused rather than resetting the cursor. Restore the matching stopped-node ledger/market backup; any alternative recovery requires reviewed evidence, not deleting a checkpoint. - [HTTP NFT lookup and custody card checks](../server/testnet/http.ts) use the same index. Wallet deck pagination accepts an optional `tip` query; stale tips return 409. Existing callers without `tip` remain compatible. There is no new public event/admin endpoint, no payment observer and no NFT-lock signer. ## Tests and measurements @@ -25,7 +24,7 @@ The final Docker test-target build passed TypeScript/Vite compilation. All **174 - additive migration of a prior-schema copy, refusing corrupt ownership and event records; - production market adapter exercised with full snapshots forbidden; - multi-page catch-up and persisted restart checkpoint; -- 128-row order/listing keyset progress, restart continuation, and a new canonical event arriving during an in-flight job; +- 32-row order/listing keyset progress, restart continuation, and a new canonical event arriving during an in-flight job; - chain revision change fencing that resets continuation and refreshes the event page before the scan continues; - injected batch-write failure rolling back order effects and keyset progress, plus cursor-write failure rolling back checkpoint advancement while preserving committed review holds; - numeric cursor reuse with a different historical hash rejected; @@ -46,7 +45,7 @@ Observed in the development Docker environment: These numbers are a reproducible **baseline**, not acceptance SLOs: host CPU/storage and production concurrency were not selected, outputs include synthetic article payloads, warm caches apply, one transaction amortizes insertion fsync, and real validation/startup/HTTP/market workloads are excluded. They do not establish a supported production envelope. -The synthetic market-journal harness [benchmark](../scripts/benchmark-market-reconciliation.ts), available as `npm run benchmark:market -- 10000 [terminal|confirmed-delivery]`, measures two bounded reconciliation paths in its own temporary SQLite directory. On the same development Docker environment, 10,000 terminal orders plus 10,000 listings completed in 776 ms (about 25,770 records/s), with pass-latency p50/p95/max of 4.56/8.03/17.73 ms, sampled event-loop-delay p95/max of 16.70/19.55 ms at 10ms resolution, and an 11,927,552-byte SQLite file. The synthetic confirmed-delivery scenario transitioned 10,000 paid orders and reserved listings to complete/sold in 2,031 ms (about 9,846 records/s), with pass-latency p50/p95/max of 12.27/24.75/35.59 ms, sampled event-loop-delay p95/max of 28.77/48.46 ms, and a 12,832,768-byte SQLite file. Its canonical transaction IDs come from an in-memory adapter fixture, not validated Ledger evidence. The event-loop samples demonstrate synchronous batch stalls locally; yielding between benchmark passes allows the monitor to sample but does not offload production work. These are not target-hardware throughput/SLO claims and exclude concurrent HTTP traffic, other server work, PostgreSQL, actual chain validation and adversarial workloads. +The synthetic market-journal harness [benchmark](../scripts/benchmark-market-reconciliation.ts), available as `npm run benchmark:market -- 10000 [terminal|confirmed-delivery]`, measures two bounded reconciliation paths in its own temporary SQLite directory. With 32-row slices, the development Docker host processed 10,000 terminal orders plus 10,000 listings in 2,549 ms (about 7,845 records/s), with pass-latency p50/p95/max of 3.44/6.48/42.13 ms, sampled event-loop-delay p95/max of 14.97/48.99 ms at 10ms resolution, and an 11,927,552-byte SQLite file. The confirmed-delivery fixture transitioned 10,000 paid orders and reserved listings to complete/sold in 4,667 ms (about 4,286 records/s), with pass-latency p50/p95/max of 6.88/14.30/62.12 ms, sampled event-loop-delay p95/max of 20.63/62.23 ms, and a 12,832,768-byte SQLite file. The 32-row size increases pass count (625 vs. 157 at 128 rows) and reduces typical delay, but these runs still show long-tail event-loop stalls; a smaller batch is not equivalent to worker isolation. The canonical transaction IDs are in-memory adapter fixtures, not validated Ledger evidence. These are not target-hardware throughput/SLO claims and exclude concurrent HTTP traffic, other server work, PostgreSQL, actual chain validation and adversarial workloads. ## Remaining gates diff --git a/docs/production-plan.md b/docs/production-plan.md index 4a074a8..da94781 100755 --- a/docs/production-plan.md +++ b/docs/production-plan.md @@ -63,7 +63,7 @@ Acceptance: exact-amount property/boundary tests, versioned signature tests, zer Can run alongside P0/P1. Addresses SEC-01/SEC-06 rather than adding more fixed caps. -**Implemented M2 increments, 2026-10-20–21:** [canonical indexes and durable market observation](canonical-indexes.md). Canonical ownership/transaction SQL indexes now serve NFT, deck, custody availability and mock-market queries; per-block undo and anchored connect/disconnect events commit with ledger updates. Market checkpoint/reorg holds persist atomically and bounded catch-up blocks new mutations. Order/listing reconciliation now advances in durable, stable-ID keyset batches of at most 128 records per table, resumes after restart, re-fences when the chain tip/revision changes, and runs from an HTTP lifecycle poller rather than relying only on page loads. Final TypeScript/Vite build and **174 offline tests passed**; focused market tests passed 23/23 and the disposable PostgreSQL integration suite passed 31/31. Synthetic 10,000-order/listing runs processed terminal records in 776 ms (event-loop-delay max 19.55 ms) and paid orders with confirmed-delivery adapter fixtures in 2,031 ms (max 48.46 ms); these measurements show synchronous batches can block the local loop, but are not target-hardware SLOs or complete M2 acceptance. Queue isolation, target-scale/concurrent throughput and latency, production financial storage/recovery, retained-DAG capacity and operator dataset/SLO targets remain open. +**Implemented M2 increments, 2026-10-20–21:** [canonical indexes and durable market observation](canonical-indexes.md). Canonical ownership/transaction SQL indexes now serve NFT, deck, custody availability and mock-market queries; per-block undo and anchored connect/disconnect events commit with ledger updates. Market checkpoint/reorg holds persist atomically and bounded catch-up blocks new mutations. Order/listing reconciliation advances in durable, stable-ID keyset batches of at most 32 records per table, resumes after restart, re-fences when the chain tip/revision changes, and drains through a yielding HTTP-lifecycle pump rather than relying only on page loads. Final TypeScript/Vite build and **174 offline tests passed**; focused market tests passed 23/23 and the disposable PostgreSQL integration suite passed 31/31. Synthetic 10,000-order/listing runs processed terminal records in 2,549 ms (event-loop-delay p95/max 14.97/48.99 ms) and paid orders with confirmed-delivery adapter fixtures in 4,667 ms (20.63/62.23 ms). Slicing reduces typical per-pass delay but increases total work and still has long-tail stalls; it is not worker-thread isolation or target-hardware evidence. Queue isolation, target-scale/concurrent throughput and latency, production financial storage/recovery, retained-DAG capacity and operator dataset/SLO targets remain open. **Next financial-storage increment:** [ADR 003 — PostgreSQL](postgres-financial-storage.md) selects and integrates the custody SQL backend with existing HTTP callers, exact constrained accounting, explicit migration/runtime roles, source-freezing SQLite import and dump/restore rehearsal. The adapter remains on the fictitious sandbox domain; it is not a production deposit/withdrawal worker. SQLite node-only backup deliberately refuses PostgreSQL-bound financial state. Production hardware/SLO tests, coordinated backup/PITR/failover, real payment schema/worker controls and independent review remain open, so this does not close M2/M4 or authorize funds. @@ -72,7 +72,7 @@ Can run alongside P0/P1. Addresses SEC-01/SEC-06 rather than adding more fixed c **Deployment increment:** [persistent local custody stack](persistent-custody-sandbox.md) now starts PostgreSQL and the actual app together, with private TLS/SCRAM networking, separate owner/runtime roles, explicit migrations and scoped persistent credential/data volumes. A signed fictitious credit and custodian identity survived container/network removal and recreation; the server rejected plaintext authentication. Existing deployments and SQLite custody were not switched. Production certificate rotation, coordinated backup/restore and external operating review remain open; all XMR remains simulated. - Indexed canonical ownership, NFT/transfer lookup and paged wallet/explorer APIs: browser signing must not require loading all history. Tip-pinned queries detect reorgs and preserve complete ownership semantics. -- Incremental market reconciliation from durable canonical/reorg events. **Local bounded implementation delivered:** rollback-safe canonical event pages, durable 128-row order/listing keyset jobs, tip/revision re-fencing, restart continuation, mutation fencing, and background catch-up polling; regression tests cover appends/reorgs and injected SQLite failures. **Still open:** move synchronous SQLite batches off the HTTP event loop, coordinate a single worker across processes, measure lag/throughput and request latency at declared order/event volumes, and demonstrate overload/recovery limits before production claims. +- Incremental market reconciliation from durable canonical/reorg events. **Local bounded implementation delivered:** rollback-safe canonical event pages, durable 32-row order/listing keyset jobs, tip/revision re-fencing, restart continuation, mutation fencing, and a cooperative background pump that yields between batches; regression tests cover appends/reorgs and injected SQLite failures. **Still open:** true worker-thread isolation (the pump still runs synchronous SQLite on the main event loop), coordination of a single worker across processes, and lag/throughput/request-latency plus overload/recovery measurements at declared order/event volumes. - A versioned production market/settlement schema with append-only financial events and balanced accounting entries for buyer liabilities, host revenue, author liabilities, seller liabilities, paid/refunded amounts and actual network fees. Derived views are rebuildable; financial history and idempotency keys are not discarded to reclaim space. - Durable inbox/outbox, processor attempts and reconciliation states; no network operation inside a long DB transaction. Store intent before external side effects; unknown POST outcomes stay quarantined until evidenced resolution. Per-market isolation, actor-scoped paged reads, queue/backpressure and tenant/global exposure budgets. - Database architecture ADR: retain a measured single-writer SQLite deployment if it meets declared limits, or choose a production DB/worker topology and supported migration. Do not claim horizontal safety merely by launching multiple workers against local state. Start with one authoritative writer per market. diff --git a/scripts/benchmark-market-reconciliation.ts b/scripts/benchmark-market-reconciliation.ts index 79e1c7a..d3b7000 100755 --- a/scripts/benchmark-market-reconciliation.ts +++ b/scripts/benchmark-market-reconciliation.ts @@ -5,7 +5,7 @@ import { join } from 'node:path'; import { monitorEventLoopDelay, performance } from 'node:perf_hooks'; import { randomUUID } from 'node:crypto'; import { setImmediate as yieldToEventLoop } from 'node:timers/promises'; -import { Market } from '../server/market/store.ts'; +import { MARKET_RECONCILIATION_BATCH_SIZE, Market } from '../server/market/store.ts'; import { CHAIN_ID, GENESIS_HASH, transactionId } from '../server/testnet/consensus.ts'; import { NETWORK } from '../shared/testnet.ts'; @@ -62,7 +62,7 @@ try { } finally { db.close(); } const latencies: number[] = []; - const maxPasses = Math.ceil(count / 128) * 2 + 4; + const maxPasses = Math.ceil(count / MARKET_RECONCILIATION_BATCH_SIZE) * 2 + 4; const eventLoopDelay = monitorEventLoopDelay({ resolution: 10 }); eventLoopDelay.enable(); const start = performance.now(); @@ -81,7 +81,7 @@ try { latencies.sort((a, b) => a - b); const percentile = (p: number) => latencies[Math.min(latencies.length - 1, Math.ceil(latencies.length * p) - 1)]; console.log(JSON.stringify({ fixture: `synthetic-${scenario}-orders-and-listings`, orders: count, listings: count, - batchSizePerTable: 128, reconciliationPasses: latencies.length, elapsedMs, + batchSizePerTable: MARKET_RECONCILIATION_BATCH_SIZE, reconciliationPasses: latencies.length, elapsedMs, rowsPerSecond: Math.round((count * 2 * 1000) / elapsedMs), passLatencyP50Ms: percentile(0.50), passLatencyP95Ms: percentile(0.95), passLatencyMaxMs: latencies.at(-1), sqliteBytes: bytes, eventLoopDelayResolutionMs: 10, eventLoopDelayP95Ms: eventLoopDelay.percentile(95) / 1e6, diff --git a/server/market/store.ts b/server/market/store.ts index 3a25a1d..0f50670 100755 --- a/server/market/store.ts +++ b/server/market/store.ts @@ -52,7 +52,8 @@ const reconciliationJobSchema = z.object({ version: z.literal(1), stage: z.enum( afterId: z.string().nullable(), observedAt: z.number().int().nonnegative().safe(), tip: z.string().regex(/^[a-f0-9]{64}$/).nullable(), revision: z.number().int().nonnegative().safe().nullable(), eventPage: canonicalEventPageSchema.nullable() }).strict(); -const RECONCILE_BATCH_SIZE = 128; +// Keep each synchronous SQLite slice small enough to limit request-loop stalls. +export const MARKET_RECONCILIATION_BATCH_SIZE = 32; const RECONCILIATION_JOB_KEY = 'canonical_reconciliation_job'; /** Node-local marketplace, NOT consensus escrow. Its SQLite journal is backed up with the ledger. */ @@ -128,7 +129,7 @@ export class Market { } private nextReconciliationRows(table: 'listings' | 'orders', afterId: string | null): { id: string; data: string }[] { return this.db.prepare(`SELECT id,data FROM ${table} WHERE (? IS NULL OR id>?) ORDER BY id LIMIT ?`) - .all(afterId, afterId, RECONCILE_BATCH_SIZE) as { id: string; data: string }[]; + .all(afterId, afterId, MARKET_RECONCILIATION_BATCH_SIZE) as { id: string; data: string }[]; } private listing(id: string): MarketListingRecord { const row = this.db.prepare('SELECT data FROM listings WHERE id=?').get(id) as { data: string } | undefined; @@ -270,10 +271,10 @@ export class Market { } if (job.stage === 'orders') { - if (rows.length < RECONCILE_BATCH_SIZE) nextJob = { ...job, stage: 'listings', afterId: null }; + if (rows.length < MARKET_RECONCILIATION_BATCH_SIZE) nextJob = { ...job, stage: 'listings', afterId: null }; else nextJob = { ...job, afterId: rows.at(-1)!.id }; } else { - if (rows.length < RECONCILE_BATCH_SIZE) { + if (rows.length < MARKET_RECONCILIATION_BATCH_SIZE) { this.db.prepare('DELETE FROM receipts WHERE expires < ?').run(job.observedAt - 86400_000); if (eventPage) { this.db.prepare('INSERT INTO market_meta(key,value) VALUES(?,?) ON CONFLICT(key) DO UPDATE SET value=excluded.value') @@ -305,7 +306,7 @@ export class Market { } // Empty/short order pages can advance to the listing stage in the same bounded call. This // avoids rejecting the first command on an empty market while still capping work at 2×128 rows. - if (!completed && job.stage === 'orders' && rows.length < RECONCILE_BATCH_SIZE) this.reconcile(); + if (!completed && job.stage === 'orders' && rows.length < MARKET_RECONCILIATION_BATCH_SIZE) this.reconcile(); } browse(offset = 0): { listings: MarketListingRecord[]; next: number | null; info: MarketInfo } { if (!Number.isSafeInteger(offset) || offset < 0 || offset > 10000) throw new Error('Invalid marketplace page'); diff --git a/server/testnet/http.ts b/server/testnet/http.ts index 2664df6..daf08c8 100644 --- a/server/testnet/http.ts +++ b/server/testnet/http.ts @@ -251,17 +251,29 @@ server.setTimeout(30000,socket=>socket.destroy()); // Advance simulated payout/expiry state even when no browser is open. No real funds are sent. const marketTimer=setInterval(()=>{try {market.browse();}catch{console.error('Marketplace reconciliation failed; operator review required.');}},15000); marketTimer.unref(); -// When durable reconciliation is behind, advance bounded keyset pages without requiring a browser request. +let stopping=false; +let catchupPaused=false; +let catchupImmediate:NodeJS.Immediate|undefined; +// Drain durable catch-up without page-load dependence. Each pass is a smaller synchronous +// SQLite slice, then setImmediate yields so HTTP and peer work can run between passes. +const advanceMarketCatchup=()=>{ + catchupImmediate=undefined; + if(stopping||catchupPaused||!market.hasReconciliationBacklog())return; + try {market.browse();} + catch {catchupPaused=true;console.error('Marketplace canonical catch-up paused; operator review required.');return;} + if(market.hasReconciliationBacklog()){catchupImmediate=setImmediate(advanceMarketCatchup);catchupImmediate.unref();} +}; const marketCatchupTimer=setInterval(()=>{ - if(!market.hasReconciliationBacklog())return; - try {market.browse();}catch{console.error('Marketplace canonical catch-up failed; operator review required.');} + if(!stopping&&!catchupPaused&&!catchupImmediate&&market.hasReconciliationBacklog()){ + catchupImmediate=setImmediate(advanceMarketCatchup);catchupImmediate.unref(); + } },250); marketCatchupTimer.unref(); -let stopping=false; for(const signal of ['SIGTERM','SIGINT']as const)process.once(signal,()=>{ if(stopping)return;stopping=true;peers.close();operations.close();clearInterval(marketTimer);clearInterval(marketCatchupTimer); + if(catchupImmediate)clearImmediate(catchupImmediate); const observerDrain=custodyObserver?.close()??Promise.resolve(); const timeout=setTimeout(()=>process.exit(1),25000);timeout.unref(); server.close(async()=>{await observerDrain;await custody?.close();market.close();ledger.close();engine.close();await lock.close();await unlink(lockPath).catch(()=>{});clearTimeout(timeout);process.exit(0);}); }); -server.on('error',async()=>{clearInterval(marketTimer);clearInterval(marketCatchupTimer);operations.close();peers.close();await custodyObserver?.close();await custody?.close();market.close();ledger.close();engine.close();await lock.close();await unlink(lockPath).catch(()=>{});process.exit(1);}); +server.on('error',async()=>{stopping=true;clearInterval(marketTimer);clearInterval(marketCatchupTimer);if(catchupImmediate)clearImmediate(catchupImmediate);operations.close();peers.close();await custodyObserver?.close();await custody?.close();market.close();ledger.close();engine.close();await lock.close();await unlink(lockPath).catch(()=>{});process.exit(1);}); diff --git a/tests/market.test.ts b/tests/market.test.ts index 7b23f9b..30f2859 100755 --- a/tests/market.test.ts +++ b/tests/market.test.ts @@ -171,18 +171,23 @@ test('market reconciliation keyset progress is bounded, durable and resumes afte try { return JSON.parse((conn.prepare('SELECT value FROM market_meta WHERE key=?').get('canonical_reconciliation_job') as { value: string }).value); } finally { conn.close(); } }; - assert.equal(progress().stage, 'orders'); assert.equal(progress().afterId, ids[127]); + assert.equal(progress().stage, 'orders'); assert.equal(progress().afterId, ids[31]); assert.equal(f.market.hasReconciliationBacklog(), true); - await assert.rejects(f.list(), /catching up/); // second 128-order batch; no new listing is admitted. - assert.equal(progress().stage, 'orders'); assert.equal(progress().afterId, ids[255]); + await assert.rejects(f.list(), /catching up/); // Second 32-order batch; no new listing is admitted. + assert.equal(progress().stage, 'orders'); assert.equal(progress().afterId, ids[63]); // The ledger advances while this durable multi-batch job is in flight. f.events.push({ sequence: 2, type: 'connect', height: 2, hash: 'f'.repeat(64), previousHash: 'd'.repeat(64), transactionIds: [], eventHash: '2'.repeat(64) }); f.market.browse(); // Tip/revision changed: rewind and bind the updated event page. assert.equal(progress().eventPage.cursor, 2); - assert.equal(progress().afterId, ids[127]); - f.market.browse(); // Second order batch of the refreshed scan. - f.market.browse(); // Remaining four orders, then at most one listing page. + assert.equal(progress().afterId, ids[31]); + const tailStatus = () => { + const conn = new DatabaseSync(join(f.dir, 'testnet', 'market.sqlite')); + try { return JSON.parse((conn.prepare('SELECT data FROM orders WHERE id=?').get(ids[259]) as { data: string }).data).status; } + finally { conn.close(); } + }; + for (let i = 0; i < 12 && tailStatus() !== 'review'; i++) f.market.browse(); + assert.equal(tailStatus(), 'review'); // Disconnect reached the tail order before checkpoint advancement. assert.equal(f.market.diagnostics().canonicalEventCursor, 0); assert.equal(f.market.hasReconciliationBacklog(), true, 'new events appended during job must stay behind the work fence'); const reviewDb = new DatabaseSync(join(f.dir, 'testnet', 'market.sqlite')); @@ -194,7 +199,7 @@ test('market reconciliation keyset progress is bounded, durable and resumes afte // Simulate process death after committed batch progress, then resume using the SQLite job cursor. await f.restart(); assert.equal(f.market.diagnostics().eventBacklog, true); - for (let i = 0; i < 12 && f.market.hasReconciliationBacklog(); i++) f.market.browse(); + for (let i = 0; i < 24 && f.market.hasReconciliationBacklog(); i++) f.market.browse(); assert.equal(f.market.hasReconciliationBacklog(), false); const conn = new DatabaseSync(join(f.dir, 'testnet', 'market.sqlite')); try { assert.equal(conn.prepare('SELECT 1 FROM market_meta WHERE key=?').get('canonical_reconciliation_job'), undefined); }