Reconciliation uses 32-row SQLite slices

docs/canonical-indexes.md

@@ -9,10 +9,9 @@ Status: implemented local infrastructure, **not full M2 acceptance or production
 - Events contain sequence, connect/disconnect type, block identity/height/parent, transaction IDs and a hash chained to the previous event. An independently generated local stream ID binds the history; it is **not consensus data or a public proof of honest observation**. Pages contain at most 64 events and no article payloads. Financial/chain history is not silently pruned.
 - On first open, original blocks are replayed and independently proof/state validated before additive local index backfill. Later opens compare indexed block identity, ownership, transaction inclusion and event continuity/anchors to that validated chain. Inconsistent state fails closed; existing consensus version, genesis, signatures and free-transfer/trade rules are unchanged.
 - [Market reconciliation](../server/market/store.ts) uses indexed current delivery/ownership queries and a persisted `(stream ID, sequence, anchor)` checkpoint. Every observed disconnect of a previously completed order causes a durable review hold, even if delivery reappeared before polling. Review is never automatically repaid. Hold updates and checkpoint advancement commit together in the market database.
-- Order and listing review now uses stable-ID keyset batches capped at 128 rows each. A strict, durable job record stores stage, continuation ID, observed time, chain tip/revision, and the bounded canonical event page. Each batch commits row effects and its continuation atomically; the canonical event cursor advances only after both tables finish. A chain tip/revision change during a multi-batch scan rewinds the job and captures a fresh event page, preventing a mixed-view scan from being marked caught up. Completion checks the current event head so events appended during the scan remain behind the write fence.
-- While any job/event backlog exists, ordinary market mutations and time-dependent payout/expiry decisions stop; diagnostic/read access remains available. Subsequent requests advance bounded work, and the HTTP lifecycle has an unref'd 250ms poller that drives catch-up independently of user page loads. The poller is not a separately isolated worker: SQLite work is synchronous on the server event loop. There is no claim of instant catch-up, a throughput SLO, or horizontal-worker safety.
+- Order and listing review now uses stable-ID keyset batches capped at 32 rows each. A strict, durable job record stores stage, continuation ID, observed time, chain tip/revision, and the bounded canonical event page. Each batch commits row effects and its continuation atomically; the canonical event cursor advances only after both tables finish. A chain tip/revision change during a multi-batch scan rewinds the job and captures a fresh event page, preventing a mixed-view scan from being marked caught up. Completion checks the current event head so events appended during the scan remain behind the write fence.
+- While any job/event backlog exists, ordinary market mutations and time-dependent payout/expiry decisions stop; diagnostic/read access remains available. Requests advance one bounded slice, and an unref'd HTTP-lifecycle pump drains remaining work through `setImmediate` turns without depending on page loads. This is cooperative yielding, **not a worker thread**: one slice still runs synchronous SQLite work on the server event loop. There is no claim of instant catch-up, a throughput SLO, or horizontal-worker safety.
 - Reconciliation restart, mid-job event append/reorg fencing, disconnect review, and injected SQLite batch/cursor failures are tested. A failed batch leaves both order effects and durable keyset progress unchanged; restart resumes committed work. Restore mismatch (different stream, different history anchor or a cursor ahead of the restored ledger) is refused rather than resetting the cursor. Restore the matching stopped-node ledger/market backup; any alternative recovery requires reviewed evidence, not deleting a checkpoint.
-- Restore mismatch (different stream, different history anchor or a cursor ahead of the restored ledger) is refused rather than resetting the cursor. Restore the matching stopped-node ledger/market backup; any alternative recovery requires reviewed evidence, not deleting a checkpoint.
 - [HTTP NFT lookup and custody card checks](../server/testnet/http.ts) use the same index. Wallet deck pagination accepts an optional `tip` query; stale tips return 409. Existing callers without `tip` remain compatible. There is no new public event/admin endpoint, no payment observer and no NFT-lock signer.
 
 ## Tests and measurements
@@ -25,7 +24,7 @@ The final Docker test-target build passed TypeScript/Vite compilation. All **174
 - additive migration of a prior-schema copy, refusing corrupt ownership and event records;
 - production market adapter exercised with full snapshots forbidden;
 - multi-page catch-up and persisted restart checkpoint;
-- 128-row order/listing keyset progress, restart continuation, and a new canonical event arriving during an in-flight job;
+- 32-row order/listing keyset progress, restart continuation, and a new canonical event arriving during an in-flight job;
 - chain revision change fencing that resets continuation and refreshes the event page before the scan continues;
 - injected batch-write failure rolling back order effects and keyset progress, plus cursor-write failure rolling back checkpoint advancement while preserving committed review holds;
 - numeric cursor reuse with a different historical hash rejected;
@@ -46,7 +45,7 @@ Observed in the development Docker environment:
 
 These numbers are a reproducible **baseline**, not acceptance SLOs: host CPU/storage and production concurrency were not selected, outputs include synthetic article payloads, warm caches apply, one transaction amortizes insertion fsync, and real validation/startup/HTTP/market workloads are excluded. They do not establish a supported production envelope.
 
-The synthetic market-journal harness [benchmark](../scripts/benchmark-market-reconciliation.ts), available as `npm run benchmark:market -- 10000 [terminal|confirmed-delivery]`, measures two bounded reconciliation paths in its own temporary SQLite directory. On the same development Docker environment, 10,000 terminal orders plus 10,000 listings completed in 776 ms (about 25,770 records/s), with pass-latency p50/p95/max of 4.56/8.03/17.73 ms, sampled event-loop-delay p95/max of 16.70/19.55 ms at 10ms resolution, and an 11,927,552-byte SQLite file. The synthetic confirmed-delivery scenario transitioned 10,000 paid orders and reserved listings to complete/sold in 2,031 ms (about 9,846 records/s), with pass-latency p50/p95/max of 12.27/24.75/35.59 ms, sampled event-loop-delay p95/max of 28.77/48.46 ms, and a 12,832,768-byte SQLite file. Its canonical transaction IDs come from an in-memory adapter fixture, not validated Ledger evidence. The event-loop samples demonstrate synchronous batch stalls locally; yielding between benchmark passes allows the monitor to sample but does not offload production work. These are not target-hardware throughput/SLO claims and exclude concurrent HTTP traffic, other server work, PostgreSQL, actual chain validation and adversarial workloads.
+The synthetic market-journal harness [benchmark](../scripts/benchmark-market-reconciliation.ts), available as `npm run benchmark:market -- 10000 [terminal|confirmed-delivery]`, measures two bounded reconciliation paths in its own temporary SQLite directory. With 32-row slices, the development Docker host processed 10,000 terminal orders plus 10,000 listings in 2,549 ms (about 7,845 records/s), with pass-latency p50/p95/max of 3.44/6.48/42.13 ms, sampled event-loop-delay p95/max of 14.97/48.99 ms at 10ms resolution, and an 11,927,552-byte SQLite file. The confirmed-delivery fixture transitioned 10,000 paid orders and reserved listings to complete/sold in 4,667 ms (about 4,286 records/s), with pass-latency p50/p95/max of 6.88/14.30/62.12 ms, sampled event-loop-delay p95/max of 20.63/62.23 ms, and a 12,832,768-byte SQLite file. The 32-row size increases pass count (625 vs. 157 at 128 rows) and reduces typical delay, but these runs still show long-tail event-loop stalls; a smaller batch is not equivalent to worker isolation. The canonical transaction IDs are in-memory adapter fixtures, not validated Ledger evidence. These are not target-hardware throughput/SLO claims and exclude concurrent HTTP traffic, other server work, PostgreSQL, actual chain validation and adversarial workloads.
 
 ## Remaining gates
 

docs/production-plan.md

@@ -63,7 +63,7 @@ Acceptance: exact-amount property/boundary tests, versioned signature tests, zer
 
 Can run alongside P0/P1. Addresses SEC-01/SEC-06 rather than adding more fixed caps.
 
-**Implemented M2 increments, 2026-10-20–21:** [canonical indexes and durable market observation](canonical-indexes.md). Canonical ownership/transaction SQL indexes now serve NFT, deck, custody availability and mock-market queries; per-block undo and anchored connect/disconnect events commit with ledger updates. Market checkpoint/reorg holds persist atomically and bounded catch-up blocks new mutations. Order/listing reconciliation now advances in durable, stable-ID keyset batches of at most 128 records per table, resumes after restart, re-fences when the chain tip/revision changes, and runs from an HTTP lifecycle poller rather than relying only on page loads. Final TypeScript/Vite build and **174 offline tests passed**; focused market tests passed 23/23 and the disposable PostgreSQL integration suite passed 31/31. Synthetic 10,000-order/listing runs processed terminal records in 776 ms (event-loop-delay max 19.55 ms) and paid orders with confirmed-delivery adapter fixtures in 2,031 ms (max 48.46 ms); these measurements show synchronous batches can block the local loop, but are not target-hardware SLOs or complete M2 acceptance. Queue isolation, target-scale/concurrent throughput and latency, production financial storage/recovery, retained-DAG capacity and operator dataset/SLO targets remain open.
+**Implemented M2 increments, 2026-10-20–21:** [canonical indexes and durable market observation](canonical-indexes.md). Canonical ownership/transaction SQL indexes now serve NFT, deck, custody availability and mock-market queries; per-block undo and anchored connect/disconnect events commit with ledger updates. Market checkpoint/reorg holds persist atomically and bounded catch-up blocks new mutations. Order/listing reconciliation advances in durable, stable-ID keyset batches of at most 32 records per table, resumes after restart, re-fences when the chain tip/revision changes, and drains through a yielding HTTP-lifecycle pump rather than relying only on page loads. Final TypeScript/Vite build and **174 offline tests passed**; focused market tests passed 23/23 and the disposable PostgreSQL integration suite passed 31/31. Synthetic 10,000-order/listing runs processed terminal records in 2,549 ms (event-loop-delay p95/max 14.97/48.99 ms) and paid orders with confirmed-delivery adapter fixtures in 4,667 ms (20.63/62.23 ms). Slicing reduces typical per-pass delay but increases total work and still has long-tail stalls; it is not worker-thread isolation or target-hardware evidence. Queue isolation, target-scale/concurrent throughput and latency, production financial storage/recovery, retained-DAG capacity and operator dataset/SLO targets remain open.
 
 **Next financial-storage increment:** [ADR 003 — PostgreSQL](postgres-financial-storage.md) selects and integrates the custody SQL backend with existing HTTP callers, exact constrained accounting, explicit migration/runtime roles, source-freezing SQLite import and dump/restore rehearsal. The adapter remains on the fictitious sandbox domain; it is not a production deposit/withdrawal worker. SQLite node-only backup deliberately refuses PostgreSQL-bound financial state. Production hardware/SLO tests, coordinated backup/PITR/failover, real payment schema/worker controls and independent review remain open, so this does not close M2/M4 or authorize funds.
 
@@ -72,7 +72,7 @@ Can run alongside P0/P1. Addresses SEC-01/SEC-06 rather than adding more fixed c
 **Deployment increment:** [persistent local custody stack](persistent-custody-sandbox.md) now starts PostgreSQL and the actual app together, with private TLS/SCRAM networking, separate owner/runtime roles, explicit migrations and scoped persistent credential/data volumes. A signed fictitious credit and custodian identity survived container/network removal and recreation; the server rejected plaintext authentication. Existing deployments and SQLite custody were not switched. Production certificate rotation, coordinated backup/restore and external operating review remain open; all XMR remains simulated.
 
 - Indexed canonical ownership, NFT/transfer lookup and paged wallet/explorer APIs: browser signing must not require loading all history. Tip-pinned queries detect reorgs and preserve complete ownership semantics.
-- Incremental market reconciliation from durable canonical/reorg events. **Local bounded implementation delivered:** rollback-safe canonical event pages, durable 128-row order/listing keyset jobs, tip/revision re-fencing, restart continuation, mutation fencing, and background catch-up polling; regression tests cover appends/reorgs and injected SQLite failures. **Still open:** move synchronous SQLite batches off the HTTP event loop, coordinate a single worker across processes, measure lag/throughput and request latency at declared order/event volumes, and demonstrate overload/recovery limits before production claims.
+- Incremental market reconciliation from durable canonical/reorg events. **Local bounded implementation delivered:** rollback-safe canonical event pages, durable 32-row order/listing keyset jobs, tip/revision re-fencing, restart continuation, mutation fencing, and a cooperative background pump that yields between batches; regression tests cover appends/reorgs and injected SQLite failures. **Still open:** true worker-thread isolation (the pump still runs synchronous SQLite on the main event loop), coordination of a single worker across processes, and lag/throughput/request-latency plus overload/recovery measurements at declared order/event volumes.
 - A versioned production market/settlement schema with append-only financial events and balanced accounting entries for buyer liabilities, host revenue, author liabilities, seller liabilities, paid/refunded amounts and actual network fees. Derived views are rebuildable; financial history and idempotency keys are not discarded to reclaim space.
 - Durable inbox/outbox, processor attempts and reconciliation states; no network operation inside a long DB transaction. Store intent before external side effects; unknown POST outcomes stay quarantined until evidenced resolution. Per-market isolation, actor-scoped paged reads, queue/backpressure and tenant/global exposure budgets.
 - Database architecture ADR: retain a measured single-writer SQLite deployment if it meets declared limits, or choose a production DB/worker topology and supported migration. Do not claim horizontal safety merely by launching multiple workers against local state. Start with one authoritative writer per market.