Bounded reconciliation increment

docs/canonical-indexes.md

@@ -1,6 +1,6 @@
 # M2 increment — canonical indexes and recoverable market observation
 
-Status: implemented local infrastructure, **not full M2 acceptance or production authorization**. Evidence date: 2026-10-20. Related: [production plan](production-plan.md), [custodial design](custodial-market-design.md).
+Status: implemented local infrastructure, **not full M2 acceptance or production authorization**. Evidence date: 2026-10-21. Related: [production plan](production-plan.md), [custodial design](custodial-market-design.md).
 
 ## Delivered
 
@@ -9,13 +9,15 @@ Status: implemented local infrastructure, **not full M2 acceptance or production
 - Events contain sequence, connect/disconnect type, block identity/height/parent, transaction IDs and a hash chained to the previous event. An independently generated local stream ID binds the history; it is **not consensus data or a public proof of honest observation**. Pages contain at most 64 events and no article payloads. Financial/chain history is not silently pruned.
 - On first open, original blocks are replayed and independently proof/state validated before additive local index backfill. Later opens compare indexed block identity, ownership, transaction inclusion and event continuity/anchors to that validated chain. Inconsistent state fails closed; existing consensus version, genesis, signatures and free-transfer/trade rules are unchanged.
 - [Market reconciliation](../server/market/store.ts) uses indexed current delivery/ownership queries and a persisted `(stream ID, sequence, anchor)` checkpoint. Every observed disconnect of a previously completed order causes a durable review hold, even if delivery reappeared before polling. Review is never automatically repaid. Hold updates and checkpoint advancement commit together in the market database.
-- Catch-up consumes one bounded event page per reconciliation pass. While behind, ordinary market mutation/payout/expiry decisions stop; diagnostic/read access remains available and exposes event cursor/head/backlog. Periodic polling and subsequent requests advance catch-up. No claim of instant catch-up or background-thread isolation.
+- Order and listing review now uses stable-ID keyset batches capped at 128 rows each. A strict, durable job record stores stage, continuation ID, observed time, chain tip/revision, and the bounded canonical event page. Each batch commits row effects and its continuation atomically; the canonical event cursor advances only after both tables finish. A chain tip/revision change during a multi-batch scan rewinds the job and captures a fresh event page, preventing a mixed-view scan from being marked caught up. Completion checks the current event head so events appended during the scan remain behind the write fence.
+- While any job/event backlog exists, ordinary market mutations and time-dependent payout/expiry decisions stop; diagnostic/read access remains available. Subsequent requests advance bounded work, and the HTTP lifecycle has an unref'd 250ms poller that drives catch-up independently of user page loads. The poller is not a separately isolated worker: SQLite work is synchronous on the server event loop. There is no claim of instant catch-up, a throughput SLO, or horizontal-worker safety.
+- Reconciliation restart, mid-job event append/reorg fencing, disconnect review, and injected SQLite batch/cursor failures are tested. A failed batch leaves both order effects and durable keyset progress unchanged; restart resumes committed work. Restore mismatch (different stream, different history anchor or a cursor ahead of the restored ledger) is refused rather than resetting the cursor. Restore the matching stopped-node ledger/market backup; any alternative recovery requires reviewed evidence, not deleting a checkpoint.
 - Restore mismatch (different stream, different history anchor or a cursor ahead of the restored ledger) is refused rather than resetting the cursor. Restore the matching stopped-node ledger/market backup; any alternative recovery requires reviewed evidence, not deleting a checkpoint.
 - [HTTP NFT lookup and custody card checks](../server/testnet/http.ts) use the same index. Wallet deck pagination accepts an optional `tip` query; stale tips return 409. Existing callers without `tip` remain compatible. There is no new public event/admin endpoint, no payment observer and no NFT-lock signer.
 
 ## Tests and measurements
 
-The final Docker test-target build passed TypeScript/Vite compilation. All **149 tests passed, zero failures/skips**, including native RandomX multi-process HTTP/peer/restart and backup rehearsal. New failure evidence includes:
+The final Docker test-target build passed TypeScript/Vite compilation. All **174 offline tests passed, zero failures/skips**, including native RandomX multi-process HTTP/peer/restart and backup rehearsal. The focused market suite passed **23 tests**; the disposable PostgreSQL suite passed **31 tests**. New failure evidence includes:
 
 - indexed ownership/sequence versus full replay for transfers, multi-card trades and forks;
 - append/disconnect/reconnect ordering, side-branch/mempool exclusion, bounded defensive-copy pages and durable restart;
@@ -23,7 +25,9 @@ The final Docker test-target build passed TypeScript/Vite compilation. All **149
 - additive migration of a prior-schema copy, refusing corrupt ownership and event records;
 - production market adapter exercised with full snapshots forbidden;
 - multi-page catch-up and persisted restart checkpoint;
-- injected cursor-write failure rolling back the payout hold, followed by restart recovery;
+- 128-row order/listing keyset progress, restart continuation, and a new canonical event arriving during an in-flight job;
+- chain revision change fencing that resets continuation and refreshes the event page before the scan continues;
+- injected batch-write failure rolling back order effects and keyset progress, plus cursor-write failure rolling back checkpoint advancement while preserving committed review holds;
 - numeric cursor reuse with a different historical hash rejected;
 - real HTTP NFT results and stale-tip deck pagination.
 
@@ -44,6 +48,6 @@ These numbers are a reproducible **baseline**, not acceptance SLOs: host CPU/sto
 
 ## Remaining gates
 
-M2 still needs target-hardware/data/SLO decisions, market-order incremental scheduling rather than scanning every existing bounded order/listing, bounded worker/event-loop latency under load, production financial database/schema and measured recovery/retention/storage growth. Ledger PoW replay and DAG state memory still scale with retained history; indexed queries do not solve those costs. Startup validation checks undo-record counts but not every possible future undo mutation against a second reconstructed undo model; comprehensive corruption/fuzz coverage and independent review remain required.
+M2 still needs target-hardware/data/SLO decisions, populated long-journal/large-event benchmarks, bounded worker/event-loop latency under load, production financial database/schema and measured recovery/retention/storage growth. The new scheduler bounds work per pass, but does not yet provide queue isolation, multi-worker coordination, or a demonstrated production throughput envelope. Ledger PoW replay and DAG state memory still scale with retained history; indexed queries do not solve those costs. Startup validation checks undo-record counts but not every possible future undo mutation against a second reconstructed undo model; comprehensive corruption/fuzz coverage and independent review remain required.
 
 Custodial reservations still stop at `awaiting-lock`. This increment does **not** consume ordinary transfers as custodial settlement evidence: doing that would bypass the reviewed sale-lock/cancel protocol. Real Monero deposit/withdrawal integration, production treasury keys, host attestations and a reviewed versioned NFT lock/completion/cancellation specification remain blocked. No real funds, credentials, external processor or genesis change were used.

docs/production-plan.md

@@ -1,6 +1,6 @@
 # Production delivery plan
 
-Status: active engineering plan; **not launch or payment authorization**. Updated 2026-10-20. The objective is a production service with explicit guarantees, measurable limits and accountable operations—not relabeling the existing testnet. Existing implementations are reusable groundwork; findings are closed only by reviewed implementation and evidence.
+Status: active engineering plan; **not launch or payment authorization**. Updated 2026-10-21. The objective is a production service with explicit guarantees, measurable limits and accountable operations—not relabeling the existing testnet. Existing implementations are reusable groundwork; findings are closed only by reviewed implementation and evidence.
 
 ## 1. Decisions agreed with the operator
 
@@ -63,7 +63,7 @@ Acceptance: exact-amount property/boundary tests, versioned signature tests, zer
 
 Can run alongside P0/P1. Addresses SEC-01/SEC-06 rather than adding more fixed caps.
 
-**Implemented M2 increment, 2026-10-20:** [canonical indexes and durable market observation](canonical-indexes.md). Canonical ownership/transaction SQL indexes now serve NFT, deck, custody availability and mock-market queries; per-block undo and anchored connect/disconnect events commit with ledger updates. Market checkpoint/reorg holds persist atomically and bounded catch-up blocks new mutations. Final TypeScript/Vite build and 149 offline tests passed. A disposable 10,000-NFT SQL baseline is recorded, **not** a target-hardware SLO or complete M2 acceptance. Production financial storage, incremental order scheduling, retained-DAG capacity and operator dataset/latency targets remain open.
+**Implemented M2 increments, 2026-10-20–21:** [canonical indexes and durable market observation](canonical-indexes.md). Canonical ownership/transaction SQL indexes now serve NFT, deck, custody availability and mock-market queries; per-block undo and anchored connect/disconnect events commit with ledger updates. Market checkpoint/reorg holds persist atomically and bounded catch-up blocks new mutations. Order/listing reconciliation now advances in durable, stable-ID keyset batches of at most 128 records per table, resumes after restart, re-fences when the chain tip/revision changes, and runs from an HTTP lifecycle poller rather than relying only on page loads. Final TypeScript/Vite build and **174 offline tests passed**; focused market tests passed 23/23 and the disposable PostgreSQL integration suite passed 31/31. A disposable 10,000-NFT SQL baseline is recorded, **not** a target-hardware SLO or complete M2 acceptance. Queue isolation, target-scale throughput/latency, production financial storage/recovery, retained-DAG capacity and operator dataset/SLO targets remain open.
 
 **Next financial-storage increment:** [ADR 003 — PostgreSQL](postgres-financial-storage.md) selects and integrates the custody SQL backend with existing HTTP callers, exact constrained accounting, explicit migration/runtime roles, source-freezing SQLite import and dump/restore rehearsal. The adapter remains on the fictitious sandbox domain; it is not a production deposit/withdrawal worker. SQLite node-only backup deliberately refuses PostgreSQL-bound financial state. Production hardware/SLO tests, coordinated backup/PITR/failover, real payment schema/worker controls and independent review remain open, so this does not close M2/M4 or authorize funds.
 
@@ -72,7 +72,7 @@ Can run alongside P0/P1. Addresses SEC-01/SEC-06 rather than adding more fixed c
 **Deployment increment:** [persistent local custody stack](persistent-custody-sandbox.md) now starts PostgreSQL and the actual app together, with private TLS/SCRAM networking, separate owner/runtime roles, explicit migrations and scoped persistent credential/data volumes. A signed fictitious credit and custodian identity survived container/network removal and recreation; the server rejected plaintext authentication. Existing deployments and SQLite custody were not switched. Production certificate rotation, coordinated backup/restore and external operating review remain open; all XMR remains simulated.
 
 - Indexed canonical ownership, NFT/transfer lookup and paged wallet/explorer APIs: browser signing must not require loading all history. Tip-pinned queries detect reorgs and preserve complete ownership semantics.
-- Incremental market reconciliation from durable canonical/reorg events. Maintain rollback-safe indexes and a recoverable cursor; catch-up after restart must not miss delivery or payout holds. Bound worker time/concurrency; move expensive work off the request event loop where appropriate.
+- Incremental market reconciliation from durable canonical/reorg events. **Local bounded implementation delivered:** rollback-safe canonical event pages, durable 128-row order/listing keyset jobs, tip/revision re-fencing, restart continuation, mutation fencing, and background catch-up polling; regression tests cover appends/reorgs and injected SQLite failures. **Still open:** move synchronous SQLite batches off the HTTP event loop, coordinate a single worker across processes, measure lag/throughput and request latency at declared order/event volumes, and demonstrate overload/recovery limits before production claims.
 - A versioned production market/settlement schema with append-only financial events and balanced accounting entries for buyer liabilities, host revenue, author liabilities, seller liabilities, paid/refunded amounts and actual network fees. Derived views are rebuildable; financial history and idempotency keys are not discarded to reclaim space.
 - Durable inbox/outbox, processor attempts and reconciliation states; no network operation inside a long DB transaction. Store intent before external side effects; unknown POST outcomes stay quarantined until evidenced resolution. Per-market isolation, actor-scoped paged reads, queue/backpressure and tenant/global exposure budgets.
 - Database architecture ADR: retain a measured single-writer SQLite deployment if it meets declared limits, or choose a production DB/worker topology and supported migration. Do not claim horizontal safety merely by launching multiple workers against local state. Start with one authoritative writer per market.
@@ -155,7 +155,7 @@ Acceptance: signed release evidence packet; recorded unresolved accepted risks w
 
 First implementation increment: write the M0 threat model/ADR research questions and P1 XMR domain specification, then build exact amount/property tests. Do not spend the next increment adding another disconnected “live” scaffold. Each milestone must integrate with its actual callers, persistence, recovery, UI and failure tests before being described as delivered.
 
-Current increment status: the operator selected custodial implementation after [settlement research](settlement-feasibility.md). M1 has a signed v3 XMR-native mock path, with historical v2 USD orders preserved. [ADR 002](custodial-market-design.md) records the implemented isolated custody accounting/UI foundation; production custody is not complete. The next M2 infrastructure increment is [canonical indexing and durable reconciliation](canonical-indexes.md), now built and regression-tested. Next engineering gates include the remaining production accounting/worker envelope and the separately reviewed NFT lock/completion/cancellation specification (M3); no live Monero integration or consensus activation is authorized by passing these tests. Migration dry-run, final fee/risk policy, capacity SLOs and independent review remain open.
+Current increment status: the operator selected custodial implementation after [settlement research](settlement-feasibility.md). M1 has a signed v3 XMR-native mock path, with historical v2 USD orders preserved. [ADR 002](custodial-market-design.md) records the implemented isolated custody accounting/UI foundation; production custody is not complete. [Canonical indexing and durable reconciliation](canonical-indexes.md) now includes bounded, restartable, chain-revision-fenced market catch-up and has passed focused, offline, and disposable PostgreSQL regression suites. This closes one incremental-scheduling implementation gap, **not** M2 acceptance. Next gates include target-scale benchmarks, off-event-loop worker design, production accounting/recovery envelope and the separately reviewed NFT lock/completion/cancellation specification (M3); no live Monero integration or consensus activation is authorized by passing these tests. Migration dry-run, final fee/risk policy, capacity SLOs and independent review remain open.
 
 ## 5. Acceptance evidence and release discipline