Bounded reconciliation increment

docs/production-plan.md

@@ -1,6 +1,6 @@
 # Production delivery plan
 
-Status: active engineering plan; **not launch or payment authorization**. Updated 2026-10-20. The objective is a production service with explicit guarantees, measurable limits and accountable operations—not relabeling the existing testnet. Existing implementations are reusable groundwork; findings are closed only by reviewed implementation and evidence.
+Status: active engineering plan; **not launch or payment authorization**. Updated 2026-10-21. The objective is a production service with explicit guarantees, measurable limits and accountable operations—not relabeling the existing testnet. Existing implementations are reusable groundwork; findings are closed only by reviewed implementation and evidence.
 
 ## 1. Decisions agreed with the operator
 
@@ -63,7 +63,7 @@ Acceptance: exact-amount property/boundary tests, versioned signature tests, zer
 
 Can run alongside P0/P1. Addresses SEC-01/SEC-06 rather than adding more fixed caps.
 
-**Implemented M2 increment, 2026-10-20:** [canonical indexes and durable market observation](canonical-indexes.md). Canonical ownership/transaction SQL indexes now serve NFT, deck, custody availability and mock-market queries; per-block undo and anchored connect/disconnect events commit with ledger updates. Market checkpoint/reorg holds persist atomically and bounded catch-up blocks new mutations. Final TypeScript/Vite build and 149 offline tests passed. A disposable 10,000-NFT SQL baseline is recorded, **not** a target-hardware SLO or complete M2 acceptance. Production financial storage, incremental order scheduling, retained-DAG capacity and operator dataset/latency targets remain open.
+**Implemented M2 increments, 2026-10-20–21:** [canonical indexes and durable market observation](canonical-indexes.md). Canonical ownership/transaction SQL indexes now serve NFT, deck, custody availability and mock-market queries; per-block undo and anchored connect/disconnect events commit with ledger updates. Market checkpoint/reorg holds persist atomically and bounded catch-up blocks new mutations. Order/listing reconciliation now advances in durable, stable-ID keyset batches of at most 128 records per table, resumes after restart, re-fences when the chain tip/revision changes, and runs from an HTTP lifecycle poller rather than relying only on page loads. Final TypeScript/Vite build and **174 offline tests passed**; focused market tests passed 23/23 and the disposable PostgreSQL integration suite passed 31/31. A disposable 10,000-NFT SQL baseline is recorded, **not** a target-hardware SLO or complete M2 acceptance. Queue isolation, target-scale throughput/latency, production financial storage/recovery, retained-DAG capacity and operator dataset/SLO targets remain open.
 
 **Next financial-storage increment:** [ADR 003 — PostgreSQL](postgres-financial-storage.md) selects and integrates the custody SQL backend with existing HTTP callers, exact constrained accounting, explicit migration/runtime roles, source-freezing SQLite import and dump/restore rehearsal. The adapter remains on the fictitious sandbox domain; it is not a production deposit/withdrawal worker. SQLite node-only backup deliberately refuses PostgreSQL-bound financial state. Production hardware/SLO tests, coordinated backup/PITR/failover, real payment schema/worker controls and independent review remain open, so this does not close M2/M4 or authorize funds.
 
@@ -72,7 +72,7 @@ Can run alongside P0/P1. Addresses SEC-01/SEC-06 rather than adding more fixed c
 **Deployment increment:** [persistent local custody stack](persistent-custody-sandbox.md) now starts PostgreSQL and the actual app together, with private TLS/SCRAM networking, separate owner/runtime roles, explicit migrations and scoped persistent credential/data volumes. A signed fictitious credit and custodian identity survived container/network removal and recreation; the server rejected plaintext authentication. Existing deployments and SQLite custody were not switched. Production certificate rotation, coordinated backup/restore and external operating review remain open; all XMR remains simulated.
 
 - Indexed canonical ownership, NFT/transfer lookup and paged wallet/explorer APIs: browser signing must not require loading all history. Tip-pinned queries detect reorgs and preserve complete ownership semantics.
-- Incremental market reconciliation from durable canonical/reorg events. Maintain rollback-safe indexes and a recoverable cursor; catch-up after restart must not miss delivery or payout holds. Bound worker time/concurrency; move expensive work off the request event loop where appropriate.
+- Incremental market reconciliation from durable canonical/reorg events. **Local bounded implementation delivered:** rollback-safe canonical event pages, durable 128-row order/listing keyset jobs, tip/revision re-fencing, restart continuation, mutation fencing, and background catch-up polling; regression tests cover appends/reorgs and injected SQLite failures. **Still open:** move synchronous SQLite batches off the HTTP event loop, coordinate a single worker across processes, measure lag/throughput and request latency at declared order/event volumes, and demonstrate overload/recovery limits before production claims.
 - A versioned production market/settlement schema with append-only financial events and balanced accounting entries for buyer liabilities, host revenue, author liabilities, seller liabilities, paid/refunded amounts and actual network fees. Derived views are rebuildable; financial history and idempotency keys are not discarded to reclaim space.
 - Durable inbox/outbox, processor attempts and reconciliation states; no network operation inside a long DB transaction. Store intent before external side effects; unknown POST outcomes stay quarantined until evidenced resolution. Per-market isolation, actor-scoped paged reads, queue/backpressure and tenant/global exposure budgets.
 - Database architecture ADR: retain a measured single-writer SQLite deployment if it meets declared limits, or choose a production DB/worker topology and supported migration. Do not claim horizontal safety merely by launching multiple workers against local state. Start with one authoritative writer per market.
@@ -155,7 +155,7 @@ Acceptance: signed release evidence packet; recorded unresolved accepted risks w
 
 First implementation increment: write the M0 threat model/ADR research questions and P1 XMR domain specification, then build exact amount/property tests. Do not spend the next increment adding another disconnected “live” scaffold. Each milestone must integrate with its actual callers, persistence, recovery, UI and failure tests before being described as delivered.
 
-Current increment status: the operator selected custodial implementation after [settlement research](settlement-feasibility.md). M1 has a signed v3 XMR-native mock path, with historical v2 USD orders preserved. [ADR 002](custodial-market-design.md) records the implemented isolated custody accounting/UI foundation; production custody is not complete. The next M2 infrastructure increment is [canonical indexing and durable reconciliation](canonical-indexes.md), now built and regression-tested. Next engineering gates include the remaining production accounting/worker envelope and the separately reviewed NFT lock/completion/cancellation specification (M3); no live Monero integration or consensus activation is authorized by passing these tests. Migration dry-run, final fee/risk policy, capacity SLOs and independent review remain open.
+Current increment status: the operator selected custodial implementation after [settlement research](settlement-feasibility.md). M1 has a signed v3 XMR-native mock path, with historical v2 USD orders preserved. [ADR 002](custodial-market-design.md) records the implemented isolated custody accounting/UI foundation; production custody is not complete. [Canonical indexing and durable reconciliation](canonical-indexes.md) now includes bounded, restartable, chain-revision-fenced market catch-up and has passed focused, offline, and disposable PostgreSQL regression suites. This closes one incremental-scheduling implementation gap, **not** M2 acceptance. Next gates include target-scale benchmarks, off-event-loop worker design, production accounting/recovery envelope and the separately reviewed NFT lock/completion/cancellation specification (M3); no live Monero integration or consensus activation is authorized by passing these tests. Migration dry-run, final fee/risk policy, capacity SLOs and independent review remain open.
 
 ## 5. Acceptance evidence and release discipline