Worker-owned reconciliation commits

docs/canonical-indexes.md

@@ -10,13 +10,13 @@ Status: implemented local infrastructure, **not full M2 acceptance or production
 - On first open, original blocks are replayed and independently proof/state validated before additive local index backfill. Later opens compare indexed block identity, ownership, transaction inclusion and event continuity/anchors to that validated chain. Inconsistent state fails closed; existing consensus version, genesis, signatures and free-transfer/trade rules are unchanged.
 - [Market reconciliation](../server/market/store.ts) uses indexed current delivery/ownership queries and a persisted `(stream ID, sequence, anchor)` checkpoint. Every observed disconnect of a previously completed order causes a durable review hold, even if delivery reappeared before polling. Review is never automatically repaid. Hold updates and checkpoint advancement commit together in the market database.
 - Order and listing review now uses stable-ID keyset batches capped at 32 rows each. A strict, durable job record stores stage, continuation ID, observed time, chain tip/revision, and the bounded canonical event page. Each batch commits row effects and its continuation atomically; the canonical event cursor advances only after both tables finish. A chain tip/revision change during a multi-batch scan rewinds the job and captures a fresh event page, preventing a mixed-view scan from being marked caught up. Completion checks the current event head so events appended during the scan remain behind the write fence.
-- While any job/event backlog exists, ordinary market mutations and time-dependent payout/expiry decisions stop; diagnostic/read access remains available. Requests advance one bounded slice, and an unref'd HTTP-lifecycle pump drains remaining work through `setImmediate` turns without depending on page loads. This is cooperative yielding, **not a worker thread**: one slice still runs synchronous SQLite work on the server event loop. There is no claim of instant catch-up, a throughput SLO, or horizontal-worker safety.
+- While any job/event backlog exists, ordinary market mutations and time-dependent payout/expiry decisions stop; diagnostic/read access remains available. Requests advance one bounded slice, and an unref'd HTTP-lifecycle pump drains remaining work without depending on page loads. The production HTTP process enables a single persistent Node worker thread to own atomic batch writes and final checkpoint commits; the main thread retains the chain queries, per-row reconciliation decisions and event/revision fences. This reduces SQLite write/commit work on the request loop but is **not complete CPU isolation**. Worker failure leaves the durable job/cursor intact, fails closed, and requires retry/restart; there is no horizontal-worker safety claim.
 - Reconciliation restart, mid-job event append/reorg fencing, disconnect review, and injected SQLite batch/cursor failures are tested. A failed batch leaves both order effects and durable keyset progress unchanged; restart resumes committed work. Restore mismatch (different stream, different history anchor or a cursor ahead of the restored ledger) is refused rather than resetting the cursor. Restore the matching stopped-node ledger/market backup; any alternative recovery requires reviewed evidence, not deleting a checkpoint.
 - [HTTP NFT lookup and custody card checks](../server/testnet/http.ts) use the same index. Wallet deck pagination accepts an optional `tip` query; stale tips return 409. Existing callers without `tip` remain compatible. There is no new public event/admin endpoint, no payment observer and no NFT-lock signer.
 
 ## Tests and measurements
 
-The final Docker test-target build passed TypeScript/Vite compilation. All **174 offline tests passed, zero failures/skips**, including native RandomX multi-process HTTP/peer/restart and backup rehearsal. The focused market suite passed **23 tests**; the disposable PostgreSQL suite passed **31 tests**. New failure evidence includes:
+The final Docker test-target build passed TypeScript/Vite compilation. All **175 offline tests passed, zero failures/skips**, including native RandomX multi-process HTTP/peer/restart and backup rehearsal. The focused market suite passed **24 tests**; the disposable PostgreSQL suite passed **31 tests** in the preceding integration run. New failure evidence includes:
 
 - indexed ownership/sequence versus full replay for transfers, multi-card trades and forks;
 - append/disconnect/reconnect ordering, side-branch/mempool exclusion, bounded defensive-copy pages and durable restart;
@@ -27,6 +27,7 @@ The final Docker test-target build passed TypeScript/Vite compilation. All **174
 - 32-row order/listing keyset progress, restart continuation, and a new canonical event arriving during an in-flight job;
 - chain revision change fencing that resets continuation and refreshes the event page before the scan continues;
 - injected batch-write failure rolling back order effects and keyset progress, plus cursor-write failure rolling back checkpoint advancement while preserving committed review holds;
+- worker-owned batch/finalize transaction with injected failure, mutation fence and restart continuation;
 - numeric cursor reuse with a different historical hash rejected;
 - real HTTP NFT results and stale-tip deck pagination.
 
@@ -45,10 +46,10 @@ Observed in the development Docker environment:
 
 These numbers are a reproducible **baseline**, not acceptance SLOs: host CPU/storage and production concurrency were not selected, outputs include synthetic article payloads, warm caches apply, one transaction amortizes insertion fsync, and real validation/startup/HTTP/market workloads are excluded. They do not establish a supported production envelope.
 
-The synthetic market-journal harness [benchmark](../scripts/benchmark-market-reconciliation.ts), available as `npm run benchmark:market -- 10000 [terminal|confirmed-delivery]`, measures two bounded reconciliation paths in its own temporary SQLite directory. With 32-row slices, the development Docker host processed 10,000 terminal orders plus 10,000 listings in 2,549 ms (about 7,845 records/s), with pass-latency p50/p95/max of 3.44/6.48/42.13 ms, sampled event-loop-delay p95/max of 14.97/48.99 ms at 10ms resolution, and an 11,927,552-byte SQLite file. The confirmed-delivery fixture transitioned 10,000 paid orders and reserved listings to complete/sold in 4,667 ms (about 4,286 records/s), with pass-latency p50/p95/max of 6.88/14.30/62.12 ms, sampled event-loop-delay p95/max of 20.63/62.23 ms, and a 12,832,768-byte SQLite file. The 32-row size increases pass count (625 vs. 157 at 128 rows) and reduces typical delay, but these runs still show long-tail event-loop stalls; a smaller batch is not equivalent to worker isolation. The canonical transaction IDs are in-memory adapter fixtures, not validated Ledger evidence. These are not target-hardware throughput/SLO claims and exclude concurrent HTTP traffic, other server work, PostgreSQL, actual chain validation and adversarial workloads.
+The synthetic market-journal harness [benchmark](../scripts/benchmark-market-reconciliation.ts), available as `npm run benchmark:market -- 10000 [terminal|confirmed-delivery] [worker]`, measures terminal and confirmed-delivery reconciliation in its own temporary SQLite directory. With synchronous 32-row commits, 10,000 terminal orders plus 10,000 listings took 2,549 ms (about 7,845 records/s; sampled event-loop-delay p95/max 14.97/48.99 ms). The confirmed-delivery path took 4,667 ms (about 4,286 records/s; 20.63/62.23 ms). With worker-owned commits, the terminal fixture took 2,782 ms (about 7,190 records/s; sampled event-loop-delay p95/max 11.17/16.11 ms), and confirmed delivery took 5,762 ms (about 3,471 records/s; 13.39/24.25 ms). Worker IPC/transaction cost increases elapsed time, while sampled maximum main-loop delay fell in these runs; measurements are noisy and not an SLO. Maximum sampled pass duration with the worker was 25.15 ms for confirmed delivery, since row evaluation and chain lookups remain on the main thread. Canonical transaction IDs are in-memory adapter fixtures, not validated Ledger evidence. These runs are not target-hardware capacity claims and exclude concurrent HTTP traffic, PostgreSQL, actual chain validation and adversarial workloads.
 
 ## Remaining gates
 
-M2 still needs target-hardware/data/SLO decisions, populated long-journal/large-event benchmarks, bounded worker/event-loop latency under load, production financial database/schema and measured recovery/retention/storage growth. The new scheduler bounds work per pass, but does not yet provide queue isolation, multi-worker coordination, or a demonstrated production throughput envelope. Ledger PoW replay and DAG state memory still scale with retained history; indexed queries do not solve those costs. Startup validation checks undo-record counts but not every possible future undo mutation against a second reconstructed undo model; comprehensive corruption/fuzz coverage and independent review remain required.
+M2 still needs target-hardware/data/SLO decisions, populated long-journal/large-event benchmarks, HTTP load tests during worker commits, production financial database/schema and measured recovery/retention/storage growth. The worker moves writes/final commits off-loop, but row processing/chain reads remain on-loop; queue isolation, multi-process fencing and a demonstrated production throughput envelope remain open. Ledger PoW replay and DAG state memory still scale with retained history; indexed queries do not solve those costs. Startup validation checks undo-record counts but not every possible future undo mutation against a second reconstructed undo model; comprehensive corruption/fuzz coverage and independent review remain required.
 
 Custodial reservations still stop at `awaiting-lock`. This increment does **not** consume ordinary transfers as custodial settlement evidence: doing that would bypass the reviewed sale-lock/cancel protocol. Real Monero deposit/withdrawal integration, production treasury keys, host attestations and a reviewed versioned NFT lock/completion/cancellation specification remain blocked. No real funds, credentials, external processor or genesis change were used.