Canonical indexes and recoverable market observation

docs/canonical-indexes.md

@@ -0,0 +1,49 @@
+# M2 increment — canonical indexes and recoverable market observation
+
+Status: implemented local infrastructure, **not full M2 acceptance or production authorization**. Evidence date: 2026-10-20. Related: [production plan](production-plan.md), [custodial design](custodial-market-design.md).
+
+## Delivered
+
+- [Canonical index](../server/testnet/canonical-index.ts) stores current NFT ownership/sequence, canonical mint/block identity and transaction inclusion in SQLite. Owner/mint ordering and transaction indexes replace whole-chain scans for deck, NFT and marketplace queries.
+- [Ledger persistence](../server/testnet/ledger.ts) commits accepted blocks, pending transactions, index connect/disconnect undo and canonical events in **one SQLite transaction**. A losing side branch or mempool submission does not create a canonical event. Disconnects run tip-first; connects run ancestor-first. No await separates the database commit from updating the in-memory tip.
+- Events contain sequence, connect/disconnect type, block identity/height/parent, transaction IDs and a hash chained to the previous event. An independently generated local stream ID binds the history; it is **not consensus data or a public proof of honest observation**. Pages contain at most 64 events and no article payloads. Financial/chain history is not silently pruned.
+- On first open, original blocks are replayed and independently proof/state validated before additive local index backfill. Later opens compare indexed block identity, ownership, transaction inclusion and event continuity/anchors to that validated chain. Inconsistent state fails closed; existing consensus version, genesis, signatures and free-transfer/trade rules are unchanged.
+- [Market reconciliation](../server/market/store.ts) uses indexed current delivery/ownership queries and a persisted `(stream ID, sequence, anchor)` checkpoint. Every observed disconnect of a previously completed order causes a durable review hold, even if delivery reappeared before polling. Review is never automatically repaid. Hold updates and checkpoint advancement commit together in the market database.
+- Catch-up consumes one bounded event page per reconciliation pass. While behind, ordinary market mutation/payout/expiry decisions stop; diagnostic/read access remains available and exposes event cursor/head/backlog. Periodic polling and subsequent requests advance catch-up. No claim of instant catch-up or background-thread isolation.
+- Restore mismatch (different stream, different history anchor or a cursor ahead of the restored ledger) is refused rather than resetting the cursor. Restore the matching stopped-node ledger/market backup; any alternative recovery requires reviewed evidence, not deleting a checkpoint.
+- [HTTP NFT lookup and custody card checks](../server/testnet/http.ts) use the same index. Wallet deck pagination accepts an optional `tip` query; stale tips return 409. Existing callers without `tip` remain compatible. There is no new public event/admin endpoint, no payment observer and no NFT-lock signer.
+
+## Tests and measurements
+
+The final Docker test-target build passed TypeScript/Vite compilation. All **149 tests passed, zero failures/skips**, including native RandomX multi-process HTTP/peer/restart and backup rehearsal. New failure evidence includes:
+
+- indexed ownership/sequence versus full replay for transfers, multi-card trades and forks;
+- append/disconnect/reconnect ordering, side-branch/mempool exclusion, bounded defensive-copy pages and durable restart;
+- injected event-write failure rolling back blocks, indexes and mempool together;
+- additive migration of a prior-schema copy, refusing corrupt ownership and event records;
+- production market adapter exercised with full snapshots forbidden;
+- multi-page catch-up and persisted restart checkpoint;
+- injected cursor-write failure rolling back the payout hold, followed by restart recovery;
+- numeric cursor reuse with a different historical hash rejected;
+- real HTTP NFT results and stale-tip deck pagination.
+
+[Benchmark script](../scripts/benchmark-canonical-index.ts), available as `npm run benchmark:index -- 10000`, creates/removes its own temporary SQL database. Input is bounded to 1–100,000 synthetic blocks; **these are not valid mined blocks** and are never submitted to a ledger/peer. It measures only derived index behavior, not PoW, network throughput or operating capacity.
+
+Observed in the development Docker environment:
+
+| Metric | Result |
+| --- | --- |
+| Fixture | 10,000 NFTs/blocks, 9,998 transfers, 16 owner identities |
+| SQL fixture build (single transaction) | 5,096 ms |
+| 1,000 query groups (NFT + 24-item owner page + optional transaction lookup), median / p95 | 0.619 / 1.167 ms |
+| Read all events, maximum 64 per page | 157 pages, 54.5 ms total |
+| Verify derived indexes and anchored history | 524 ms |
+| Database plus WAL before close | 76,856,784 bytes |
+
+These numbers are a reproducible **baseline**, not acceptance SLOs: host CPU/storage and production concurrency were not selected, outputs include synthetic article payloads, warm caches apply, one transaction amortizes insertion fsync, and real validation/startup/HTTP/market workloads are excluded. They do not establish a supported production envelope.
+
+## Remaining gates
+
+M2 still needs target-hardware/data/SLO decisions, market-order incremental scheduling rather than scanning every existing bounded order/listing, bounded worker/event-loop latency under load, production financial database/schema and measured recovery/retention/storage growth. Ledger PoW replay and DAG state memory still scale with retained history; indexed queries do not solve those costs. Startup validation checks undo-record counts but not every possible future undo mutation against a second reconstructed undo model; comprehensive corruption/fuzz coverage and independent review remain required.
+
+Custodial reservations still stop at `awaiting-lock`. This increment does **not** consume ordinary transfers as custodial settlement evidence: doing that would bypass the reviewed sale-lock/cancel protocol. Real Monero deposit/withdrawal integration, production treasury keys, host attestations and a reviewed versioned NFT lock/completion/cancellation specification remain blocked. No real funds, credentials, external processor or genesis change were used.

docs/production-plan.md

@@ -63,6 +63,8 @@ Acceptance: exact-amount property/boundary tests, versioned signature tests, zer
 
 Can run alongside P0/P1. Addresses SEC-01/SEC-06 rather than adding more fixed caps.
 
+**Implemented M2 increment, 2026-10-20:** [canonical indexes and durable market observation](canonical-indexes.md). Canonical ownership/transaction SQL indexes now serve NFT, deck, custody availability and mock-market queries; per-block undo and anchored connect/disconnect events commit with ledger updates. Market checkpoint/reorg holds persist atomically and bounded catch-up blocks new mutations. Final TypeScript/Vite build and 149 offline tests passed. A disposable 10,000-NFT SQL baseline is recorded, **not** a target-hardware SLO or complete M2 acceptance. Production financial storage, incremental order scheduling, retained-DAG capacity and operator dataset/latency targets remain open.
+
 - Indexed canonical ownership, NFT/transfer lookup and paged wallet/explorer APIs: browser signing must not require loading all history. Tip-pinned queries detect reorgs and preserve complete ownership semantics.
 - Incremental market reconciliation from durable canonical/reorg events. Maintain rollback-safe indexes and a recoverable cursor; catch-up after restart must not miss delivery or payout holds. Bound worker time/concurrency; move expensive work off the request event loop where appropriate.
 - A versioned production market/settlement schema with append-only financial events and balanced accounting entries for buyer liabilities, host revenue, author liabilities, seller liabilities, paid/refunded amounts and actual network fees. Derived views are rebuildable; financial history and idempotency keys are not discarded to reclaim space.
@@ -138,7 +140,7 @@ Acceptance: signed release evidence packet; recorded unresolved accepted risks w
 
 First implementation increment: write the M0 threat model/ADR research questions and P1 XMR domain specification, then build exact amount/property tests. Do not spend the next increment adding another disconnected “live” scaffold. Each milestone must integrate with its actual callers, persistence, recovery, UI and failure tests before being described as delivered.
 
-Next increment status: M0 source investigation and [concrete settlement/evidence choices](settlement-feasibility.md) remain open. M1 now has a signed v3 XMR-native mock marketplace path; the public USD-to-XMR rate was removed. Version-2 historical USD orders are preserved and can continue under their original terms; open USD listings are legacy read-only until seller retirement/relisting. This is not live settlement, a completed migration dry-run, or production milestone acceptance; payment and payouts remain mock-only, and live auctions remain a separate release gate.
+Current increment status: the operator selected custodial implementation after [settlement research](settlement-feasibility.md). M1 has a signed v3 XMR-native mock path, with historical v2 USD orders preserved. [ADR 002](custodial-market-design.md) records the implemented isolated custody accounting/UI foundation; production custody is not complete. The next M2 infrastructure increment is [canonical indexing and durable reconciliation](canonical-indexes.md), now built and regression-tested. Next engineering gates include the remaining production accounting/worker envelope and the separately reviewed NFT lock/completion/cancellation specification (M3); no live Monero integration or consensus activation is authorized by passing these tests. Migration dry-run, final fee/risk policy, capacity SLOs and independent review remain open.
 
 ## 5. Acceptance evidence and release discipline
 
@@ -178,4 +180,4 @@ Not every value is needed to begin engineering; each becomes a blocking decision
 - [Resource controls](node-resources.md), [wallet lifecycle](wallet-security.md), [calibration evidence](consensus-calibration.md), [operations/recovery](operations-recovery.md).
 - [Offline SHKeeper contract/state-machine work](shkeeper-settlement.md) — reusable tests/ideas, not the selected production architecture.
 
-This plan supersedes the earlier **production pricing direction** of USD listings, but does not claim the current USD/mock runtime has already migrated. Keep historical evidence/versioned terms intact; update live-facing copy when the integrated XMR-native implementation is actually delivered.
+This plan supersedes the earlier USD production-pricing direction. New mock listings already use native XMR; historical USD terms remain intact. Custody and M2 implementation status are explicitly bounded above; no sandbox or infrastructure milestone implies live settlement or launch readiness.