wikideck/docs/canonical-indexes.md

# M2 increment — canonical indexes and recoverable market observation

Status: implemented local infrastructure, **not full M2 acceptance or production authorization**. Evidence date: 2026-10-21. Related: [production plan](production-plan.md), [custodial design](custodial-market-design.md).

## Delivered

- [Canonical index](../server/testnet/canonical-index.ts) stores current NFT ownership/sequence, canonical mint/block identity and transaction inclusion in SQLite. Owner/mint ordering and transaction indexes replace whole-chain scans for deck, NFT and marketplace queries.
- [Ledger persistence](../server/testnet/ledger.ts) commits accepted blocks, pending transactions, index connect/disconnect undo and canonical events in **one SQLite transaction**. A losing side branch or mempool submission does not create a canonical event. Disconnects run tip-first; connects run ancestor-first. No await separates the database commit from updating the in-memory tip.
- Events contain sequence, connect/disconnect type, block identity/height/parent, transaction IDs and a hash chained to the previous event. An independently generated local stream ID binds the history; it is **not consensus data or a public proof of honest observation**. Pages contain at most 64 events and no article payloads. Financial/chain history is not silently pruned.
- On first open, original blocks are replayed and independently proof/state validated before additive local index backfill. Later opens compare indexed block identity, ownership, transaction inclusion and event continuity/anchors to that validated chain. Inconsistent state fails closed; existing consensus version, genesis, signatures and free-transfer/trade rules are unchanged.
- [Market reconciliation](../server/market/store.ts) uses indexed current delivery/ownership queries and a persisted `(stream ID, sequence, anchor)` checkpoint. Every observed disconnect of a previously completed order causes a durable review hold, even if delivery reappeared before polling. Review is never automatically repaid. Hold updates and checkpoint advancement commit together in the market database.
- Order and listing review now uses stable-ID keyset batches capped at 32 rows each. A strict, durable job record stores stage, continuation ID, observed time, chain tip/revision, and the bounded canonical event page. Each batch commits row effects and its continuation atomically; the canonical event cursor advances only after both tables finish. A chain tip/revision change during a multi-batch scan rewinds the job and captures a fresh event page, preventing a mixed-view scan from being marked caught up. Completion checks the current event head so events appended during the scan remain behind the write fence.
- While any job/event backlog exists, ordinary market mutations and time-dependent payout/expiry decisions stop; diagnostic/read access remains available. Requests advance one bounded slice, and an unref'd HTTP-lifecycle pump drains remaining work without depending on page loads. The production HTTP process enables a single persistent Node worker thread to select rows, read canonical transaction/NFT/mempool facts from its own SQLite read-only Ledger connection under a consistent read transaction, evaluate order/listing transitions, and atomically persist market row changes plus job progress. The main thread retains event-page acquisition and post-batch tip/revision/event-head/mempool fencing; if observations changed, it rewinds before cursor advancement. This moves fact collection and per-row JSON/status work off-loop, but final fences still run on the request event loop. Worker failure leaves the durable job/cursor intact, fails closed, and requires process restart to retry; there is no horizontal-worker safety claim.
- Reconciliation restart, mid-job event append/reorg fencing, disconnect review, and injected SQLite batch/cursor failures are tested. A failed batch leaves both order effects and durable keyset progress unchanged; restart resumes committed work. Restore mismatch (different stream, different history anchor or a cursor ahead of the restored ledger) is refused rather than resetting the cursor. Restore the matching stopped-node ledger/market backup; any alternative recovery requires reviewed evidence, not deleting a checkpoint.
- [HTTP NFT lookup and custody card checks](../server/testnet/http.ts) use the same index. Wallet deck pagination accepts an optional `tip` query; stale tips return 409. Existing callers without `tip` remain compatible. There is no new public event/admin endpoint, no payment observer and no NFT-lock signer.

## Tests and measurements

The final Docker test-target build passed TypeScript/Vite compilation. All **183 offline tests passed, zero failures/skips**, including native RandomX multi-process HTTP/peer/restart, backup rehearsal and five offline Monero observer contract/security tests. The focused market suite passed **27 tests**. The disposable PostgreSQL suite most recently passed **33/33**, including v1→v2 schema migration and observation persistence.

- indexed ownership/sequence versus full replay for transfers, multi-card trades and forks;
- append/disconnect/reconnect ordering, side-branch/mempool exclusion, bounded defensive-copy pages and durable restart;
- injected event-write failure rolling back blocks, indexes and mempool together;
- additive migration of a prior-schema copy, refusing corrupt ownership and event records;
- production market adapter exercised with full snapshots forbidden;
- multi-page catch-up and persisted restart checkpoint;
- 32-row order/listing keyset progress, restart continuation, and a new canonical event arriving during an in-flight job;
- chain revision change fencing that resets continuation and refreshes the event page before the scan continues;
- injected batch-write failure rolling back order effects and keyset progress, plus cursor-write failure rolling back checkpoint advancement while preserving committed review holds;
- worker-owned row evaluation/batch/finalize transaction with injected failure, mutation fence and restart continuation;
- canonical event arrival during an in-flight worker batch forces a fresh bounded event page before the cursor reaches the new head;
- separate worker read-only Ledger connection supplies canonical NFT and delivery facts and re-fences when an event arrives during that snapshot;
- main-thread timer fires while a large worker reconciliation backlog remains in flight;
- numeric cursor reuse with a different historical hash rejected;
- real HTTP NFT results and stale-tip deck pagination.

[Benchmark script](../scripts/benchmark-canonical-index.ts), available as `npm run benchmark:index -- 10000`, creates/removes its own temporary SQL database. Input is bounded to 1–100,000 synthetic blocks; **these are not valid mined blocks** and are never submitted to a ledger/peer. It measures only derived index behavior, not PoW, network throughput or operating capacity.

Observed in the development Docker environment:

| Metric | Result |
| --- | --- |
| Fixture | 10,000 NFTs/blocks, 9,998 transfers, 16 owner identities |
| SQL fixture build (single transaction) | 5,096 ms |
| 1,000 query groups (NFT + 24-item owner page + optional transaction lookup), median / p95 | 0.619 / 1.167 ms |
| Read all events, maximum 64 per page | 157 pages, 54.5 ms total |
| Verify derived indexes and anchored history | 524 ms |
| Database plus WAL before close | 76,856,784 bytes |

These numbers are a reproducible **baseline**, not acceptance SLOs: host CPU/storage and production concurrency were not selected, outputs include synthetic article payloads, warm caches apply, one transaction amortizes insertion fsync, and real validation/startup/HTTP/market workloads are excluded. They do not establish a supported production envelope.

The synthetic market-journal harness [benchmark](../scripts/benchmark-market-reconciliation.ts), available as `npm run benchmark:market -- 10000 [terminal|confirmed-delivery] [worker] [http]`, measures terminal and confirmed-delivery reconciliation in its own temporary SQLite directory. Both modes use the same Ledger-shaped fixture; worker mode reads it through a separate read-only SQLite connection and performs market row selection, canonical transaction/NFT fact reads, status evaluation and market commits on the worker. In the latest development Docker run, synchronous 32-row batches processed 10,000 terminal orders plus 10,000 listings in 4,100 ms (about 4,878 records/s; sampled event-loop-delay p95/max 18.43/21.95 ms), and confirmed delivery in 5,853 ms (about 3,417 records/s; 21.64/55.67 ms). Worker-owned selection/evaluation/fact reads/commits took 4,508 ms for terminal rows (about 4,437 records/s; delay p95/max 10.93/13.40 ms) and 4,088 ms for confirmed delivery (about 4,893 records/s; 10.80/12.48 ms). Thus worker execution reduced sampled delay, improved this delivery fixture, and slightly slowed terminal-only work. A new optional loopback HTTP mode issues 160 health/market requests during catch-up: worker run request p50/p95/max was 285/322/324 ms, versus 1,416/2,844/2,971 ms for synchronous reconciliation. These two runs are preliminary, local, synthetic, and lack repeated confidence intervals or a representative arrival distribution; the latency is not an SLO. SQLite fixture data is not validated Ledger evidence. All measurements exclude real concurrent users, PostgreSQL and adversarial workloads.

## Remaining gates

M2 still needs target-hardware/data/SLO decisions, repeated concurrent HTTP tests with realistic request mix/rates, multi-instance/restart/in-flight-worker fencing evidence, production financial database/schema and measured recovery/retention/storage growth. The worker removes row selection, canonical fact reads and per-row evaluation from the request event loop, but event-page acquisition and post-batch consistency fencing remain on-loop; horizontally shared job ownership has not been implemented. Ledger PoW replay and DAG state memory still scale with retained history; indexed queries do not solve those costs. Startup validation checks undo-record counts but not every possible future undo mutation against a second reconstructed undo model; comprehensive corruption/fuzz coverage and independent review remain required.

Custodial reservations still stop at `awaiting-lock`. This increment does **not** consume ordinary transfers as custodial settlement evidence: doing that would bypass the reviewed sale-lock/cancel protocol. Real Monero deposit/withdrawal integration, production treasury keys, host attestations and a reviewed versioned NFT lock/completion/cancellation specification remain blocked. No real funds, credentials, external processor or genesis change were used.