wikideck/docs/canonical-indexes.md

M2 increment — canonical indexes and recoverable market observation

Status: implemented local infrastructure, not full M2 acceptance or production authorization. Evidence date: 2026-10-21. Related: production plan, custodial design.

Delivered

• Canonical index stores current NFT ownership/sequence, canonical mint/block identity and transaction inclusion in SQLite. Owner/mint ordering and transaction indexes replace whole-chain scans for deck, NFT and marketplace queries.
• Ledger persistence commits accepted blocks, pending transactions, index connect/disconnect undo and canonical events in one SQLite transaction. A losing side branch or mempool submission does not create a canonical event. Disconnects run tip-first; connects run ancestor-first. No await separates the database commit from updating the in-memory tip.
• Events contain sequence, connect/disconnect type, block identity/height/parent, transaction IDs and a hash chained to the previous event. An independently generated local stream ID binds the history; it is not consensus data or a public proof of honest observation. Pages contain at most 64 events and no article payloads. Financial/chain history is not silently pruned.
• On first open, original blocks are replayed and independently proof/state validated before additive local index backfill. Later opens compare indexed block identity, ownership, transaction inclusion and event continuity/anchors to that validated chain. Inconsistent state fails closed; existing consensus version, genesis, signatures and free-transfer/trade rules are unchanged.
• Market reconciliation uses indexed current delivery/ownership queries and a persisted (stream ID, sequence, anchor) checkpoint. Every observed disconnect of a previously completed order causes a durable review hold, even if delivery reappeared before polling. Review is never automatically repaid. Hold updates and checkpoint advancement commit together in the market database.
• Order and listing review now uses stable-ID keyset batches capped at 32 rows each. A strict, durable job record stores stage, continuation ID, observed time, chain tip/revision, and the bounded canonical event page. Each batch commits row effects and its continuation atomically; the canonical event cursor advances only after both tables finish. A chain tip/revision change during a multi-batch scan rewinds the job and captures a fresh event page, preventing a mixed-view scan from being marked caught up. Completion checks the current event head so events appended during the scan remain behind the write fence.
• While any job/event backlog exists, ordinary market mutations and time-dependent payout/expiry decisions stop; diagnostic/read access remains available. Requests advance one bounded slice, and an unref'd HTTP-lifecycle pump drains remaining work without depending on page loads. The production HTTP process enables a single persistent Node worker thread to select rows, read canonical transaction/NFT/mempool facts from its own SQLite read-only Ledger connection under a consistent read transaction, evaluate order/listing transitions, and atomically persist market row changes plus job progress. The main thread retains event-page acquisition and post-batch tip/revision/event-head/mempool fencing; if observations changed, it rewinds before cursor advancement. This moves fact collection and per-row JSON/status work off-loop, but final fences still run on the request event loop. Worker failure leaves the durable job/cursor intact, fails closed, and requires process restart to retry; there is no horizontal-worker safety claim.
• Reconciliation restart, mid-job event append/reorg fencing, disconnect review, and injected SQLite batch/cursor failures are tested. A failed batch leaves both order effects and durable keyset progress unchanged; restart resumes committed work. Restore mismatch (different stream, different history anchor or a cursor ahead of the restored ledger) is refused rather than resetting the cursor. Restore the matching stopped-node ledger/market backup; any alternative recovery requires reviewed evidence, not deleting a checkpoint.
• HTTP NFT lookup and custody card checks use the same index. Wallet deck pagination accepts an optional tip query; stale tips return 409. Existing callers without tip remain compatible. There is no new public event/admin endpoint, no payment observer and no NFT-lock signer.

Tests and measurements

The final Docker test-target build passed TypeScript/Vite compilation. All 183 offline tests passed, zero failures/skips, including native RandomX multi-process HTTP/peer/restart, backup rehearsal and five offline Monero observer contract/security tests. The focused market suite passed 27 tests. The disposable PostgreSQL suite most recently passed 33/33, including v1→v2 schema migration and observation persistence.

• indexed ownership/sequence versus full replay for transfers, multi-card trades and forks;
• append/disconnect/reconnect ordering, side-branch/mempool exclusion, bounded defensive-copy pages and durable restart;
• injected event-write failure rolling back blocks, indexes and mempool together;
• additive migration of a prior-schema copy, refusing corrupt ownership and event records;
• production market adapter exercised with full snapshots forbidden;
• multi-page catch-up and persisted restart checkpoint;
• 32-row order/listing keyset progress, restart continuation, and a new canonical event arriving during an in-flight job;
• chain revision change fencing that resets continuation and refreshes the event page before the scan continues;
• injected batch-write failure rolling back order effects and keyset progress, plus cursor-write failure rolling back checkpoint advancement while preserving committed review holds;
• worker-owned row evaluation/batch/finalize transaction with injected failure, mutation fence and restart continuation;
• canonical event arrival during an in-flight worker batch forces a fresh bounded event page before the cursor reaches the new head;
• separate worker read-only Ledger connection supplies canonical NFT and delivery facts and re-fences when an event arrives during that snapshot;
• main-thread timer fires while a large worker reconciliation backlog remains in flight;
• numeric cursor reuse with a different historical hash rejected;
• real HTTP NFT results and stale-tip deck pagination.

Benchmark script, available as npm run benchmark:index -- 10000, creates/removes its own temporary SQL database. Input is bounded to 1–100,000 synthetic blocks; these are not valid mined blocks and are never submitted to a ledger/peer. It measures only derived index behavior, not PoW, network throughput or operating capacity.

Observed in the development Docker environment:

Metric	Result
Fixture	10,000 NFTs/blocks, 9,998 transfers, 16 owner identities
SQL fixture build (single transaction)	5,096 ms
1,000 query groups (NFT + 24-item owner page + optional transaction lookup), median / p95	0.619 / 1.167 ms
Read all events, maximum 64 per page	157 pages, 54.5 ms total
Verify derived indexes and anchored history	524 ms
Database plus WAL before close	76,856,784 bytes

These numbers are a reproducible baseline, not acceptance SLOs: host CPU/storage and production concurrency were not selected, outputs include synthetic article payloads, warm caches apply, one transaction amortizes insertion fsync, and real validation/startup/HTTP/market workloads are excluded. They do not establish a supported production envelope.

The synthetic market-journal harness benchmark, available as npm run benchmark:market -- 10000 [terminal|confirmed-delivery] [worker] [http], measures terminal and confirmed-delivery reconciliation in its own temporary SQLite directory. Both modes use the same Ledger-shaped fixture; worker mode reads it through a separate read-only SQLite connection and performs market row selection, canonical transaction/NFT fact reads, status evaluation and market commits on the worker. In the latest development Docker run, synchronous 32-row batches processed 10,000 terminal orders plus 10,000 listings in 4,100 ms (about 4,878 records/s; sampled event-loop-delay p95/max 18.43/21.95 ms), and confirmed delivery in 5,853 ms (about 3,417 records/s; 21.64/55.67 ms). Worker-owned selection/evaluation/fact reads/commits took 4,508 ms for terminal rows (about 4,437 records/s; delay p95/max 10.93/13.40 ms) and 4,088 ms for confirmed delivery (about 4,893 records/s; 10.80/12.48 ms). Thus worker execution reduced sampled delay, improved this delivery fixture, and slightly slowed terminal-only work. A new optional loopback HTTP mode issues 160 health/market requests during catch-up: worker run request p50/p95/max was 285/322/324 ms, versus 1,416/2,844/2,971 ms for synchronous reconciliation. These two runs are preliminary, local, synthetic, and lack repeated confidence intervals or a representative arrival distribution; the latency is not an SLO. SQLite fixture data is not validated Ledger evidence. All measurements exclude real concurrent users, PostgreSQL and adversarial workloads.

Remaining gates

M2 still needs target-hardware/data/SLO decisions, repeated concurrent HTTP tests with realistic request mix/rates, multi-instance/restart/in-flight-worker fencing evidence, production financial database/schema and measured recovery/retention/storage growth. The worker removes row selection, canonical fact reads and per-row evaluation from the request event loop, but event-page acquisition and post-batch consistency fencing remain on-loop; horizontally shared job ownership has not been implemented. Ledger PoW replay and DAG state memory still scale with retained history; indexed queries do not solve those costs. Startup validation checks undo-record counts but not every possible future undo mutation against a second reconstructed undo model; comprehensive corruption/fuzz coverage and independent review remain required.

Custodial reservations still stop at awaiting-lock. This increment does not consume ordinary transfers as custodial settlement evidence: doing that would bypass the reviewed sale-lock/cancel protocol. Real Monero deposit/withdrawal integration, production treasury keys, host attestations and a reviewed versioned NFT lock/completion/cancellation specification remain blocked. No real funds, credentials, external processor or genesis change were used.