More scalable data storage

docs/testnet-consensus.md

@@ -17,7 +17,7 @@ implementation-specific, and changing them can create a different chain.
 
 ## Network, genesis and article selection
 
-- Network: `wikideck-randomx-testnet-4`, block version **4**. Genesis has no blocks, NFTs or work; first mined height is 1. No privileged issuance/admin mint key. Version 4 has a new chain identity; prior-version ledger files and wallet backups are not compatible.
+- Network: `wikideck-randomx-testnet-5`, block version **5**. Genesis has no blocks, NFTs or work; first mined height is 1. No privileged issuance/admin mint key. Version 5 has a new chain identity; prior-version ledger files and wallet backups are not compatible.
 - `GENESIS_TIME = 1792411200` (Unix seconds). Let `S(x)` mean lowercase hex SHA-256 of UTF-8 `x`, and `J(x)` mean compact JavaScript `JSON.stringify(x)`; field order below is consensus-critical.
 - `MAX_TARGET` is `0fff` followed by 60 `f` digits. `CHAIN_ID = S(J([NETWORK, CONSENSUS, GENESIS_TIME, MAX_TARGET]))`, using the constants object's insertion order in the implementation. `CONSENSUS` identifies the random Wikipedia source and mint-statistics snapshot format.
 - `GENESIS_HASH = S(NETWORK + ':genesis:' + CHAIN_ID)`; `INITIAL_KEY = S(NETWORK + ':initial-key:' + CHAIN_ID)`. Nodes independently derive the same manifest and reject other network/chain IDs.
@@ -42,8 +42,8 @@ The miner signs `J([NETWORK, 'block-claim', blockID])` using ECDSA P-256/SHA-256
 
 ## Targets, timestamps and epoch boundaries
 
-- Desired spacing: **30 seconds**; retarget interval: **10 ancestors**. Heights 1–10 use the initial target; height 11 first retargets. Otherwise inherit the parent target.
-- At each retarget, compare the last and first timestamps of the previous 10 blocks: expected span is **9 × 30 = 270 seconds**. Clamp elapsed to `floor(270/4)` through `270*4`, multiply the parent target by elapsed/270 with integer division, then clamp target to `[1, MAX_TARGET]`. Selection uses work, not height or wall-clock estimates.
+- Desired spacing: **15 minutes (900 seconds)**; retarget interval: **10 ancestors**. Heights 1–10 use the initial target; height 11 first retargets. Otherwise inherit the parent target.
+- At each retarget, compare the last and first timestamps of the previous 10 blocks: expected span is **9 × 900 = 8,100 seconds**. Clamp elapsed to `floor(8,100/4) = 2,025` through `8,100*4 = 32,400` seconds, multiply the parent target by elapsed/8,100 with integer division, then clamp target to `[1, MAX_TARGET]`. Selection uses work, not height or wall-clock estimates.
 - A timestamp must exceed the median of up to **11** ancestors (sorted upper-middle for even counts; genesis time if none) and must be at most **120 seconds** ahead of validation time. Keep operator clocks synchronized.
 - Epoch key source height is `floor(max(0, height - 64) / 2048) * 2048`. Zero means `INITIAL_KEY`; otherwise use the ancestor block hash at that source height. Thus heights **1–2111** use the initial key, **2112–4159** use height 2048's hash, and height **4160** switches to height 4096's hash. Forks derive keys from their own ancestors.
 
@@ -63,9 +63,9 @@ Per-block work is `floor(2^256 / (target + 1))`; accumulate from genesis work ze
 
 On reorg, reconstruct ownership from the winning branch. Orphan mints disappear from canonical ownership; disconnected transfers are reconsidered before existing pending entries, in order, only if still valid and not already committed. Conflicting, immature or orphan-issuance transfers are discarded. No checkpoints/finality or rollback protection beyond observed work; an isolated node has no peer-backed security.
 
-The [ledger](../server/testnet/ledger.ts) stores the entire DAG and pending pool in `data/testnet/ledger.json`, under the `WIKIDECK_DATA_DIR` testnet subdirectory. The file is created when the node first writes ledger data. Defaults use project data; Docker uses `/app/data`. Disk caps: **128 MiB, 10,000 total blocks including side branches**; blocks: **64 KiB and 64 transfers**. Each in-memory DAG node stores a full NFT-state snapshot: potentially **quadratic memory growth**, plus full-file rewrite cost. These are experimental stop limits, not scalable storage or a public DoS defense.
+The [ledger](../server/testnet/ledger.ts) is stored in `data/testnet/ledger.sqlite`, under the `WIKIDECK_DATA_DIR` testnet subdirectory (Docker uses `/app/data`). SQLite WAL mode with full synchronous commits stores each accepted block once in an indexed table and replaces the bounded mempool in the same transaction; writes no longer serialize/rewrite the entire DAG, and there is no 128 MiB/10,000-block ledger cap. Existing v5 `ledger.json` files are validated and imported transactionally once; the source JSON is left untouched as a recovery copy. Back up the complete stopped-node data directory, including SQLite database/WAL files. Do not copy only `ledger.sqlite` while the node is running.
 
-Mutations are serialized (queue cap 256); persist via a synchronized temporary file and atomic rename before exposing new state. Restart independently revalidates every saved proof and transaction; malformed, oversized or mismatched files fail closed without automatic reset. HTTP startup exclusively creates a data-directory lock named testnet-node.lock. It is removed on normal shutdown, **not automatically after a crash**: stop and verify all owning processes have exited before manual stale-lock removal. Never bypass a live lock or share a writer directory. Back up stopped-node volumes/files and wallet exports separately; destructive volume reset cannot recover assets from a wallet alone.
+The in-memory DAG keeps block records and cumulative work, while each block's NFT state is a persistent overlay containing only that block's changes rather than a full state copy. State-root calculation still visits the complete NFT state, and startup still revalidates every stored proof; CPU and startup work therefore grow with chain size. This is materially more storage-efficient, but not yet an archival-scale chain database or a public DoS defense. Blocks remain individually limited to **64 KiB and 64 transfers**; the mempool remains capped at **256 transfers**. Mutations are serialized (queue cap 256). HTTP startup exclusively creates a data-directory lock named `testnet-node.lock`. It is removed on normal shutdown, **not automatically after a crash**: stop and verify all owning processes have exited before manual stale-lock removal. Never bypass a live lock or share a writer directory. Back up node data and wallet exports separately; deleting a volume cannot be recovered from a wallet alone.
 
 ## Fixed peers and HTTP surface