More scalable data storage

README.md

@@ -22,9 +22,9 @@ Stop the nodes with:
 docker compose -f compose.testnet.yaml down
 ```
 
-The volumes persist across stops and rebuilds. **Do not add `-v` unless you intend to delete all three ledgers.** Back up wallets separately. Protocol v4 uses a new chain identity; earlier v2/v3 ledgers and encrypted backups are not compatible. To keep old Docker data intact while trying v4, use a separate Compose project name, for example `docker compose -p wikideck-v4 -f compose.testnet.yaml up --build -d`, and use the same `-p wikideck-v4` for subsequent Compose commands. The default `docker compose up --build -d` uses [compose.yaml](compose.yaml): it starts one isolated node on port 3001 with a different volume.
+The target block interval is **15 minutes**. The volumes persist across stops and rebuilds. **Do not add `-v` unless you intend to delete all three ledgers.** Back up wallets separately. Protocol v5 uses a new chain identity; earlier v2/v3/v4 ledgers and encrypted backups are not compatible. To keep old Docker data intact while trying v5, use a separate Compose project name, for example `docker compose -p wikideck-v5 -f compose.testnet.yaml up --build -d`, and use the same `-p wikideck-v5` for subsequent Compose commands. The default `docker compose up --build -d` uses [compose.yaml](compose.yaml): it starts one isolated node on port 3001 with a different volume.
 
-For an existing host install, keep the old data directory intact and set `WIKIDECK_DATA_DIR` to a new empty directory before starting protocol v4. The node intentionally fails closed on an old ledger; it does not reset or migrate it.
+For an existing host install, keep the old data directory intact and set `WIKIDECK_DATA_DIR` to a new empty directory before starting protocol v5. The node intentionally fails closed on an old-protocol ledger; it does not reset or migrate it. An existing v5 JSON ledger is imported into SQLite on first startup and left in place as a recovery copy.
 
 ## Run on your host
 
@@ -97,7 +97,7 @@ This mines three real blocks using disposable identities. It changes the testnet
 
 ## Network and safety limits
 
-Nodes use only explicitly configured peers; there is no discovery service or public bootstrap network. The peer, storage, and HTTP configuration is documented in [docs/testnet-consensus.md](docs/testnet-consensus.md). The implementation is experimental: it has no fees or economic security, and remains vulnerable to resource exhaustion, isolation/eclipsing, censorship, majority-hashpower attacks, and reorganizations. Storage limits are stop limits rather than a scalable production design. HTTPS, container hardening, and passing tests do not constitute a security audit.
+Nodes use only explicitly configured peers; there is no discovery service or public bootstrap network. The peer, storage, and HTTP configuration is documented in [docs/testnet-consensus.md](docs/testnet-consensus.md). The implementation is experimental: it has no fees or economic security, and remains vulnerable to resource exhaustion, isolation/eclipsing, censorship, majority-hashpower attacks, and reorganizations. The ledger now uses transactional SQLite/WAL storage and sparse per-block NFT-state deltas, but nodes still retain DAG block records in memory, rebuild state roots over the full NFT set, and revalidate all proofs at startup. It is not yet an archival-scale production design. HTTPS, container hardening, and passing tests do not constitute a security audit.
 
 Before considering any broader deployment, the protocol and cryptography would need independent audits, adversarial network testing, hardened abuse controls, and tested recovery procedures. Do not expose a public node or use valuable assets on this experimental network.
 

docs/testnet-consensus.md

@@ -17,7 +17,7 @@ implementation-specific, and changing them can create a different chain.
 
 ## Network, genesis and article selection
 
-- Network: `wikideck-randomx-testnet-4`, block version **4**. Genesis has no blocks, NFTs or work; first mined height is 1. No privileged issuance/admin mint key. Version 4 has a new chain identity; prior-version ledger files and wallet backups are not compatible.
+- Network: `wikideck-randomx-testnet-5`, block version **5**. Genesis has no blocks, NFTs or work; first mined height is 1. No privileged issuance/admin mint key. Version 5 has a new chain identity; prior-version ledger files and wallet backups are not compatible.
 - `GENESIS_TIME = 1792411200` (Unix seconds). Let `S(x)` mean lowercase hex SHA-256 of UTF-8 `x`, and `J(x)` mean compact JavaScript `JSON.stringify(x)`; field order below is consensus-critical.
 - `MAX_TARGET` is `0fff` followed by 60 `f` digits. `CHAIN_ID = S(J([NETWORK, CONSENSUS, GENESIS_TIME, MAX_TARGET]))`, using the constants object's insertion order in the implementation. `CONSENSUS` identifies the random Wikipedia source and mint-statistics snapshot format.
 - `GENESIS_HASH = S(NETWORK + ':genesis:' + CHAIN_ID)`; `INITIAL_KEY = S(NETWORK + ':initial-key:' + CHAIN_ID)`. Nodes independently derive the same manifest and reject other network/chain IDs.
@@ -42,8 +42,8 @@ The miner signs `J([NETWORK, 'block-claim', blockID])` using ECDSA P-256/SHA-256
 
 ## Targets, timestamps and epoch boundaries
 
-- Desired spacing: **30 seconds**; retarget interval: **10 ancestors**. Heights 1–10 use the initial target; height 11 first retargets. Otherwise inherit the parent target.
+- Desired spacing: **15 minutes (900 seconds)**; retarget interval: **10 ancestors**. Heights 1–10 use the initial target; height 11 first retargets. Otherwise inherit the parent target.
-- At each retarget, compare the last and first timestamps of the previous 10 blocks: expected span is **9 × 30 = 270 seconds**. Clamp elapsed to `floor(270/4)` through `270*4`, multiply the parent target by elapsed/270 with integer division, then clamp target to `[1, MAX_TARGET]`. Selection uses work, not height or wall-clock estimates.
+- At each retarget, compare the last and first timestamps of the previous 10 blocks: expected span is **9 × 900 = 8,100 seconds**. Clamp elapsed to `floor(8,100/4) = 2,025` through `8,100*4 = 32,400` seconds, multiply the parent target by elapsed/8,100 with integer division, then clamp target to `[1, MAX_TARGET]`. Selection uses work, not height or wall-clock estimates.
 - A timestamp must exceed the median of up to **11** ancestors (sorted upper-middle for even counts; genesis time if none) and must be at most **120 seconds** ahead of validation time. Keep operator clocks synchronized.
 - Epoch key source height is `floor(max(0, height - 64) / 2048) * 2048`. Zero means `INITIAL_KEY`; otherwise use the ancestor block hash at that source height. Thus heights **1–2111** use the initial key, **2112–4159** use height 2048's hash, and height **4160** switches to height 4096's hash. Forks derive keys from their own ancestors.
 
@@ -63,9 +63,9 @@ Per-block work is `floor(2^256 / (target + 1))`; accumulate from genesis work ze
 
 On reorg, reconstruct ownership from the winning branch. Orphan mints disappear from canonical ownership; disconnected transfers are reconsidered before existing pending entries, in order, only if still valid and not already committed. Conflicting, immature or orphan-issuance transfers are discarded. No checkpoints/finality or rollback protection beyond observed work; an isolated node has no peer-backed security.
 
-The [ledger](../server/testnet/ledger.ts) stores the entire DAG and pending pool in `data/testnet/ledger.json`, under the `WIKIDECK_DATA_DIR` testnet subdirectory. The file is created when the node first writes ledger data. Defaults use project data; Docker uses `/app/data`. Disk caps: **128 MiB, 10,000 total blocks including side branches**; blocks: **64 KiB and 64 transfers**. Each in-memory DAG node stores a full NFT-state snapshot: potentially **quadratic memory growth**, plus full-file rewrite cost. These are experimental stop limits, not scalable storage or a public DoS defense.
+The [ledger](../server/testnet/ledger.ts) is stored in `data/testnet/ledger.sqlite`, under the `WIKIDECK_DATA_DIR` testnet subdirectory (Docker uses `/app/data`). SQLite WAL mode with full synchronous commits stores each accepted block once in an indexed table and replaces the bounded mempool in the same transaction; writes no longer serialize/rewrite the entire DAG, and there is no 128 MiB/10,000-block ledger cap. Existing v5 `ledger.json` files are validated and imported transactionally once; the source JSON is left untouched as a recovery copy. Back up the complete stopped-node data directory, including SQLite database/WAL files. Do not copy only `ledger.sqlite` while the node is running.
 
-Mutations are serialized (queue cap 256); persist via a synchronized temporary file and atomic rename before exposing new state. Restart independently revalidates every saved proof and transaction; malformed, oversized or mismatched files fail closed without automatic reset. HTTP startup exclusively creates a data-directory lock named testnet-node.lock. It is removed on normal shutdown, **not automatically after a crash**: stop and verify all owning processes have exited before manual stale-lock removal. Never bypass a live lock or share a writer directory. Back up stopped-node volumes/files and wallet exports separately; destructive volume reset cannot recover assets from a wallet alone.
+The in-memory DAG keeps block records and cumulative work, while each block's NFT state is a persistent overlay containing only that block's changes rather than a full state copy. State-root calculation still visits the complete NFT state, and startup still revalidates every stored proof; CPU and startup work therefore grow with chain size. This is materially more storage-efficient, but not yet an archival-scale chain database or a public DoS defense. Blocks remain individually limited to **64 KiB and 64 transfers**; the mempool remains capped at **256 transfers**. Mutations are serialized (queue cap 256). HTTP startup exclusively creates a data-directory lock named `testnet-node.lock`. It is removed on normal shutdown, **not automatically after a crash**: stop and verify all owning processes have exited before manual stale-lock removal. Never bypass a live lock or share a writer directory. Back up node data and wallet exports separately; deleting a volume cannot be recovered from a wallet alone.
 
 ## Fixed peers and HTTP surface
 

server/testnet/consensus.ts

@@ -1,7 +1,8 @@
 import { createHash, createPublicKey, verify } from 'node:crypto';
 import { z } from 'zod';
 import { NETWORK, CONSENSUS, headerMessage, blockClaimMessage, nftTransferMessage, statsSnapshotTuple,
-  type BlockHeader, type TestnetBlock, type NftTransfer, type NftState, type MiningTemplate } from '../../shared/testnet.ts';
+  type BlockHeader, type TestnetBlock, type NftTransfer, type NftState, type NftStateLayer,
+  type NftStateView, type MiningTemplate } from '../../shared/testnet.ts';
 import type { Article } from '../../shared/protocol.ts';
 import { deriveStats, type MintStatsSnapshot } from '../../shared/enrichment.ts';
 import type { PowEngine } from './randomx.ts';
@@ -70,7 +71,7 @@ const statsSnapshotSchema = z.object({ capturedAt: z.string().datetime(), revisi
       ctx.addIssue({ code: 'custom', message: 'Derived statistics do not match the page metrics', path: ['stats'] });
     }
   });
-export const blockSchema = z.object({ version: z.literal(4), network: z.literal(NETWORK), chainId: z.literal(CHAIN_ID),
+export const blockSchema = z.object({ version: z.literal(5), network: z.literal(NETWORK), chainId: z.literal(CHAIN_ID),
   height: integer, previousHash: hex, createdAt: integer, article: articleSchema, statsSnapshot: statsSnapshotSchema, owner: hex,
   publicKey: key, nonce: integer, target: hex, key: hex, transactionsRoot: hex, stateRoot: hex,
   hash: hex, tokenId: hex, powHash: hex, signature, transactions: z.array(transferSchema).max(CONSENSUS.maxTransactions) }).strict();
@@ -102,9 +103,32 @@ export function merkleRoot(leaves: string[]): string {
   // Commit the count as well: odd-leaf padding must not make [a,b,c] equal [a,b,c,c].
   return sha256(JSON.stringify([NETWORK,'merkle',leaves.length,layer[0]]));
 }
-export function stateRoot(state: NftState): string {
+export function materializeState(state: NftStateView): NftState {
-  return merkleRoot(Object.keys(state).sort().map(id => {
+  const layers: NftState[] = [];
-    const n = state[id]; return sha256(JSON.stringify([id, n.owner, n.sequence, n.mintedHeight, n.pageId]));
+  let cursor: NftStateView = state;
+  while (isStateLayer(cursor)) {
+    layers.push(cursor.changes);
+    cursor = cursor.parent;
+  }
+  const result: NftState = Object.assign(Object.create(null) as NftState, cursor as NftState);
+  for (let i = layers.length - 1; i >= 0; i--) Object.assign(result, layers[i]);
+  return result;
+}
+function stateEntry(state: NftStateView, id: string): NftState[string] | undefined {
+  let cursor: NftStateView = state;
+  while (isStateLayer(cursor)) {
+    if (Object.hasOwn(cursor.changes, id)) return cursor.changes[id];
+    cursor = cursor.parent;
+  }
+  return (cursor as NftState)[id];
+}
+function isStateLayer(state: NftStateView): state is NftStateLayer {
+  return (state as NftStateLayer).kind === 'layer';
+}
+export function stateRoot(state: NftStateView): string {
+  const materialized = materializeState(state);
+  return merkleRoot(Object.keys(materialized).sort().map(id => {
+    const n = materialized[id]; return sha256(JSON.stringify([id, n.owner, n.sequence, n.mintedHeight, n.pageId]));
   }));
 }
 export function mintId(h: Pick<BlockHeader, 'previousHash' | 'height' | 'owner' | 'article' | 'statsSnapshot'>): string {
@@ -143,35 +167,49 @@ export function applyTransfer(state: NftState, tx: NftTransfer, height: number):
   if (!verifySignature(tx.publicKey, tx.from, nftTransferMessage(tx), tx.signature)) throw new Error('Invalid transfer signature');
   state[tx.cardHash] = { ...nft, owner: tx.to, sequence: tx.sequence + 1 };
 }
-export function transition(parent: NftState, header: BlockHeader, txs: NftTransfer[]): NftState {
+export function applyTransferOverlay(state: NftStateView, tx: NftTransfer, height: number): NftStateLayer {
-  const next = structuredClone(parent);
+  if (!transferSchema.safeParse(tx).success) throw new Error('Invalid transfer encoding/network');
+  const nft = stateEntry(state, tx.cardHash);
+  if (!nft || nft.owner !== tx.from || tx.from === tx.to || nft.sequence !== tx.sequence) {
+    throw new Error('Invalid NFT owner/sequence');
+  }
+  if (height - nft.mintedHeight < CONSENSUS.coinbaseMaturity) throw new Error('NFT coinbase is immature');
+  if (!verifySignature(tx.publicKey, tx.from, nftTransferMessage(tx), tx.signature)) throw new Error('Invalid transfer signature');
+  const changes: NftState = Object.create(null) as NftState;
+  changes[tx.cardHash] = { ...nft, owner: tx.to, sequence: tx.sequence + 1 };
+  return { kind: 'layer', parent: state, changes };
+}
+export function transition(parent: NftStateView, header: BlockHeader, txs: NftTransfer[]): NftStateLayer {
+  let next = parent;
   const ids = new Set<string>();
   for (const tx of txs) {
     const id = transferId(tx);
     if (ids.has(id)) throw new Error('Duplicate transfer');
-    ids.add(id); applyTransfer(next, tx, header.height);
+    ids.add(id);
+    next = applyTransferOverlay(next, tx, header.height);
   }
   const id = mintId(header);
-  if (next[id]) throw new Error('Duplicate NFT issuance');
+  if (stateEntry(next, id)) throw new Error('Duplicate NFT issuance');
-  next[id] = { owner: header.owner, sequence: 0, mintedHeight: header.height, pageId: header.article.pageId };
+  const changes: NftState = Object.create(null) as NftState;
-  return next;
+  changes[id] = { owner: header.owner, sequence: 0, mintedHeight: header.height, pageId: header.article.pageId };
+  return { kind: 'layer', parent: next, changes };
 }
-export function makeTemplate(ancestors: TestnetBlock[], state: NftState, owner: string,
+export function makeTemplate(ancestors: TestnetBlock[], state: NftStateView, owner: string,
   publicKey: string, transactions: NftTransfer[], article: Article, statsSnapshot: MintStatsSnapshot,
   now = Math.floor(Date.now()/1000)): MiningTemplate {
   if (!hex.safeParse(owner).success || address(publicKey) !== owner) throw new Error('Invalid miner identity');
   const parsedArticle = articleSchema.parse(article);
   const parsedStats = statsSnapshotSchema.parse(statsSnapshot);
   const previousHash = ancestors.at(-1)?.hash ?? GENESIS_HASH;
-  const header: BlockHeader = { version: 4, network: NETWORK, chainId: CHAIN_ID,
+  const header: BlockHeader = { version: 5, network: NETWORK, chainId: CHAIN_ID,
     height: ancestors.length + 1, previousHash, createdAt: Math.max(now, medianTime(ancestors) + 1),
     article: parsedArticle, statsSnapshot: parsedStats, owner, publicKey, nonce: 0, target: expectedTarget(ancestors),
     key: expectedKey(ancestors, ancestors.length + 1), transactionsRoot: merkleRoot(transactions.map(transferId)), stateRoot: '' };
   header.stateRoot = stateRoot(transition(state, header, transactions));
   return { header, transactions, blob: miningBlob(header).toString('hex'), nonceOffset: 32 };
 }
-export async function validateBlock(raw: unknown, ancestors: TestnetBlock[], parentState: NftState,
+export async function validateBlock(raw: unknown, ancestors: TestnetBlock[], parentState: NftStateView,
-  pow: PowEngine, now = Math.floor(Date.now()/1000)): Promise<{block:TestnetBlock;state:NftState}> {
+  pow: PowEngine, now = Math.floor(Date.now()/1000)): Promise<{block:TestnetBlock;state:NftStateView}> {
   if (Buffer.byteLength(JSON.stringify(raw)) > CONSENSUS.maxBlockBytes) throw new Error('Block size limit exceeded');
   const b = blockSchema.parse(raw);
   if (b.height !== ancestors.length + 1 || b.previousHash !== (ancestors.at(-1)?.hash ?? GENESIS_HASH)) throw new Error('Invalid parent/height');

server/testnet/http.ts

@@ -30,7 +30,7 @@ catch(e) {engine.close();await lock.close();await unlink(lockPath);throw e;}
 let peers: PeerNetwork;
 try { peers = new PeerNetwork({ ledger, urls:(process.env.WIKIDECK_PEERS ?? '').split(',').map(s=>s.trim()).filter(Boolean),
   allowPrivate:process.env.WIKIDECK_ALLOW_PRIVATE_PEERS === '1' }); }
-catch(e) {engine.close();await lock.close();await unlink(lockPath);throw e;}
+catch(e) {ledger.close();engine.close();await lock.close();await unlink(lockPath);throw e;}
 const app = express();
 app.disable('x-powered-by');
 app.use(helmet({contentSecurityPolicy:{directives:{defaultSrc:["'self'"],scriptSrc:["'self'"],
@@ -63,7 +63,7 @@ app.get('/api/wikipedia/:pageId', rateLimit({windowMs:60000,limit:60,standardHea
 app.get('/p2p/status',(_req,res)=>res.json({network:NETWORK,chainId:CHAIN_ID,size:ledger.size,tip:ledger.snapshot().tip}));
 app.get('/p2p/blocks',(req,res)=>{
   const offset=Number(req.query.offset??0);
-  if(!Number.isSafeInteger(offset)||offset<0||offset>10000){res.status(400).json({error:'Invalid offset'});return;}
+  if(!Number.isSafeInteger(offset)||offset<0){res.status(400).json({error:'Invalid offset'});return;}
   const blocks=ledger.inventory(offset,32);
   res.json({blocks,next:offset+blocks.length<ledger.size?offset+blocks.length:null});
 });
@@ -119,6 +119,6 @@ let stopping=false;
 for(const signal of ['SIGTERM','SIGINT']as const)process.once(signal,()=>{
   if(stopping)return;stopping=true;peers.close();
   const timeout=setTimeout(()=>process.exit(1),25000);timeout.unref();
-  server.close(async()=>{engine.close();await lock.close();await unlink(lockPath).catch(()=>{});clearTimeout(timeout);process.exit(0);});
+  server.close(async()=>{ledger.close();engine.close();await lock.close();await unlink(lockPath).catch(()=>{});clearTimeout(timeout);process.exit(0);});
 });
-server.on('error',async()=>{peers.close();engine.close();await lock.close();await unlink(lockPath).catch(()=>{});process.exit(1);});
+server.on('error',async()=>{peers.close();ledger.close();engine.close();await lock.close();await unlink(lockPath).catch(()=>{});process.exit(1);});

server/testnet/ledger.ts

@@ -1,88 +1,155 @@
-import { mkdir, open, rename, unlink } from 'node:fs/promises';
+import { chmod, mkdir, readFile, stat } from 'node:fs/promises';
 import { join } from 'node:path';
-import { randomUUID } from 'node:crypto';
+import { DatabaseSync } from 'node:sqlite';
-import { setTimeout as delay } from 'node:timers/promises';
 import { z } from 'zod';
-import { NETWORK, CONSENSUS, type TestnetBlock, type NftTransfer, type NftState,
+import { NETWORK, CONSENSUS, type TestnetBlock, type NftTransfer,
-  type MiningTemplate, type TestnetSnapshot } from '../../shared/testnet.ts';
+  type NftStateView, type MiningTemplate, type TestnetSnapshot } from '../../shared/testnet.ts';
-import { CHAIN_ID, GENESIS_HASH, applyTransfer, blockSchema, makeTemplate,
+import { CHAIN_ID, GENESIS_HASH, applyTransferOverlay, blockSchema, makeTemplate,
   transferId, transferSchema, validateBlock, workFor } from './consensus.ts';
 import type { PowEngine } from './randomx.ts';
 import type { Article } from '../../shared/protocol.ts';
 import type { MintStatsSnapshot } from '../../shared/enrichment.ts';
 
-const MAX_BLOCKS = 10_000;
 const MAX_PENDING = 256;
-const MAX_FILE = 128 * 1024 * 1024;
+const MAX_LEGACY_FILE = 128 * 1024 * 1024;
 const diskSchema = z.object({ version: z.literal(1), network: z.literal(NETWORK),
-  chainId: z.literal(CHAIN_ID), blocks: z.array(blockSchema).max(MAX_BLOCKS),
+  chainId: z.literal(CHAIN_ID), blocks: z.array(blockSchema).max(10_000),
   pendingTransfers: z.array(transferSchema).max(MAX_PENDING) }).strict();
-type Node = { block: TestnetBlock; state: NftState; work: bigint };
+type Node = { block: TestnetBlock; state: NftStateView; work: bigint };
+const DATABASE_SCHEMA_VERSION = '1';
 
-/** Experimental, bounded DAG. The supplied engine is always used, including on restart.
+/** SQLite-backed DAG. Blocks append transactionally; pending rows are replaced transactionally.
- * One process must own a data directory; there is no cross-process writer locking.
+ * Each block state retains only its NFT delta, sharing the parent's state overlay.
  */
 export class Ledger {
   private readonly nodes = new Map<string, Node>();
   private tip = GENESIS_HASH;
   private pending: NftTransfer[] = [];
+  private pendingState: NftStateView = {};
   private queue: Promise<unknown> = Promise.resolve();
   private queued = 0;
   /** Application-owned connectivity metadata; never part of consensus or persistence. */
   peers = 0;
 
   private readonly file: string;
+  private readonly db: DatabaseSync;
   private readonly pow: PowEngine;
-  private constructor(file: string, pow: PowEngine) { this.file = file; this.pow = pow; }
+  private closed = false;
+  private constructor(file: string, db: DatabaseSync, pow: PowEngine) { this.file = file; this.db = db; this.pow = pow; }
 
   static async open(dir: string, pow: PowEngine): Promise<Ledger> {
     const directory = join(dir, 'testnet');
     await mkdir(directory, { recursive: true, mode: 0o700 });
-    const ledger = new Ledger(join(directory, 'ledger.json'), pow);
+    const file = join(directory, 'ledger.sqlite');
-    let handle;
+    let databaseExists = false;
-    try { handle = await open(ledger.file, 'r'); }
+    try { databaseExists = (await stat(file)).size > 0; }
-    catch (error) {
+    catch (error) { if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error; }
-      if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error;
+    const db = new DatabaseSync(file);
-      await ledger.persist([], []);
-      return ledger;
-    }
-    let text: string;
     try {
-      if ((await handle.stat()).size > MAX_FILE) throw new Error('Testnet ledger file exceeds 128 MiB');
+      const tableExists = (name: string) => Boolean(db.prepare(
-      // Bounded reads also protect against a file growing after stat().
+        "SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ?").get(name));
-      const chunks: Buffer[] = [];
+      if (databaseExists && !tableExists('metadata')) {
-      let size = 0;
+        throw new Error('SQLite ledger metadata missing; refusing to initialize over unknown data');
-      for (;;) {
-        const buffer = Buffer.alloc(Math.min(64 * 1024, MAX_FILE + 1 - size));
-        const { bytesRead } = await handle.read(buffer, 0, buffer.length, null);
-        if (!bytesRead) break;
-        size += bytesRead;
-        if (size > MAX_FILE) throw new Error('Testnet ledger file exceeds 128 MiB');
-        chunks.push(buffer.subarray(0, bytesRead));
       }
-      text = Buffer.concat(chunks).toString('utf8');
+      const readMeta = databaseExists ? db.prepare('SELECT value FROM metadata WHERE key = ?') : undefined;
-    } finally { await handle.close(); }
+      if (!databaseExists) {
-    const disk = diskSchema.parse(JSON.parse(text)); // Mismatches/corruption fail closed, never reset.
+        db.exec('BEGIN IMMEDIATE');
-    for (const raw of disk.blocks) {
+        try {
-      if (ledger.nodes.has(raw.hash)) throw new Error('Duplicate persisted block');
+          db.exec(`CREATE TABLE metadata (key TEXT PRIMARY KEY, value TEXT NOT NULL) WITHOUT ROWID;
-      const node = await ledger.validate(raw);
+            CREATE TABLE blocks (
-      ledger.nodes.set(node.block.hash, node);
+              sequence INTEGER PRIMARY KEY AUTOINCREMENT,
-      if (node.work > ledger.tipWork()) ledger.tip = node.block.hash;
+              hash TEXT NOT NULL UNIQUE,
+              previous_hash TEXT NOT NULL,
+              block_json TEXT NOT NULL
+            );
+            CREATE INDEX blocks_by_parent ON blocks(previous_hash);
+            CREATE TABLE pending_transfers (
+              position INTEGER PRIMARY KEY,
+              id TEXT NOT NULL UNIQUE,
+              transaction_json TEXT NOT NULL
+            );`);
+          const insertMeta = db.prepare('INSERT INTO metadata(key, value) VALUES(?, ?)');
+          insertMeta.run('schema_version', DATABASE_SCHEMA_VERSION);
+          insertMeta.run('network', NETWORK);
+          insertMeta.run('chain_id', CHAIN_ID);
+          db.exec('COMMIT');
+        } catch (error) { db.exec('ROLLBACK'); throw error; }
+      } else {
+        if (!tableExists('blocks') || !tableExists('pending_transfers')) {
+          throw new Error('SQLite ledger schema is incomplete; refusing to repair stored data automatically');
+        }
+        const schemaVersion = readMeta!.get('schema_version') as { value: string } | undefined;
+        const storedNetwork = readMeta!.get('network') as { value: string } | undefined;
+        const storedChain = readMeta!.get('chain_id') as { value: string } | undefined;
+        if (schemaVersion?.value !== DATABASE_SCHEMA_VERSION || storedNetwork?.value !== NETWORK || storedChain?.value !== CHAIN_ID) {
+          throw new Error('SQLite ledger schema or chain mismatch; refusing to reset stored data');
+        }
+      }
+      if (process.platform !== 'win32') await chmod(file, 0o600);
+      db.exec('PRAGMA journal_mode = WAL; PRAGMA synchronous = FULL; PRAGMA foreign_keys = ON; PRAGMA busy_timeout = 5000;');
+
+      const migrated = db.prepare('SELECT value FROM metadata WHERE key = ?').get('legacy_json_migrated') as { value: string } | undefined;
+      if (!migrated) {
+        const legacyFile = join(directory, 'ledger.json');
+        let legacy: unknown;
+        try {
+          const info = await stat(legacyFile);
+          if (info.size > MAX_LEGACY_FILE) throw new Error('Legacy testnet ledger exceeds 128 MiB; preserve it and migrate offline');
+          legacy = JSON.parse(await readFile(legacyFile, 'utf8'));
+        } catch (error) {
+          if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error;
+        }
+        const insertMeta = db.prepare('INSERT OR REPLACE INTO metadata(key, value) VALUES(?, ?)');
+        db.exec('BEGIN IMMEDIATE');
+        try {
+          if (legacy !== undefined) {
+            const oldDisk = diskSchema.parse(legacy);
+            const insertBlock = db.prepare('INSERT INTO blocks(hash, previous_hash, block_json) VALUES(?, ?, ?)');
+            for (const block of oldDisk.blocks) insertBlock.run(block.hash, block.previousHash, JSON.stringify(block));
+            const insertPending = db.prepare('INSERT INTO pending_transfers(position, id, transaction_json) VALUES(?, ?, ?)');
+            oldDisk.pendingTransfers.forEach((tx, index) => insertPending.run(index, transferId(tx), JSON.stringify(tx)));
+          }
+          insertMeta.run('legacy_json_migrated', '1');
+          db.exec('COMMIT');
+        } catch (error) { db.exec('ROLLBACK'); throw error; }
+      }
+
+      const ledger = new Ledger(file, db, pow);
+      for (const row of db.prepare('SELECT block_json FROM blocks ORDER BY sequence').iterate() as Iterable<{ block_json: string }>) {
+        const raw = blockSchema.parse(JSON.parse(row.block_json));
+        if (ledger.nodes.has(raw.hash)) throw new Error('Duplicate persisted block');
+        const node = await ledger.validate(raw);
+        ledger.nodes.set(node.block.hash, node);
+        if (node.work > ledger.tipWork()) ledger.tip = node.block.hash;
+      }
+      let state = ledger.tipState();
+      const seen = new Set(ledger.canonical().flatMap(b => b.transactions.map(transferId)));
+      for (const row of db.prepare('SELECT id, transaction_json FROM pending_transfers ORDER BY position').iterate() as Iterable<{ id: string; transaction_json: string }>) {
+        const tx = transferSchema.parse(JSON.parse(row.transaction_json));
+        const id = transferId(tx);
+        if (row.id !== id || seen.has(id)) throw new Error('Duplicate or mismatched persisted transfer');
+        state = applyTransferOverlay(state, tx, ledger.height() + 1);
+        seen.add(id);
+        ledger.pending.push(tx);
+      }
+      ledger.pendingState = state;
+      return ledger;
+    } catch (error) {
+      db.close();
+      throw error;
     }
-    const state = structuredClone(ledger.tipState());
-    const seen = new Set(ledger.canonical().flatMap(b => b.transactions.map(transferId)));
-    for (const tx of disk.pendingTransfers) {
-      const id = transferId(tx);
-      if (seen.has(id)) throw new Error('Duplicate persisted transfer');
-      applyTransfer(state, tx, ledger.height() + 1);
-      seen.add(id);
-    }
-    ledger.pending = disk.pendingTransfers;
-    return ledger;
   }
 
   get size(): number { return this.nodes.size; }
-  get allBlocks(): TestnetBlock[] { return this.inventory(0, MAX_BLOCKS); }
+  get allBlocks(): TestnetBlock[] {
+    const rows = this.db.prepare('SELECT block_json FROM blocks ORDER BY sequence').all() as { block_json: string }[];
+    return rows.map(row => blockSchema.parse(JSON.parse(row.block_json)));
+  }
+  close(): void {
+    if (this.closed) return;
+    this.closed = true;
+    this.db.close();
+  }
   block(hash: string): TestnetBlock | undefined {
     const block = this.nodes.get(hash)?.block;
     return block ? structuredClone(block) : undefined;
@@ -91,10 +158,12 @@ export class Ledger {
    * HTTP callers should request limit=32 and return next=offset+length or null.
    */
   inventory(offset: number, limit: number): TestnetBlock[] {
-    if (!Number.isSafeInteger(offset) || offset < 0 || !Number.isSafeInteger(limit) || limit < 1 || limit > MAX_BLOCKS) {
+    if (!Number.isSafeInteger(offset) || offset < 0 || !Number.isSafeInteger(limit) || limit < 1 || limit > 1000) {
       throw new Error('Invalid inventory range');
     }
-    return structuredClone(Array.from(this.nodes.values()).slice(offset, offset + limit).map(n => n.block));
+    const rows = this.db.prepare('SELECT block_json FROM blocks ORDER BY sequence LIMIT ? OFFSET ?')
+      .all(limit, offset) as { block_json: string }[];
+    return rows.map(row => blockSchema.parse(JSON.parse(row.block_json)));
   }
 
   async submit(raw: unknown): Promise<{ accepted: boolean; reorg: boolean }> {
@@ -102,20 +171,21 @@ export class Ledger {
     return this.serialize(async () => {
       const parsed = blockSchema.parse(input);
       if (this.nodes.has(parsed.hash)) return { accepted: false, reorg: false };
-      if (this.size >= MAX_BLOCKS) throw new Error('Experimental testnet limit: 10000 stored blocks');
       const node = await this.validate(parsed);
       const wins = node.work > this.tipWork();
       const oldChain = this.canonical();
       const chain = wins ? [...this.branch(node.block.previousHash), node.block] : oldChain;
       const hashes = new Set(chain.map(b => b.hash));
       const disconnected = wins ? oldChain.filter(b => !hashes.has(b.hash)) : [];
-      const pending = wins ? this.filterPending(node.state, chain,
+      const filtered = wins ? this.filterPending(node.state, chain,
-        [...disconnected.flatMap(b => b.transactions), ...this.pending]) : this.pending;
+        [...disconnected.flatMap(b => b.transactions), ...this.pending]) : undefined;
+      const pending = filtered?.pending ?? this.pending;
       // Nothing visible changes until the atomic replacement succeeds.
-      await this.persist([...Array.from(this.nodes.values(), n => n.block), node.block], pending);
+      await this.persist(node.block, pending);
       this.nodes.set(node.block.hash, node);
       if (wins) this.tip = node.block.hash;
       this.pending = pending;
+      if (filtered) this.pendingState = filtered.state;
       return { accepted: true, reorg: disconnected.length > 0 };
     });
   }
@@ -127,12 +197,11 @@ export class Ledger {
       const id = transferId(tx);
       if (this.pending.some(t => transferId(t) === id)) return id;
       if (this.pending.length >= MAX_PENDING) throw new Error('Testnet mempool limit: 256 transfers');
-      const state = structuredClone(this.tipState());
+      const nextPendingState = applyTransferOverlay(this.pendingState, tx, this.height() + 1);
-      for (const pending of this.pending) applyTransfer(state, pending, this.height() + 1);
-      applyTransfer(state, tx, this.height() + 1); // No speculative issuance.
       const pending = [...this.pending, tx];
-      await this.persist(Array.from(this.nodes.values(), n => n.block), pending);
+      await this.persist(undefined, pending);
       this.pending = pending;
+      this.pendingState = nextPendingState;
       return id;
     });
   }
@@ -152,7 +221,7 @@ export class Ledger {
   }
 
   private height(): number { return this.nodes.get(this.tip)?.block.height ?? 0; }
-  private tipState(): NftState { return this.nodes.get(this.tip)?.state ?? {}; }
+  private tipState(): NftStateView { return this.nodes.get(this.tip)?.state ?? {}; }
   private tipWork(): bigint { return this.nodes.get(this.tip)?.work ?? 0n; }
   private branch(hash: string): TestnetBlock[] {
     const result: TestnetBlock[] = [];
@@ -171,19 +240,20 @@ export class Ledger {
     const { block, state } = await validateBlock(raw, ancestors, parent?.state ?? {}, this.pow);
     return { block, state, work: (parent?.work ?? 0n) + workFor(block.target) };
   }
-  private filterPending(parent: NftState, chain: TestnetBlock[], candidates: NftTransfer[]): NftTransfer[] {
+  private filterPending(parent: NftStateView, chain: TestnetBlock[], candidates: NftTransfer[]):
-    const state = structuredClone(parent);
+    { pending: NftTransfer[]; state: NftStateView } {
+    let state = parent;
     const seen = new Set(chain.flatMap(b => b.transactions.map(transferId)));
     const pending: NftTransfer[] = [];
     for (const tx of candidates) {
       if (pending.length === MAX_PENDING) break;
       const id = transferId(tx);
       if (seen.has(id)) continue;
-      try { applyTransfer(state, tx, chain.length + 1); }
+      try { state = applyTransferOverlay(state, tx, chain.length + 1); }
       catch { continue; } // Disconnected issuance, immature coins and conflicts are not replayable.
       seen.add(id); pending.push(tx);
     }
-    return pending;
+    return { pending, state };
   }
   private serialize<T>(operation: () => Promise<T>): Promise<T> {
     if (this.queued >= 256) return Promise.reject(new Error('Testnet mutation queue full'));
@@ -192,25 +262,18 @@ export class Ledger {
     this.queue = result.catch(() => undefined).finally(() => { this.queued--; });
     return result;
   }
-  private async persist(blocks: TestnetBlock[], pendingTransfers: NftTransfer[]): Promise<void> {
+  private async persist(block: TestnetBlock | undefined, pendingTransfers: NftTransfer[]): Promise<void> {
-    const data = JSON.stringify({ version: 1, network: NETWORK, chainId: CHAIN_ID, blocks, pendingTransfers });
+    this.db.exec('BEGIN IMMEDIATE');
-    if (Buffer.byteLength(data) > MAX_FILE) throw new Error('Testnet ledger file exceeds 128 MiB');
-    const temp = `${this.file}.${randomUUID()}.tmp`;
     try {
-      const handle = await open(temp, 'wx', 0o600);
+      if (block) this.db.prepare('INSERT INTO blocks(hash, previous_hash, block_json) VALUES(?, ?, ?)')
-      try { await handle.writeFile(data); await handle.sync(); }
+        .run(block.hash, block.previousHash, JSON.stringify(block));
-      finally { await handle.close(); }
+      this.db.exec('DELETE FROM pending_transfers');
-      // Windows scanners can briefly hold the destination open without delete sharing.
+      const insertPending = this.db.prepare('INSERT INTO pending_transfers(position, id, transaction_json) VALUES(?, ?, ?)');
-      // Retry only transient sharing failures; never unlink the old durable ledger.
+      pendingTransfers.forEach((tx, index) => insertPending.run(index, transferId(tx), JSON.stringify(tx)));
-      for (let attempt = 0; ; attempt++) {
+      this.db.exec('COMMIT');
-        try { await rename(temp, this.file); break; }
+    } catch (error) {
-        catch (error) {
+      this.db.exec('ROLLBACK');
-          const code = (error as NodeJS.ErrnoException).code;
+      throw error;
-          if (process.platform !== 'win32' || attempt >= 5 ||
+    }
-            !['EPERM', 'EACCES', 'EBUSY'].includes(code ?? '')) throw error;
-          await delay(10 * 2 ** attempt);
-        }
-      }
-    } finally { await unlink(temp).catch(() => undefined); }
   }
 }

server/testnet/miner.ts

@@ -3,7 +3,7 @@ import { resolve } from 'node:path';
 import { stdin, stdout } from 'node:process';
 import { randomInt } from 'node:crypto';
 import { decryptWallet } from '../../shared/wallet-backup.ts';
-import { blockClaimMessage, NETWORK, type MiningTemplate, type TestnetBlock } from '../../shared/testnet.ts';
+import { blockClaimMessage, CONSENSUS, NETWORK, type MiningTemplate, type TestnetBlock } from '../../shared/testnet.ts';
 import { CHAIN_ID, sha256, miningBlob, mintId, blockSchema } from './consensus.ts';
 import { RandomX } from './randomx.ts';
 
@@ -83,7 +83,7 @@ try {
         header.nonce += 16;
         stdout.write(`\r${(attempts / ((Date.now()-started)/1000)).toFixed(1)} H/s · searching block #${header.height}   `);
         const status = await api('/p2p/status');
-        if (status.tip !== header.previousHash || Date.now()/1000 - header.createdAt > 60) { console.log('\nRefreshing stale work'); break; }
+        if (status.tip !== header.previousHash || Date.now()/1000 - header.createdAt > CONSENSUS.targetSeconds * 2) { console.log('\nRefreshing stale work'); break; }
         // Explicit idle interval; native miner never starts with the web page.
         await new Promise(resolve => setTimeout(resolve,100));
       }

server/testnet/peers.ts

@@ -24,9 +24,9 @@ function publicAddress(ip: string): boolean {
   return family === 6 && globalV6.check(ip, 'ipv6') && !excluded.check(ip, 'ipv6');
 }
 const remoteStatus = z.object({ network: z.literal(NETWORK), chainId: z.literal(CHAIN_ID),
-  size: z.number().int().min(0).max(10_000), tip: z.string().regex(/^[a-f0-9]{64}$/).optional() }).strict();
+  size: z.number().int().min(0).max(Number.MAX_SAFE_INTEGER), tip: z.string().regex(/^[a-f0-9]{64}$/).optional() }).strict();
 const pageSchema = z.object({ blocks: z.array(blockSchema).max(32),
-  next: z.number().int().min(0).max(10_000).nullable() }).strict();
+  next: z.number().int().min(0).max(Number.MAX_SAFE_INTEGER).nullable() }).strict();
 const poolSchema = z.object({ transactions: z.array(transferSchema).max(256) }).strict();
 type Peer = { url: string; connected: boolean; error?: string; offset: number; busy: boolean };
 
@@ -121,8 +121,8 @@ export class PeerNetwork {
     if (peer.offset < status.size) {
       const page = pageSchema.parse(await this.request(peer, `/p2p/blocks?offset=${peer.offset}`));
       const end = peer.offset + page.blocks.length;
-      if (!page.blocks.length || end > 10_000 ||
+      if (!page.blocks.length || !Number.isSafeInteger(end) ||
-        (page.next !== null && (page.next !== end || end >= 10_000)) ||
+        (page.next !== null && page.next !== end) ||
         (page.next === null && end < status.size)) throw new Error('Invalid peer pagination');
       for (const block of page.blocks) {
         if (this.closed) throw new Error('Peer network closed');

server/wiki.ts

@@ -74,7 +74,7 @@ export class WikipediaArticleSource {
             generator: 'random', grnnamespace: '0', grnlimit: '10', prop: 'extracts|pageimages|info|categories',
             exintro: '1', explaintext: '1', exchars: '900', piprop: 'thumbnail', pithumbsize: '800', cllimit: '20' }).toString();
           const response = await this.fetcher(url, { signal: AbortSignal.timeout(3500), redirect: 'error',
-            headers: { User-Agent: 'Wikideck/0.1 (experimental Wikipedia article cards)', Accept: 'application/json' } });
+            headers: { 'User-Agent': 'Wikideck/0.1 (experimental Wikipedia article cards)', Accept: 'application/json' } });
           const data = responseSchema.parse(await readJson(response));
           for (const rawPage of data.query.pages) {
             const parsedPage = pageSchema.safeParse(rawPage);

shared/testnet.ts

@@ -1,9 +1,9 @@
 import type { Article } from './protocol.ts';
 import type { MintStatsSnapshot } from './enrichment.ts';
 
-export const NETWORK = 'wikideck-randomx-testnet-4' as const;
+export const NETWORK = 'wikideck-randomx-testnet-5' as const;
 export const CONSENSUS = {
-  version: 4, targetSeconds: 30, retargetInterval: 10, medianWindow: 11,
+  version: 5, targetSeconds: 15 * 60, retargetInterval: 10, medianWindow: 11,
   commitments: 'sha256-domain-separated-counted-merkle-v1', randomx: '1.2.1', powEndian: 'big',
   maxFutureSeconds: 120, epochBlocks: 2048, epochDelay: 64,
   maxTransactions: 64, coinbaseMaturity: 2, maxBlockBytes: 64 * 1024,
@@ -14,7 +14,7 @@ export type NftTransfer = {
   sequence: number; publicKey: string; signature: string;
 };
 export type BlockHeader = {
-  version: 4; network: typeof NETWORK; chainId: string; height: number;
+  version: 5; network: typeof NETWORK; chainId: string; height: number;
   previousHash: string; createdAt: number; article: Article; statsSnapshot: MintStatsSnapshot;
   owner: string; publicKey: string; nonce: number; target: string; key: string;
   transactionsRoot: string; stateRoot: string;
@@ -26,6 +26,9 @@ export type MiningTemplate = {
   header: BlockHeader; transactions: NftTransfer[]; blob: string; nonceOffset: number;
 };
 export type NftState = Record<string, { owner: string; sequence: number; mintedHeight: number; pageId: number }>;
+/** Persistent overlay: a block stores only NFT entries it changed, sharing its parent's state. */
+export type NftStateLayer = { kind: 'layer'; parent: NftStateView; changes: NftState };
+export type NftStateView = NftState | NftStateLayer;
 export type TestnetSnapshot = {
   network: typeof NETWORK; chainId: string; genesis: string; tip: string; cumulativeWork: string;
   blocks: TestnetBlock[]; transfers: NftTransfer[]; pendingTransfers: NftTransfer[];

shared/wallet-backup.ts

@@ -1,5 +1,5 @@
 // Browser/Node WebCrypto only: this module never reads storage or sends data.
-const NETWORK = 'wikideck-randomx-testnet-4';
+const NETWORK = 'wikideck-randomx-testnet-5';
 const ITERATIONS = 600_000;
 const MAX_BYTES = 32 * 1024;
 const encoder = new TextEncoder();
@@ -7,7 +7,7 @@ const encoder = new TextEncoder();
 export type WalletBackup = {
   format: 'wikideck-wallet-backup';
   version: 1;
-  network: 'wikideck-randomx-testnet-4';
+  network: 'wikideck-randomx-testnet-5';
   kdf: 'PBKDF2-SHA256';
   iterations: 600000;
   cipher: 'AES-256-GCM';

src/WalletPanel.tsx

@@ -83,7 +83,7 @@ export function WalletPanel({ wallet, onRestore, onNotice }: Props) {
 
   return <section aria-labelledby={`${id}-title`} aria-busy={busy}>
     <h2 id={`${id}-title`}>Wallet backup &amp; recovery</h2>
-    <p>This is a recoverable P-256 wallet for wikideck-randomx-testnet-4, separate from your old demo wallet.</p>
+    <p>This is a recoverable P-256 wallet for wikideck-randomx-testnet-5, separate from your old demo wallet.</p>
     <p>The exportable private key is stored locally in this browser’s IndexedDB, not encrypted by this password.
       Scripts running on this origin or someone with access to your browser profile may access it.
       Clearing site data without a backup loses access.</p>

tests/api.test.ts

@@ -7,6 +7,7 @@ import { tmpdir } from 'node:os';
 import { delimiter, join, resolve } from 'node:path';
 import { fileURLToPath } from 'node:url';
 import { generateKeyPairSync, sign } from 'node:crypto';
+import { DatabaseSync } from 'node:sqlite';
 import { setTimeout as delay } from 'node:timers/promises';
 import { RandomX } from '../server/testnet/randomx.ts';
 import { address, CHAIN_ID, GENESIS_HASH, MAX_TARGET, miningBlob, mintId, sha256,
@@ -171,7 +172,15 @@ test('native RandomX HTTP: three blocks, mined transfer, peer gossip/sync and du
       assert.deepEqual(await json(base, '/api/blocks', 201, block), { accepted: true, reorg: false });
       assert.deepEqual(await json(base, '/api/blocks', 200, block), { accepted: false, reorg: false });
     }
-    const persisted = (dir: string) => readFile(join(dir, 'testnet', 'ledger.json'), 'utf8');
+    const persisted = (dir: string) => {
+      const db = new DatabaseSync(join(dir, 'testnet', 'ledger.sqlite'));
+      try {
+        return JSON.stringify({
+          blocks: db.prepare('SELECT hash, previous_hash, block_json FROM blocks ORDER BY sequence').all(),
+          pending: db.prepare('SELECT position, id, transaction_json FROM pending_transfers ORDER BY position').all(),
+        });
+      } finally { db.close(); }
+    };
 
     try {
       engine = new RandomX({ binary }); // Always the actual reference RandomX engine, never a test double.
@@ -209,7 +218,7 @@ test('native RandomX HTTP: three blocks, mined transfer, peer gossip/sync and du
       for (const path of ['/api/missing', '/p2p/missing', `/api/nfts/${'0'.repeat(64)}`]) {
         await (await request(first.base, path, 404)).text();
       }
-      for (const offset of ['-1', '1.5', '10001', 'nope']) {
+      for (const offset of ['-1', '1.5', '9007199254740992', 'nope']) {
         await (await request(first.base, `/p2p/blocks?offset=${offset}`, 400)).text();
       }
 

tests/testnet-consensus.test.ts

@@ -1,7 +1,7 @@
 import test from 'node:test';
 import assert from 'node:assert/strict';
 import { merkleRoot, sha256, expectedKey, INITIAL_KEY, expectedTarget, MAX_TARGET, CHAIN_ID } from '../server/testnet/consensus.ts';
-import { NETWORK, type TestnetBlock } from '../shared/testnet.ts';
+import { CONSENSUS, NETWORK, type TestnetBlock } from '../shared/testnet.ts';
 
 test('Merkle commitments distinguish padded leaves, ordering, and node/leaf domains',()=>{
   const a=sha256('a'),b=sha256('b'),c=sha256('c');
@@ -20,7 +20,7 @@ test('RandomX keys rotate only after the protocol delay using an ancestor, never
 });
 test('difficulty retargets deterministically at exact windows and is bounded',()=>{
   assert.equal(expectedTarget([]),MAX_TARGET);
-  const blocks=Array.from({length:10},(_,i)=>({target:MAX_TARGET,createdAt:1000+i*30}as TestnetBlock));
+  const blocks=Array.from({length:10},(_,i)=>({target:MAX_TARGET,createdAt:1000+i*CONSENSUS.targetSeconds}as TestnetBlock));
   assert.equal(expectedTarget(blocks.slice(0,9)),MAX_TARGET);
   assert.equal(expectedTarget(blocks),MAX_TARGET);
   const fast=blocks.map((b,i)=>({...b,createdAt:1000+i}));

tests/testnet-ledger.test.ts

@@ -6,13 +6,15 @@ import { tmpdir } from 'node:os';
 import { join } from 'node:path';
 import { createServer, type IncomingMessage, type ServerResponse } from 'node:http';
 import dns from 'node:dns/promises';
+import { DatabaseSync } from 'node:sqlite';
 import { setTimeout as delay } from 'node:timers/promises';
 import { Ledger } from '../server/testnet/ledger.ts';
 import { PeerNetwork } from '../server/testnet/peers.ts';
 import { CHAIN_ID, GENESIS_HASH, GENESIS_TIME, MAX_TARGET, address, makeTemplate, miningBlob,
   mintId, sha256, stateRoot, transferId, transition, workFor } from '../server/testnet/consensus.ts';
 import { NETWORK, CONSENSUS, blockClaimMessage, nftTransferMessage, testnetOwnerOf,
-  type MiningTemplate, type NftState, type NftTransfer, type TestnetBlock } from '../shared/testnet.ts';
+  type MiningTemplate, type NftState, type NftStateLayer, type NftStateView,
+  type NftTransfer, type TestnetBlock } from '../shared/testnet.ts';
 import { deriveStats } from '../shared/enrichment.ts';
 import type { PowEngine } from '../server/testnet/randomx.ts';
 
@@ -47,7 +49,7 @@ async function mine(template: MiningTemplate, miner: Identity = alice): Promise<
   }
   throw new Error('Test nonce search exhausted');
 }
-type Branch = { blocks: TestnetBlock[]; state: NftState };
+type Branch = { blocks: TestnetBlock[]; state: NftStateView };
 function branch(): Branch { return { blocks: [], state: {} }; }
 function articleFor(_ancestors: TestnetBlock[]) {
   const pageId = 50_000;
@@ -77,10 +79,10 @@ function transfer(cardHash: string, from: Identity = alice, to: Identity = bob,
 }
 async function fixture(t: { after: (fn: () => Promise<void>) => void }) {
   const dir = await mkdtemp(join(tmpdir(), 'wikideck-testnet-'));
-  t.after(() => rm(dir, { recursive: true, force: true }));
   const engine = new FakePow();
   const ledger = await Ledger.open(dir, engine);
-  return { dir, ledger, engine, file: join(dir, 'testnet', 'ledger.json') };
+  t.after(async () => { ledger.close(); engine.close(); await rm(dir, { recursive: true, force: true }); });
+  return { dir, ledger, engine, file: join(dir, 'testnet', 'ledger.sqlite') };
 }
 
 test('testnet genesis is empty, manifest-bound, isolated and defensively copied', async t => {
@@ -99,10 +101,15 @@ test('testnet genesis is empty, manifest-bound, isolated and defensively copied'
   assert.deepEqual(initial.pendingTransfers, []);
   assert.equal(stateRoot({}), sha256(`${NETWORK}:empty`));
   await assert.rejects(readFile(join(dir, 'chain.json')), /ENOENT/);
-  assert.equal(JSON.parse(await readFile(file, 'utf8')).version, 1);
+  assert.equal((await stat(file)).isFile(), true);
-  if (process.platform !== 'win32') assert.equal((await stat(file)).mode & 0o777, 0o600);
+  const database = new DatabaseSync(file);
+  assert.equal((database.prepare('SELECT COUNT(*) AS count FROM blocks').get() as { count: number }).count, 0);
+  database.close();
   const candidate = await mine(ledger.template(alice.owner, alice.publicKey, articleFor([]), statsSnapshot()));
   assert.deepEqual(await ledger.submit(candidate), { accepted: true, reorg: false });
+  const persisted = new DatabaseSync(file);
+  assert.equal((persisted.prepare('SELECT COUNT(*) AS count FROM blocks').get() as { count: number }).count, 1);
+  persisted.close();
   candidate.article.title = 'caller mutation';
   const copy = ledger.block(candidate.hash)!;
   copy.article.title = 'getter mutation';
@@ -114,8 +121,9 @@ test('testnet genesis is empty, manifest-bound, isolated and defensively copied'
 
 test('DAG retains equal-work branches, serializes duplicates and selects strictly greater cumulative work', async t => {
   const { ledger, dir } = await fixture(t);
+  t.mock.method(Date, 'now', () => (GENESIS_TIME + CONSENSUS.targetSeconds * 20) * 1000);
   const slow = branch(), fast = branch();
-  for (let i = 0; i < 12; i++) await ledger.submit(await extend(slow, alice, [], 30));
+  for (let i = 0; i < 12; i++) await ledger.submit(await extend(slow, alice, [], CONSENSUS.targetSeconds));
   for (let i = 0; i < 10; i++) {
     const result = await ledger.submit(await extend(fast, bob, [], 1));
     assert.equal(result.reorg, false);
@@ -136,6 +144,7 @@ test('DAG retains equal-work branches, serializes duplicates and selects strictl
   for (const b of stored) { assert.ok(seen.has(b.previousHash)); seen.add(b.hash); }
   assert.deepEqual(ledger.inventory(10, 3), stored.slice(10, 13));
   const restart = await Ledger.open(dir, new FakePow());
+  t.after(async () => { restart.close(); });
   assert.deepEqual(restart.snapshot(), ledger.snapshot());
   assert.deepEqual(restart.allBlocks, stored);
 });
@@ -148,6 +157,7 @@ test('equal-work ties preserve the current tip across restart and only extension
   assert.deepEqual(await ledger.submit(b1), { accepted: true, reorg: false });
   assert.equal(ledger.snapshot().tip, a1.hash);
   const restart = await Ledger.open(dir, new FakePow());
+  t.after(async () => { restart.close(); });
   assert.equal(restart.snapshot().tip, a1.hash);
   const b2 = await extend(b, bob);
   assert.deepEqual(await restart.submit(b2), { accepted: true, reorg: true });
@@ -169,6 +179,7 @@ test('NFT maturity, ordered mempool, block commitments and replay protection', a
   assert.deepEqual(ledger.snapshot().pendingTransfers, [tx, tx2]);
   assert.equal(testnetOwnerOf(ledger.snapshot(), minted.tokenId), alice.owner); // Unconfirmed is not ownership.
   const restarted = await Ledger.open(dir, new FakePow());
+  t.after(async () => { restarted.close(); });
   assert.deepEqual(restarted.snapshot().pendingTransfers, [tx, tx2]);
   const block = await mine(restarted.template(alice.owner, alice.publicKey,
     articleFor(restarted.snapshot().blocks), statsSnapshot()));
@@ -203,7 +214,9 @@ test('reorg rolls back NFT ownership and requeues only still-valid disconnected
   assert.deepEqual(snapshot.transfers, []);
   assert.deepEqual(snapshot.pendingTransfers, [tx, tx2]); // Orphan issuance is discarded.
   assert.ok(ledger.block(committed.hash));
-  assert.deepEqual((await Ledger.open(dir, new FakePow())).snapshot(), snapshot);
+  const restored = await Ledger.open(dir, new FakePow());
+  t.after(async () => { restored.close(); });
+  assert.deepEqual(restored.snapshot(), snapshot);
   const next = await mine(ledger.template(bob.owner, bob.publicKey,
     articleFor(ledger.snapshot().blocks), statsSnapshot()), bob);
   await ledger.submit(next);
@@ -252,53 +265,88 @@ test('invalid parent, manifest, proof, article/stat snapshots, and commitments l
   assert.equal(ledger.size, 2);
 });
 
-test('startup rechecks every proof, rejects tampering and mismatched manifests without resetting disk', async t => {
+test('block state is stored as a one-entry delta that shares its parent state layer', async () => {
+  const parent: NftState = Object.assign(Object.create(null) as NftState, {
+    ['a'.repeat(64)]: { owner: alice.owner, sequence: 0, mintedHeight: 1, pageId: 50_000 },
+  });
+  const chain: Branch = { blocks: [], state: parent };
+  await extend(chain);
+  const layer = chain.state as NftStateLayer;
+  assert.equal(layer.kind, 'layer');
+  assert.equal(layer.parent, parent);
+  assert.equal(Object.keys(layer.changes).length, 1);
+});
+
+test('SQLite startup rechecks every proof and fails closed on database corruption or manifest mismatch', async t => {
   const { dir, ledger, file } = await fixture(t);
   const chain = branch();
   await ledger.submit(await extend(chain));
   await ledger.submit(await extend(chain));
   const checking = new FakePow();
-  await Ledger.open(dir, checking);
+  const verified = await Ledger.open(dir, checking);
+  t.after(async () => { verified.close(); });
   assert.equal(checking.calls, 2);
-  const original = await readFile(file, 'utf8');
+  const originalDb = new DatabaseSync(file);
-  const disk = JSON.parse(original);
+  const row = originalDb.prepare('SELECT hash, block_json FROM blocks ORDER BY sequence LIMIT 1').get() as { hash: string; block_json: string };
-  for (const changed of [ { ...disk, chainId: 'ff'.repeat(32) }, { ...disk, network: 'wrong' },
+  const originalBlock = JSON.parse(row.block_json) as TestnetBlock;
-    { ...disk, version: 2 }, { ...disk, blocks: [...disk.blocks].reverse() },
+  originalDb.close();
-    { ...disk, blocks: [disk.blocks[0], disk.blocks[0]] },
+  verified.close();
-    { ...disk, blocks: [{ ...disk.blocks[0], stateRoot: '00'.repeat(32) }] },
+  ledger.close();
-    { ...disk, pendingTransfers: [transfer(chain.blocks[0].tokenId, bob)] } ]) {
+
-    const text = JSON.stringify(changed);
+  const malformed = new DatabaseSync(file);
-    await writeFile(file, text);
+  malformed.prepare('UPDATE blocks SET block_json = ? WHERE hash = ?').run('{truncated', row.hash);
-    await assert.rejects(Ledger.open(dir, new FakePow()));
+  malformed.close();
-    assert.equal(await readFile(file, 'utf8'), text);
-  }
-  await writeFile(file, original);
-  await assert.rejects(Ledger.open(dir, { hash: async () => 'ff'.repeat(32), close() {} }), /RandomX proof/);
-  await writeFile(file, '{truncated');
   await assert.rejects(Ledger.open(dir, new FakePow()));
+  const restoreMalformed = new DatabaseSync(file);
+  restoreMalformed.prepare('UPDATE blocks SET block_json = ? WHERE hash = ?').run(row.block_json, row.hash);
+  restoreMalformed.close();
+
+  const tampered = new DatabaseSync(file);
+  tampered.prepare('UPDATE blocks SET block_json = ? WHERE hash = ?')
+    .run(JSON.stringify({ ...originalBlock, stateRoot: '00'.repeat(32) }), row.hash);
+  tampered.close();
+  await assert.rejects(Ledger.open(dir, new FakePow()), /state commitment/);
+  const restoreTampered = new DatabaseSync(file);
+  restoreTampered.prepare('UPDATE blocks SET block_json = ? WHERE hash = ?').run(row.block_json, row.hash);
+  restoreTampered.prepare("UPDATE metadata SET value = 'wrong' WHERE key = 'network'").run();
+  restoreTampered.close();
+  await assert.rejects(Ledger.open(dir, new FakePow()), /mismatch/);
+  const restoreManifest = new DatabaseSync(file);
+  restoreManifest.prepare('UPDATE metadata SET value = ? WHERE key = ?').run(NETWORK, 'network');
+  restoreManifest.close();
+  await assert.rejects(Ledger.open(dir, { hash: async () => 'ff'.repeat(32), close() {} }), /RandomX proof/);
 });
 
-test('failed atomic replacement never commits memory, and queued mutations recover', async t => {
+test('failed SQLite transaction never commits memory, and later mutations recover', async t => {
   const { ledger, file } = await fixture(t);
-  const saved = `${file}.saved`;
+  const database = new DatabaseSync(file);
-  await rename(file, saved);
+  database.exec("CREATE TRIGGER reject_block BEFORE INSERT ON blocks BEGIN SELECT RAISE(ABORT, 'test persistence failure'); END;");
-  await mkdir(file);
+  database.close();
   const valid = await extend(branch());
   await assert.rejects(ledger.submit(valid));
   assert.equal(ledger.size, 0);
   assert.equal(ledger.snapshot().tip, GENESIS_HASH);
-  await rm(file, { recursive: true });
+  const removeTrigger = new DatabaseSync(file);
-  await rename(saved, file);
+  removeTrigger.exec('DROP TRIGGER reject_block');
+  removeTrigger.close();
   await ledger.submit(valid);
   assert.equal(ledger.size, 1);
 });
 
-test('bounded persistence rejects oversized files before reading or replaying proofs', async t => {
+test('legacy JSON ledger imports once into SQLite without altering its source file', async t => {
-  const { dir, file } = await fixture(t);
+  const dir = await mkdtemp(join(tmpdir(), 'wikideck-legacy-ledger-'));
-  await truncate(file, 128 * 1024 * 1024 + 1);
+  const block = await extend(branch());
-  const engine = new FakePow();
+  const original = { version: 1, network: NETWORK, chainId: CHAIN_ID, blocks: [block], pendingTransfers: [] };
-  await assert.rejects(Ledger.open(dir, engine), /128 MiB/);
+  const legacyFile = join(dir, 'testnet', 'ledger.json');
-  assert.equal(engine.calls, 0);
+  await mkdir(join(dir, 'testnet'), { recursive: true });
+  await writeFile(legacyFile, JSON.stringify(original));
+  const ledger = await Ledger.open(dir, new FakePow());
+  t.after(async () => { ledger.close(); await rm(dir, { recursive: true, force: true }); });
+  assert.deepEqual(ledger.snapshot().blocks, [block]);
+  assert.deepEqual(JSON.parse(await readFile(legacyFile, 'utf8')), original);
+  const database = new DatabaseSync(join(dir, 'testnet', 'ledger.sqlite'));
+  assert.equal((database.prepare("SELECT value FROM metadata WHERE key = 'legacy_json_migrated'").get() as { value: string }).value, '1');
+  database.close();
 });
 
 test('mempool is ordered, capped at 256, and templates include at most 64 without new issuance', async t => {
@@ -321,6 +369,7 @@ test('mempool is ordered, capped at 256, and templates include at most 64 withou
   assert.deepEqual(ledger.snapshot().pendingTransfers, transactions.slice(64));
   await assert.rejects(ledger.submitTransfer(transfer(block.tokenId)), /immature/);
   const restarted = await Ledger.open(dir, new FakePow());
+  t.after(async () => { restarted.close(); });
   assert.deepEqual(restarted.snapshot(), ledger.snapshot());
 });
 
@@ -374,6 +423,22 @@ test('peer configuration rejects unsafe origins; explicit private permission ena
   assert.equal(ledger.snapshot().peers, 0);
 });
 
+
+test('unknown SQLite files are rejected without adding or replacing tables', async t => {
+  const dir = await mkdtemp(join(tmpdir(), 'wikideck-foreign-sqlite-'));
+  const file = join(dir, 'testnet', 'ledger.sqlite');
+  await mkdir(join(dir, 'testnet'), { recursive: true });
+  const foreign = new DatabaseSync(file);
+  foreign.exec('CREATE TABLE operator_data(value TEXT NOT NULL); INSERT INTO operator_data VALUES(\'keep\');');
+  foreign.close();
+  t.after(async () => { await rm(dir, { recursive: true, force: true }); });
+  await assert.rejects(Ledger.open(dir, new FakePow()), /metadata missing/);
+  const verify = new DatabaseSync(file);
+  assert.deepEqual((verify.prepare("SELECT name FROM sqlite_master WHERE type = 'table' ORDER BY name").all() as { name: string }[])
+    .map(row => row.name), ['operator_data']);
+  assert.equal((verify.prepare('SELECT value FROM operator_data').get() as { value: string }).value, 'keep');
+  verify.close();
+});
 test('peers verify DNS, manifests, pagination, payloads and consensus; redirects are not followed', async t => {
   const { ledger } = await fixture(t);
   const chain = branch();

tests/wallet-backup.test.ts

@@ -54,7 +54,7 @@ test('Node WebCrypto backup roundtrip preserves SPKI SHA256 identity and signing
     key: createPublicKey({ key: spki, format: 'der', type: 'spki' }), dsaEncoding: 'ieee-p1363',
   }, Buffer.from(signature, 'base64')), true);
   assert.equal(backup.iterations, 600000);
-  assert.equal(backup.network, 'wikideck-randomx-testnet-4');
+  assert.equal(backup.network, 'wikideck-randomx-testnet-5');
   assert.equal(Buffer.from(backup.salt, 'base64').length, 16);
   assert.equal(Buffer.from(backup.iv, 'base64').length, 12);
   assert.ok(Buffer.byteLength(JSON.stringify(backup)) < 32 * 1024);