wikideck/docs/testnet-consensus.md

# Experimental testnet consensus

Implementation reference for the current tree, **not a public-mainnet specification or security certification**. See [README](../README.md) for setup/wallet commands and [native-engine details](randomx-native.md) for RandomX tooling. Consensus code: [shared serialization](../shared/testnet.ts), [validation](../server/testnet/consensus.ts), [DAG/persistence](../server/testnet/ledger.ts), [peer transport](../server/testnet/peers.ts), [HTTP](../server/testnet/http.ts), [miner](../server/testnet/miner.ts).

This document is for operators and developers who need the exact current protocol
rules. It is not a quick-start guide: consensus serialization and constants are
implementation-specific, and changing them can create a different chain.

## Contents

- [Network, genesis and article selection](#network-genesis-and-article-selection)
- [Header, work and three distinct identities](#header-work-and-three-distinct-identities)
- [Targets, timestamps and epoch boundaries](#targets-timestamps-and-epoch-boundaries)
- [Native NFT transition and commitments](#native-nft-transition-and-commitments)
- [Fork choice, reorgs and storage](#fork-choice-reorgs-and-storage)
- [Fixed peers and HTTP surface](#fixed-peers-and-http-surface)

## Network, genesis and article selection

- Network: `wikideck-randomx-testnet-6`, block version **6**. Genesis has no blocks, NFTs or work; first mined height is 1. No privileged issuance/admin mint key. Version 6 adds atomic NFT trade bundles and has a new chain identity; prior-version ledger files and wallet backups are not compatible.
- `GENESIS_TIME = 1792411200` (Unix seconds). Let `S(x)` mean lowercase hex SHA-256 of UTF-8 `x`, and `J(x)` mean compact JavaScript `JSON.stringify(x)`; field order below is consensus-critical.
- `MAX_TARGET` is `0fff` followed by 60 `f` digits. `CHAIN_ID = S(J([NETWORK, CONSENSUS, GENESIS_TIME, MAX_TARGET]))`, using the constants object's insertion order in the implementation. `CONSENSUS` identifies the random Wikipedia source and mint-statistics snapshot format.
- `GENESIS_HASH = S(NETWORK + ':genesis:' + CHAIN_ID)`; `INITIAL_KEY = S(NETWORK + ':initial-key:' + CHAIN_ID)`. Nodes independently derive the same manifest and reject other network/chain IDs.
- [Starter samples](../shared/catalog.ts) remain only as interface previews; they are not eligible for minting. Each mining-template request asks the fixed English Wikipedia API endpoint for up to ten random namespace-0 pages, with a plain-text extract and thumbnail. It makes at most three requests and returns a page with a usable extract and an HTTPS image on `upload.wikimedia.org` or `thumb.wikimedia.org`. A page may be minted repeatedly. There is no static fallback: if no pictured article or its statistics are available, template creation fails and the miner can retry.
- The selected `Article` snapshot and `statsSnapshot` are copied into the block header and covered by the miner's block signature and block ID. `articleTuple = [pageId, revisionId, title, extract, image-or-empty-string, url, source, category]`. The at-mint statistics contain capture time, metadata revision, page bytes, language-link count/truncation, six-month pageviews when available, and the derived Attention/Depth/Reach scores. Missing pageviews are represented as unavailable, never invented zeroes. The node forces a fresh enrichment request for this snapshot; it is captured when the mining template is prepared, since changing it after work begins would invalidate the proof. Current UI statistics are fetched separately and may reflect later edits.
- A repeated page still creates a distinct NFT: token identity includes the parent block, height, and owner. Validators check article/image and stats schemas, the page URL, Wikimedia HTTPS host, pageview summaries, derived scores, and block commitments, but do not constrain repeated page IDs.
- Validators do **not** contact Wikipedia during consensus validation. They verify the committed snapshot's structure and internal metric summaries, not the API provenance or ongoing existence of the page/image. A miner can construct a syntactically valid snapshot itself; this experimental design avoids making block acceptance dependent on an external service being online. The node's template endpoint fetches the article and fresh statistics for ordinary mining clients.

## Header, work and three distinct identities

`headerMessage(h)` is `J([NETWORK, version, chainId, height, previousHash, createdAt, articleTuple(article), statsSnapshotTuple, owner, publicKey, target, key, transactionsRoot, stateRoot])`. `statsSnapshotTuple` commits capture time, revision, size/language counts, ordered monthly views and summary values, plus the three derived scores. It excludes the nonce. Hash its UTF-8 bytes with SHA-256 to obtain **32 raw bytes**, then append the nonce encoded as **8 unsigned little-endian bytes**. The resulting **40-byte mining blob** has nonce offset **32** (not a 32-bit nonce). Node's numeric interface restricts nonces to nonnegative safe integers; the native line protocol supports uint64.

| Identifier | Exact construction / meaning |
| --- | --- |
| Block ID (`hash`) | SHA-256 of the raw mining blob, lowercase hex. Does not include ECDSA signature bytes or the separate `powHash` field. |
| RandomX proof (`powHash`) | Native RandomX of the mining blob using the 32-byte epoch key. Determines work eligibility and game rarity/stats, not the NFT ID. |
| NFT ID (`tokenId`) | `S(J([NETWORK, CHAIN_ID, 'mint', previousHash, height, owner, article.pageId, statsSnapshotTuple]))`. The capture time and page statistics help distinguish repeated mints of the same page. Independent of nonce/proof/signature, avoiding a circular dependency through state commitment. |

Interpret RandomX output and target as **unsigned big-endian** integers; accept `powHash <= target` including equality. This differs from Monero's target convention. The initial/easiest threshold accepts exactly **1/16** of uniformly distributed hashes (expected 16 attempts). No fake or browser PoW is accepted by the production node.

The miner signs `J([NETWORK, 'block-claim', blockID])` using ECDSA P-256/SHA-256, IEEE-P1363 encoding. Address is SHA-256 of canonical SPKI DER public-key bytes. Nodes check public-key/address consistency, signature, body schema, exact parent/height, article, target, key, roots, mint ID, block ID and recomputed native proof. Signatures remain required even though their bytes are excluded from block identity.

## Targets, timestamps and epoch boundaries

- Desired spacing: **15 minutes (900 seconds)**; retarget interval: **10 ancestors**. Heights 1–10 use the initial target; height 11 first retargets. Otherwise inherit the parent target.
- At each retarget, compare the last and first timestamps of the previous 10 blocks: expected span is **9 × 900 = 8,100 seconds**. Clamp elapsed to `floor(8,100/4) = 2,025` through `8,100*4 = 32,400` seconds, multiply the parent target by elapsed/8,100 with integer division, then clamp target to `[1, MAX_TARGET]`. Selection uses work, not height or wall-clock estimates.
- A timestamp must exceed the median of up to **11** ancestors (sorted upper-middle for even counts; genesis time if none) and must be at most **120 seconds** ahead of validation time. Keep operator clocks synchronized.
- Epoch key source height is `floor(max(0, height - 64) / 2048) * 2048`. Zero means `INITIAL_KEY`; otherwise use the ancestor block hash at that source height. Thus heights **1–2111** use the initial key, **2112–4159** use height 2048's hash, and height **4160** switches to height 4096's hash. Forks derive keys from their own ancestors.

## Native NFT transition and commitments

Each block executes its ordered transactions against its parent's NFT state, **then mints one NFT** to the block owner. State entries contain owner, sequence (initially 0), minted height and page ID. Transactions cannot spend this block's new issuance.

An individual transfer carries network, chain ID, `cardHash` (**NFT token ID**, despite the legacy field name), from, to, sequence, public key and signature. Its signed message is `J([NETWORK, 'nft-transfer', chainId, cardHash, from, to, sequence])`. **Transfer ID is SHA-256 of that message, excluding ECDSA signature bytes** to avoid signature-malleability identity changes. Require the current owner, matching public key, a different recipient, exact current sequence, valid signature, and inclusion height minus mint height **>= 2**; increment sequence on application. Replays and duplicate IDs within a block fail.

A trade is one `type: 'trade'` transaction with proposer and counterparty addresses, ordered offered/requested NFT legs (token ID plus current sequence), and one P-256 signature from each party over `J([NETWORK, 'nft-trade', chainId, proposer, counterparty, offeredLegs, requestedLegs])`, where each leg is `[cardHash, sequence]`. Both public keys must derive the respective party addresses. Each side must contain 1–8 distinct NFTs; duplicates across either side are rejected. At inclusion, validation checks **all** selected NFTs against the same pre-trade state, owners, sequences, maturity, and both signatures before writing any changes. If any leg is invalid, the entire trade is rejected; if valid, all offered ownerships move to the counterparty and all requested ownerships to the proposer, with each sequence incremented once. A trade counts as one mempool and block transaction and is committed by one transaction ID. It cannot be partially included or partially replayed. Offers are not reservations; stale offers fail validation.

`transactionsRoot` commits the **ordered transaction IDs** (individual transfer IDs or atomic trade IDs). `stateRoot` commits the complete **post-transaction, post-mint** state: sort token IDs lexicographically; each input leaf is `S(J([id, owner, sequence, mintedHeight, pageId]))`. `merkleRoot` returns `S(NETWORK + ':empty')` for no leaves. For nonempty input, hash each raw 32-byte input leaf with byte prefix `0x00`; hash pairs of raw children with byte prefix `0x01`, duplicating an odd final child at each layer. Finally return `S(J([NETWORK, 'merkle', originalLeafCount, treeRootHex]))`. Count commitment prevents odd-padding ambiguity; leaf/node prefixes separate domains. These commitment semantics, the RandomX version, and big-endian proof convention are pinned in the genesis consensus manifest.

The mempool holds at most **256** ordered transactions (transfers or trade bundles), validating sequentially against the next inclusion height; a template takes the first **64 transaction units**. A 202 response changes **no canonical ownership**. Mint at height `h` is eligible for inclusion at `h+2` (pool admission can happen at tip `h+1`). Confirmations are `tipHeight - mintedHeight + 1`, not a finality guarantee.

## Fork choice, reorgs and storage

Per-block work is `floor(2^256 / (target + 1))`; accumulate from genesis work zero. Keep all valid known-parent branches. Adopt a tip only for **strictly greater cumulative work**; equal-work ties retain the current tip, including across replay of persisted insertion order. A shorter, harder branch can win.

On reorg, reconstruct ownership from the winning branch. Orphan mints disappear from canonical ownership; disconnected transactions are reconsidered before existing pending entries, in order, only if still valid and not already committed. A trade is requeued as one unit only when every leg remains valid; otherwise the complete bundle is discarded. No checkpoints/finality or rollback protection beyond observed work; an isolated node has no peer-backed security.

The [ledger](../server/testnet/ledger.ts) is stored in `data/testnet/ledger.sqlite`, under the `WIKIDECK_DATA_DIR` testnet subdirectory (Docker uses `/app/data`). SQLite WAL mode with full synchronous commits stores each accepted block once in an indexed table and replaces the bounded mempool in the same transaction; writes no longer serialize/rewrite the entire DAG, and there is no 128 MiB/10,000-block ledger cap. A matching v6 legacy `ledger.json` is schema-checked and imported transactionally once; the source JSON is left untouched as a recovery copy. Earlier-protocol ledgers are not compatible and fail closed. Back up the complete stopped-node data directory, including SQLite database/WAL files. Do not copy only `ledger.sqlite` while the node is running.

The in-memory DAG keeps block records and cumulative work, while each block's NFT state is a persistent overlay containing only that block's changes rather than a full state copy. State-root calculation still visits the complete NFT state, and startup still revalidates every stored proof; CPU and startup work therefore grow with chain size. This is materially more storage-efficient, but not yet an archival-scale chain database or a public DoS defense. Blocks remain individually limited to **64 KiB and 64 transaction units**; the mempool remains capped at **256 transaction units**. Mutations are serialized (queue cap 256). HTTP startup exclusively creates a data-directory lock named `testnet-node.lock`. It is removed on normal shutdown, **not automatically after a crash**: stop and verify all owning processes have exited before manual stale-lock removal. Never bypass a live lock or share a writer directory. Back up node data and wallet exports separately; deleting a volume cannot be recovered from a wallet alone.

## Fixed peers and HTTP surface

Peers are **explicit operator-configured URL origins**, up to 16, not discovered from network input. Default: HTTPS/public addresses only, checked and pinned during DNS lookup; no credentials, paths, queries, fragments or redirects. `WIKIDECK_ALLOW_PRIVATE_PEERS=1` permits private/HTTP destinations only by explicit operator choice (used for local Docker tests). No bootnodes, DHT, automatic public deployment or claimed Sybil resistance.

Sync polls every **5 seconds**, checks network/chain ID, imports at most **32** topologically ordered DAG blocks per peer/pass, and validates remote pool entries locally. Accepted blocks and transactions (including complete trade bundles) are gossiped to configured peers; busy-peer gossip is dropped, with polling for catch-up. Responses are capped at 2 MiB with a 10-second deadline. Peer count is current outbound connectivity metadata, not consensus membership or a security score.

| Route | Purpose |
| --- | --- |
| `GET /healthz` | Process health/network, not sync or consensus security. |
| `GET /api/state`, `/api/chain` | Canonical blocks, mined/pending transactions, tip/work and peer count. |
| `GET /api/summary`, `/api/canonical?tip=<hash>&offset=0&limit=16` | Bounded state metadata and current-tip-pinned canonical pages. Full legacy snapshots are limited to 128 blocks and the response-byte budget. See [resource controls](node-resources.md). |
| `GET /api/network` | Selected profile, pinned chain/genesis, candidate launch status and mock-payment boundary. |
| `GET /api/market?offset=0`, `POST /api/market/command` | Node-local mock-payment marketplace; strict signed listing, bid, checkout, delivery and private activity commands. Not consensus escrow. See [marketplace operations](marketplace.md). |
| `GET /api/nfts/:id`, `/api/peers` | Canonical NFT owner/sequence/confirmations, article and at-mint statistics; configured peer status. |
| `GET /api/decks/:address?offset=0&limit=24` | Public canonical deck for a wallet address at one tip; paged, maximum 24 cards per response. This reveals public address holdings. |
| `GET /api/wikipedia/:pageId` | Cached current enrichment for positive English Wikipedia page IDs: categories, views, size, languages and display-only scores. Never a consensus input. |
| `POST /api/mining/template` | Rate-limited public miner identity; fetches a random article with a picture plus fresh at-mint statistics. Returns 503 when either cannot be fetched. |
| `POST /api/blocks` | Independently validates a signed block, proof, article snapshot, required Wikimedia image URL, and statistics snapshot consistency. Repeated page IDs are accepted. |
| `POST /api/transfer` | Signed single-card transfer admission; 202 means pending. |
| `POST /api/trades` | Validates and admits one signed, atomic multi-card trade bundle; 202 means the whole bundle is pending. |
| `GET /p2p/status`, `/p2p/blocks?offset=0`, `/p2p/mempool` | Manifest/size/tip, 32-block inventory pages, pending transactions. |
| `POST /p2p/block`, `/p2p/transaction` | Same validation as local block/transaction submission; transaction endpoint accepts a transfer or atomic trade. |

POSTs require JSON, enforce supplied browser origins and signatures where applicable; peers without an Origin header are not authenticated by this check. HTTP parsing is capped at 68 KiB; API/P2P rate limits are 300/minute and block/transaction/trade submissions additionally 30/minute. Marketplace commands additionally allow 60/minute. There are no consensus fees, peer authentication or economic spam deterrents; market fees are off-chain simulations only. Same-origin browser RPC trusts the node; it does not independently verify these commitments. Deploy no funds: majority-hashpower, eclipse, censorship, reorg and resource-exhaustion risks require independent audits and adversarial testing before any broader launch.

Testnet storage and chain identities above remain unchanged. The [gated mainnet candidate](mainnet-launch.md) selects separate chain/signature domains, storage, lock and wallet identities; it is not a running or certified production network and still rejects real payment modes.