randomx

docs/randomx-native.md

@@ -0,0 +1,194 @@
+# Native RandomX engine
+
+This is the CPU proof engine used by the Wikideck testnet node and explicit opt-in
+native miner. Browser mining has been removed from the active application. The
+Docker image builds this pinned implementation and runs its known-vector test.
+There is no GPU path, simulated hash, or production fallback hash.
+
+## Source and build
+
+[RandomX v1.2.1](https://github.com/tevador/RandomX/tree/102f8acf90a7649ada410de5499a7ec62e49e1da)
+is fetched by CMake FetchContent from the official HTTPS Git repository, pinned to
+**102f8acf90a7649ada410de5499a7ec62e49e1da** (also the upstream v1.2.1 tag).
+The first configuration requires network access and Git; later builds reuse the
+checkout. Keep upstream copyright/license notices when distributing binaries.
+Do not override FetchContent sources with untrusted local code.
+
+Requirements: CMake 3.24+, Git, a C/C++17 toolchain, and a 64-bit host for fast mode.
+MSVC needs its C++ and MASM tools in a developer terminal. MinGW needs its compiler
+and make on PATH; select the `MinGW Makefiles` generator. Linux/macOS need their
+normal C/C++ build tools. No administrator privileges or huge pages are required.
+On MSVC, upstream's pinned build invokes its own PowerShell assembly-generation
+script; review upstream build tooling before running under restrictive policies.
+
+From the repository root:
+
+```sh
+cmake -S native -B native/build -DCMAKE_BUILD_TYPE=Release -DWIKIDECK_RANDOMX_TESTS=OFF
+cmake --build native/build --config Release --target wikideck-randomx --parallel 2
+```
+
+Convenience scripts (two compiler jobs by default):
+
+```sh
+sh scripts/build-randomx.sh
+sh scripts/build-randomx.sh --test
+```
+
+```powershell
+./scripts/build-randomx.ps1
+./scripts/build-randomx.ps1 -Generator 'MinGW Makefiles' -Test
+```
+
+The scripts propagate configure/build/test failures. PowerShell execution policy
+must permit the script; do not bypass a managed policy. POSIX customization:
+`JOBS=2`, `RANDOMX_BUILD_DIR=/absolute/build/path`; PowerShell customization:
+`-Jobs 2`, `-BuildDirectory <path>`. Select a toolchain/generator through the normal
+CMake environment or a manual configuration. Don't switch generators in an
+existing build directory.
+
+All generators place the executable directly under [native/build](../native/build)
+(with `.exe` on Windows), including Release builds with multi-config generators.
+Upstream tests/benchmark/codegen are excluded from the default build: v1.2.1 does
+not provide a tests-disable option. No benchmark or dataset-heavy fast-mode test
+runs automatically.
+
+## Modes and resource use
+
+Run `wikideck-randomx` with no arguments for **light** mode (256 MiB cache plus
+approximately 2 MiB VM scratchpad and runtime overhead). `--fast` adds the real
+approximately 2080 MiB dataset; the 256 MiB cache is retained for key changes, so
+budget over 2.3 GiB per process. Dataset initialization is single-threaded and can
+be expensive. Neither mode allocates RandomX memory until its first valid request.
+Cache, dataset, and VM are reused for the same key. A changed key reinitializes the
+cache and, in fast mode, the entire dataset before creating a fresh VM.
+
+Flags start with `randomx_get_flags()` (CPU-supported AES/JIT/Argon2 optimization).
+`RANDOMX_FLAG_SECURE` is explicitly added for VM JIT W^X, and upstream is compiled
+with `RANDOMX_FORCE_SECURE` for cache JIT policy too. Platforms without JIT use the
+reference interpreter; allocation/JIT failures are fatal, never a fake PoW result.
+No large pages are requested. RAII releases VM, dataset, then cache on normal EOF
+and caught errors. Public PoW key/cache data is not secret and is released, not
+claimed to be securely erased from physical memory.
+
+## Line protocol
+
+One ASCII command per LF-terminated line; CRLF is also accepted. Exactly one ASCII
+response line, flushed immediately, per valid command:
+
+```text
+hash <key64hex> <blobhex>
+<hash64hex>
+
+mine <key64hex> <blobhex> <target64hex> <nonceOffsetDecimal> <maxAttemptsDecimal>
+found <hash64hex> <nonceDecimal>
+exhausted
+```
+
+`found` and `exhausted` are alternatives, not two responses. Input hex may use
+either case; output hashes are lowercase in the RandomX byte order. Keys and
+targets are exactly 32 bytes (64 hex characters); blobs are 1..8192 bytes.
+Tokens use exactly one ASCII space, without leading/trailing whitespace. Commands
+are capped at 16600 bytes before LF. Invalid characters, malformed/overflowing
+unsigned decimal, extra tokens, invalid bounds, or a partial final line cause a
+concise stderr error and nonzero exit, with no fabricated response on stdout.
+Empty stdin EOF exits successfully. There are no stdout banners or diagnostic logs.
+
+Mining requires `maxAttempts` in 1..10000 and `nonceOffset + 8 <= blob.size`.
+The initial nonce is read from the blob's existing **eight little-endian bytes**.
+The first attempt uses that nonce, then increments by one and rewrites LE8 for
+each attempt. Only those eight bytes change. At uint64 maximum, search ends
+without wrapping. The native response contains an exact unsigned decimal nonce.
+
+**Target convention: interpret the 32 output bytes and target bytes as unsigned
+BIG-ENDIAN integers; accept `hash <= target`. This intentionally differs from
+Monero's convention.** In particular, equal hashes pass. Never reverse the hash
+or target when using this interface.
+
+## Node API
+
+[RandomX](../server/testnet/randomx.ts) exports:
+
+```ts
+interface PowEngine {
+  hash(key: string, blob: Uint8Array): Promise<string>;
+  close(): void;
+}
+class RandomX implements PowEngine {
+  constructor(options?: { binary?: string; fast?: boolean });
+  hash(key: string, blob: Uint8Array): Promise<string>;
+  mine(key: string, blob: Uint8Array, target: string,
+       nonceOffset: number, maxAttempts: number):
+    Promise<{ hash: string; nonce: number } | null>;
+  close(): void;
+}
+```
+
+Binary precedence: `options.binary`, `RANDOMX_BINARY`, then the executable under
+the repository's native build directory (resolved relative to this module,
+independent of working directory). Relative explicit paths are resolved against
+the caller's working directory. Configure only a trusted executable: this is a
+native-code trust boundary, not a sandbox. The adapter uses `spawn` with a fixed
+argument array, `shell: false`, and pipe stdio; user data only enters stdin after
+strict validation. Missing/non-executable files throw in the constructor. Other
+spawn failures are retained and reject current/future requests.
+
+Each instance owns one long-lived child, one active command, and at most 64 total
+active/queued requests. Calls are FIFO, including mixed hash/mine calls and key
+changes. Inputs are snapshotted before queuing. The 120-second deadline starts
+when each command is sent (includes cold cache/dataset initialization); queued
+requests wait for earlier work. Keep mining batches small enough to finish on the
+host CPU. The process is killed on timeout, malformed/unsolicited/oversized output,
+stream error, or crash, rejecting all outstanding work and subsequent calls.
+Responses and retained stderr are bounded. No automatic retry or process restart.
+
+`mine` returns `null` for exhaustion and validates returned hash syntax, target,
+and nonce range (it does not independently recompute a native result). Because the
+API returns `number`, its entire requested nonce range must fit
+`Number.MAX_SAFE_INTEGER`; higher uint64 nonces are supported only by the direct
+native protocol. Invalid inputs reject without sending native work.
+
+Always call `close()` in `finally`. It is idempotent, rejects pending/queued work,
+and sends EOF for graceful RAII cleanup; a busy child is forcibly killed after
+one second. A failed/closed instance cannot be reopened.
+
+## Real upstream known-vector check
+
+The opt-in `--self-test` executable argument validates upstream's exact
+[Hash test 1a/2a](https://github.com/tevador/RandomX/blob/102f8acf90a7649ada410de5499a7ec62e49e1da/src/tests/tests.cpp):
+
+- Key: ASCII `test key 000` (12 bytes, **not NUL-terminated or zero-padded**).
+- Input: ASCII `This is a test` (14 bytes, no terminator).
+- Expected RandomX hash:
+  `639183aae1bf4c9a35884cb46b09cad9175f04efd7684e7262a0ac1c2f0b4e3f`.
+
+The self-test uses an internal API so the line protocol retains its strict
+32-byte-key requirement. Padding this upstream key to 32 bytes would change the
+hash and is not a valid reproduction of the vector. Self-test success is reported
+on stderr only, with exit zero; mismatch exits nonzero. `--fast --self-test` checks
+the same real vector in dataset mode, but requires the full memory allocation.
+
+```sh
+cmake -S native -B native/build -DCMAKE_BUILD_TYPE=Release -DWIKIDECK_RANDOMX_TESTS=ON
+cmake --build native/build --config Release --target wikideck-randomx --parallel 2
+ctest --test-dir native/build -C Release --output-on-failure
+```
+
+This test is OFF by default and is separate from the application's tests. Without
+a built native binary, no native engine or native test is run by the application.
+
+## Validation record
+
+Verified on Windows x64 with CMake 3.26.4, MinGW GCC 13.1.0 and Node 24.22.0:
+
+- Release native build with two compiler jobs; opt-in CTest upstream vector passed.
+- Production TypeScript/application build passed.
+- Ad-hoc actual-native light-mode checks passed: FIFO hashes, same-key reuse,
+  key changes and restoration, mining success/exhaustion, target equality and
+  big-endian comparison, LE8 nonce handling, 8192-byte blob, strict invalid-input
+  rejection, native uint64 nonce output, missing binary, queue bounds, crash
+  rejection, idempotent close, and stdin EOF.
+- Both build scripts passed syntax checks.
+
+Fast-mode allocation/runtime and Linux/macOS/MSVC builds were not run. The memory
+and initialization cost of the full dataset remains an explicit opt-in.

docs/testnet-consensus.md

@@ -0,0 +1,71 @@
+# Experimental testnet consensus
+
+Implementation reference for the current tree, **not a public-mainnet specification or security certification**. See [README](../README.md) for setup/wallet commands and [native-engine details](randomx-native.md) for RandomX tooling. Consensus code: [shared serialization](../shared/testnet.ts), [validation](../server/testnet/consensus.ts), [DAG/persistence](../server/testnet/ledger.ts), [peer transport](../server/testnet/peers.ts), [HTTP](../server/testnet/http.ts), [miner](../server/testnet/miner.ts).
+
+## Network, genesis and catalog
+
+- Network: `wikideck-randomx-testnet-2`, block version **2**. Genesis has no blocks, NFTs or work; first mined height is 1. No privileged issuance/admin mint key.
+- `GENESIS_TIME = 1792411200` (Unix seconds). Let `S(x)` mean lowercase hex SHA-256 of UTF-8 `x`, and `J(x)` mean compact JavaScript `JSON.stringify(x)`; field order below is consensus-critical.
+- `MAX_TARGET` is `0fff` followed by 60 `f` digits. `CHAIN_ID = S(J([NETWORK, CONSENSUS, GENESIS_TIME, MAX_TARGET, STARTER_ARTICLES.map(articleTuple)]))`, using the constants object's insertion order in the implementation.
+- `GENESIS_HASH = S(NETWORK + ':genesis:' + CHAIN_ID)`; `INITIAL_KEY = S(NETWORK + ':initial-key:' + CHAIN_ID)`. Nodes independently derive the same manifest and reject other network/chain IDs.
+- The [catalog](../shared/catalog.ts) contains four fixed **original summaries**: Albert Einstein (736), Moon (19331), Octopus (22780), Periodic table (23053). No live Wikipedia request determines consensus metadata. `articleTuple = [pageId, revisionId, title, extract, image-or-empty-string, url, source, category]`.
+- The next catalog index is the big-endian integer `S(CHAIN_ID + ':article:' + previousHash)` modulo 4. Validation requires the exact selected tuple. NFT state stores that article's same `pageId`. Changing the catalog/constants currently changes genesis: coordinate a new chain, not a live catalog expansion or demo-asset migration.
+
+## Header, work and three distinct identities
+
+`headerMessage(h)` is `J([NETWORK, version, chainId, height, previousHash, createdAt, articleTuple(article), owner, publicKey, target, key, transactionsRoot, stateRoot])`. It excludes the nonce. Hash its UTF-8 bytes with SHA-256 to obtain **32 raw bytes**, then append the nonce encoded as **8 unsigned little-endian bytes**. The resulting **40-byte mining blob** has nonce offset **32** (not a 32-bit nonce). Node's numeric interface restricts nonces to nonnegative safe integers; the native line protocol supports uint64.
+
+| Identifier | Exact construction / meaning |
+| --- | --- |
+| Block ID (`hash`) | SHA-256 of the raw mining blob, lowercase hex. Does not include ECDSA signature bytes or the separate `powHash` field. |
+| RandomX proof (`powHash`) | Native RandomX of the mining blob using the 32-byte epoch key. Determines work eligibility and game rarity/stats, not the NFT ID. |
+| NFT ID (`tokenId`) | `S(J([NETWORK, CHAIN_ID, 'mint', previousHash, height, owner, article.pageId]))`. Independent of nonce/proof/signature, avoiding a circular dependency through state commitment. |
+
+Interpret RandomX output and target as **unsigned big-endian** integers; accept `powHash <= target` including equality. This differs from Monero's target convention. The initial/easiest threshold accepts exactly **1/16** of uniformly distributed hashes (expected 16 attempts). No fake or browser PoW is accepted by the production node.
+
+The miner signs `J([NETWORK, 'block-claim', blockID])` using ECDSA P-256/SHA-256, IEEE-P1363 encoding. Address is SHA-256 of canonical SPKI DER public-key bytes. Nodes check public-key/address consistency, signature, body schema, exact parent/height, article, target, key, roots, mint ID, block ID and recomputed native proof. Signatures remain required even though their bytes are excluded from block identity.
+
+## Targets, timestamps and epoch boundaries
+
+- Desired spacing: **30 seconds**; retarget interval: **10 ancestors**. Heights 1–10 use the initial target; height 11 first retargets. Otherwise inherit the parent target.
+- At each retarget, compare the last and first timestamps of the previous 10 blocks: expected span is **9 × 30 = 270 seconds**. Clamp elapsed to `floor(270/4)` through `270*4`, multiply the parent target by elapsed/270 with integer division, then clamp target to `[1, MAX_TARGET]`. Selection uses work, not height or wall-clock estimates.
+- A timestamp must exceed the median of up to **11** ancestors (sorted upper-middle for even counts; genesis time if none) and must be at most **120 seconds** ahead of validation time. Keep operator clocks synchronized.
+- Epoch key source height is `floor(max(0, height - 64) / 2048) * 2048`. Zero means `INITIAL_KEY`; otherwise use the ancestor block hash at that source height. Thus heights **1–2111** use the initial key, **2112–4159** use height 2048's hash, and height **4160** switches to height 4096's hash. Forks derive keys from their own ancestors.
+
+## Native NFT transition and commitments
+
+Each block executes its ordered transfers against its parent's NFT state, **then mints one NFT** to the block owner. State entries contain owner, sequence (initially 0), minted height and page ID. Transfers cannot spend this block's new issuance.
+
+A transfer carries network, chain ID, `cardHash` (**NFT token ID**, despite the legacy field name), from, to, sequence, public key and signature. Its signed message is `J([NETWORK, 'nft-transfer', chainId, cardHash, from, to, sequence])`. **Transfer ID is SHA-256 of that message, excluding ECDSA signature bytes** to avoid signature-malleability identity changes. Require the current owner, matching public key, a different recipient, exact current sequence, valid signature, and inclusion height minus mint height **>= 2**; increment sequence on application. Replays and duplicate IDs within a block fail.
+
+`transactionsRoot` commits the **ordered transfer IDs**. `stateRoot` commits the complete **post-transfer, post-mint** state: sort token IDs lexicographically; each input leaf is `S(J([id, owner, sequence, mintedHeight, pageId]))`. `merkleRoot` returns `S(NETWORK + ':empty')` for no leaves. For nonempty input, hash each raw 32-byte input leaf with byte prefix `0x00`; hash pairs of raw children with byte prefix `0x01`, duplicating an odd final child at each layer. Finally return `S(J([NETWORK, 'merkle', originalLeafCount, treeRootHex]))`. Count commitment prevents odd-padding ambiguity; leaf/node prefixes separate domains. These commitment semantics, the RandomX version, and big-endian proof convention are pinned in the genesis consensus manifest.
+
+The mempool holds at most **256** ordered transfers, validating sequentially against the next inclusion height; a template takes the first **64**. The protocol can accept chained transfers of the same NFT in order; the browser conservatively blocks a second pending transfer for a card. A 202 response changes **no canonical ownership**. Mint at height `h` is eligible for inclusion at `h+2` (pool admission can happen at tip `h+1`). Confirmations are `tipHeight - mintedHeight + 1`, not a finality guarantee.
+
+## Fork choice, reorgs and storage
+
+Per-block work is `floor(2^256 / (target + 1))`; accumulate from genesis work zero. Keep all valid known-parent branches. Adopt a tip only for **strictly greater cumulative work**; equal-work ties retain the current tip, including across replay of persisted insertion order. A shorter, harder branch can win.
+
+On reorg, reconstruct ownership from the winning branch. Orphan mints disappear from canonical ownership; disconnected transfers are reconsidered before existing pending entries, in order, only if still valid and not already committed. Conflicting, immature or orphan-issuance transfers are discarded. No checkpoints/finality or rollback protection beyond observed work; an isolated node has no peer-backed security.
+
+The [ledger](../server/testnet/ledger.ts) stores the entire DAG and pending pool in [the testnet ledger](../data/testnet/ledger.json), under `WIKIDECK_DATA_DIR`'s testnet subdirectory. Defaults use project data; Docker uses /app/data. Disk caps: **128 MiB, 10,000 total blocks including side branches**; blocks: **64 KiB and 64 transfers**. Each in-memory DAG node stores a full NFT-state snapshot: potentially **quadratic memory growth**, plus full-file rewrite cost. These are experimental stop limits, not scalable storage or a public DoS defense.
+
+Mutations are serialized (queue cap 256); persist via a synchronized temporary file and atomic rename before exposing new state. Restart independently revalidates every saved proof and transaction; malformed, oversized or mismatched files fail closed without automatic reset. HTTP startup exclusively creates a data-directory lock named testnet-node.lock. It is removed on normal shutdown, **not automatically after a crash**: stop and verify all owning processes have exited before manual stale-lock removal. Never bypass a live lock or share a writer directory. Back up stopped-node volumes/files and wallet exports separately; destructive volume reset cannot recover assets from a wallet alone.
+
+## Fixed peers and HTTP surface
+
+Peers are **explicit operator-configured URL origins**, up to 16, not discovered from network input. Default: HTTPS/public addresses only, checked and pinned during DNS lookup; no credentials, paths, queries, fragments or redirects. `WIKIDECK_ALLOW_PRIVATE_PEERS=1` permits private/HTTP destinations only by explicit operator choice (used for local Docker tests). No bootnodes, DHT, automatic public deployment or claimed Sybil resistance.
+
+Sync polls every **5 seconds**, checks network/chain ID, imports at most **32** topologically ordered DAG blocks per peer/pass, and validates remote pool entries locally. Accepted blocks/transfers are gossiped to available configured peers; busy-peer gossip is dropped, with polling for catch-up. Responses are capped at 2 MiB with a 10-second deadline. Peer count is current outbound connectivity metadata, not consensus membership or a security score.
+
+| Route | Purpose |
+| --- | --- |
+| `GET /healthz` | Process health/network, not sync or consensus security. |
+| `GET /api/state`, `/api/chain` | Canonical blocks, mined/pending transfers, tip/work and next article. |
+| `GET /api/nfts/:id`, `/api/peers` | Canonical NFT owner/sequence/confirmations/metadata; configured peer status. |
+| `POST /api/mining/template`, `/api/blocks` | Public miner identity to template; independently validated signed block submission. |
+| `POST /api/transfer` | Signed transfer admission; 202 means pending. |
+| `GET /p2p/status`, `/p2p/blocks?offset=0`, `/p2p/mempool` | Manifest/size/tip, 32-block inventory pages, pending transactions. |
+| `POST /p2p/block`, `/p2p/transaction` | Same validation as local block/transfer submission. |
+
+POSTs require JSON, enforce supplied browser origins and signatures where applicable; peers without an Origin header are not authenticated by this check. HTTP parsing is capped at 68 KiB; API/P2P rate limits are 300/minute and block/transfer POSTs additionally 30/minute. There are no fees, peer authentication or economic spam deterrents. Same-origin browser RPC trusts the node; it does not independently verify these commitments. Deploy no funds: majority-hashpower, eclipse, censorship, reorg and resource-exhaustion risks require independent audits and adversarial testing before any broader launch.