diff --git a/README.md b/README.md index 8ee6b65..fb4641a 100644 --- a/README.md +++ b/README.md @@ -107,6 +107,8 @@ This mines three real blocks using disposable identities. It changes the testnet ## Network and safety limits +[Stage-1 node resource controls](docs/node-resources.md) bound global RPC admission, response bytes and canonical page sizes. The UI reuses paged history; legacy full-state responses stop beyond 128 blocks. A consent-gated loopback-only smoke-load script is available as `test:rpc-load`. This does not resolve long-chain/DAG storage limits or establish production capacity. + Nodes use only explicitly configured peers; there is no discovery service or public bootstrap network. The peer, storage, and HTTP configuration is documented in [docs/testnet-consensus.md](docs/testnet-consensus.md). The implementation is experimental: it has no consensus transaction fees or established economic security (market fees are off-chain simulations), and remains vulnerable to resource exhaustion, isolation/eclipsing, censorship, majority-hashpower attacks, and reorganizations. The ledger now uses transactional SQLite/WAL storage and sparse per-block NFT-state deltas, but nodes still retain DAG block records in memory, rebuild state roots over the full NFT set, and revalidate all proofs at startup. It is not yet an archival-scale production design. HTTPS, container hardening, and passing tests do not constitute a security audit. Before considering any broader deployment, the protocol and cryptography would need independent audits, adversarial network testing, hardened abuse controls, and tested recovery procedures. Do not expose a public node or use valuable assets on this experimental network. diff --git a/docs/node-resources.md b/docs/node-resources.md new file mode 100755 index 0000000..a151018 --- /dev/null +++ b/docs/node-resources.md @@ -0,0 +1,51 @@ +# Node resource controls — stage 1 + +This stage bounds public RPC transport/admission and browser state loading without changing consensus, chain identity, launch approvals or the mock-only payment boundary. **It does not make the node production-ready.** + +## Admission and transport + +API and P2P traffic have separate process-local token buckets. API saturation does not spend P2P tokens. Both lanes still share the event loop, ledger mutation queue, native engine and host resources: this is admission separation, not CPU/process isolation. Budgets are global per process rather than per wallet/IP, so changing wallet or source IP cannot bypass the global limit. Existing per-IP limits remain in place; no permissive proxy trust was enabled. + +Admission precedes JSON parsing, signature verification and route work. Excess traffic receives 503 plus `Retry-After: 1`; there is no unbounded retry queue. JSON request bodies remain capped at 68 KiB and compressed request inflation is disabled. Response JSON is capped before transmission. Byte budget is charged on serialized JSON bytes, not network/TLS framing. Small error responses are not charged recursively when rejecting a payload/budget; edge limits are still needed. + +| Variable | Default | Accepted range | +| --- | --- | --- | +| `WIKIDECK_RPC_REQUESTS_PER_SECOND` | 120 | 1–1000 | +| `WIKIDECK_RPC_WRITES_PER_SECOND` | 10 | 1–100 | +| `WIKIDECK_RPC_MAX_ACTIVE` | 32 | 1–256 | +| `WIKIDECK_RPC_RESPONSE_BYTES` | 2097152 (2 MiB) | 128 KiB–2 MiB | +| `WIKIDECK_RPC_BYTES_PER_SECOND` | 8388608 (8 MiB/s) | 2–128 MiB/s | +| `WIKIDECK_RPC_LEGACY_BLOCKS` | 128 | 0–128 | + +Token bursts equal one second of the configured rate. P2P uses a separate fixed lane of 40 requests/s, 10 writes/s and 16 active HTTP responses, with the same configured byte limits. Active slots are released on response finish/connection close; asynchronous route work already admitted may still be completing after disconnection. Existing bounded native, ledger and Wikimedia queues/concurrency remain necessary. + +HTTP has at most 256 accepted connections, 10-second headers timeout, 15-second request timeout, 5-second idle keepalive and a 30-second socket inactivity timeout. These are process transport protections, not a substitute for proxy timeouts/connection limits, global ingress limits, OS limits or load-balanced per-instance policy. Body parsing is no longer ahead of admission. Health bypasses the API lanes for liveness, but still shares server connections and CPU. + +Configuration is validated before opening node resources. Lower limits can shed legitimate traffic, including sync; test settings against the intended topology before deploying. Do not raise limits merely to hide overload. Custom settings require container environment forwarding when deploying with Compose. + +## Bounded state API + +- `GET /api/summary`: network/chain/genesis, canonical tip/height/work, process-local revision, peers/maturity and bounded pending transactions. No full-chain copy. Revision is only a local cache invalidation counter, not consensus or a durable sequence. +- `GET /api/canonical?tip=&offset=0&limit=16`: up to **16 canonical blocks**, pinned to the current tip. Tip changes return 409 so the caller restarts. Offset cannot exceed current height. Each page uses the cached canonical branch and clones only its slice. The first page after a new tip may rebuild that branch in O(height). +- Legacy `/api/state` and `/api/chain` still return full snapshots only up to the configured legacy height and response-byte cap. Beyond it, they return 413 with paging instructions **before full snapshot cloning**. They never silently truncate ownership history. Legacy scripts/API clients must migrate to paging for longer chains. +- `/p2p/status` and `/p2p/mempool` now obtain only the fields needed instead of cloning the chain. P2P block inventory semantics are unchanged. + +The browser and trade signing refresh use bounded streaming response reads (2 MiB/page), pagination/network/tip/parent continuity checks, one restart on tip conflict, shared in-flight loads, and append-history cache reuse. A final summary confirms the loaded tip. These checks are **not independent PoW/consensus verification**: the UI still trusts the node. + +The existing snapshot-based UI still assembles complete ownership history, capped at **10,000 blocks or 64 MiB of encoded history** with a 30-second fetch deadline. If exceeded, it fails explicitly rather than presenting partial ownership as complete. That is a client resource guard, not a solution for an archival-sized wallet/explorer. Full state objects and parsed JS memory cost more than encoded JSON. A truly indexed wallet and fully paged explorer remain required before closing SEC-01. An unchanged refresh requests two summaries and no block pages; append loading verifies the cached prefix and requests only new blocks. + +## Marketplace read improvements + +Public listing pages query 25 indexed matching rows for a 24-row page plus next-page detection instead of loading the entire listing table. Actor activity uses indexed seller/buyer expressions. Browse/activity reconciliation results are reused for at most one second while the production ledger tip/revision are unchanged. Any observed ledger revision invalidates reuse. Every state-changing market command still reconciles freshly before validating terms; no cached eligibility authorizes a sale. + +Canonical transaction identities are indexed once per reconciliation pass rather than searched through the full chain for each order. Unchanged rows are not rewritten to SQLite. These reduce repeated work, but **reconciliation still clones/scans canonical history and all journal rows**, ownership checks still scan history, private activity is not paginated, and receipt/journal lifecycle needs further work. The response cap can reject an actor's oversized activity response; it is not silently truncated. These limitations keep SEC-06 open. + +## Tests and safe load tooling + +Final validation: **86 tests passed, 0 failed, 0 skipped** in the Node 24 Docker test image, including real RandomX HTTP integration. TypeScript/Vite build passed. The suite includes deterministic token refill/write/active/byte limits; a held-response 40-request loopback burst (4 admitted, 36 shed); health/P2P-lane availability during API saturation; canonical paging defensive copies/tip conflicts/legacy guards; client append caching, malformed pages/continuity, bounded history, single-flight loading and response size checks; non-recursive oversized-response errors; and paging against the real RandomX HTTP fixture. + +`test:rpc-load` invokes the read-only [local smoke-load script](../scripts/rpc-load.ts). It requires `--consent`, accepts only **literal loopback IP origins** (not remote hosts or DNS names), at most 32 workers and 1,000 total requests. Options: `--node`, `--concurrency`, `--requests`. It reads summary/market endpoints and checks health, with response/time bounds. It never submits a block, bid, payment or transfer. Use only a disposable isolated node: even read traffic consumes CPU/IO. Output includes status counts, bytes and p50/p95 latency; rejected load is visible, not counted as successful capacity. + +Initial smoke measurement on a disposable **empty-chain/empty-market**, loopback-only Docker node limited to **1 CPU / 1 GiB**: 100 read requests, 8 workers, 100 HTTP 200s, 40,348 JSON bytes, 1,837 ms total, p50 99 ms, p95 463 ms, final health HTTP 200. This is a single local run, not a performance guarantee; it does not measure hashing load, a populated journal, long-chain costs or hostile Internet traffic. The deterministic 40-request held-response test separately verifies overload shedding (4 admitted, 36 rejected) and peer/health-lane availability. Final test totals are recorded in the completion report. + +Neither measurement establishes that a realistic long-chain public deployment meets its latency/throughput SLO. Production-scale chain/journal fixtures, slow clients, adversarial payloads, sustained concurrency, disk pressure and multi-host recovery measurements remain outstanding. No launch-readiness finding is marked resolved by this stage. diff --git a/docs/operator-launch.md b/docs/operator-launch.md index 6e2286c..8c47940 100755 --- a/docs/operator-launch.md +++ b/docs/operator-launch.md @@ -2,6 +2,8 @@ **Current decision: HOLD / NO-GO.** Review [the security findings](security-review.md) before scheduling production. The automated preflight intentionally exits nonzero while known production blockers remain. An environment variable, successful container startup or genesis timestamp cannot substitute for closing these findings. +[Stage-1 node resource controls and local load tooling](node-resources.md) are implemented. This is partial remediation, not capacity certification or closure of the public-node hold point. + ## 1. Safe inspection — does not launch anything Use Node.js 24 and a clean reviewed checkout. For mainnet assessment on a host, select `WIKIDECK_NETWORK=mainnet` in the process environment, then run the `launch:preflight` package script. It runs the node entry point with `--launch-preflight`, prints JSON and exits. `--network-manifest` remains available for identity inspection only. diff --git a/docs/security-review.md b/docs/security-review.md index 13e0750..3a5f327 100755 --- a/docs/security-review.md +++ b/docs/security-review.md @@ -2,6 +2,8 @@ ## Decision: NO-GO for public production or valuable assets +**Stage-1 follow-up:** [Node resource controls](node-resources.md) adds global admission/byte budgets, bounded canonical pages, browser cache/history limits and marketplace read improvements. SEC-01 and SEC-06 remain open: full-history client/reconciliation, DAG/disk growth and production-scale capacity evidence are not solved. Original source line references below describe review-time locations and may shift as remediation proceeds. + This is an internal, source-assisted review with automated regression and dependency checks, **not an independent security audit, penetration test, formal consensus proof, or production certification**. The candidate has not been launched and no approval/attestation was set. Do not interpret passing tests, the existence of a mainnet profile, or the arrival of its candidate genesis date as permission to launch. Scope: current consensus/ledger/P2P/RandomX integration, HTTP entry point, browser wallet/backups, marketplace authorization/accounting/reconciliation, Docker configuration and candidate startup gates. The operating hosts, real reverse proxy, public peers, SHKeeper instance, wallets and funds were **not** accessed or assessed. No active Internet-facing adversarial load test or independent native-code audit was performed. diff --git a/docs/testnet-consensus.md b/docs/testnet-consensus.md index 7febf40..9fcd3c8 100644 --- a/docs/testnet-consensus.md +++ b/docs/testnet-consensus.md @@ -79,6 +79,7 @@ Sync polls every **5 seconds**, checks network/chain ID, imports at most **32** | --- | --- | | `GET /healthz` | Process health/network, not sync or consensus security. | | `GET /api/state`, `/api/chain` | Canonical blocks, mined/pending transactions, tip/work and peer count. | +| `GET /api/summary`, `/api/canonical?tip=&offset=0&limit=16` | Bounded state metadata and current-tip-pinned canonical pages. Full legacy snapshots are limited to 128 blocks and the response-byte budget. See [resource controls](node-resources.md). | | `GET /api/network` | Selected profile, pinned chain/genesis, candidate launch status and mock-payment boundary. | | `GET /api/market?offset=0`, `POST /api/market/command` | Node-local mock-payment marketplace; strict signed listing, bid, checkout, delivery and private activity commands. Not consensus escrow. See [marketplace operations](marketplace.md). | | `GET /api/nfts/:id`, `/api/peers` | Canonical NFT owner/sequence/confirmations, article and at-mint statistics; configured peer status. | diff --git a/package.json b/package.json index ec99f1d..e929a55 100644 --- a/package.json +++ b/package.json @@ -8,6 +8,7 @@ "build": "tsc --noEmit && vite build", "start": "tsx server/index.ts", "launch:preflight": "node --experimental-strip-types server/index.ts --launch-preflight", + "test:rpc-load": "node --experimental-strip-types scripts/rpc-load.ts", "miner": "tsx server/testnet/miner.ts", "native:configure": "cmake -S native -B native/build -DCMAKE_BUILD_TYPE=Release -DWIKIDECK_RANDOMX_TESTS=ON", "native:build": "cmake --build native/build --config Release --parallel 2", diff --git a/scripts/rpc-load.ts b/scripts/rpc-load.ts new file mode 100755 index 0000000..083aab6 --- /dev/null +++ b/scripts/rpc-load.ts @@ -0,0 +1,30 @@ +// Opt-in loopback-only, read-only smoke load. Not a production capacity certification. +const args=process.argv.slice(2); +const argument=(key:string,fallback:string)=>{const i=args.indexOf(key);return i<0?fallback:args[i+1];}; +if(!args.includes('--consent'))throw new Error('Explicit --consent required; use only an isolated disposable local node.'); +const endpoint=new URL(argument('--node','http://127.0.0.1:3001')); +if(!['127.0.0.1','[::1]'].includes(endpoint.hostname)||!['http:','https:'].includes(endpoint.protocol)|| + endpoint.username||endpoint.password||endpoint.pathname!=='/'||endpoint.search||endpoint.hash)throw new Error('Only literal loopback HTTP(S) origins are allowed. No remote load targets.'); +const concurrency=Number(argument('--concurrency','8')),count=Number(argument('--requests','100')); +if(!Number.isInteger(concurrency)||concurrency<1||concurrency>32||!Number.isInteger(count)||count<1||count>1000)throw new Error('Concurrency 1–32; requests 1–1000.'); +let next=0,totalBytes=0;const statuses:Record={};const latencies:number[]=[]; +const started=performance.now(); +await Promise.all(Array.from({length:concurrency},async()=>{ + while(next2*1024*1024)throw new Error('Response too large');}}finally{await reader.cancel();reader.releaseLock();} + totalBytes+=bytes;statuses[response.status]=(statuses[response.status]??0)+1; + }catch{statuses.error=(statuses.error??0)+1;} + latencies.push(performance.now()-start); + } +})); +latencies.sort((a,b)=>a-b); +const health=await fetch(new URL('/healthz',endpoint),{redirect:'error',signal:AbortSignal.timeout(5000)}); +console.log(JSON.stringify({scope:'loopback read-only smoke, not production benchmark',requests:count,concurrency, + elapsedMs:Math.round(performance.now()-started),statuses,totalBytes, + p50Ms:Math.round(latencies[Math.floor(latencies.length*.5)]??0),p95Ms:Math.round(latencies[Math.min(latencies.length-1,Math.floor(latencies.length*.95))]??0), + healthStatus:health.status},null,2)); +await health.body?.cancel(); +if(statuses.error||health.status!==200)process.exitCode=1; diff --git a/server/market/store.ts b/server/market/store.ts index 2163f1f..e265bb1 100755 --- a/server/market/store.ts +++ b/server/market/store.ts @@ -9,7 +9,7 @@ import { ADDITIONAL_MARKET_FEE, marketCommandSchema, marketMessage, type MarketC import { CHAIN_ID, transactionId, transferSchema, verifySignature } from '../testnet/consensus.ts'; import { mockQuote, paymentMode } from './payments.ts'; -type Chain = { snapshot(): TestnetSnapshot; submitTransfer(raw: unknown): Promise }; +type Chain = { snapshot(): TestnetSnapshot; summary?(): { tip: string; revision: number }; submitTransfer(raw: unknown): Promise }; type Config = { feeBps: number; confirmations: number; usdCentsPerXmr: number }; export function marketConfig(env: NodeJS.ProcessEnv = process.env): Config { paymentMode(env); @@ -31,6 +31,7 @@ const DELIVERY_WINDOW = 48 * 60 * 60_000; export class Market { private queue: Promise = Promise.resolve(); private queued = 0; + private reconciled: { tip: string; revision: number; at: number } | undefined; private readonly db: DatabaseSync; private readonly chain: Chain; readonly config: Config; @@ -52,6 +53,10 @@ export class Market { CREATE UNIQUE INDEX IF NOT EXISTS one_active_listing ON listings(card) WHERE status IN ('open','reserved'); CREATE TABLE IF NOT EXISTS orders(id TEXT PRIMARY KEY, data TEXT NOT NULL); CREATE TABLE IF NOT EXISTS receipts(id TEXT PRIMARY KEY, digest TEXT NOT NULL, expires INTEGER NOT NULL, data TEXT NOT NULL);`); + db.exec(`CREATE INDEX IF NOT EXISTS listings_by_status ON listings(status); + CREATE INDEX IF NOT EXISTS listings_by_seller ON listings(json_extract(data,'$.seller')); + CREATE INDEX IF NOT EXISTS orders_by_seller ON orders(json_extract(data,'$.seller')); + CREATE INDEX IF NOT EXISTS orders_by_buyer ON orders(json_extract(data,'$.buyer'));`); const meta = db.prepare('SELECT value FROM market_meta WHERE key=?').get('identity') as { value: string } | undefined; let id: string; if (meta) { @@ -88,11 +93,11 @@ export class Market { return JSON.parse(row.data); } private saveListing(l: MarketListing) { - this.db.prepare('INSERT INTO listings VALUES(?,?,?,?) ON CONFLICT(id) DO UPDATE SET status=excluded.status,data=excluded.data') + this.db.prepare('INSERT INTO listings VALUES(?,?,?,?) ON CONFLICT(id) DO UPDATE SET status=excluded.status,data=excluded.data WHERE listings.data<>excluded.data') .run(l.id, l.cardHash, l.status, JSON.stringify(l)); } private saveOrder(o: MarketOrder) { - this.db.prepare('INSERT INTO orders VALUES(?,?) ON CONFLICT(id) DO UPDATE SET data=excluded.data').run(o.id, JSON.stringify(o)); + this.db.prepare('INSERT INTO orders VALUES(?,?) ON CONFLICT(id) DO UPDATE SET data=excluded.data WHERE orders.data<>excluded.data').run(o.id, JSON.stringify(o)); } private atomic(fn: () => T): T { this.db.exec('BEGIN IMMEDIATE'); @@ -114,12 +119,15 @@ export class Market { /** Reconcile from canonical transaction identity, never from a mempool acknowledgement. */ private reconcile() { const s = this.chain.snapshot(), now = this.now(); + const deliveries = new Map(); + for (const b of s.blocks) for (const tx of b.transactions) deliveries.set(transactionId(tx), b); + const pendingIds = new Set(s.pendingTransfers.map(transactionId)); this.atomic(() => { for (const o of this.rows('orders')) { const l = this.listing(o.listingId); if (o.status === 'refunded' || o.status === 'expired') continue; const deliveryId = this.deliveryId(o); - const delivery = s.blocks.find(b => b.transactions.some(tx => transactionId(tx) === deliveryId)); + const delivery = deliveries.get(deliveryId); const confirmations = delivery ? s.blocks.length - delivery.height + 1 : 0; o.deliveryConfirmations = confirmations; if (o.status === 'complete' || o.status === 'review') { @@ -143,7 +151,7 @@ export class Market { } else if (o.status === 'awaiting-payment' && now >= o.payBy) { o.status = 'expired'; l.status = l.kind === 'fixed' ? 'open' : 'expired'; } else if (o.status !== 'awaiting-payment') { - const pending = s.pendingTransfers.some(tx => transactionId(tx) === deliveryId); + const pending = pendingIds.has(deliveryId); const conflict = s.pendingTransfers.some(tx => transactionCardIds(tx).includes(o.cardHash) && transactionId(tx) !== deliveryId); if (!this.card(s, o.cardHash, o.seller, o.sequence) || conflict || (now >= o.deliverBy && !pending)) o.status = 'refund-required'; else if (o.status === 'delivery-pending' && !pending) o.status = 'paid'; @@ -160,13 +168,18 @@ export class Market { } this.db.prepare('DELETE FROM receipts WHERE expires < ?').run(now - 86400_000); }); + const summary = this.chain.summary?.(); + if (summary) this.reconciled = { ...summary, at: now }; } browse(offset = 0): { listings: MarketListing[]; next: number | null; info: MarketInfo } { if (!Number.isSafeInteger(offset) || offset < 0 || offset > 10000) throw new Error('Invalid marketplace page'); // Reconciliation must not interleave with delivery submission while its chain await is outstanding. - if (!this.queued) this.reconcile(); - const all = this.rows('listings').filter(l => l.status === 'open' || l.status === 'reserved'); - return { listings: all.slice(offset, offset + 24), next: offset + 24 < all.length ? offset + 24 : null, info: this.info() }; + const summary = this.chain.summary?.(); + const last = this.reconciled; + if (!this.queued && (!summary || !last || summary.tip !== last.tip || summary.revision !== last.revision || this.now() - last.at >= 1000)) this.reconcile(); + const rows = this.db.prepare("SELECT data FROM listings WHERE status IN ('open','reserved') ORDER BY rowid DESC LIMIT 25 OFFSET ?") + .all(offset) as { data: string }[]; + return { listings: rows.slice(0,24).map(r=>JSON.parse(r.data)), next: rows.length>24?offset+24:null, info: this.info() }; } execute(raw: unknown): Promise { const cmd = marketCommandSchema.parse(raw); @@ -189,10 +202,15 @@ export class Market { if (old.digest !== digest) throw new Error('Nonce already used for different terms'); return JSON.parse(old.data); } - this.reconcile(); const a = c.action; - if (a.type === 'activity') return { orders: this.rows('orders').filter(o => o.seller === c.actor || o.buyer === c.actor), - listings: this.rows('listings').filter(l => l.seller === c.actor) }; + const summary = this.chain.summary?.(), last = this.reconciled; + if (a.type !== 'activity' || !summary || !last || summary.tip !== last.tip || summary.revision !== last.revision || + this.now() - last.at >= 1000) this.reconcile(); + if (a.type === 'activity') return { + orders: (this.db.prepare("SELECT data FROM orders WHERE json_extract(data,'$.seller')=? OR json_extract(data,'$.buyer')=? ORDER BY rowid DESC") + .all(c.actor,c.actor) as { data: string }[]).map(r=>JSON.parse(r.data)), + listings: (this.db.prepare("SELECT data FROM listings WHERE json_extract(data,'$.seller')=? ORDER BY rowid DESC") + .all(c.actor) as { data: string }[]).map(r=>JSON.parse(r.data)) }; if (a.type === 'deliver') { const o = this.order(a.orderId), tx = transferSchema.parse(a.transfer); if (o.seller !== c.actor || !['paid', 'delivery-pending'].includes(o.status)) throw new Error('Only the paid order seller can deliver'); diff --git a/server/preflight.ts b/server/preflight.ts index 7878394..e47360c 100755 --- a/server/preflight.ts +++ b/server/preflight.ts @@ -24,7 +24,7 @@ export function launchPreflight(env: NodeJS.ProcessEnv = process.env, now = Date } } checks.push( - { id: 'public-node-resource-controls', status: 'blocked', detail: 'Full-chain RPC, in-memory DAG and synchronous journal scans need a reviewed capacity/abuse strategy and load-test evidence.' }, + { id: 'public-node-resource-controls', status: 'blocked', detail: 'Stage-1 global RPC budgets, bounded pages/client history and read caching exist. In-memory DAG growth, full-history UI/reconciliation and production-scale load/capacity evidence remain unresolved.' }, { id: 'consensus-launch-calibration', status: 'blocked', detail: 'Initial RandomX difficulty, retarget behavior, final genesis and economic security have not been calibrated and independently reviewed for production.' }, { id: 'wallet-key-protection', status: 'blocked', detail: 'Browser signing keys remain extractable and persist unencrypted in IndexedDB; production wallet policy and recovery review are required.' }, { id: 'live-market-settlement', status: 'blocked', detail: 'No live SHKeeper adapter or reviewed payout/refund/reorg system exists. Mock payment cannot constitute real payment for valuable cards.' }, diff --git a/server/resources.ts b/server/resources.ts new file mode 100755 index 0000000..68aa4e5 --- /dev/null +++ b/server/resources.ts @@ -0,0 +1,89 @@ +import type { RequestHandler } from 'express'; + +export class RpcError extends Error { + readonly status: number; + constructor(status: number, message: string) { super(message); this.status = status; } +} +export type ResourceConfig = { requestsPerSecond: number; writesPerSecond: number; active: number; + responseBytes: number; bytesPerSecond: number; legacyBlocks: number }; +export function resourceConfig(env: NodeJS.ProcessEnv = process.env): ResourceConfig { + const number = (name: string, fallback: number, min: number, max: number) => { + const text = env[name] ?? String(fallback), n = Number(text); + if (!/^\d+$/.test(text) || !Number.isSafeInteger(n) || n < min || n > max) throw new Error(`Invalid ${name}`); + return n; + }; + return { requestsPerSecond: number('WIKIDECK_RPC_REQUESTS_PER_SECOND', 120, 1, 1000), + writesPerSecond: number('WIKIDECK_RPC_WRITES_PER_SECOND', 10, 1, 100), + active: number('WIKIDECK_RPC_MAX_ACTIVE', 32, 1, 256), + responseBytes: number('WIKIDECK_RPC_RESPONSE_BYTES', 2 * 1024 * 1024, 128 * 1024, 2 * 1024 * 1024), + bytesPerSecond: number('WIKIDECK_RPC_BYTES_PER_SECOND', 8 * 1024 * 1024, 2 * 1024 * 1024, 128 * 1024 * 1024), + legacyBlocks: number('WIKIDECK_RPC_LEGACY_BLOCKS', 128, 0, 128) }; +} + +/** Bounded, constant-storage global budgets. No per-client map or retry queue. + * Peer traffic has its own lane so a saturated browser lane cannot consume its tokens. */ +export class RpcBudget { + private tokens: number; + private writeTokens: number; + private byteTokens: number; + private updated: number; + private active = 0; + private rejected = 0; + readonly config: ResourceConfig; + constructor(config: ResourceConfig, privateClock: () => number = () => performance.now()) { + this.config = config; + this.clock = privateClock; + this.tokens = config.requestsPerSecond; this.writeTokens = config.writesPerSecond; + this.byteTokens = config.bytesPerSecond; this.updated = this.clock(); + } + private readonly clock: () => number; + private refill() { + const now = this.clock(), elapsed = Math.max(0, now - this.updated) / 1000; + this.updated = now; + this.tokens = Math.min(this.config.requestsPerSecond, this.tokens + elapsed * this.config.requestsPerSecond); + this.writeTokens = Math.min(this.config.writesPerSecond, this.writeTokens + elapsed * this.config.writesPerSecond); + this.byteTokens = Math.min(this.config.bytesPerSecond, this.byteTokens + elapsed * this.config.bytesPerSecond); + } + acquire(write: boolean): () => void { + this.refill(); + if (this.active >= this.config.active || this.tokens < 1 || (write && this.writeTokens < 1)) { + this.rejected++; throw new RpcError(503, 'Node RPC admission budget exhausted; retry later.'); + } + this.tokens--; if (write) this.writeTokens--; + this.active++; + let released = false; + return () => { if (!released) { this.active--; released = true; } }; + } + encode(value: unknown): string { + const encoded = JSON.stringify(value), bytes = Buffer.byteLength(encoded); + if (bytes > this.config.responseBytes) { + this.rejected++; throw new RpcError(413, 'RPC response limit exceeded; use smaller paged requests.'); + } + this.refill(); + if (bytes > this.byteTokens) { this.rejected++; throw new RpcError(503, 'Node RPC response-byte budget exhausted; retry later.'); } + this.byteTokens -= bytes; + return encoded; + } + status() { return { active: this.active, rejected: this.rejected }; } + middleware(): RequestHandler { + return (req, res, next) => { + res.setHeader('Cache-Control', 'no-store'); + let release: () => void; + try { release = this.acquire(req.method !== 'GET' && req.method !== 'HEAD'); } + catch (error) { + res.setHeader('Retry-After', '1'); + res.status(503).json({ error: (error as Error).message }); return; + } + res.once('finish', release); res.once('close', release); + // Express.json normally serializes before send; intercept here so no oversized payload is sent. + const originalJson = res.json.bind(res); + res.json = (value: unknown) => { + try { + const encoded = this.encode(value); + return res.type('application/json').send(encoded); + } catch (error) { res.json = originalJson; throw error; } + }; + next(); + }; + } +} diff --git a/server/testnet/http.ts b/server/testnet/http.ts index bca0132..8586fd7 100644 --- a/server/testnet/http.ts +++ b/server/testnet/http.ts @@ -18,10 +18,14 @@ import { Market, marketConfig } from '../market/store.ts'; import { marketCommandSchema } from '../../shared/market.ts'; import { IS_MAINNET, PROFILE } from '../../shared/network.ts'; import { assertLaunchAllowed, launchManifest } from '../launch.ts'; +import { resourceConfig, RpcBudget, RpcError } from '../resources.ts'; const root = fileURLToPath(new URL('../../', import.meta.url)); assertLaunchAllowed(); const paymentConfig = marketConfig(); // Fail before opening node resources if live payments are requested. +const limits = resourceConfig(); +const apiBudget = new RpcBudget(limits); +const peerBudget = new RpcBudget({ ...limits, requestsPerSecond: 40, writesPerSecond: 10, active: 16 }); const dataDir = process.env.WIKIDECK_DATA_DIR ?? resolve(root,'data'); await mkdir(dataDir,{recursive:true}); // Never permit two processes to rewrite the same ledger. A stale lock requires operator review. @@ -48,7 +52,9 @@ app.use(helmet({contentSecurityPolicy:{directives:{defaultSrc:["'self'"],scriptS connectSrc:["'self'"],upgradeInsecureRequests:null}},crossOriginEmbedderPolicy:false, strictTransportSecurity:process.env.NODE_ENV==='production'})); app.get('/healthz',(_req,res)=>{res.setHeader('Cache-Control','no-store');res.json({status:'ok',network:NETWORK});}); -app.use(express.json({limit:'68kb',strict:true})); +// Admission comes before body parsing and signature/schema work. Disable compressed request inflation. +app.use('/api',apiBudget.middleware()); +app.use('/p2p',peerBudget.middleware()); app.use(['/api','/p2p'],rateLimit({windowMs:60000,limit:300,standardHeaders:'draft-8',legacyHeaders:false})); app.use(['/api','/p2p'],(req,res,next)=>{ res.setHeader('Cache-Control','no-store'); @@ -60,9 +66,16 @@ app.use(['/api','/p2p'],(req,res,next)=>{ } next(); }); -app.get('/api/state',(_req,res)=>res.json(ledger.snapshot())); +app.use(['/api','/p2p'],express.json({limit:'68kb',strict:true,inflate:false})); +app.get('/api/state',(_req,res)=>res.json(ledger.legacySnapshot(limits.legacyBlocks))); +app.get('/api/summary',(_req,res)=>res.json(ledger.summary())); +app.get('/api/canonical',(req,res)=>{ + const q=z.object({tip:z.string().regex(/^[a-f0-9]{64}$/),offset:z.string().regex(/^\d{1,12}$/).optional(), + limit:z.string().regex(/^\d{1,2}$/).optional()}).strict().parse(req.query); + res.json(ledger.canonicalPage(q.tip,Number(q.offset??0),Number(q.limit??16))); +}); app.get('/api/network',(_req,res)=>res.json(launchManifest())); -app.get('/api/chain',(_req,res)=>res.json(ledger.snapshot())); +app.get('/api/chain',(_req,res)=>res.json(ledger.legacySnapshot(limits.legacyBlocks))); app.get('/api/peers',(_req,res)=>res.json({peers:peers.status()})); app.get('/api/market', (req,res)=>res.json(market.browse(Number(req.query.offset ?? 0)))); app.post('/api/market/command',rateLimit({windowMs:60000,limit:60,standardHeaders:'draft-8',legacyHeaders:false}),async(req,res)=>{ @@ -72,6 +85,7 @@ app.post('/api/market/command',rateLimit({windowMs:60000,limit:60,standardHeader res.json(result); if(command.action.type==='deliver')void peers.broadcastTransfer(command.action.transfer).catch(()=>{}); } catch(error) { + if(error instanceof RpcError)throw error; // Marketplace storage/payment internals must never leak through the generic node error handler. if(error instanceof Error && !/SQLITE|constraint|database/i.test(error.message))res.status(400).json({error:error.message.slice(0,200)}); else res.status(409).json({error:'Marketplace state conflict. Refresh and retry.'}); @@ -95,14 +109,14 @@ app.get('/api/wikipedia/:pageId', rateLimit({windowMs:60000,limit:60,standardHea try {res.json(await wikipediaEnrichment.get(pageId));} catch {res.status(503).json({error:'Wikipedia enrichment is temporarily unavailable. NFT data is unaffected.'});} }); -app.get('/p2p/status',(_req,res)=>res.json({network:NETWORK,chainId:CHAIN_ID,size:ledger.size,tip:ledger.snapshot().tip})); +app.get('/p2p/status',(_req,res)=>res.json(ledger.peerStatus())); app.get('/p2p/blocks',(req,res)=>{ const offset=Number(req.query.offset??0); if(!Number.isSafeInteger(offset)||offset<0){res.status(400).json({error:'Invalid offset'});return;} const blocks=ledger.inventory(offset,32); res.json({blocks,next:offset+blocks.lengthres.json({transactions:ledger.snapshot().pendingTransfers})); +app.get('/p2p/mempool',(_req,res)=>res.json({transactions:ledger.pendingTransactions()})); app.get('/api/nfts/:id', (req,res)=>{ const s=ledger.snapshot();const block=s.blocks.find(b=>b.tokenId===req.params.id); if(!block){res.status(404).json({error:'NFT not found on the canonical chain'});return;} @@ -154,13 +168,19 @@ app.use((err:Error & {status?:number},_req:express.Request,res:express.Response, const input=err instanceof z.ZodError||err instanceof SyntaxError; const message=input?'Invalid request encoding':err.message; // No keys/passwords are accepted by any node endpoint. - res.status(err.status===413?413:400).json({error:message.slice(0,200)}); + if(err instanceof RpcError&&err.status===503)res.setHeader('Retry-After','1'); + res.status(err instanceof RpcError?err.status:err.status===413?413:err.status===415?415:400).json({error:message.slice(0,200)}); }); const port=Number(process.env.PORT??3001),host=process.env.HOST??'127.0.0.1'; const server=app.listen(port,host,()=>{ const a=server.address();console.log(`Wikideck RandomX ${IS_MAINNET?'mainnet candidate':'testnet'}: http://${host}:${typeof a==='object'&&a?a.port:port} · chain ${CHAIN_ID} · ${ledger.snapshot().blocks.length} blocks`); peers.start(); }); +server.maxConnections=256; +server.headersTimeout=10000; +server.requestTimeout=15000; +server.keepAliveTimeout=5000; +server.setTimeout(30000,socket=>socket.destroy()); // Advance simulated payout/expiry state even when no browser is open. No real funds are sent. const marketTimer=setInterval(()=>{try {market.browse();}catch{console.error('Marketplace reconciliation failed; operator review required.');}},15000); marketTimer.unref(); diff --git a/server/testnet/ledger.ts b/server/testnet/ledger.ts index 40ed398..3da99a0 100644 --- a/server/testnet/ledger.ts +++ b/server/testnet/ledger.ts @@ -4,7 +4,7 @@ import { DatabaseSync } from 'node:sqlite'; import { z } from 'zod'; import { NETWORK, CONSENSUS, type TestnetBlock, type NftState, type NftStateView, type TestnetTransaction, type MiningTemplate, - type TestnetDeck, type TestnetSnapshot } from '../../shared/testnet.ts'; + type TestnetDeck, type TestnetSnapshot, type TestnetSummary, type CanonicalPage } from '../../shared/testnet.ts'; import { CHAIN_ID, GENESIS_HASH, applyTransactionOverlay, blockSchema, makeTemplate, applyTransferOverlay, materializeState, transactionId, transactionSchema, transferSchema, validateBlock, workFor } from './consensus.ts'; @@ -12,6 +12,7 @@ import type { PowEngine } from './randomx.ts'; import type { Article } from '../../shared/protocol.ts'; import type { MintStatsSnapshot } from '../../shared/enrichment.ts'; import { PROFILE } from '../../shared/network.ts'; +import { RpcError } from '../resources.ts'; const MAX_PENDING = 256; const MAX_DECK_PAGE = 24; @@ -39,6 +40,8 @@ export class Ledger { private readonly db: DatabaseSync; private readonly pow: PowEngine; private closed = false; + private revision = 0; + private canonicalCache: { tip: string; blocks: TestnetBlock[] } | undefined; private constructor(file: string, db: DatabaseSync, pow: PowEngine) { this.file = file; this.db = db; this.pow = pow; } static async open(dir: string, pow: PowEngine): Promise { @@ -190,6 +193,7 @@ export class Ledger { if (wins) this.tip = node.block.hash; this.pending = pending; if (filtered) this.pendingState = filtered.state; + this.revision++; return { accepted: true, reorg: disconnected.length > 0 }; }); } @@ -211,6 +215,7 @@ export class Ledger { await this.persist(undefined, pending); this.pending = pending; this.pendingState = nextPendingState; + this.revision++; return id; }); } @@ -228,6 +233,27 @@ export class Ledger { peers: this.peers, algorithm: 'RandomX', maturity: CONSENSUS.coinbaseMaturity }); } + /** Small metadata/mempool response. No full-chain traversal or clone. */ + summary(): TestnetSummary { + return { network: NETWORK, chainId: CHAIN_ID, genesis: GENESIS_HASH, tip: this.tip, + height: this.height(), revision: this.revision, cumulativeWork: this.tipWork().toString(), + pendingTransfers: structuredClone(this.pending), peers: this.peers, algorithm: 'RandomX', maturity: CONSENSUS.coinbaseMaturity }; + } + peerStatus() { return { network: NETWORK, chainId: CHAIN_ID, size: this.size, tip: this.tip }; } + pendingTransactions(): TestnetTransaction[] { return structuredClone(this.pending); } + legacySnapshot(maxBlocks: number): TestnetSnapshot { + // Reject before any full-chain copy/serialization. Never truncate ownership history silently. + if (this.height() > maxBlocks) throw new RpcError(413, 'Full snapshot disabled at this height; use /api/summary and /api/canonical pages.'); + return this.snapshot(); + } + canonicalPage(tip: string, offset: number, limit = 16): CanonicalPage { + if (tip !== this.tip) throw new RpcError(409, 'Canonical tip changed; restart paged loading.'); + if (!Number.isSafeInteger(offset) || offset < 0 || offset > this.height() || + !Number.isSafeInteger(limit) || limit < 1 || limit > 16) throw new RpcError(400, 'Invalid canonical page'); + const blocks = structuredClone(this.canonical().slice(offset, offset + limit)); + return { network: NETWORK, chainId: CHAIN_ID, tip: this.tip, height: this.height(), offset, blocks, + next: offset + blocks.length < this.height() ? offset + blocks.length : null }; + } deck(address: string, offset = 0, limit = MAX_DECK_PAGE): TestnetDeck { if (!Number.isSafeInteger(offset) || offset < 0 || !Number.isSafeInteger(limit) || limit < 1 || limit > MAX_DECK_PAGE) { throw new Error('Invalid deck page'); @@ -259,7 +285,10 @@ export class Ledger { } return result.reverse(); } - private canonical(): TestnetBlock[] { return this.branch(this.tip); } + private canonical(): TestnetBlock[] { + if (this.canonicalCache?.tip !== this.tip) this.canonicalCache = { tip: this.tip, blocks: this.branch(this.tip) }; + return this.canonicalCache.blocks; // Internal immutable block references only; public APIs clone their slice. + } private async validate(raw: TestnetBlock): Promise { const ancestors = this.branch(raw.previousHash); const parent = this.nodes.get(raw.previousHash); diff --git a/shared/rpc-client.ts b/shared/rpc-client.ts new file mode 100755 index 0000000..44126ef --- /dev/null +++ b/shared/rpc-client.ts @@ -0,0 +1,99 @@ +import { NETWORK, type CanonicalPage, type TestnetBlock, type TestnetSnapshot, type TestnetSummary } from './testnet.ts'; + +const MAX_PAGE_BYTES = 2 * 1024 * 1024; +const MAX_HISTORY_BYTES = 64 * 1024 * 1024; +const MAX_CLIENT_BLOCKS = 10000; +const hex = /^[a-f0-9]{64}$/; +/** Bounded transport and continuity checks, NOT independent consensus/SPV verification. */ +export class StateClient { + private cache: { chainId: string; blocks: TestnetBlock[]; bytes: number } | undefined; + private inFlight: Promise | undefined; + private readonly fetcher: typeof fetch; + constructor(fetcher: typeof fetch = fetch) { this.fetcher = fetcher; } + async json(path: string, signal: AbortSignal): Promise { + const response = await this.fetcher(`/api/${path}`, { cache: 'no-store', redirect: 'error', signal }); + if (!/^application\/json(?:\s*;|$)/i.test(response.headers.get('content-type') ?? '') || !response.body) throw new Error('Node must return JSON.'); + const announced = response.headers.get('content-length'); + if (announced !== null && (!/^\d+$/.test(announced) || Number(announced) > MAX_PAGE_BYTES)) { + await response.body.cancel(); throw new Error('Node response exceeds client page limit.'); + } + const reader = response.body.getReader(), chunks: Uint8Array[] = []; + let bytes = 0; + try { + for (;;) { + const item = await reader.read(); if (item.done) break; + bytes += item.value.byteLength; + if (bytes > MAX_PAGE_BYTES) throw new Error('Node response exceeds client page limit.'); + chunks.push(item.value); + } + } finally { await reader.cancel(); reader.releaseLock(); } + const joined = new Uint8Array(bytes); let offset = 0; + for (const chunk of chunks) { joined.set(chunk, offset); offset += chunk.byteLength; } + const data = JSON.parse(new TextDecoder('utf-8', { fatal: true }).decode(joined)); + if (!response.ok) throw Object.assign(new Error(typeof data.error === 'string' ? data.error : 'Node RPC failed.'), { status: response.status }); + return data; + } + load(): Promise { + if (this.inFlight) return this.inFlight; + const result = this.loadFresh(); + this.inFlight = result; + void result.finally(() => { if (this.inFlight === result) this.inFlight = undefined; }).catch(() => {}); + return result; + } + private async summary(signal: AbortSignal) { + const s = await this.json('summary', signal); + if (s.network !== NETWORK || s.algorithm !== 'RandomX' || !hex.test(s.chainId) || !hex.test(s.tip) || !hex.test(s.genesis) || + !Number.isSafeInteger(s.height) || s.height < 0 || !Array.isArray(s.pendingTransfers) || s.pendingTransfers.length > 256 || + !/^\d{1,256}$/.test(s.cumulativeWork)) throw new Error('Incompatible node summary.'); + if (s.height > MAX_CLIENT_BLOCKS) throw new Error('This node exceeds the browser history limit (10,000 blocks). A paged explorer/wallet is required; no partial ownership state will be shown.'); + return s; + } + private async page(s: TestnetSummary, offset: number, signal: AbortSignal, limit = 16) { + const p = await this.json(`canonical?tip=${s.tip}&offset=${offset}&limit=${limit}`, signal); + const count = Math.min(limit, s.height - offset), next = offset + count < s.height ? offset + count : null; + if (p.network !== s.network || p.chainId !== s.chainId || p.tip !== s.tip || p.height !== s.height || p.offset !== offset || + p.next !== next || !Array.isArray(p.blocks) || p.blocks.length !== count) throw new Error('Invalid canonical pagination.'); + for (let i = 0; i < p.blocks.length; i++) { + const b = p.blocks[i]; + if (b.network !== s.network || b.chainId !== s.chainId || b.height !== offset + i + 1 || !hex.test(b.hash) || + !Array.isArray(b.transactions) || b.transactions.length > 64) throw new Error('Invalid canonical block page.'); + } + return p; + } + private async loadFresh(): Promise { + const signal = AbortSignal.timeout(30000); + for (let attempt = 0; attempt < 2; attempt++) { + try { + const s = await this.summary(signal); + let blocks: TestnetBlock[] = [], bytes = 0; + if (this.cache?.chainId === s.chainId && this.cache.blocks.length <= s.height) { + const cached = this.cache; + if (!cached.blocks.length || cached.blocks.at(-1)!.hash === s.tip || + (await this.page(s, cached.blocks.length - 1, signal, 1)).blocks[0]?.hash === cached.blocks.at(-1)!.hash) { + blocks = cached.blocks.slice(); bytes = cached.bytes; + } + } + while (blocks.length < s.height) { + const p = await this.page(s, blocks.length, signal); + const encodedBytes = new TextEncoder().encode(JSON.stringify(p.blocks)).byteLength; + bytes += encodedBytes; + if (bytes > MAX_HISTORY_BYTES) throw new Error('Browser history exceeds 64 MiB; no incomplete ownership state will be used.'); + for (const b of p.blocks) { + if (b.previousHash !== (blocks.at(-1)?.hash ?? s.genesis)) throw new Error('Canonical block continuity mismatch.'); + blocks.push(b); + } + } + if ((blocks.at(-1)?.hash ?? s.genesis) !== s.tip) throw new Error('Canonical tip does not match loaded history.'); + const final = await this.summary(signal); + if (final.chainId !== s.chainId || final.tip !== s.tip || final.height !== s.height) throw Object.assign(new Error('Canonical tip changed during loading.'), { status: 409 }); + this.cache = { chainId: s.chainId, blocks, bytes }; + const { height: _height, revision: _revision, ...metadata } = final; + return { ...metadata, blocks: blocks.slice(), transfers: blocks.flatMap(b => b.transactions) }; + } catch (error) { + if (attempt === 0 && (error as { status?: number }).status === 409) { this.cache = undefined; continue; } + throw error; + } + } + throw new Error('Could not load a stable canonical chain.'); + } +} diff --git a/shared/testnet.ts b/shared/testnet.ts index af20124..63f3f00 100644 --- a/shared/testnet.ts +++ b/shared/testnet.ts @@ -46,6 +46,9 @@ export type TestnetSnapshot = { blocks: TestnetBlock[]; transfers: TestnetTransaction[]; pendingTransfers: TestnetTransaction[]; peers: number; algorithm: 'RandomX'; maturity: number; }; +export type TestnetSummary = Omit & { height: number; revision: number }; +export type CanonicalPage = { network: typeof NETWORK; chainId: string; tip: string; height: number; + offset: number; next: number | null; blocks: TestnetBlock[] }; export type TestnetDeck = { network: typeof NETWORK; chainId: string; address: string; tip: string; offset: number; limit: number; total: number; next: number | null; diff --git a/src/App.tsx b/src/App.tsx index 29cb5dd..2c4a781 100644 --- a/src/App.tsx +++ b/src/App.tsx @@ -13,6 +13,7 @@ import { ThemePicker } from './ThemePicker.tsx'; import { TradeDesk } from './TradeDesk.tsx'; import { Marketplace } from './Marketplace.tsx'; import { IS_MAINNET, NETWORK_LABEL } from '../shared/network.ts'; +import { StateClient } from '../shared/rpc-client.ts'; import './App.css'; import './testnet.css'; @@ -59,12 +60,13 @@ export default function App() { const transferInFlight = useRef(false); const refreshVersion = useRef(0); const minerPanel = useRef(null); + const stateClient = useRef(new StateClient()); useModalFocus(Boolean(dialog || selected)); const refresh = async () => { const version = ++refreshVersion.current; try { - const data = await request('state'); + const data = await stateClient.current.load(); if (data.network !== NETWORK || data.algorithm !== 'RandomX' || typeof data.chainId !== 'string' || !Array.isArray(data.blocks) || !Array.isArray(data.transfers) || !Array.isArray(data.pendingTransfers)) throw new Error('Incompatible node. This interface requires the Wikideck RandomX testnet API.'); diff --git a/src/TradeDesk.tsx b/src/TradeDesk.tsx index a1749b4..ae2214f 100755 --- a/src/TradeDesk.tsx +++ b/src/TradeDesk.tsx @@ -4,6 +4,8 @@ import { CONSENSUS, NETWORK, isNftTrade, nftTradeMessage, type NftTrade, type Te import { tradeOfferSchema, type TradeOffer } from '../shared/trade-offer.ts'; import type { Wallet } from './wallet.ts'; import './trade.css'; +import { StateClient } from '../shared/rpc-client.ts'; +const stateClient = new StateClient(); type Props = { wallet?: Wallet; snapshot?: TestnetSnapshot; rpcAvailable: boolean; onNotice: (message: string) => void }; type OwnedCard = TestnetDeck['nfts'][number]; @@ -124,7 +126,7 @@ export function TradeDesk({ wallet, snapshot, rpcAvailable, onNotice }: Props) { } setBusy(true); setOfferError(''); try { - const fresh = await api('state'); + const fresh = await stateClient.load(); if (fresh.chainId !== snapshot.chainId || remoteDeck.chainId !== fresh.chainId) throw new Error('Chain changed. Refresh and rebuild the trade.'); const offeredLegs = [...offered].map(cardHash => { const card = myDeck.find(item => item.block.tokenId === cardHash); @@ -166,7 +168,7 @@ export function TradeDesk({ wallet, snapshot, rpcAvailable, onNotice }: Props) { if (offer.network !== NETWORK || offer.chainId !== snapshot.chainId) throw new Error('Offer is for another network or chain.'); if (offer.counterparty !== wallet.address) throw new Error('This offer is addressed to a different wallet.'); if (!await verifyOfferSignature(offer)) throw new Error('The proposer signature is invalid; do not accept this offer.'); - const fresh = await api('state'); + const fresh = await stateClient.load(); if (fresh.chainId !== offer.chainId) throw new Error('The chain changed. Reload and review the offer again.'); for (const leg of offer.offered) { const block = fresh.blocks.find(item => item.tokenId === leg.cardHash); diff --git a/tests/api.test.ts b/tests/api.test.ts index c0cf719..3600e44 100644 --- a/tests/api.test.ts +++ b/tests/api.test.ts @@ -16,6 +16,7 @@ import { NETWORK, CONSENSUS, blockClaimMessage, nftTransferMessage, type MiningTemplate, type NftTransfer, type TestnetBlock, type TestnetSnapshot } from '../shared/testnet.ts'; import { marketMessage, type MarketAction, type MarketInfo, type MarketListing, type MarketOrder, type MarketActivity } from '../shared/market.ts'; +import { StateClient } from '../shared/rpc-client.ts'; const root = fileURLToPath(new URL('../', import.meta.url)); const wikipediaMock = resolve(root, 'tests', 'mock-wikipedia.mjs'); @@ -203,6 +204,10 @@ test('native RandomX HTTP: three blocks, mined transfer, peer gossip/sync and du await (await request(first.base, '/api/wikipedia/not-a-page', 400)).text(); await (await request(first.base, '/api/wikipedia/9999999999', 503)).text(); const initial = await snapshot(first.base); + const pagedClient=new StateClient(((path: string,init?: RequestInit)=>fetch(new URL(path,first.base),init)) as typeof fetch); + assert.deepEqual(await pagedClient.load(),initial); + await (await request(first.base,'/api/canonical?tip=bad',400)).text(); + await (await request(first.base,`/api/canonical?tip=${GENESIS_HASH}&limit=17`,400)).text(); assert.equal(initial.network, NETWORK); assert.equal(initial.chainId, CHAIN_ID); assert.equal(initial.genesis, GENESIS_HASH); @@ -358,6 +363,7 @@ test('native RandomX HTTP: three blocks, mined transfer, peer gossip/sync and du } first = await start(firstDir); // Ledger.open revalidates every persisted block using native RandomX. const restored = await snapshot(first.base); + assert.deepEqual(await pagedClient.load(),restored); const restoredMarket = await json<{ info: MarketInfo }>(first.base, '/api/market'); assert.equal(restoredMarket.info.marketId, market.info.marketId); assert.equal((await marketCommand(first.base, market.info, bob, { type: 'activity' })).orders[0].id, order.id); diff --git a/tests/resources.test.ts b/tests/resources.test.ts new file mode 100755 index 0000000..b836c3b --- /dev/null +++ b/tests/resources.test.ts @@ -0,0 +1,73 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import express from 'express'; +import { once } from 'node:events'; +import { RpcBudget, RpcError, resourceConfig } from '../server/resources.ts'; + +test('global admission has bounded concurrency, write tokens, refill and idempotent release', () => { + let time = 0; + const config = { ...resourceConfig({}), requestsPerSecond: 3, writesPerSecond: 1, active: 2 }; + const budget = new RpcBudget(config, () => time); + const a = budget.acquire(true); + assert.throws(() => budget.acquire(true), /admission/); + const b = budget.acquire(false); + assert.throws(() => budget.acquire(false), /admission/); + a(); a(); b(); + assert.equal(budget.status().active, 0); + budget.acquire(false)(); + assert.throws(() => budget.acquire(false), /admission/); + time = 1000; + budget.acquire(true)(); +}); + +test('response bytes are bounded globally and refill without per-client storage', () => { + let time = 0; + const budget = new RpcBudget({ ...resourceConfig({}), responseBytes: 50, bytesPerSecond: 60 }, () => time); + assert.throws(() => budget.encode('x'.repeat(50)), /response limit/); + budget.encode('x'.repeat(28)); budget.encode('x'.repeat(28)); + assert.throws(() => budget.encode('a'), /byte budget/); + time = 1000; assert.equal(budget.encode('a'), '"a"'); + for (const value of ['-1','0','Infinity','1.2','']) assert.throws(() => resourceConfig({ WIKIDECK_RPC_MAX_ACTIVE: value })); +}); + +test('isolated HTTP burst sheds excess work, preserves peer/health lanes and releases slots', async t => { + const app = express(), cfg = { ...resourceConfig({}), active: 4, requestsPerSecond: 1000 }; + const publicBudget = new RpcBudget(cfg), peerBudget = new RpcBudget(cfg); + let release!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + app.get('/healthz', (_req,res) => { res.json({ status:'ok' }); }); + app.use('/api',publicBudget.middleware()); app.use('/p2p',peerBudget.middleware()); + app.get('/api/hold',async (_req,res)=>{ await gate; res.json({ok:true}); }); + app.get('/p2p/status',(_req,res)=>{ res.json({ok:true}); }); + const server=app.listen(0,'127.0.0.1'); await once(server,'listening'); + const address=server.address(); assert.ok(address&&typeof address==='object'); + const base=`http://127.0.0.1:${address.port}`; + t.after(()=>{release();server.closeAllConnections();server.close();}); + const requests=Array.from({length:40},()=>fetch(`${base}/api/hold`,{signal:AbortSignal.timeout(5000)})); + // The first rejected response proves requests reached admission while accepted requests remain held. + const first=await Promise.race(requests); assert.equal(first.status,503); + assert.equal((await fetch(`${base}/p2p/status`)).status,200); + assert.equal((await fetch(`${base}/healthz`)).status,200); + release(); + const responses=await Promise.all(requests); + const counts=responses.reduce((out,r)=>{out[r.status]=(out[r.status]??0)+1;return out;},{} as Record); + assert.equal(counts[200],4); assert.equal(counts[503],36); + for(const r of responses)await r.text(); + assert.equal(publicBudget.status().active,0); + t.diagnostic(JSON.stringify({requests:40,accepted:counts[200],shed:counts[503],peerLane:'available',health:'available'})); +}); + +test('oversized response errors restore JSON handling and do not recurse', async t => { + const app=express(),budget=new RpcBudget({...resourceConfig({}),responseBytes:128}); + app.use(budget.middleware()); + app.get('/large',(_req,res)=>{res.json({data:'x'.repeat(1000)});}); + app.use((error:Error,_req:express.Request,res:express.Response,_next:express.NextFunction)=>{ + res.status(error instanceof RpcError?error.status:500).json({error:error.message}); + }); + const server=app.listen(0,'127.0.0.1');await once(server,'listening'); + t.after(()=>{server.closeAllConnections();server.close();}); + const a=server.address();assert.ok(a&&typeof a==='object'); + const r=await fetch(`http://127.0.0.1:${a.port}/large`,{signal:AbortSignal.timeout(5000)}); + assert.equal(r.status,413);assert.match((await r.json()).error,/response limit/); + assert.equal(budget.status().active,0); +}); diff --git a/tests/rpc-client.test.ts b/tests/rpc-client.test.ts new file mode 100755 index 0000000..bfe2631 --- /dev/null +++ b/tests/rpc-client.test.ts @@ -0,0 +1,48 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { StateClient } from '../shared/rpc-client.ts'; +import { NETWORK, type TestnetBlock } from '../shared/testnet.ts'; + +const genesis='0'.repeat(64),chainId='f'.repeat(64); +const block=(height:number,previousHash:string):TestnetBlock=>({ network:NETWORK,chainId,height, + hash:height.toString(16).padStart(64,'0'),previousHash,transactions:[] }) as unknown as TestnetBlock; +function fixture(count:number){ + const blocks:TestnetBlock[]=[];for(let i=0;i{ + calls.push(path); const url=new URL(path,'http://localhost'); + const tip=blocks.at(-1)?.hash??genesis; + if(url.pathname==='/api/summary')return Response.json({network:NETWORK,chainId,genesis,tip,height:blocks.length, + cumulativeWork:String(blocks.length),pendingTransfers:[],peers:0,algorithm:'RandomX',maturity:2,revision:0}); + if(conflict){conflict=false;return Response.json({error:'changed'},{status:409});} + const offset=Number(url.searchParams.get('offset')),limit=Number(url.searchParams.get('limit')); + const page=blocks.slice(offset,offset+limit); + return Response.json({network:NETWORK,chainId,tip,height:blocks.length,offset,blocks:page, + next:malformed?offset:(offset+page.length{malformed=true;},conflict:()=>{conflict=true;}}; +} +test('paged state loads contiguous history and reuses cache when unchanged or appended',async()=>{ + const f=fixture(35);const initial=await f.client.load();assert.equal(initial.blocks.length,35); + assert.equal(f.calls.filter(p=>p.includes('canonical')).length,3); + f.calls.length=0;await f.client.load();assert.equal(f.calls.length,2); + f.blocks.push(block(36,f.blocks.at(-1)!.hash));f.calls.length=0; + assert.equal((await f.client.load()).blocks.length,36); + assert.equal(f.calls.filter(p=>p.includes('canonical')).length,2); // prefix check + append +}); +test('paged state rejects malformed pagination, continuity and excessive history',async()=>{ + const bad=fixture(17);bad.malform();await assert.rejects(bad.client.load(),/pagination/); + const broken=fixture(3);broken.blocks[1].previousHash='a'.repeat(64);await assert.rejects(broken.client.load(),/continuity/); + const big=fixture(10001);await assert.rejects(big.client.load(),/history limit/); +}); +test('tip conflict restarts once; parallel refreshes share bounded work',async()=>{ + const f=fixture(20);f.conflict(); + const [a,b]=await Promise.all([f.client.load(),f.client.load()]); + assert.equal(a,b);assert.equal(a.blocks.length,20); + assert.equal(f.calls.filter(p=>p==='/api/summary').length,3); +}); +test('oversized node responses are rejected before JSON parsing',async()=>{ + const client=new StateClient((async()=>new Response('{}',{headers:{'Content-Type':'application/json','Content-Length':'99999999'}})) as typeof fetch); + await assert.rejects(client.load(),/page limit/); +}); diff --git a/tests/testnet-ledger.test.ts b/tests/testnet-ledger.test.ts index 62ca394..3def78e 100644 --- a/tests/testnet-ledger.test.ts +++ b/tests/testnet-ledger.test.ts @@ -124,6 +124,16 @@ test('testnet genesis is empty, manifest-bound, isolated and defensively copied' copy.article.title = 'getter mutation'; assert.notEqual(ledger.block(candidate.hash)!.article.title, copy.article.title); assert.equal(ledger.size, 1); + const summary=ledger.summary(); + assert.equal(summary.height,1);assert.equal(summary.tip,candidate.hash); + const page=ledger.canonicalPage(summary.tip,0,1); + assert.equal(page.blocks.length,1);assert.equal(page.next,null); + page.blocks[0].article.title='mutated page'; + assert.notEqual(ledger.canonicalPage(summary.tip,0).blocks[0].article.title,'mutated page'); + assert.throws(()=>ledger.canonicalPage(GENESIS_HASH,0),/tip changed/); + assert.throws(()=>ledger.canonicalPage(summary.tip,0,17),/Invalid/); + assert.throws(()=>ledger.legacySnapshot(0),/Full snapshot disabled/); + assert.equal(ledger.legacySnapshot(1).blocks.length,1); assert.throws(() => ledger.inventory(-1, 32)); assert.throws(() => ledger.inventory(0, 0)); });