diff --git a/README.md b/README.md index bc47e2f..24fcdfe 100644 --- a/README.md +++ b/README.md @@ -75,6 +75,14 @@ Use another profile or a second wallet to receive a transfer. Signing submits it Metadata is a **fixed four-article catalog of original starter summaries**, not a live Wikipedia API feed. The catalog is genesis-bound; expanding it currently requires a new genesis/chain. Page IDs agree across catalog and state. See [consensus, identifiers, peer RPC and limits](docs/testnet-consensus.md). +### Live Wikipedia card insights + +Cards also display **optional off-chain enrichment**, separate from the minted snapshot: human-readable description, visible Wikipedia categories (up to 50), article size in bytes, other-language links (up to 500, labeled as a lower bound if truncated), Wikidata ID, current revision, and six complete UTC months of Wikimedia page views. The details show the latest monthly count, daily average, six-month total, month-over-month change, and a readership chart. Counts use English Wikipedia, all access methods, and Analytics' `user` traffic classification; classified automated traffic is excluded. Article renames can affect title-based history. Missing/incomplete API data is shown as unavailable, never invented zeroes. + +Three **live scores** range from 0–100, rounded and capped: **Attention** = log₁₀(latest complete month's views + 1) / 6 × 100; **Depth** = log₁₀(article bytes + 1) / 6 × 100; **Reach** = log₁₀(language links + 1) / 3 × 100. Depth is a size proxy, not article quality. These scores may change; the original proof-derived Power/Defense/Knowledge, rarity, NFT ownership, and consensus do not. Live revision and retrieval time are shown explicitly, with Wikipedia/Wikimedia attribution. + +`GET /api/wikipedia/:pageId` only enriches current catalog IDs. The node contacts fixed Wikimedia HTTPS hosts with no redirects, bounded time/response sizes, four concurrent jobs, per-article request deduplication, and a bounded cache (up to six hours; shorter for partial failures). API outages do not block mining, validation, or transfers. No live enrichment is written into blocks, NFT state, or the genesis manifest; existing chains remain compatible. Rebuild the Docker image to serve this feature from containers. + ## Connect your own independent nodes Operators must provision their own nodes, separate storage and connectivity; multiple containers on one machine are not independent security domains. Configure `WIKIDECK_PEERS` as a comma-separated list of fixed peer **origins**, for example `https://peer-a.example.org,https://peer-b.example.org` after replacing these placeholders with actual operator-controlled endpoints. Default policy requires HTTPS and public IP/DNS answers; credentials, paths, queries, fragments and redirects are rejected. At most 16 peers; no bootnodes, DHT, discovery or peer-list expansion. diff --git a/compose.testnet.yaml b/compose.testnet.yaml index a957707..e67f38b 100644 --- a/compose.testnet.yaml +++ b/compose.testnet.yaml @@ -57,4 +57,4 @@ services: volumes: testnet-a: testnet-b: - testnet-c: \ No newline at end of file + testnet-c: diff --git a/docs/randomx-native.md b/docs/randomx-native.md index 3479144..66892b1 100644 --- a/docs/randomx-native.md +++ b/docs/randomx-native.md @@ -191,4 +191,4 @@ Verified on Windows x64 with CMake 3.26.4, MinGW GCC 13.1.0 and Node 24.22.0: - Both build scripts passed syntax checks. Fast-mode allocation/runtime and Linux/macOS/MSVC builds were not run. The memory -and initialization cost of the full dataset remains an explicit opt-in. \ No newline at end of file +and initialization cost of the full dataset remains an explicit opt-in. diff --git a/docs/testnet-consensus.md b/docs/testnet-consensus.md index 8b6b7c1..6c51543 100644 --- a/docs/testnet-consensus.md +++ b/docs/testnet-consensus.md @@ -63,9 +63,10 @@ Sync polls every **5 seconds**, checks network/chain ID, imports at most **32** | `GET /healthz` | Process health/network, not sync or consensus security. | | `GET /api/state`, `/api/chain` | Canonical blocks, mined/pending transfers, tip/work and next article. | | `GET /api/nfts/:id`, `/api/peers` | Canonical NFT owner/sequence/confirmations/metadata; configured peer status. | +| `GET /api/wikipedia/:pageId` | Optional cached live enrichment for catalog IDs: categories, views, size, languages and display-only scores. Never a consensus input. | | `POST /api/mining/template`, `/api/blocks` | Public miner identity to template; independently validated signed block submission. | | `POST /api/transfer` | Signed transfer admission; 202 means pending. | | `GET /p2p/status`, `/p2p/blocks?offset=0`, `/p2p/mempool` | Manifest/size/tip, 32-block inventory pages, pending transactions. | | `POST /p2p/block`, `/p2p/transaction` | Same validation as local block/transfer submission. | -POSTs require JSON, enforce supplied browser origins and signatures where applicable; peers without an Origin header are not authenticated by this check. HTTP parsing is capped at 68 KiB; API/P2P rate limits are 300/minute and block/transfer POSTs additionally 30/minute. There are no fees, peer authentication or economic spam deterrents. Same-origin browser RPC trusts the node; it does not independently verify these commitments. Deploy no funds: majority-hashpower, eclipse, censorship, reorg and resource-exhaustion risks require independent audits and adversarial testing before any broader launch. \ No newline at end of file +POSTs require JSON, enforce supplied browser origins and signatures where applicable; peers without an Origin header are not authenticated by this check. HTTP parsing is capped at 68 KiB; API/P2P rate limits are 300/minute and block/transfer POSTs additionally 30/minute. There are no fees, peer authentication or economic spam deterrents. Same-origin browser RPC trusts the node; it does not independently verify these commitments. Deploy no funds: majority-hashpower, eclipse, censorship, reorg and resource-exhaustion risks require independent audits and adversarial testing before any broader launch. diff --git a/native/CMakeLists.txt b/native/CMakeLists.txt index 5297109..9bc5577 100644 --- a/native/CMakeLists.txt +++ b/native/CMakeLists.txt @@ -45,4 +45,4 @@ if(WIKIDECK_RANDOMX_TESTS) enable_testing() add_test(NAME randomx-upstream-vector COMMAND wikideck-randomx --self-test) set_tests_properties(randomx-upstream-vector PROPERTIES TIMEOUT 120) -endif() \ No newline at end of file +endif() diff --git a/native/randomx-engine.cpp b/native/randomx-engine.cpp index 4e28f20..381353e 100644 --- a/native/randomx-engine.cpp +++ b/native/randomx-engine.cpp @@ -211,4 +211,4 @@ int main(int argc, char** argv) { std::cerr << "randomx: native failure\n"; return 1; } -} \ No newline at end of file +} diff --git a/scripts/build-randomx.ps1 b/scripts/build-randomx.ps1 index abd01f1..313fc7d 100644 --- a/scripts/build-randomx.ps1 +++ b/scripts/build-randomx.ps1 @@ -22,4 +22,4 @@ if ($Test) { & ctest --test-dir $BuildDirectory -C Release --output-on-failure if ($LASTEXITCODE -ne 0) { throw 'RandomX known-vector test failed.' } } -Write-Host "Built native RandomX engine in $BuildDirectory" \ No newline at end of file +Write-Host "Built native RandomX engine in $BuildDirectory" diff --git a/scripts/build-randomx.sh b/scripts/build-randomx.sh index 579744a..1a9a0e2 100644 --- a/scripts/build-randomx.sh +++ b/scripts/build-randomx.sh @@ -18,4 +18,4 @@ command -v git >/dev/null 2>&1 || { printf '%s\n' 'Git is required' >&2; exit 1; cmake -S "$root/native" -B "$build" -DCMAKE_BUILD_TYPE=Release "-DWIKIDECK_RANDOMX_TESTS=$tests" cmake --build "$build" --config Release --target wikideck-randomx --parallel "$jobs" if [ "$tests" = ON ]; then ctest --test-dir "$build" -C Release --output-on-failure; fi -printf 'Built native RandomX engine in %s\n' "$build" \ No newline at end of file +printf 'Built native RandomX engine in %s\n' "$build" diff --git a/scripts/testnet-smoke.ts b/scripts/testnet-smoke.ts index d692928..1005136 100644 --- a/scripts/testnet-smoke.ts +++ b/scripts/testnet-smoke.ts @@ -56,4 +56,4 @@ try { if(Date.now()>deadline)throw new Error('Peer convergence timeout'); await delay(500); } -}finally{engine.close();} \ No newline at end of file +}finally{engine.close();} diff --git a/server/enrichment.ts b/server/enrichment.ts new file mode 100644 index 0000000..9647a87 --- /dev/null +++ b/server/enrichment.ts @@ -0,0 +1,165 @@ +import { z } from 'zod'; +import { deriveStats, type PageEnrichment } from '../shared/enrichment.ts'; + +const MAX_BYTES = 1024 * 1024; +const integer = z.number().int().min(0).max(Number.MAX_SAFE_INTEGER); +const text = z.string().max(4096); +const pageSchema = z.object({ + pageid: integer.positive(), ns: z.literal(0), title: text.min(1), + missing: z.never().optional(), invalid: z.never().optional(), + lastrevid: integer.positive(), length: integer, + description: text.optional(), + pageprops: z.object({ wikibase_item: z.string().regex(/^Q[1-9]\d{0,19}$/).optional() }).optional(), + categories: z.array(z.object({ ns: z.literal(14), title: text.min(1) })).max(50).optional(), + langlinks: z.array(z.object({ lang: z.string().min(1).max(64) })).max(500).optional(), +}); +const metadataSchema = z.object({ + query: z.object({ pages: z.array(pageSchema).length(1) }), + continue: z.object({ clcontinue: text.optional(), llcontinue: text.optional() }).optional(), +}); +const viewsSchema = z.object({ items: z.array(z.object({ + timestamp: z.string().regex(/^\d{6}0100$/), views: integer, + project: z.literal('en.wikipedia'), access: z.literal('all-access'), + agent: z.literal('user'), granularity: z.literal('monthly'), +})).max(6) }); + +type CacheEntry = { expires: number; value: PageEnrichment } | { expires: number; error: Error }; +type Options = { fetch?: typeof globalThis.fetch; now?: () => Date; ttlMs?: number; maxEntries?: number }; + +function monthWindow(now: Date): string[] { + return Array.from({ length: 6 }, (_, index) => + new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth() - 6 + index, 1)).toISOString().slice(0, 7)); +} + +/** Fixed-source read-only enrichment; no consensus, persistence, URL or header inputs. */ +export class WikipediaEnrichment { + private readonly fetcher: typeof globalThis.fetch; + private readonly now: () => Date; + private readonly ttlMs: number; + private readonly maxEntries: number; + private readonly cache = new Map(); + private readonly pending = new Map>(); + private active = 0; + + constructor(options: Options = {}) { + this.fetcher = options.fetch ?? globalThis.fetch; + this.now = options.now ?? (() => new Date()); + this.ttlMs = options.ttlMs ?? 6 * 60 * 60 * 1000; + this.maxEntries = options.maxEntries ?? 128; + if (!Number.isSafeInteger(this.ttlMs) || this.ttlMs <= 0 || + !Number.isSafeInteger(this.maxEntries) || this.maxEntries <= 0 || this.maxEntries > 128) { + throw new RangeError('Invalid enrichment cache options'); + } + } + + async get(pageId: number): Promise { + if (!Number.isSafeInteger(pageId) || pageId <= 0) throw new RangeError('Invalid Wikipedia page ID'); + const now = this.now(); + const cached = this.cache.get(pageId); + if (cached && cached.expires > now.getTime() && + (!('value' in cached) || cached.value.pageviews.latestMonth === monthWindow(now)[5])) { + this.cache.delete(pageId); + this.cache.set(pageId, cached); + if ('error' in cached) throw cached.error; + return structuredClone(cached.value); + } + this.cache.delete(pageId); + const existing = this.pending.get(pageId); + if (existing) return structuredClone(await existing); + // Reject immediately: no unbounded waiting queue. Four active jobs also keep + // the pending-map ceiling of eight conservative (one upstream request/job). + if (this.pending.size >= 8 || this.active >= 4) throw new Error('Wikipedia enrichment busy'); + this.active++; + const job = this.load(pageId, now).then(value => { + const ttl = value.pageviews.status === 'available' ? this.ttlMs : Math.min(this.ttlMs, 5 * 60 * 1000); + this.remember(pageId, { value, expires: this.now().getTime() + ttl }); + return value; + }, () => { + const error = new Error('Wikipedia page enrichment unavailable'); + this.remember(pageId, { error, expires: this.now().getTime() + 30_000 }); + throw error; + }).finally(() => { this.active--; this.pending.delete(pageId); }); + this.pending.set(pageId, job); + return structuredClone(await job); + } + + private remember(pageId: number, entry: CacheEntry): void { + this.cache.delete(pageId); + this.cache.set(pageId, entry); + while (this.cache.size > this.maxEntries) this.cache.delete(this.cache.keys().next().value!); + } + + private async json(url: URL): Promise { + const signal = AbortSignal.timeout(6000); + const response = await this.fetcher(url, { signal, redirect: 'error', + headers: { Accept: 'application/json', 'User-Agent': 'Wikideck/0.1 (Wikipedia off-chain page enrichment)' } }); + const length = response.headers.get('content-length'); + if (!response.ok || response.redirected || + !/^application\/json(?:\s*;|\s*$)/i.test(response.headers.get('content-type') ?? '') || + (length !== null && (!/^\d+$/.test(length) || Number(length) > MAX_BYTES)) || !response.body) { + await response.body?.cancel(); + throw new Error('Invalid upstream response'); + } + const reader = response.body.getReader(); + const chunks: Uint8Array[] = []; + let size = 0; + try { + for (;;) { + signal.throwIfAborted(); + const { done, value } = await reader.read(); + signal.throwIfAborted(); + if (done) break; + size += value.byteLength; + if (size > MAX_BYTES) throw new Error('Upstream response too large'); + chunks.push(value); + } + return JSON.parse(new TextDecoder('utf-8', { fatal: true }).decode(Buffer.concat(chunks))); + } finally { + await reader.cancel(); + reader.releaseLock(); + } + } + + private async load(pageId: number, now: Date): Promise { + const url = new URL('https://en.wikipedia.org/w/api.php'); + url.search = new URLSearchParams({ action: 'query', format: 'json', formatversion: '2', + pageids: String(pageId), prop: 'info|categories|pageprops|description|langlinks', + clshow: '!hidden', cllimit: '50', lllimit: '500', ppprop: 'wikibase_item' }).toString(); + const metadata = metadataSchema.parse(await this.json(url)); + const page = metadata.query.pages[0]; + if (page.pageid !== pageId) throw new Error('Wikipedia page ID mismatch'); + const months = monthWindow(now); + const latestMonth = months[5]; + let pageviews: PageEnrichment['pageviews'] = { status: 'unavailable', months: [], latestMonth, + total: null, dailyAverage: null, trendPercent: null }; + try { + const title = encodeURIComponent(page.title.replace(/ /g, '_')); + const stamp = (month: string) => `${month.replace('-', '')}0100`; + const endpoint = new URL(`https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia.org/all-access/user/${title}/monthly/${stamp(months[0])}/${stamp(latestMonth)}`); + const result = viewsSchema.parse(await this.json(endpoint)); + const byMonth = new Map(); + for (const item of result.items) { + const month = `${item.timestamp.slice(0, 4)}-${item.timestamp.slice(4, 6)}`; + if (!months.includes(month) || byMonth.has(month)) throw new Error('Invalid pageview month'); + byMonth.set(month, item.views); + } + if (byMonth.size !== 6) throw new Error('Incomplete pageviews'); + const ordered = months.map(month => ({ month, views: byMonth.get(month)! })); + const total = ordered.reduce((sum, item) => sum + item.views, 0); + if (!Number.isSafeInteger(total)) throw new Error('Pageview total overflow'); + const latest = ordered[5].views, prior = ordered[4].views; + const days = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), 0)).getUTCDate(); + pageviews = { status: 'available', months: ordered, latestMonth, total, + dailyAverage: latest / days, trendPercent: prior > 0 ? (latest - prior) / prior * 100 : null }; + } catch { /* Metadata remains useful; unknown views must not become fake zeroes. */ } + return { pageId, title: page.title, description: page.description ?? null, + wikidataId: page.pageprops?.wikibase_item ?? null, revisionId: page.lastrevid, pageBytes: page.length, + languageCount: page.langlinks?.length ?? 0, languagesTruncated: metadata.continue?.llcontinue !== undefined, + categories: (page.categories ?? []).map(item => item.title.replace(/^Category:/, '')), + categoriesTruncated: metadata.continue?.clcontinue !== undefined, fetchedAt: now.toISOString(), pageviews, + stats: deriveStats(page.length, page.langlinks?.length ?? 0, + pageviews.status === 'available' ? pageviews.months[5].views : null) }; + } +} + +export const wikipediaEnrichment = new WikipediaEnrichment(); diff --git a/server/index.ts b/server/index.ts index 1f75864..65b8932 100644 --- a/server/index.ts +++ b/server/index.ts @@ -1,2 +1,2 @@ // A node independently validates RandomX work and follows cumulative-work consensus. -import './testnet/http.ts'; \ No newline at end of file +import './testnet/http.ts'; diff --git a/server/testnet/consensus.ts b/server/testnet/consensus.ts index 962a270..986504f 100644 --- a/server/testnet/consensus.ts +++ b/server/testnet/consensus.ts @@ -139,4 +139,4 @@ export async function validateBlock(raw: unknown, ancestors: TestnetBlock[], par if (BigInt(`0x${b.powHash}`) > BigInt(`0x${b.target}`)) throw new Error('Insufficient work'); if (await pow.hash(b.key, blob) !== b.powHash) throw new Error('Invalid RandomX proof'); return { block: b, state: next }; -} \ No newline at end of file +} diff --git a/server/testnet/http.ts b/server/testnet/http.ts index ac482f5..2ae005d 100644 --- a/server/testnet/http.ts +++ b/server/testnet/http.ts @@ -10,6 +10,8 @@ import { CHAIN_ID, blockSchema, transferSchema } from './consensus.ts'; import { RandomX } from './randomx.ts'; import { Ledger } from './ledger.ts'; import { PeerNetwork } from './peers.ts'; +import { wikipediaEnrichment } from '../enrichment.ts'; +import { STARTER_ARTICLES } from '../../shared/catalog.ts'; const root = fileURLToPath(new URL('../../', import.meta.url)); const dataDir = process.env.WIKIDECK_DATA_DIR ?? resolve(root,'data'); @@ -50,6 +52,16 @@ app.use(['/api','/p2p'],(req,res,next)=>{ app.get('/api/state',(_req,res)=>res.json(ledger.snapshot())); app.get('/api/chain',(_req,res)=>res.json(ledger.snapshot())); app.get('/api/peers',(_req,res)=>res.json({peers:peers.status()})); +// Optional live data is intentionally not used by genesis, block validation or NFT ownership. +app.get('/api/wikipedia/:pageId', rateLimit({windowMs:60000,limit:60,standardHeaders:'draft-8',legacyHeaders:false}), async(req,res)=>{ + if(typeof req.params.pageId!=='string'||!/^[1-9]\d{0,9}$/.test(req.params.pageId)){res.status(400).json({error:'Invalid Wikipedia page ID'});return;} + const pageId=Number(req.params.pageId); + if(!STARTER_ARTICLES.some(a=>a.pageId===pageId)){ + res.status(404).json({error:'Page is not in the current consensus catalog'});return; + } + try {res.json(await wikipediaEnrichment.get(pageId));} + catch {res.status(503).json({error:'Wikipedia enrichment is temporarily unavailable. NFT data is unaffected.'});} +}); app.get('/p2p/status',(_req,res)=>res.json({network:NETWORK,chainId:CHAIN_ID,size:ledger.size,tip:ledger.snapshot().tip})); app.get('/p2p/blocks',(req,res)=>{ const offset=Number(req.query.offset??0); @@ -103,4 +115,4 @@ for(const signal of ['SIGTERM','SIGINT']as const)process.once(signal,()=>{ const timeout=setTimeout(()=>process.exit(1),25000);timeout.unref(); server.close(async()=>{engine.close();await lock.close();await unlink(lockPath).catch(()=>{});clearTimeout(timeout);process.exit(0);}); }); -server.on('error',async()=>{peers.close();engine.close();await lock.close();await unlink(lockPath).catch(()=>{});process.exit(1);}); \ No newline at end of file +server.on('error',async()=>{peers.close();engine.close();await lock.close();await unlink(lockPath).catch(()=>{});process.exit(1);}); diff --git a/server/testnet/ledger.ts b/server/testnet/ledger.ts index 697b01e..5b6c0ae 100644 --- a/server/testnet/ledger.ts +++ b/server/testnet/ledger.ts @@ -210,4 +210,4 @@ export class Ledger { } } finally { await unlink(temp).catch(() => undefined); } } -} \ No newline at end of file +} diff --git a/server/testnet/miner.ts b/server/testnet/miner.ts index bd781e9..db9cb8f 100644 --- a/server/testnet/miner.ts +++ b/server/testnet/miner.ts @@ -90,4 +90,4 @@ try { } } } catch(e) { if(!stopping) { console.error((e as Error).message); process.exitCode=1; } } -finally { engine.close(); process.off('SIGINT',stop);process.off('SIGTERM',stop); } \ No newline at end of file +finally { engine.close(); process.off('SIGINT',stop);process.off('SIGTERM',stop); } diff --git a/server/testnet/peers.ts b/server/testnet/peers.ts index c7224c5..5f527b3 100644 --- a/server/testnet/peers.ts +++ b/server/testnet/peers.ts @@ -192,4 +192,4 @@ export class PeerNetwork { }); } finally { clearTimeout(timeout); this.active.delete(controller); } } -} \ No newline at end of file +} diff --git a/server/testnet/randomx.ts b/server/testnet/randomx.ts index 8050a55..55bf2a8 100644 --- a/server/testnet/randomx.ts +++ b/server/testnet/randomx.ts @@ -205,4 +205,4 @@ export class RandomX implements PowEngine { this.shutdownTimer.unref(); } } -} \ No newline at end of file +} diff --git a/shared/enrichment.ts b/shared/enrichment.ts new file mode 100644 index 0000000..784df8c --- /dev/null +++ b/shared/enrichment.ts @@ -0,0 +1,36 @@ +/** Live, off-chain Wikipedia data. Text fields are plain strings, never HTML. */ +export interface PageEnrichment { + pageId: number; + title: string; + description: string | null; + wikidataId: string | null; + revisionId: number; + pageBytes: number; + /** Number of returned interlanguage links (a lower bound when truncated). */ + languageCount: number; + languagesTruncated: boolean; + categories: string[]; + categoriesTruncated: boolean; + fetchedAt: string; + pageviews: { + status: 'available' | 'unavailable'; + /** Six completed UTC months, oldest first, identified as YYYY-MM. */ + months: { month: string; views: number }[]; + latestMonth: string; + total: number | null; + /** Latest completed month's views divided by its actual UTC day count. */ + dailyAverage: number | null; + trendPercent: number | null; + }; + stats: { attention: number | null; depth: number; reach: number }; +} + +/** Attention uses the latest completed month's views, not the six-month total. */ +export function deriveStats(pageBytes: number, languageCount: number, monthlyViews: number | null): PageEnrichment['stats'] { + const score = (value: number, scale: number) => { + if (!Number.isFinite(value) || value < 0) throw new RangeError('Invalid enrichment score input'); + return Math.min(100, Math.round(Math.log10(value + 1) / scale * 100)); + }; + return { attention: monthlyViews === null ? null : score(monthlyViews, 6), + depth: score(pageBytes, 6), reach: score(languageCount, 3) }; +} diff --git a/shared/testnet.ts b/shared/testnet.ts index beac797..26fcdb7 100644 --- a/shared/testnet.ts +++ b/shared/testnet.ts @@ -46,4 +46,4 @@ export function testnetOwnerOf(state: Pick b.tokenId === hash)?.owner; for (const t of state.transfers) if (t.cardHash === hash) owner = t.to; return owner; -} \ No newline at end of file +} diff --git a/shared/wallet-backup.ts b/shared/wallet-backup.ts index bad2f61..2ba4563 100644 --- a/shared/wallet-backup.ts +++ b/shared/wallet-backup.ts @@ -142,4 +142,4 @@ export async function decryptWallet(backup: unknown, password: string): Promise< } catch { throw new Error('Cannot unlock backup: incorrect password, damaged backup, or mismatched wallet keys.'); } finally { plaintext?.fill(0); } -} \ No newline at end of file +} diff --git a/src/Card.tsx b/src/Card.tsx index 6dec57d..57dc200 100644 --- a/src/Card.tsx +++ b/src/Card.tsx @@ -2,6 +2,8 @@ import { ArrowUpRight, BookOpen, Globe2, Shield, Sparkles, Swords } from 'lucide import { rarity, stats, type Article } from '../shared/protocol.ts'; import type { TestnetBlock } from '../shared/testnet.ts'; import { STARTER_ARTICLES } from '../shared/catalog.ts'; +import { useEnrichment } from './useEnrichment.ts'; +import { CardInsights, EnrichmentScores } from './CardInsights.tsx'; // hash is the RandomX proof hash for game stats; block.tokenId is NFT identity. export type CardData = { article: Article; hash: string; block?: TestnetBlock }; @@ -19,15 +21,17 @@ export function Card({ card, onClick }: { card: CardData; onClick?: () => void } const { article, hash } = card; const tier = rarity(hash); const s = stats(hash); - return ; + {!onClick&&}; } diff --git a/src/CardInsights.tsx b/src/CardInsights.tsx new file mode 100644 index 0000000..5bab114 --- /dev/null +++ b/src/CardInsights.tsx @@ -0,0 +1,44 @@ +import { BookOpen, Eye, Globe2, Tags, TrendingUp } from 'lucide-react'; +import { useEnrichment } from './useEnrichment.ts'; +import type { PageEnrichment } from '../shared/enrichment.ts'; +import './enrichment.css'; + +const compact=new Intl.NumberFormat('en',{notation:'compact',maximumFractionDigits:1}); +const full=new Intl.NumberFormat('en',{maximumFractionDigits:0}); +const month=(value:string)=>new Date(`${value}-01T00:00:00Z`).toLocaleDateString('en',{month:'short',year:'numeric',timeZone:'UTC'}); +const score=(value:number|null)=>value===null?'—':`${value}`; +export function EnrichmentScores({data}: {data:PageEnrichment}){ + return
+ {score(data.stats.attention)}Attention + {score(data.stats.depth)}Depth + {score(data.stats.reach)}Reach +
; +} +export function CardInsights({pageId}:{pageId:number}){ + const {data,error,retry}=useEnrichment(pageId); + if(!data)return

Wikipedia insights

{error?'Live metadata is unavailable. The NFT and its original game stats are unchanged.':'Loading Wikipedia metadata and monthly readership…'}

{error&&}
; + const views=data.pageviews; + const latest=views.months.at(-1); + const maximum=Math.max(1,...views.months.map(m=>m.views)); + const trend=views.trendPercent; + return
+

Wikipedia insights

LIVE · OFF-CHAIN
+ {data.description&&

{data.description}

} + +
+
Monthly views
{latest?full.format(latest.views):'Unavailable'}{month(views.latestMonth)} · human traffic
+
Monthly change
{trend===null?'Unavailable':`${trend>=0?'+':''}${trend.toFixed(1)}%`}vs. preceding month
+
6-month views
{views.total===null?'Unavailable':full.format(views.total)}
+
Daily average
{views.dailyAverage===null?'Unavailable':full.format(views.dailyAverage)}latest complete month
+
Article size
{full.format(data.pageBytes)} bytes
+
Languages
{data.languagesTruncated?'≥ ':''}{full.format(data.languageCount)}other-language links
+
Live revision
{data.revisionId}
+
Wikidata
{data.wikidataId?{data.wikidataId}:'Not linked'}
+
+ {views.status==='available'?
Monthly readership · six complete UTC months
{views.months.map(item=>
{compact.format(item.views)}
{month(item.month)}
)}
{views.months.map(item=>)}
Monthly human page views
{month(item.month)}{item.views}
:

Page views are unavailable or incomplete. Unknown counts are not treated as zero.

} +

Wikipedia categories{data.categoriesTruncated?' · first 50':''}

+
{data.categories.length?data.categories.map(category=>{category}):No visible categories returned}
+
How are these scores calculated?

Scores range from 0–100, capped and rounded: Attention = log₁₀(monthly views + 1) ÷ 6 × 100; Depth = log₁₀(article bytes + 1) ÷ 6 × 100; Reach = log₁₀(language links + 1) ÷ 3 × 100. Depth measures size, not accuracy or quality. Scores can change and do not affect proof-derived rarity, NFT ownership, or consensus.

+

Retrieved {new Date(data.fetchedAt).toLocaleString()}. Live revision may differ from the minted snapshot. Sources: Wikipedia contributors (CC BY-SA 4.0) and Wikimedia Analytics (CC0). Views use all-access, user traffic and exclude classified bots. Article moves may affect title-based view history.

+
; +} diff --git a/src/WalletPanel.tsx b/src/WalletPanel.tsx index bb96149..a9af599 100644 --- a/src/WalletPanel.tsx +++ b/src/WalletPanel.tsx @@ -122,4 +122,4 @@ export function WalletPanel({ wallet, onRestore, onNotice }: Props) { ; } -export default WalletPanel; \ No newline at end of file +export default WalletPanel; diff --git a/src/enrichment.css b/src/enrichment.css new file mode 100644 index 0000000..a0eb09b --- /dev/null +++ b/src/enrichment.css @@ -0,0 +1,2 @@ +.card-modal{grid-template-areas:"card detail" "insights insights"}.card-modal .detail-card{display:contents}.card-modal .detail-card>.wiki-card{grid-area:card;align-self:start}.card-modal .detail-copy{grid-area:detail;min-width:0}.card-modal .article-insights{grid-area:insights;margin-top:0}.card-modal .article-fields{grid-template-columns:repeat(4,minmax(0,1fr))}@media(max-width:560px){.card-modal .detail-card>.wiki-card{margin-bottom:24px}.card-modal .article-insights{margin:24px 0}.card-modal .article-fields{grid-template-columns:repeat(2,minmax(0,1fr))}} +.card-enrichment{border-top:1px solid var(--theme-border);margin-top:12px;padding-top:10px;min-height:68px}.card-readership{display:flex;justify-content:space-between;gap:6px;font-size:8px;color:var(--ctp-subtext0)}.card-readership>span:first-child{font-size:7px;letter-spacing:.7px;color:var(--ctp-teal)}.enrichment-scores{display:grid;grid-template-columns:repeat(3,minmax(0,1fr));gap:8px;margin-top:10px}.enrichment-scores>span{display:grid;grid-template-columns:auto auto;justify-content:center;align-items:center;gap:3px 5px;font-size:11px;color:var(--ctp-lavender)}.enrichment-scores b{font-weight:600;color:var(--ctp-text)}.enrichment-scores small{grid-column:1/-1;text-align:center;font-size:8px;color:var(--ctp-subtext0)}.card-enrichment-status{color:var(--ctp-subtext0);font-size:9px}.hero-card .card-enrichment{display:none}.article-insights{margin-top:24px;padding:20px;background:var(--ctp-mantle);border:1px solid var(--ctp-surface1);border-radius:10px;text-align:left;pointer-events:auto}.article-insights h3{font-size:18px;letter-spacing:-.3px;font-weight:550;color:var(--ctp-text)}.article-insights p{font-size:11px;line-height:1.85;color:var(--ctp-subtext0);margin-top:12px}.insights-heading{display:flex;flex-wrap:wrap;align-items:center;justify-content:space-between;gap:9px}.insights-heading>span{font-size:8px;letter-spacing:1px;color:var(--ctp-teal)}.article-insights .enrichment-scores{padding:16px 0;border-bottom:1px solid var(--theme-border)}.article-insights .enrichment-scores b{font-size:21px}.article-insights .enrichment-scores small{font-size:10px}.article-fields{display:grid;grid-template-columns:repeat(2,minmax(0,1fr));gap:18px 14px;margin:20px 0}.article-fields dt{display:flex;align-items:center;gap:5px;font-size:10px;color:var(--ctp-subtext0)}.article-fields dd{margin:6px 0 0;font-size:14px;color:var(--ctp-text);overflow-wrap:anywhere}.article-fields dd small{display:block;font-size:9px;color:var(--ctp-subtext0);margin-top:4px}.article-insights a{color:var(--ctp-blue);text-decoration:underline;text-underline-offset:3px}.pageview-history{margin:22px 0}.pageview-history figcaption{font-size:10px;color:var(--ctp-subtext1);margin-bottom:14px}.pageview-bars{display:grid;grid-template-columns:repeat(6,minmax(0,1fr));gap:7px}.pageview-bars>div{text-align:center;min-width:0}.pageview-bars span{font-size:8px;color:var(--ctp-subtext1)}.pageview-track{height:80px;display:flex;align-items:flex-end;margin-top:5px;background:var(--ctp-base);border-radius:4px;overflow:hidden}.pageview-track i{display:block;width:100%;background:var(--ctp-mauve);border-radius:3px 3px 0 0}.pageview-bars small{display:block;margin-top:7px;font-size:8px;color:var(--ctp-subtext0);line-height:1.6}.insights-category-heading{display:flex;align-items:center;gap:6px;color:var(--ctp-subtext1);font-size:11px;margin:20px 0 12px}.article-categories{display:flex;flex-wrap:wrap;gap:6px;max-height:170px;overflow:auto}.article-categories a,.article-categories>span{font-size:9px;line-height:1.6;border-radius:4px;background:var(--ctp-surface0);padding:4px 7px;text-decoration:none;color:var(--ctp-subtext1);overflow-wrap:anywhere}.article-categories a:hover{color:var(--ctp-mauve)}.insights-method{margin-top:20px;color:var(--ctp-lavender);font-size:10px;line-height:1.8}.insights-method summary{cursor:pointer}.article-insights p.insights-attribution{font-size:9px;overflow-wrap:anywhere}.visually-hidden{position:absolute!important;width:1px;height:1px;padding:0;margin:-1px;overflow:hidden;clip:rect(0,0,0,0);white-space:nowrap;border:0}.card-modal .detail-card{align-self:start}.card-modal .detail-copy{align-self:start}.card-modal .detail-card .wiki-card{pointer-events:none}@media(max-width:560px){.card-readership{flex-direction:column;font-size:8px;gap:4px}.enrichment-scores{gap:4px}.enrichment-scores>span{font-size:10px}.enrichment-scores small{font-size:7px}.article-insights{padding:16px}.card-modal .detail-card{max-width:none}.card-modal .detail-card>.wiki-card{max-width:230px;margin-inline:auto}} diff --git a/src/testnet.css b/src/testnet.css index a317f16..c281d4c 100644 --- a/src/testnet.css +++ b/src/testnet.css @@ -59,4 +59,4 @@ .native-mining .miner-footnote { font-size: 11px; } .wallet-recovery fieldset { padding: 14px; } .wallet-recovery h2 { font-size: 20px; } -} \ No newline at end of file +} diff --git a/src/useEnrichment.ts b/src/useEnrichment.ts new file mode 100644 index 0000000..72ba6c9 --- /dev/null +++ b/src/useEnrichment.ts @@ -0,0 +1,28 @@ +import { useEffect, useState } from 'react'; +import type { PageEnrichment } from '../shared/enrichment.ts'; + +// One request per article shared by hero, collection and detail views. No wallet data is sent. +const cache = new Map }>(); +function load(pageId: number): Promise { + const existing=cache.get(pageId); + if(existing && existing.expires>Date.now())return existing.job; + const job=(async()=>{ + const response=await fetch(`/api/wikipedia/${pageId}`,{signal:AbortSignal.timeout(15000),cache:'no-store'}); + if(!response.ok)throw new Error('Live Wikipedia data unavailable'); + return await response.json() as PageEnrichment; + })(); + cache.set(pageId,{expires:Date.now()+5*60*1000,job}); + if(cache.size>128)cache.delete(cache.keys().next().value!); + return job; +} +export function useEnrichment(pageId: number) { + const [data,setData]=useState(); + const [error,setError]=useState(false); + const [attempt,setAttempt]=useState(0); + useEffect(()=>{ + let active=true;setData(undefined);setError(false); + load(pageId).then(value=>{if(active)setData(value);},()=>{if(active)setError(true);}); + return()=>{active=false;}; + },[pageId,attempt]); + return {data,error,retry:()=>{cache.delete(pageId);setAttempt(n=>n+1);}}; +} diff --git a/tests/api.test.ts b/tests/api.test.ts index 6b76dc4..96b9a4b 100644 --- a/tests/api.test.ts +++ b/tests/api.test.ts @@ -181,6 +181,9 @@ test('native RandomX HTTP: three blocks, mined transfer, peer gossip/sync and du assert.deepEqual(await health.json(), { status: 'ok', network: NETWORK }); assert.equal(health.headers.get('cache-control'), 'no-store'); assert.equal(health.headers.has('ratelimit'), false); + await (await request(first.base, '/api/wikipedia/0', 400)).text(); + await (await request(first.base, '/api/wikipedia/not-a-page', 400)).text(); + await (await request(first.base, '/api/wikipedia/9999999999', 404)).text(); const initial = await snapshot(first.base); assert.equal(initial.network, NETWORK); assert.equal(initial.chainId, CHAIN_ID); diff --git a/tests/enrichment.test.ts b/tests/enrichment.test.ts new file mode 100644 index 0000000..a17755c --- /dev/null +++ b/tests/enrichment.test.ts @@ -0,0 +1,305 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { WikipediaEnrichment } from '../server/enrichment.ts'; +import { deriveStats } from '../shared/enrichment.ts'; + +const JANUARY = new Date('2025-01-15T23:59:59Z'); +const MONTHS = ['2024-07', '2024-08', '2024-09', '2024-10', '2024-11', '2024-12']; +function metadata(pageId = 736, title = 'Albert Einstein') { + return { query: { pages: [{ pageid: pageId, ns: 0, title, lastrevid: 1234, length: 100_000, + description: '', pageprops: { wikibase_item: 'Q937' }, + categories: [{ ns: 14, title: 'Category:Physicists' }], langlinks: [{ lang: 'de', title: 'Albert Einstein' }] }] }, + continue: { clcontinue: '736|next', llcontinue: '736|next', continue: '||' } }; +} +function views(months = MONTHS, counts = [10, 20, 30, 40, 50, 100]) { + return { items: months.map((month, index) => ({ timestamp: `${month.replace('-', '')}0100`, + views: counts[index], project: 'en.wikipedia', access: 'all-access', agent: 'user', granularity: 'monthly' })) }; +} +function json(value: unknown) { return new Response(JSON.stringify(value), { headers: { 'Content-Type': 'application/json; charset=utf-8' } }); } +function mockFetch(handler: (url: URL, init: RequestInit) => Response | Promise) { + const calls: { url: URL; init: RequestInit }[] = []; + const fetcher: typeof fetch = async (input, init) => { + const url = new URL(String(input)); + calls.push({ url, init: init! }); + return handler(url, init!); + }; + return { fetcher, calls }; +} +function fixture(options: { now?: () => Date; ttlMs?: number; maxEntries?: number } = {}) { + const mock = mockFetch(url => json(url.hostname === 'en.wikipedia.org' + ? metadata(Number(url.searchParams.get('pageids'))) : views())); + return { ...mock, service: new WikipediaEnrichment({ fetch: mock.fetcher, now: () => JANUARY, ...options }) }; +} + +test('fixed HTTPS sources, encoded title, hidden categories, lower bounds and plain strings', async () => { + const title = 'A / B? #é%'; + const mock = mockFetch(url => json(url.hostname === 'en.wikipedia.org' + ? metadata(736, title) : { items: views().items.reverse() })); + const service = new WikipediaEnrichment({ fetch: mock.fetcher, now: () => JANUARY }); + const result = await service.get(736); + assert.equal(mock.calls.length, 2); + const query = mock.calls[0].url; + assert.equal(query.origin, 'https://en.wikipedia.org'); + assert.equal(query.pathname, '/w/api.php'); + assert.equal(query.searchParams.get('pageids'), '736'); + assert.equal(query.searchParams.get('formatversion'), '2'); + assert.equal(query.searchParams.get('prop'), 'info|categories|pageprops|description|langlinks'); + assert.equal(query.searchParams.get('clshow'), '!hidden'); + assert.equal(query.searchParams.get('cllimit'), '50'); + assert.equal(query.searchParams.get('lllimit'), '500'); + const rest = mock.calls[1].url; + assert.equal(rest.origin, 'https://wikimedia.org'); + assert.equal(rest.pathname, `/api/rest_v1/metrics/pageviews/per-article/en.wikipedia.org/all-access/user/${encodeURIComponent(title.replace(/ /g, '_'))}/monthly/2024070100/2024120100`); + for (const { init } of mock.calls) { + assert.equal(init.redirect, 'error'); + assert.ok(init.signal instanceof AbortSignal); + assert.equal(new Headers(init.headers).get('accept'), 'application/json'); + assert.match(new Headers(init.headers).get('user-agent')!, /Wikideck/); + assert.equal(new Headers(init.headers).has('origin'), false); // Server-side requests need no CORS workaround. + } + assert.equal(result.title, title); + assert.equal(result.description, ''); + assert.equal(result.wikidataId, 'Q937'); + assert.equal(result.revisionId, 1234); + assert.equal(result.pageBytes, 100_000); + assert.deepEqual(result.categories, ['Physicists']); + assert.equal(result.categoriesTruncated, true); + assert.equal(result.languageCount, 1); + assert.equal(result.languagesTruncated, true); + assert.equal(result.fetchedAt, JANUARY.toISOString()); + assert.deepEqual(result.pageviews.months.map(item => item.month), MONTHS); + assert.equal(result.pageviews.total, 250); + assert.equal(result.pageviews.dailyAverage, 100 / 31); + assert.equal(result.pageviews.trendPercent, 100); + assert.deepEqual(result.stats, deriveStats(100_000, 1, 100)); +}); + +test('UTC completed-month windows cross December/January and use leap February day counts', async () => { + for (const [date, months, days] of [ + ['2024-03-01T00:01:00Z', ['2023-09', '2023-10', '2023-11', '2023-12', '2024-01', '2024-02'], 29], + ['2025-03-01T00:00:00Z', ['2024-09', '2024-10', '2024-11', '2024-12', '2025-01', '2025-02'], 28], + ['2025-01-01T00:01:00+01:00', ['2024-06', '2024-07', '2024-08', '2024-09', '2024-10', '2024-11'], 30], + ] as const) { + const mock = mockFetch(url => json(url.hostname === 'en.wikipedia.org' ? metadata() : views([...months]))); + const result = await new WikipediaEnrichment({ fetch: mock.fetcher, now: () => new Date(date) }).get(736); + assert.deepEqual(result.pageviews.months.map(item => item.month), months); + assert.equal(result.pageviews.dailyAverage, 100 / days); + } +}); + +test('missing optional metadata has explicit defaults, not invented descriptions or counts', async () => { + const mock = mockFetch(url => json(url.hostname === 'en.wikipedia.org' + ? { query: { pages: [{ pageid: 736, ns: 0, title: 'A', lastrevid: 1, length: 0 }] } } : views())); + const result = await new WikipediaEnrichment({ fetch: mock.fetcher, now: () => JANUARY }).get(736); + assert.equal(result.description, null); + assert.equal(result.wikidataId, null); + assert.equal(result.languageCount, 0); + assert.equal(result.languagesTruncated, false); + assert.equal(result.categoriesTruncated, false); + assert.deepEqual(result.categories, []); +}); + +test('reject invalid IDs before fetching and malformed, missing, cross-page or non-article metadata', async () => { + const { service, calls } = fixture(); + for (const id of [0, -1, 1.2, NaN, Infinity, Number.MAX_SAFE_INTEGER + 1, '736' as unknown as number]) { + await assert.rejects(service.get(id), /Invalid Wikipedia page ID/); + } + assert.equal(calls.length, 0); + const invalid = [ + { pageid: 737 }, { ns: 14 }, { missing: true }, { invalid: true }, { lastrevid: -1 }, + { length: Number.MAX_SAFE_INTEGER + 1 }, { title: 'a'.repeat(4097) }, + { categories: Array.from({ length: 51 }, () => ({ ns: 14, title: 'Category:A' })) }, + { langlinks: Array.from({ length: 501 }, () => ({ lang: 'en' })) }, + { pageprops: { wikibase_item: 'https://evil.example' } }, + ]; + for (const patch of invalid) { + const raw = metadata(); + Object.assign(raw.query.pages[0], patch); + const mock = mockFetch(() => json(raw)); + await assert.rejects(new WikipediaEnrichment({ fetch: mock.fetcher }).get(736), /unavailable/); + assert.equal(mock.calls.length, 1); + } + for (const raw of [{}, { query: { pages: [] } }, { query: { pages: [metadata().query.pages[0], metadata().query.pages[0]] } }]) { + const mock = mockFetch(() => json(raw)); + await assert.rejects(new WikipediaEnrichment({ fetch: mock.fetcher }).get(736)); + } +}); + +test('pageviews failure, missing months, duplicates and invalid values retain metadata as unavailable', async () => { + const failures: (() => Response)[] = [ + () => new Response('', { status: 503 }), () => json({ items: [] }), + () => json({ items: views().items.slice(1) }), + () => json({ items: [...views().items.slice(0, 5), views().items[0]] }), + () => json(views(['2024-06', ...MONTHS.slice(1)])), + ...[-1, 0.5, Number.MAX_SAFE_INTEGER + 1].map(value => () => json(views(MONTHS, [value, 0, 0, 0, 0, 0]))), + () => json(views(MONTHS, Array(6).fill(Number.MAX_SAFE_INTEGER))), + () => json({ items: views().items.map(item => ({ ...item, agent: 'spider' })) }), + () => json({ items: views().items.map(item => ({ ...item, timestamp: '2024120200' })) }), + () => { throw new Error('Network failure'); }, + ]; + for (const failure of failures) { + const mock = mockFetch(url => url.hostname === 'en.wikipedia.org' ? json(metadata()) : failure()); + const result = await new WikipediaEnrichment({ fetch: mock.fetcher, now: () => JANUARY }).get(736); + assert.equal(result.title, 'Albert Einstein'); + assert.deepEqual(result.pageviews, { status: 'unavailable', months: [], latestMonth: '2024-12', + total: null, dailyAverage: null, trendPercent: null }); + assert.equal(result.stats.attention, null); + assert.ok(result.stats.depth > 0); + } +}); + +test('six real zero months remain available; zero prior month has no trend', async () => { + const mock = mockFetch(url => json(url.hostname === 'en.wikipedia.org' ? metadata() : views(MONTHS, Array(6).fill(0)))); + const result = await new WikipediaEnrichment({ fetch: mock.fetcher, now: () => JANUARY }).get(736); + assert.equal(result.pageviews.status, 'available'); + assert.equal(result.pageviews.total, 0); + assert.equal(result.pageviews.dailyAverage, 0); + assert.equal(result.pageviews.trendPercent, null); + assert.equal(result.stats.attention, 0); +}); + +test('logarithmic scores are rounded, capped, nullable and reject invalid inputs', () => { + assert.deepEqual(deriveStats(0, 0, null), { attention: null, depth: 0, reach: 0 }); + assert.deepEqual(deriveStats(999, 9, 99), { attention: 33, depth: 50, reach: 33 }); + assert.deepEqual(deriveStats(Number.MAX_SAFE_INTEGER, 1e9, 1e15), { attention: 100, depth: 100, reach: 100 }); + for (const value of [-1, NaN, Infinity]) { + assert.throws(() => deriveStats(value, 0, 0), RangeError); + assert.throws(() => deriveStats(0, value, 0), RangeError); + assert.throws(() => deriveStats(0, 0, value), RangeError); + } +}); + +test('six-hour TTL, deduplication and mutation isolation', async () => { + let now = JANUARY.getTime(); + const { service, calls } = fixture({ now: () => new Date(now) }); + const [first, second] = await Promise.all([service.get(736), service.get(736)]); + assert.equal(calls.length, 2); + first.categories.push('not cached'); + second.pageviews.months[0].views = 999; + assert.deepEqual((await service.get(736)).categories, ['Physicists']); + assert.equal((await service.get(736)).pageviews.months[0].views, 10); + now += 6 * 60 * 60 * 1000 - 1; + await service.get(736); + assert.equal(calls.length, 2); + now++; + await service.get(736); + assert.equal(calls.length, 4); +}); + +test('LRU bound and configured TTL are enforced', async () => { + let now = JANUARY.getTime(); + const { service, calls } = fixture({ maxEntries: 2, ttlMs: 100, now: () => new Date(now) }); + await service.get(1); await service.get(2); await service.get(1); await service.get(3); + assert.equal(calls.length, 6); + await service.get(1); + assert.equal(calls.length, 6); + await service.get(2); + assert.equal(calls.length, 8); + now += 100; + await service.get(2); + assert.equal(calls.length, 10); + for (const options of [{ maxEntries: 129 }, { maxEntries: 0 }, { ttlMs: 0 }, { ttlMs: Infinity }]) { + assert.throws(() => new WikipediaEnrichment(options), RangeError); + } +}); + +test('negative cache expires after 30 seconds; unavailable views retry after five minutes', async () => { + let now = JANUARY.getTime(); + let failPage = true, failViews = true; + const mock = mockFetch(url => { + if (url.hostname === 'en.wikipedia.org') return failPage ? json({}) : json(metadata()); + return failViews ? new Response('', { status: 503 }) : json(views()); + }); + const service = new WikipediaEnrichment({ fetch: mock.fetcher, now: () => new Date(now) }); + await assert.rejects(service.get(736)); + failPage = false; + now += 29_999; + await assert.rejects(service.get(736)); + assert.equal(mock.calls.length, 1); + now++; + assert.equal((await service.get(736)).pageviews.status, 'unavailable'); + failViews = false; + now += 299_999; + assert.equal((await service.get(736)).pageviews.status, 'unavailable'); + assert.equal(mock.calls.length, 3); + now++; + assert.equal((await service.get(736)).pageviews.status, 'available'); + assert.equal(mock.calls.length, 5); +}); + +test('cache rolls over at UTC month boundaries even before its TTL', async () => { + let now = new Date('2024-12-31T23:59:00Z'); + const { service, calls } = fixture({ now: () => now }); + await service.get(736); + now = new Date('2025-01-01T00:00:00Z'); + const result = await service.get(736); + assert.equal(calls.length, 4); + assert.equal(result.pageviews.latestMonth, '2024-12'); + assert.equal(result.pageviews.status, 'available'); +}); + +test('four concurrent jobs, bounded pending requests, busy rejection without a queue and recovery', async () => { + const releases: (() => void)[] = []; + const mock = mockFetch(async url => { + if (url.hostname !== 'en.wikipedia.org') return json(views()); + await new Promise(resolve => releases.push(resolve)); + return json(metadata(Number(url.searchParams.get('pageids')))); + }); + const service = new WikipediaEnrichment({ fetch: mock.fetcher, now: () => JANUARY }); + const jobs = [1, 2, 3, 4].map(id => service.get(id)); + const duplicate = service.get(1); + for (let id = 5; id <= 20; id++) await assert.rejects(service.get(id), /busy/); + assert.equal(mock.calls.length, 4); + releases.splice(0).forEach(release => release()); + await Promise.all([...jobs, duplicate]); + assert.equal(mock.calls.length, 8); + const next = service.get(5); + releases.splice(0).forEach(release => release()); + await next; + assert.equal(mock.calls.length, 10); +}); + +test('JSON-only response contracts, redirects, announced and streamed one-MiB byte limits', async () => { + const badResponses = [ + () => new Response('{}', { headers: { 'Content-Type': 'text/html' } }), + () => new Response('{}'), + () => new Response('{', { headers: { 'Content-Type': 'application/json' } }), + () => new Response(null, { status: 302, headers: { Location: 'https://evil.example' } }), + () => new Response('{}', { headers: { 'Content-Type': 'application/json', 'Content-Length': '1048577' } }), + () => new Response('é'.repeat(600_000), { headers: { 'Content-Type': 'application/json' } }), + () => { const response = json(metadata()); Object.defineProperty(response, 'redirected', { value: true }); return response; }, + ]; + for (const makeResponse of badResponses) { + const mock = mockFetch(() => makeResponse()); + await assert.rejects(new WikipediaEnrichment({ fetch: mock.fetcher }).get(736), /unavailable/); + assert.equal(mock.calls.length, 1); + } + let cancelled = false; + const mock = mockFetch(() => new Response(new ReadableStream({ + pull(controller) { controller.enqueue(new Uint8Array(600_000)); }, + cancel() { cancelled = true; }, + }), { headers: { 'Content-Type': 'application/json' } })); + await assert.rejects(new WikipediaEnrichment({ fetch: mock.fetcher }).get(736)); + assert.equal(cancelled, true); +}); + +test('every fetch receives a six-second timeout and aborted views retain metadata', async t => { + const signals: AbortController[] = []; + t.mock.method(AbortSignal, 'timeout', (ms: number) => { + assert.equal(ms, 6000); + const controller = new AbortController(); + signals.push(controller); + return controller.signal; + }); + const mock = mockFetch((url, init) => { + if (url.hostname === 'en.wikipedia.org') return json(metadata()); + return new Promise((_resolve, reject) => { + init.signal!.addEventListener('abort', () => reject(init.signal!.reason), { once: true }); + signals[1].abort(new DOMException('Timeout', 'TimeoutError')); + }); + }); + const result = await new WikipediaEnrichment({ fetch: mock.fetcher, now: () => JANUARY }).get(736); + assert.equal(signals.length, 2); + assert.equal(result.pageviews.status, 'unavailable'); + assert.equal(result.pageId, 736); +}); diff --git a/tests/testnet-consensus.test.ts b/tests/testnet-consensus.test.ts index a405137..62f73ee 100644 --- a/tests/testnet-consensus.test.ts +++ b/tests/testnet-consensus.test.ts @@ -26,4 +26,4 @@ test('difficulty retargets deterministically at exact windows and is bounded',() const fast=blocks.map((b,i)=>({...b,createdAt:1000+i})); assert.ok(BigInt(`0x${expectedTarget(fast)}`)