From f544019bfe99a2dbd4606dd020708a8d7a31eac4 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 17 Nov 2026 21:51:20 -0500 Subject: [PATCH 1/2] Polite Wikimedia client, mint-path fix, disk cache and mining prefetch pool (#62) - server/wikimedia/client.ts: one process-wide client. Per-host token bucket (1 req/s, burst 3), 1 in flight per host, mint > interactive > background lanes with a bounded queue, maxlag=5 on Action API requests, Retry-After and backoff, a circuit breaker after 3 consecutive 429s, a daily cap, and a per-egress-IP budget (WIKIDECK_WIKIMEDIA_NODES_PER_IP). - server/wikimedia/cache.ts: node:sqlite HTTP cache with ETag/If-None-Match, stale-while-revalidate on the background lane. Mint reads never use it. - Mint fix: a 429/5xx/timeout/non-JSON reply on pageviews now fails getFresh() (template 503, miner retries) instead of minting attention: null forever. - server/wikimedia/pool.ts: per-tip article+snapshot prefetch pool for /api/mining/template, about 3 requests per block regardless of miner count. - User-Agent with an operator contact (WIKIDECK_OPERATOR_CONTACT or PUBLIC_ORIGIN) and a start-up warning when it is unset. - Mint request parameters are byte-identical apart from maxlag (golden test). No consensus, block or protocol changes. Tests use fake fetch only. Co-Authored-By: Claude Opus 5.5 --- docs/night2/WIKIMEDIA.md | 68 +++++ server/enrichment.ts | 99 +++++-- server/testnet/http.ts | 12 +- server/wiki.ts | 32 ++- server/wikimedia-user-agent.ts | 47 +++- server/wikimedia/cache.ts | 74 ++++++ server/wikimedia/client.ts | 441 +++++++++++++++++++++++++++++++ server/wikimedia/index.ts | 36 +++ server/wikimedia/pool.ts | 88 ++++++ tests/enrichment.test.ts | 2 +- tests/testnet-user-agent.test.ts | 26 +- tests/wiki.test.ts | 5 +- tests/wikimedia-client.test.ts | 236 +++++++++++++++++ tests/wikimedia-helpers.ts | 52 ++++ tests/wikimedia-mint.test.ts | 104 ++++++++ tests/wikimedia-pool.test.ts | 74 ++++++ 16 files changed, 1336 insertions(+), 60 deletions(-) create mode 100644 docs/night2/WIKIMEDIA.md create mode 100644 server/wikimedia/cache.ts create mode 100644 server/wikimedia/client.ts create mode 100644 server/wikimedia/index.ts create mode 100644 server/wikimedia/pool.ts create mode 100644 tests/wikimedia-client.test.ts create mode 100644 tests/wikimedia-helpers.ts create mode 100644 tests/wikimedia-mint.test.ts create mode 100644 tests/wikimedia-pool.test.ts diff --git a/docs/night2/WIKIMEDIA.md b/docs/night2/WIKIMEDIA.md new file mode 100644 index 0000000..241f24c --- /dev/null +++ b/docs/night2/WIKIMEDIA.md @@ -0,0 +1,68 @@ +# Polite Wikimedia client (Night 2, issue #62) + +Every server-side Wikimedia request goes through one process-wide `WikimediaClient` +(`server/wikimedia/client.ts`). The design comes from engineer review §4 and data review §5. + +## What it does + +| Behaviour | Default | +|---|---| +| Rate per upstream host (`en.wikipedia.org`, `wikimedia.org`, `www.wikidata.org`) | token bucket, 1 req/s, burst 3 | +| Concurrency | 1 request in flight per host | +| Priority lanes | `mint` → `interactive` → `background`; the queue holds 64 waiters, and background work is shed first | +| Longest queue wait | mint 10 s, interactive 5 s, background 120 s | +| `maxlag` | `maxlag=5` on every Action API request; a maxlag reply waits for `Retry-After` and retries up to 2 times | +| 429 / 503 | honour `Retry-After` (seconds or HTTP date), else back off 2ⁿ s with jitter, capped at 10 min, by pausing the whole host | +| Circuit breaker | 3 consecutive 429s stop that host for 10 min. Requests are refused at once without touching Wikimedia | +| Daily cap | 20,000 requests a day per egress IP. Background work stops at 80 %, interactive at 100 %; mint has a 10 % reserve | +| Disk cache | `$WIKIDECK_DATA_DIR/wikimedia-cache.sqlite`: ETag / Last-Modified, `If-None-Match`, stale-while-revalidate on the background lane, stale copy served on refusal | +| User-Agent | `WikiDeck/ (+https://gitea.eom.dev/WikiDeck/wikideck; ) node/` | + +Cache lifetimes: article metadata 24 h, monthly pageviews 7 days (the URL names six closed +months, which do not change). Random articles and all mint reads are never cached. + +## Operator settings + +| Variable | Meaning | +|---|---| +| `WIKIDECK_OPERATOR_CONTACT` | **Set this.** An email address or URL, put in the User-Agent. Without it (and without `PUBLIC_ORIGIN`), the node logs a warning at start-up. Wikimedia policy requires a contact, and contact-less clients are the first to be blocked. | +| `PUBLIC_ORIGIN` | Used as the contact when `WIKIDECK_OPERATOR_CONTACT` is unset. | +| `WIKIDECK_USER_AGENT` | Overrides the whole User-Agent string (sanitized; kept from Night 1). | +| `WIKIDECK_WIKIMEDIA_RPS` | Requests per second per host for the **whole egress IP** (default 1). | +| `WIKIDECK_WIKIMEDIA_BURST` | Token-bucket burst (default 3). | +| `WIKIDECK_WIKIMEDIA_CONCURRENCY` | In-flight requests per host (default 1). | +| `WIKIDECK_WIKIMEDIA_DAILY_CAP` | Requests per UTC day for the whole egress IP (default 20000). | +| `WIKIDECK_WIKIMEDIA_NODES_PER_IP` | How many nodes share this egress IP. The rate and the daily cap are divided by it. **The testnet runs several nodes behind one IP: set it to the node count** (e.g. 3 for `deploy/night/compose.lan.yaml`). | + +The testnet's `WIKIDECK_USER_AGENT` already names the project URL, which satisfies the policy. + +Counters (`lane.*`, `http.429`, `breaker.open`, `cache.hit`, `refused.*`, …), per-host pause state and the +mint pool's state are in the private operations listener's `/status`, under `wikimedia`. + +## Mint path: transport only + +- `random()` (W1) and `getFresh()` (W2 and W3) keep their request parameters byte-identical. The only + addition is `maxlag=5`, which changes only error behaviour. `tests/wikimedia-mint.test.ts` pins the exact URLs. +- Mint reads use the `mint` lane and **never** read or write the disk cache, and never join a display read. +- **Fixed:** a 429, 5xx, timeout, network error or non-JSON reply on the pageviews call used to mint a card with + `attention: null` forever (`enrichment.ts`). Now `getFresh()` rejects, the template returns 503 and the miner + retries. `unavailable` is minted only for a genuine 404 (no data) or missing months. +- No block, transfer, trade or validation code changed. Validation never calls Wikimedia. + +## Mining prefetch pool (`server/wikimedia/pool.ts`) + +Each template used to cost 3+ requests, all in a burst at each new tip, so load scaled with the number of miners. +Now every template on the same tip gets the same fresh article+snapshot pair (miners race for one block anyway), +and a new tip takes the next prefetched pair. Load is about one pair, or 3 requests, per block, whatever the number +of miners. + +- Pairs come from the mint-path reads above, and the snapshot is stamped when the pair is fetched, so `capturedAt` is true. +- Pairs older than 5 minutes are dropped, never served. +- The pool (2 pairs) refills only after use, so an idle node makes no requests. + +## Card faces + +`/api/wikipedia/:id` → `get()`: memory LRU, then the disk cache (fresh, or stale with a background +revalidation), and only then the interactive lane. A grid of cards therefore no longer means a burst +of 2 upstream requests per card per render. `TrendCache` can use `getBackground()` to stay off the +interactive lane. diff --git a/server/enrichment.ts b/server/enrichment.ts index fa7175a..ed55cdf 100644 --- a/server/enrichment.ts +++ b/server/enrichment.ts @@ -1,5 +1,5 @@ import { z } from 'zod'; -import { ENRICHMENT_USER_AGENT, wikimediaUserAgent } from './wikimedia-user-agent.ts'; +import { WikimediaClient, WikimediaUnavailable, wikimediaClient, type CachePolicy, type Lane } from './wikimedia/client.ts'; import { deriveStats, type PageEnrichment } from '../shared/enrichment.ts'; const MAX_BYTES = 1024 * 1024; @@ -25,7 +25,23 @@ const viewsSchema = z.object({ items: z.array(z.object({ })).max(6) }); type CacheEntry = { expires: number; value: PageEnrichment } | { expires: number; error: Error }; -type Options = { fetch?: typeof globalThis.fetch; now?: () => Date; ttlMs?: number; maxEntries?: number }; +type Options = { fetch?: typeof globalThis.fetch; client?: WikimediaClient; now?: () => Date; ttlMs?: number; maxEntries?: number }; +type Mode = 'mint' | 'display' | 'background'; + +/** Disk-cache lifetimes (engineer review §4). Mint reads never use the cache. */ +const METADATA_CACHE: CachePolicy = { ttlMs: 24 * 3600_000, staleWhileRevalidate: true, maxBytes: MAX_BYTES }; +// The URL names its six completed months, which are immutable; a week bounds late corrections. +const PAGEVIEWS_CACHE: CachePolicy = { ttlMs: 7 * 24 * 3600_000, staleWhileRevalidate: true, maxBytes: MAX_BYTES }; +const NOT_FOUND = Symbol('not found'); + +/** The upstream did not answer properly (throttled, 5xx, timeout, network, non-JSON). Not "no data". */ +export class UpstreamFailure extends Error { + constructor(message: string) { super(message); this.name = 'UpstreamFailure'; } +} +function transportFailure(error: unknown): boolean { + return error instanceof WikimediaUnavailable || error instanceof UpstreamFailure || + (error instanceof DOMException && (error.name === 'TimeoutError' || error.name === 'AbortError')); +} function monthWindow(now: Date): string[] { return Array.from({ length: 6 }, (_, index) => @@ -34,16 +50,17 @@ function monthWindow(now: Date): string[] { /** Fixed-source read-only enrichment; no consensus, persistence, URL or header inputs. */ export class WikipediaEnrichment { - private readonly fetcher: typeof globalThis.fetch; + private readonly client: WikimediaClient; private readonly now: () => Date; private readonly ttlMs: number; private readonly maxEntries: number; private readonly cache = new Map(); - private readonly pending = new Map>(); + private readonly pending = new Map>(); private active = 0; + private mintActive = 0; constructor(options: Options = {}) { - this.fetcher = options.fetch ?? globalThis.fetch; + this.client = options.client ?? (options.fetch ? WikimediaClient.unthrottled(options.fetch) : wikimediaClient); this.now = options.now ?? (() => new Date()); this.ttlMs = options.ttlMs ?? 6 * 60 * 60 * 1000; this.maxEntries = options.maxEntries ?? 128; @@ -53,16 +70,27 @@ export class WikipediaEnrichment { } } + /** Card faces (`/api/wikipedia/:id`): memory, then the disk cache, then the interactive lane. */ async get(pageId: number): Promise { - return this.loadCached(pageId, false); + return this.loadCached(pageId, 'display'); } - /** Bypass cached values when the result is being pinned into a newly minted block. */ + /** Like get(), on the background lane: for trends and other work nobody is waiting on. */ + async getBackground(pageId: number): Promise { + return this.loadCached(pageId, 'background'); + } + + /** + * Mint data: bypasses every cache and uses the mint lane. If pageviews fail in transport + * (429, 5xx, timeout, throttled), this rejects so the template fails and the miner retries; + * `unavailable` is recorded only for a genuine 404 or missing data. + */ async getFresh(pageId: number): Promise { - return this.loadCached(pageId, true); + return this.loadCached(pageId, 'mint'); } - private async loadCached(pageId: number, fresh: boolean): Promise { + private async loadCached(pageId: number, mode: Mode): Promise { + const fresh = mode === 'mint'; if (!Number.isSafeInteger(pageId) || pageId <= 0) throw new RangeError('Invalid Wikipedia page ID'); const now = this.now(); const cached = this.cache.get(pageId); @@ -73,23 +101,29 @@ export class WikipediaEnrichment { if ('error' in cached) throw cached.error; return structuredClone(cached.value); } - this.cache.delete(pageId); - const existing = this.pending.get(pageId); + if (!fresh) this.cache.delete(pageId); + // Mint jobs never join a display job: that one may be served from the disk cache. + const pendingKey = `${fresh ? 'mint' : 'cached'}:${pageId}`; + const existing = this.pending.get(pendingKey); if (existing) return structuredClone(await existing); // Reject immediately: no unbounded waiting queue. Four active jobs also keep // the pending-map ceiling of eight conservative (one upstream request/job). - if (this.pending.size >= 8 || this.active >= 4) throw new Error('Wikipedia enrichment busy'); - this.active++; - const job = this.load(pageId, now).then(value => { + // Mint jobs have their own small allowance so card faces can never starve a template. + if (fresh ? this.mintActive >= 2 : this.pending.size - this.mintActive >= 8 || this.active >= 4) { + throw new Error('Wikipedia enrichment busy'); + } + if (fresh) this.mintActive++; else this.active++; + const job = this.load(pageId, now, mode).then(value => { const ttl = value.pageviews.status === 'available' ? this.ttlMs : Math.min(this.ttlMs, 5 * 60 * 1000); this.remember(pageId, { value, expires: this.now().getTime() + ttl }); return value; }, () => { const error = new Error('Wikipedia page enrichment unavailable'); - this.remember(pageId, { error, expires: this.now().getTime() + 30_000 }); + // A failed mint read says nothing new about display data; keep any cached value. + if (!fresh) this.remember(pageId, { error, expires: this.now().getTime() + 30_000 }); throw error; - }).finally(() => { this.active--; this.pending.delete(pageId); }); - this.pending.set(pageId, job); + }).finally(() => { if (fresh) this.mintActive--; else this.active--; this.pending.delete(pendingKey); }); + this.pending.set(pendingKey, job); return structuredClone(await job); } @@ -99,16 +133,19 @@ export class WikipediaEnrichment { while (this.cache.size > this.maxEntries) this.cache.delete(this.cache.keys().next().value!); } - private async json(url: URL): Promise { - const signal = AbortSignal.timeout(6000); - const response = await this.fetcher(url, { signal, redirect: 'error', - headers: { Accept: 'application/json', 'User-Agent': wikimediaUserAgent(ENRICHMENT_USER_AGENT) } }); + private async json(url: URL, lane: Lane, cache: CachePolicy | undefined, allowNotFound = false): Promise { + const { response, signal } = await this.client.request(url, { lane, timeoutMs: 6000, cache }); + if (allowNotFound && response.status === 404) { await response.body?.cancel(); return NOT_FOUND; } + if (!response.ok || response.redirected) { + await response.body?.cancel(); + throw new UpstreamFailure(`Upstream HTTP ${response.status}`); + } const length = response.headers.get('content-length'); - if (!response.ok || response.redirected || + if ( !/^application\/json(?:\s*;|\s*$)/i.test(response.headers.get('content-type') ?? '') || (length !== null && (!/^\d+$/.test(length) || Number(length) > MAX_BYTES)) || !response.body) { await response.body?.cancel(); - throw new Error('Invalid upstream response'); + throw new UpstreamFailure('Invalid upstream response'); } const reader = response.body.getReader(); const chunks: Uint8Array[] = []; @@ -130,12 +167,14 @@ export class WikipediaEnrichment { } } - private async load(pageId: number, now: Date): Promise { + private async load(pageId: number, now: Date, mode: Mode): Promise { + const lane: Lane = mode === 'mint' ? 'mint' : mode === 'display' ? 'interactive' : 'background'; + const fresh = mode === 'mint'; const url = new URL('https://en.wikipedia.org/w/api.php'); url.search = new URLSearchParams({ action: 'query', format: 'json', formatversion: '2', pageids: String(pageId), prop: 'info|categories|pageprops|description|langlinks', clshow: '!hidden', cllimit: '50', lllimit: '500', ppprop: 'wikibase_item' }).toString(); - const metadata = metadataSchema.parse(await this.json(url)); + const metadata = metadataSchema.parse(await this.json(url, lane, fresh ? undefined : METADATA_CACHE)); const page = metadata.query.pages[0]; if (page.pageid !== pageId) throw new Error('Wikipedia page ID mismatch'); const months = monthWindow(now); @@ -146,7 +185,9 @@ export class WikipediaEnrichment { const title = encodeURIComponent(page.title.replace(/ /g, '_')); const stamp = (month: string) => `${month.replace('-', '')}0100`; const endpoint = new URL(`https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia.org/all-access/user/${title}/monthly/${stamp(months[0])}/${stamp(latestMonth)}`); - const result = viewsSchema.parse(await this.json(endpoint)); + const raw = await this.json(endpoint, lane, fresh ? undefined : PAGEVIEWS_CACHE, true); + if (raw === NOT_FOUND) throw new Error('No pageview data'); + const result = viewsSchema.parse(raw); const byMonth = new Map(); for (const item of result.items) { const month = `${item.timestamp.slice(0, 4)}-${item.timestamp.slice(4, 6)}`; @@ -161,7 +202,11 @@ export class WikipediaEnrichment { const days = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), 0)).getUTCDate(); pageviews = { status: 'available', months: ordered, latestMonth, total, dailyAverage: latest / days, trendPercent: prior > 0 ? (latest - prior) / prior * 100 : null }; - } catch { /* Metadata remains useful; unknown views must not become fake zeroes. */ } + } catch (error) { + // Mint: a throttled or failed upstream must not pin `attention: null` into a block forever. + if (fresh && transportFailure(error)) throw error; + /* Metadata remains useful; unknown views must not become fake zeroes. */ + } return { pageId, title: page.title, description: page.description ?? null, wikidataId: page.pageprops?.wikibase_item ?? null, revisionId: page.lastrevid, pageBytes: page.length, languageCount: page.langlinks?.length ?? 0, languagesTruncated: metadata.continue?.llcontinue !== undefined, diff --git a/server/testnet/http.ts b/server/testnet/http.ts index be6d366..681f73f 100644 --- a/server/testnet/http.ts +++ b/server/testnet/http.ts @@ -11,8 +11,7 @@ import { RandomX } from './randomx.ts'; import { Ledger } from './ledger.ts'; import { PeerNetwork } from './peers.ts'; import { wikipediaEnrichment } from '../enrichment.ts'; -import { snapshotStats } from '../../shared/enrichment.ts'; -import { wikipediaArticles } from '../wiki.ts'; +import { configureWikimedia, mintPool, wikimediaStatus } from '../wikimedia/index.ts'; import { IS_MAINNET, PROFILE } from '../../shared/network.ts'; import { assertLaunchAllowed, launchManifest } from '../launch.ts'; import { resourceConfig, RpcBudget, RpcError } from '../resources.ts'; @@ -28,6 +27,7 @@ const apiBudget = new RpcBudget(limits); const peerBudget = new RpcBudget({ ...limits, requestsPerSecond: 40, writesPerSecond: 10, active: 16 }); const dataDir = process.env.WIKIDECK_DATA_DIR ?? resolve(root,'data'); await mkdir(dataDir,{recursive:true}); +configureWikimedia({dataDir}); // Never permit two processes to rewrite the same ledger. A stale lock requires operator review. const lockPath = resolve(dataDir,`${PROFILE}-node.lock`); const lock = await open(lockPath,'wx',0o600).catch(()=>{throw new Error('Data directory locked. Stop its other node; remove stale testnet-node.lock only after verifying no node is running.');}); @@ -119,9 +119,9 @@ const templateLimit=rateLimit({windowMs:60000,limit:20,standardHeaders:'draft-8' app.post('/api/mining/template', templateLimit, async (req,res)=>{ const body=z.object({owner:z.string().regex(/^[a-f0-9]{64}$/),publicKey:z.string().max(256)}).strict().parse(req.body); try { - const article=await wikipediaArticles.random(); - const metrics=await wikipediaEnrichment.getFresh(article.pageId); - res.json(ledger.template(body.owner,body.publicKey,article,snapshotStats(metrics))); + // One fresh article+snapshot per tip, shared by every miner (server/wikimedia/pool.ts). + const {article,statsSnapshot}=await mintPool.take(ledger.summary().tip); + res.json(ledger.template(body.owner,body.publicKey,article,statsSnapshot)); } catch { res.status(503).json({error:'Could not prepare a Wikipedia article and current statistics. Retry shortly.'}); } @@ -170,7 +170,7 @@ const server=app.listen(port,host,()=>{ const operations=startOperations(opsPort,()=>({network:NETWORK, ledger:ledger.diagnostics(),native:engine.diagnostics(), peers:{configured:peers.status().length,connected:peers.status().filter(p=>p.connected).length}, - rpc:{api:apiBudget.status(),p2p:peerBudget.status()}}),resolve(dataDir,PROFILE)); + rpc:{api:apiBudget.status(),p2p:peerBudget.status()},wikimedia:wikimediaStatus()}),resolve(dataDir,PROFILE)); server.maxConnections=256; server.headersTimeout=10000; server.requestTimeout=15000; diff --git a/server/wiki.ts b/server/wiki.ts index 4b5ccaf..d0a7ee5 100644 --- a/server/wiki.ts +++ b/server/wiki.ts @@ -1,6 +1,6 @@ import { z } from 'zod'; import type { Article } from '../shared/protocol.ts'; -import { ARTICLE_USER_AGENT, wikimediaUserAgent } from './wikimedia-user-agent.ts'; +import { WikimediaClient, WikimediaUnavailable, wikimediaClient } from './wikimedia/client.ts'; const pageSchema = z.object({ pageid: z.number().int().positive().max(Number.MAX_SAFE_INTEGER), ns: z.literal(0), @@ -58,11 +58,23 @@ function pictureUrl(value: string | undefined): string | undefined { } catch { return undefined; } } +/** The exact W1 request. Its parameters decide minted article content: keep them byte-identical. */ +export function randomArticlesUrl(): URL { + const url = new URL('https://en.wikipedia.org/w/api.php'); + url.search = new URLSearchParams({ action: 'query', format: 'json', formatversion: '2', + generator: 'random', grnnamespace: '0', grnlimit: '10', prop: 'extracts|pageimages|info|categories', + exintro: '1', explaintext: '1', exchars: '900', piprop: 'thumbnail', pithumbsize: '800', cllimit: '20' }).toString(); + return url; +} + export class WikipediaArticleSource { - private readonly fetcher: typeof globalThis.fetch; + private readonly client: WikimediaClient; private active = 0; - constructor(fetcher: typeof globalThis.fetch = globalThis.fetch) { this.fetcher = fetcher; } + /** A bare fetch (tests) gets an unthrottled pass-through client; production uses the shared client. */ + constructor(source: typeof globalThis.fetch | WikimediaClient = wikimediaClient) { + this.client = source instanceof WikimediaClient ? source : WikimediaClient.unthrottled(source); + } async random(): Promise
{ if (this.active >= 4) throw new Error('Wikipedia article service is busy'); @@ -70,12 +82,8 @@ export class WikipediaArticleSource { try { for (let attempt = 0; attempt < MAX_ATTEMPTS; attempt++) { try { - const url = new URL('https://en.wikipedia.org/w/api.php'); - url.search = new URLSearchParams({ action: 'query', format: 'json', formatversion: '2', - generator: 'random', grnnamespace: '0', grnlimit: '10', prop: 'extracts|pageimages|info|categories', - exintro: '1', explaintext: '1', exchars: '900', piprop: 'thumbnail', pithumbsize: '800', cllimit: '20' }).toString(); - const response = await this.fetcher(url, { signal: AbortSignal.timeout(3500), redirect: 'error', - headers: { 'User-Agent': wikimediaUserAgent(ARTICLE_USER_AGENT), Accept: 'application/json' } }); + // Mint lane, never cached: random articles become immutable chain data. + const { response } = await this.client.request(randomArticlesUrl(), { lane: 'mint', timeoutMs: 3500 }); const data = responseSchema.parse(await readJson(response)); for (const rawPage of data.query.pages) { const parsedPage = pageSchema.safeParse(rawPage); @@ -88,7 +96,11 @@ export class WikipediaArticleSource { url: `https://en.wikipedia.org/?curid=${page.pageid}`, revisionId: page.lastrevid, source: 'live', category: category(page.title, page.categories?.map(c => c.title) ?? []) }; } - } catch { /* Try another random API response, but keep the number of attempts bounded. */ } + } catch (error) { + // Throttled, paused or failing upstream: stop now rather than hammer it with retries. + if (error instanceof WikimediaUnavailable) throw error; + /* Otherwise try another random API response, but keep the number of attempts bounded. */ + } } throw new Error('Wikipedia did not return an article with a picture after three attempts'); } finally { diff --git a/server/wikimedia-user-agent.ts b/server/wikimedia-user-agent.ts index 39eb229..491a40b 100644 --- a/server/wikimedia-user-agent.ts +++ b/server/wikimedia-user-agent.ts @@ -1,13 +1,48 @@ -// User-Agent for Wikimedia API requests. Wikimedia asks automated clients to identify -// themselves with contact information; operators can opt in with WIKIDECK_USER_AGENT. -// Unset (or blank), each caller keeps its historical default string unchanged. -export const ARTICLE_USER_AGENT = 'Wikideck/0.1 (experimental Wikipedia article cards)'; -export const ENRICHMENT_USER_AGENT = 'Wikideck/0.1 (Wikipedia off-chain page enrichment)'; +// User-Agent for Wikimedia API requests. Wikimedia's User-Agent policy asks automated clients for a +// descriptive agent with contact details: `Tool/version (+https://site; contact) lib/version`. +// Operators set WIKIDECK_OPERATOR_CONTACT (an email address or URL); PUBLIC_ORIGIN is the fallback. +// WIKIDECK_USER_AGENT still overrides the whole string. +import { readFileSync } from 'node:fs'; + +export const PROJECT_URL = 'https://gitea.eom.dev/WikiDeck/wikideck'; const MAX_LENGTH = 256; +const SAFE = /^[\x20-\x7e]+$/; + +function packageVersion(): string { + try { + const raw = JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8')) as { version?: unknown }; + return typeof raw.version === 'string' && /^[\w.+-]{1,32}$/.test(raw.version) ? raw.version : '0'; + } catch { return '0'; } +} +const VERSION = packageVersion(); + +/** The operator contact, or undefined when none (or only an unsafe value) is configured. */ +export function operatorContact(env: Record = process.env): string | undefined { + for (const value of [env.WIKIDECK_OPERATOR_CONTACT, env.PUBLIC_ORIGIN]) { + const trimmed = value?.trim(); + if (trimmed && trimmed.length <= 128 && SAFE.test(trimmed) && !/[();]/.test(trimmed)) return trimmed; + } + return undefined; +} + +/** `WikiDeck/ (+; ) node/`. */ +export function defaultUserAgent(env: Record = process.env): string { + const contact = operatorContact(env); + return `WikiDeck/${VERSION} (+${PROJECT_URL}${contact ? `; ${contact}` : ''}) node/${process.versions.node}`; +} export function wikimediaUserAgent(fallback: string, env: Record = process.env): string { const configured = env.WIKIDECK_USER_AGENT?.trim(); // Reject header-injection/control characters and oversized values; fall back to the default. - if (!configured || configured.length > MAX_LENGTH || !/^[\x20-\x7e]+$/.test(configured)) return fallback; + if (!configured || configured.length > MAX_LENGTH || !SAFE.test(configured)) return fallback; return configured; } + +/** Start-up check: Wikimedia throttles or blocks clients without contact details first. */ +export function warnIfNoContact(env: Record = process.env, + warn: (message: string) => void = console.warn): boolean { + if (wikimediaUserAgent('', env) || operatorContact(env)) return false; + warn('[wikimedia] WARNING: no operator contact in the Wikimedia User-Agent. Set WIKIDECK_OPERATOR_CONTACT ' + + '(an email address or URL) or PUBLIC_ORIGIN; Wikimedia policy requires it and may block this node.'); + return true; +} diff --git a/server/wikimedia/cache.ts b/server/wikimedia/cache.ts new file mode 100644 index 0000000..6ebd42e --- /dev/null +++ b/server/wikimedia/cache.ts @@ -0,0 +1,74 @@ +/** + * On-disk HTTP cache for off-chain Wikimedia reads (`$WIKIDECK_DATA_DIR/wikimedia-cache.sqlite`). + * It stores the raw body with its ETag / Last-Modified so stale entries can be revalidated with + * conditional requests. Mint-path requests never use it (see client.ts). + */ +import { DatabaseSync } from 'node:sqlite'; + +export type StoredEntry = { + status: number; body: Buffer; contentType: string; etag: string | null; lastModified: string | null; + expiresAt: number; storedAt: number; +}; +export type CacheEntry = StoredEntry & { toResponse(): Response }; + +/** Canonical key: origin + path + sorted query, without `maxlag` (it changes only error behaviour). */ +export function cacheKey(url: URL): string { + const params = [...url.searchParams].filter(([name]) => name !== 'maxlag') + .sort(([a, x], [b, y]) => a < b ? -1 : a > b ? 1 : x < y ? -1 : x > y ? 1 : 0); + return `${url.origin}${url.pathname}?${new URLSearchParams(params).toString()}`; +} + +function withResponse(entry: StoredEntry): CacheEntry { + return { ...entry, toResponse: () => new Response(new Uint8Array(entry.body), { status: entry.status, + headers: { 'Content-Type': entry.contentType, 'Content-Length': String(entry.body.byteLength), + ...(entry.etag ? { ETag: entry.etag } : {}) } }) }; +} + +type Row = { status: number; body: Uint8Array; content_type: string; etag: string | null; + last_modified: string | null; expires_at: number; stored_at: number }; + +export class WikimediaCache { + private readonly db: DatabaseSync; + private readonly maxRows: number; + private writes = 0; + + /** `path` may be ':memory:' (tests). */ + constructor(path: string, options: { maxRows?: number } = {}) { + this.maxRows = options.maxRows ?? 100_000; + this.db = new DatabaseSync(path); + this.db.exec(`PRAGMA journal_mode=WAL; PRAGMA synchronous=NORMAL; + CREATE TABLE IF NOT EXISTS http_cache (key TEXT PRIMARY KEY, status INTEGER NOT NULL, body BLOB NOT NULL, + content_type TEXT NOT NULL, etag TEXT, last_modified TEXT, expires_at INTEGER NOT NULL, stored_at INTEGER NOT NULL); + CREATE INDEX IF NOT EXISTS http_cache_stored ON http_cache(stored_at);`); + } + + get(key: string): CacheEntry | undefined { + const row = this.db.prepare('SELECT * FROM http_cache WHERE key = ?').get(key) as Row | undefined; + if (!row) return undefined; + return withResponse({ status: row.status, body: Buffer.from(row.body), contentType: row.content_type, + etag: row.etag, lastModified: row.last_modified, expiresAt: row.expires_at, storedAt: row.stored_at }); + } + + put(key: string, entry: StoredEntry): CacheEntry { + this.db.prepare(`INSERT OR REPLACE INTO http_cache (key, status, body, content_type, etag, last_modified, expires_at, stored_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?)`).run(key, entry.status, entry.body, entry.contentType, entry.etag, + entry.lastModified, entry.expiresAt, entry.storedAt); + if (++this.writes % 500 === 0) this.prune(); + return withResponse(entry); + } + + touch(key: string, expiresAt: number): void { + this.db.prepare('UPDATE http_cache SET expires_at = ? WHERE key = ?').run(expiresAt, key); + } + + delete(key: string): void { this.db.prepare('DELETE FROM http_cache WHERE key = ?').run(key); } + + size(): number { return (this.db.prepare('SELECT COUNT(*) AS n FROM http_cache').get() as { n: number }).n; } + + prune(): void { + this.db.prepare(`DELETE FROM http_cache WHERE key IN (SELECT key FROM http_cache ORDER BY stored_at DESC LIMIT -1 OFFSET ?)`) + .run(this.maxRows); + } + + close(): void { this.db.close(); } +} diff --git a/server/wikimedia/client.ts b/server/wikimedia/client.ts new file mode 100644 index 0000000..e63336d --- /dev/null +++ b/server/wikimedia/client.ts @@ -0,0 +1,441 @@ +/** + * One process-wide, polite Wikimedia HTTP client (engineer review §4, data review §5). + * + * Every server-side Wikimedia request goes through `WikimediaClient.request()`: + * - a token bucket per upstream host (default 1 req/s, burst 3) and 1 request in flight per host; + * - priority lanes: `mint` before `interactive` before `background`, with a bounded queue; + * - `maxlag=5` on Action API requests (it changes only error behaviour, never content); + * - `Retry-After` honoured on 429/503/maxlag by pausing the whole host, with exponential backoff; + * - a circuit breaker: 3 consecutive 429s stop all requests to that host for 10 minutes; + * - a daily request cap, and a budget shared by the nodes behind one egress IP; + * - an optional on-disk cache (see cache.ts) with ETag / If-None-Match and stale-while-revalidate. + * + * The client is transport only. It never rewrites request parameters other than adding `maxlag`, + * and requests made with `cache: undefined` (all mint-path requests) never read or write the cache. + * Nothing here is reachable from block validation, so it cannot affect consensus. + */ +import { WikimediaCache, cacheKey } from './cache.ts'; +import { wikimediaUserAgent, defaultUserAgent } from '../wikimedia-user-agent.ts'; + +export type Lane = 'mint' | 'interactive' | 'background'; +const LANES: Lane[] = ['mint', 'interactive', 'background']; + +export type Clock = { + now(): number; + setTimeout(fn: () => void, ms: number): unknown; + clearTimeout(handle: unknown): void; +}; +const realClock: Clock = { + now: () => Date.now(), + setTimeout: (fn, ms) => { const t = setTimeout(fn, ms); t.unref?.(); return t; }, + clearTimeout: handle => clearTimeout(handle as ReturnType), +}; + +export type ClientConfig = { + /** Sustained requests per second per upstream host, for this whole egress IP. */ + requestsPerSecond: number; + burst: number; + concurrencyPerHost: number; + /** Requests per UTC day for this whole egress IP. */ + dailyCap: number; + /** Nodes sharing one egress IP; the rate and daily cap are divided between them. */ + nodesPerIp: number; + queueLimit: number; + /** Longest wait in the queue before a request fails, per lane. */ + maxWaitMs: Record; + breakerThreshold: number; + breakerMs: number; + maxBackoffMs: number; + maxlag: number; + /** Retries of a single request after a maxlag reply (HTTP 200/503 with MediaWiki-API-Error: maxlag). */ + maxlagRetries: number; +}; + +export const DEFAULT_CONFIG: ClientConfig = { + requestsPerSecond: 1, burst: 3, concurrencyPerHost: 1, dailyCap: 20_000, nodesPerIp: 1, queueLimit: 64, + maxWaitMs: { mint: 10_000, interactive: 5_000, background: 120_000 }, + breakerThreshold: 3, breakerMs: 10 * 60_000, maxBackoffMs: 10 * 60_000, maxlag: 5, maxlagRetries: 2, +}; + +/** Unthrottled settings for tests and tools that inject their own fetch. */ +export const UNTHROTTLED: Partial = { + requestsPerSecond: 1e9, burst: 1e9, concurrencyPerHost: 1e9, dailyCap: Number.MAX_SAFE_INTEGER, queueLimit: 1e9, + breakerThreshold: Number.MAX_SAFE_INTEGER, maxlagRetries: 0, maxBackoffMs: 0, +}; + +function positive(value: string | undefined, fallback: number, integer = false): number { + if (value === undefined || value.trim() === '') return fallback; + const n = Number(value); + if (!Number.isFinite(n) || n <= 0 || (integer && !Number.isSafeInteger(n))) { + console.warn(`[wikimedia] ignoring invalid setting ${JSON.stringify(value)}; using ${fallback}`); + return fallback; + } + return n; +} + +/** Operator settings. Documented in docs/night2/WIKIMEDIA.md. */ +export function configFromEnv(env: Record = process.env): ClientConfig { + return { ...DEFAULT_CONFIG, + requestsPerSecond: positive(env.WIKIDECK_WIKIMEDIA_RPS, DEFAULT_CONFIG.requestsPerSecond), + burst: positive(env.WIKIDECK_WIKIMEDIA_BURST, DEFAULT_CONFIG.burst), + concurrencyPerHost: positive(env.WIKIDECK_WIKIMEDIA_CONCURRENCY, DEFAULT_CONFIG.concurrencyPerHost, true), + dailyCap: positive(env.WIKIDECK_WIKIMEDIA_DAILY_CAP, DEFAULT_CONFIG.dailyCap, true), + nodesPerIp: positive(env.WIKIDECK_WIKIMEDIA_NODES_PER_IP, DEFAULT_CONFIG.nodesPerIp, true), + }; +} + +/** Thrown for every refusal or transport failure: queue full, cap reached, circuit open, timeout, network. */ +export class WikimediaUnavailable extends Error { + readonly reason: string; + readonly status?: number; + constructor(reason: string, status?: number) { + super(`Wikimedia unavailable: ${reason}`); + this.name = 'WikimediaUnavailable'; + this.reason = reason; this.status = status; + } +} + +export type CachePolicy = { + ttlMs: number; + /** Serve a stale entry at once and revalidate it on the background lane. */ + staleWhileRevalidate?: boolean; + /** Largest body stored, in bytes. */ + maxBytes?: number; +}; + +export type RequestOptions = { + lane: Lane; + /** Starts when the request is dispatched, not while it waits in the queue. */ + timeoutMs: number; + headers?: Record; + /** Omit for mint data: the request then bypasses the cache completely. */ + cache?: CachePolicy; +}; + +export type ClientResponse = { response: Response; signal: AbortSignal; fromCache: boolean }; + +type Waiter = { + lane: Lane; host: string; enqueuedAt: number; + run: () => void; fail: (error: Error) => void; timer?: unknown; +}; +type HostState = { + tokens: number; refilledAt: number; inFlight: number; + pausedUntil: number; breakerUntil: number; consecutive429: number; backoffExponent: number; +}; + +export type ClientStatus = { + day: string; requestsToday: number; dailyCap: number; requestsPerSecond: number; queued: number; + counters: Record; + hosts: Record; +}; + +function parseRetryAfter(value: string | null, now: number): number | undefined { + if (!value) return undefined; + if (/^\d+$/.test(value.trim())) return Number(value.trim()) * 1000; + const at = Date.parse(value); + return Number.isNaN(at) ? undefined : Math.max(0, at - now); +} + +function isActionApi(url: URL): boolean { + return url.pathname === '/w/api.php' && /(^|\.)(wikipedia|wikidata|wikimedia)\.org$/.test(url.hostname); +} + +export class WikimediaClient { + readonly config: ClientConfig; + private readonly fetcher: typeof globalThis.fetch; + private readonly clock: Clock; + private readonly userAgent: () => string; + private cache: WikimediaCache | undefined; + private readonly hosts = new Map(); + private readonly queues = new Map(LANES.map(lane => [lane, []])); + private readonly revalidating = new Set(); + private pumpTimer: unknown; + private pumpAt = Infinity; + private day = ''; + private requestsToday = 0; + readonly counters: Record = {}; + + constructor(options: { fetch?: typeof globalThis.fetch; config?: Partial; clock?: Clock; + cache?: WikimediaCache; userAgent?: () => string } = {}) { + this.fetcher = options.fetch ?? ((...args) => globalThis.fetch(...args)); + this.config = { ...DEFAULT_CONFIG, ...options.config, + maxWaitMs: { ...DEFAULT_CONFIG.maxWaitMs, ...options.config?.maxWaitMs } }; + this.clock = options.clock ?? realClock; + this.userAgent = options.userAgent ?? (() => wikimediaUserAgent(defaultUserAgent())); + this.cache = options.cache; + } + + /** A pass-through client for injected fetchers (tests): no rate limit, no breaker, no cache. */ + static unthrottled(fetcher: typeof globalThis.fetch): WikimediaClient { + return new WikimediaClient({ fetch: fetcher, config: UNTHROTTLED }); + } + + attachCache(cache: WikimediaCache | undefined): void { this.cache = cache; } + + private count(name: string): void { this.counters[name] = (this.counters[name] ?? 0) + 1; } + + private get rate(): number { return this.config.requestsPerSecond / this.config.nodesPerIp; } + private get cap(): number { return Math.max(1, Math.floor(this.config.dailyCap / this.config.nodesPerIp)); } + + private host(name: string): HostState { + let state = this.hosts.get(name); + if (!state) { + state = { tokens: this.config.burst, refilledAt: this.clock.now(), inFlight: 0, + pausedUntil: 0, breakerUntil: 0, consecutive429: 0, backoffExponent: 0 }; + this.hosts.set(name, state); + } + return state; + } + + private refill(state: HostState, now: number): void { + state.tokens = Math.min(this.config.burst, state.tokens + (now - state.refilledAt) / 1000 * this.rate); + state.refilledAt = now; + } + + /** Daily budget by lane: background stops at 80 %, interactive at 100 %, mint has a 10 % reserve. */ + private overCap(lane: Lane): boolean { + const today = new Date(this.clock.now()).toISOString().slice(0, 10); + if (today !== this.day) { this.day = today; this.requestsToday = 0; } + const limit = lane === 'background' ? this.cap * 0.8 : lane === 'interactive' ? this.cap : this.cap * 1.1; + return this.requestsToday >= limit; + } + + status(): ClientStatus { + const now = this.clock.now(); + return { day: this.day, requestsToday: this.requestsToday, dailyCap: this.cap, requestsPerSecond: this.rate, + queued: LANES.reduce((n, lane) => n + this.queues.get(lane)!.length, 0), counters: { ...this.counters }, + hosts: Object.fromEntries([...this.hosts].map(([name, s]) => [name, + { inFlight: s.inFlight, pausedForMs: Math.max(0, s.pausedUntil - now), circuitOpen: s.breakerUntil > now, + consecutive429: s.consecutive429 }])) }; + } + + async request(input: URL, options: RequestOptions): Promise { + const url = new URL(input); + if (url.protocol !== 'https:') throw new RangeError('Wikimedia requests must use HTTPS'); + if (isActionApi(url) && !url.searchParams.has('maxlag')) url.searchParams.append('maxlag', String(this.config.maxlag)); + const key = options.cache && this.cache ? cacheKey(url) : undefined; + const entry = key ? this.cache?.get(key) : undefined; + const now = this.clock.now(); + if (entry && entry.expiresAt > now) { + this.count('cache.hit'); + return { response: entry.toResponse(), signal: new AbortController().signal, fromCache: true }; + } + if (entry && options.cache?.staleWhileRevalidate) { + this.count('cache.stale'); + this.revalidate(url, key!, options); + return { response: entry.toResponse(), signal: new AbortController().signal, fromCache: true }; + } + try { + return await this.network(url, options, key); + } catch (error) { + // Prefer a stale copy over an error for anything that is allowed to use the cache. + if (entry && error instanceof WikimediaUnavailable) { + this.count('cache.staleOnError'); + return { response: entry.toResponse(), signal: new AbortController().signal, fromCache: true }; + } + throw error; + } + } + + private revalidate(url: URL, key: string, options: RequestOptions): void { + if (this.revalidating.has(key)) return; + this.revalidating.add(key); + void this.network(url, { ...options, lane: 'background' }, key) + .then(({ response }) => response.body?.cancel(), () => this.count('cache.revalidateFailed')) + .finally(() => this.revalidating.delete(key)); + } + + private async network(url: URL, options: RequestOptions, key: string | undefined): Promise { + for (let attempt = 0; ; attempt++) { + const result = await this.dispatch(url, options, key); + if (result !== 'maxlag') return result; + if (attempt >= this.config.maxlagRetries) throw new WikimediaUnavailable('maxlag'); + } + } + + /** Wait for a slot, send one request, and classify the reply. */ + private async dispatch(url: URL, options: RequestOptions, key: string | undefined): Promise { + const hostName = url.hostname; + const wait = this.acquire(hostName, options.lane); + if (wait) await wait; // No await on the fast path: an idle host dispatches synchronously. + const state = this.host(hostName); + const headers = new Headers({ Accept: 'application/json', 'Accept-Encoding': 'gzip', ...options.headers, + 'User-Agent': this.userAgent() }); + const entry = key ? this.cache?.get(key) : undefined; + if (entry?.etag) headers.set('If-None-Match', entry.etag); + if (entry?.lastModified) headers.set('If-Modified-Since', entry.lastModified); + const signal = AbortSignal.timeout(options.timeoutMs); + this.requestsToday++; + this.count(`lane.${options.lane}`); + let response: Response; + try { + response = await this.fetcher(url, { signal, redirect: 'error', headers }); + } catch (error) { + this.release(state); + this.count('error.network'); + throw new WikimediaUnavailable(signal.aborted ? 'timeout' : `network: ${(error as Error).message}`); + } + const now = this.clock.now(); + const retryAfter = parseRetryAfter(response.headers.get('retry-after'), now); + const apiError = response.headers.get('mediawiki-api-error'); + if (apiError === 'maxlag') { + await response.body?.cancel(); + this.count('maxlag'); + this.pause(state, Math.min(this.config.maxBackoffMs, retryAfter ?? 5000), false); + this.release(state); + return 'maxlag'; + } + if (response.status === 429 || response.status === 503) { + await response.body?.cancel(); + this.count(`http.${response.status}`); + if (response.status === 429) state.consecutive429++; + const backoff = Math.min(this.config.maxBackoffMs, 2 ** state.backoffExponent * 1000 * (1 + Math.random() * 0.25)); + state.backoffExponent = Math.min(state.backoffExponent + 1, 20); + if (state.consecutive429 >= this.config.breakerThreshold) { + this.pause(state, this.config.breakerMs, true); + this.count('breaker.open'); + } else { + this.pause(state, Math.min(this.config.maxBackoffMs, Math.max(retryAfter ?? 0, backoff)), false); + } + this.release(state); + throw new WikimediaUnavailable(`HTTP ${response.status}`, response.status); + } + state.consecutive429 = 0; + state.backoffExponent = 0; + this.release(state); + if (response.status === 304 && entry && key) { + this.count('cache.revalidated'); + this.cache?.touch(key, now + options.cache!.ttlMs); + return { response: entry.toResponse(), signal, fromCache: true }; + } + if (key && this.cache && response.status === 200 && !response.redirected && options.cache && + /^application\/json(?:\s*;|\s*$)/i.test(response.headers.get('content-type') ?? '')) { + // Store the raw body; callers re-validate it exactly as they would a network reply. + const body = await this.readLimited(response, options.cache.maxBytes ?? 1024 * 1024, signal); + const stored = this.cache.put(key, { status: response.status, body, + contentType: response.headers.get('content-type') ?? '', etag: response.headers.get('etag'), + lastModified: response.headers.get('last-modified'), expiresAt: now + options.cache.ttlMs, storedAt: now }); + return { response: stored.toResponse(), signal, fromCache: false }; + } + if (response.status >= 500) this.count(`http.${response.status}`); + return { response, signal, fromCache: false }; + } + + private async readLimited(response: Response, maxBytes: number, signal: AbortSignal): Promise { + const length = response.headers.get('content-length'); + if (length !== null && (!/^\d+$/.test(length) || Number(length) > maxBytes) || !response.body) { + await response.body?.cancel(); + throw new WikimediaUnavailable('oversized or empty response'); + } + const reader = response.body.getReader(); + const chunks: Uint8Array[] = []; + let size = 0; + try { + for (;;) { + if (signal.aborted) throw new WikimediaUnavailable('timeout'); + const { done, value } = await reader.read(); + if (done) break; + size += value.byteLength; + if (size > maxBytes) throw new WikimediaUnavailable('oversized response'); + chunks.push(value); + } + return Buffer.concat(chunks); + } catch (error) { + throw error instanceof WikimediaUnavailable ? error : new WikimediaUnavailable('read failed'); + } finally { + await reader.cancel().catch(() => {}); + reader.releaseLock(); + } + } + + private pause(state: HostState, ms: number, breaker: boolean): void { + const until = this.clock.now() + ms; + if (until > state.pausedUntil) { + if (state.pausedUntil <= this.clock.now()) { + console.warn(`[wikimedia] ${breaker ? 'circuit open' : 'pausing'} for ${Math.ceil(ms / 1000)} s`); + } + state.pausedUntil = until; + } + if (breaker) { state.consecutive429 = 0; state.breakerUntil = until; } + } + + private release(state: HostState): void { + state.inFlight--; + this.pump(); + } + + /** Undefined when a slot was taken at once; otherwise a promise for the slot. Throws on refusal. */ + private acquire(hostName: string, lane: Lane): Promise | undefined { + const state = this.host(hostName); + const now = this.clock.now(); + if (this.overCap(lane)) { this.count('refused.cap'); throw new WikimediaUnavailable('daily cap reached'); } + // A paused host refuses at once when the pause outlasts this lane's patience. + if (state.pausedUntil - now > this.config.maxWaitMs[lane]) { + this.count('refused.paused'); + throw new WikimediaUnavailable(state.breakerUntil > now ? 'circuit open' : 'host paused'); + } + this.refill(state, now); + if (state.pausedUntil <= now && state.inFlight < this.config.concurrencyPerHost && state.tokens >= 1 && + !LANES.some(l => this.queues.get(l)!.some(w => w.host === hostName))) { + state.tokens -= 1; + state.inFlight++; + return undefined; + } + const queued = LANES.reduce((n, l) => n + this.queues.get(l)!.length, 0); + if (queued >= this.config.queueLimit) { + // Background work is shed first; a mint request evicts the newest background waiter. + const background = this.queues.get('background')!; + if (lane === 'background' || background.length === 0) { + this.count('refused.queue'); + throw new WikimediaUnavailable('queue full'); + } + const evicted = background.pop()!; + this.clock.clearTimeout(evicted.timer); + evicted.fail(new WikimediaUnavailable('queue full')); + } + return new Promise((resolve, reject) => { + const waiter: Waiter = { lane, host: hostName, enqueuedAt: now, + run: () => { this.clock.clearTimeout(waiter.timer); resolve(); }, + fail: error => reject(error) }; + waiter.timer = this.clock.setTimeout(() => { + const queue = this.queues.get(lane)!; + const index = queue.indexOf(waiter); + if (index >= 0) { queue.splice(index, 1); this.count('refused.wait'); reject(new WikimediaUnavailable('queue wait')); } + }, this.config.maxWaitMs[lane]); + this.queues.get(lane)!.push(waiter); + this.pump(); + }); + } + + /** Start every waiter that may run now, highest lane first; otherwise arm a timer for the next slot. */ + private pump(): void { + const now = this.clock.now(); + let next = Infinity; + for (const lane of LANES) { + const queue = this.queues.get(lane)!; + for (let i = 0; i < queue.length;) { + const waiter = queue[i]; + // A host is served strictly by priority: lower lanes wait while a higher lane waits for the same host. + if (LANES.slice(0, LANES.indexOf(lane)).some(l => this.queues.get(l)!.some(w => w.host === waiter.host))) { i++; continue; } + const state = this.host(waiter.host); + this.refill(state, now); + if (state.pausedUntil > now) { next = Math.min(next, state.pausedUntil); i++; continue; } + if (state.inFlight >= this.config.concurrencyPerHost) { i++; continue; } + if (state.tokens < 1) { next = Math.min(next, now + (1 - state.tokens) / this.rate * 1000); i++; continue; } + state.tokens -= 1; + state.inFlight++; + queue.splice(i, 1); + waiter.run(); + } + } + if (next < Infinity && next < this.pumpAt) { + if (this.pumpTimer !== undefined) this.clock.clearTimeout(this.pumpTimer); + this.pumpAt = next; + this.pumpTimer = this.clock.setTimeout(() => { this.pumpTimer = undefined; this.pumpAt = Infinity; this.pump(); }, + Math.max(1, Math.ceil(next - now))); + } + } +} + +/** The process-wide client. `configureWikimedia()` (server start-up) attaches the disk cache. */ +export const wikimediaClient = new WikimediaClient({ config: configFromEnv() }); diff --git a/server/wikimedia/index.ts b/server/wikimedia/index.ts new file mode 100644 index 0000000..6b7e7d7 --- /dev/null +++ b/server/wikimedia/index.ts @@ -0,0 +1,36 @@ +/** + * Node start-up wiring for the polite Wikimedia client: the disk cache, the User-Agent warning, + * the mining prefetch pool and an ops status snapshot. See docs/night2/WIKIMEDIA.md. + */ +import { resolve } from 'node:path'; +import { WikimediaCache } from './cache.ts'; +import { wikimediaClient } from './client.ts'; +import { MintTemplatePool } from './pool.ts'; +import { warnIfNoContact } from '../wikimedia-user-agent.ts'; +import { wikipediaArticles } from '../wiki.ts'; +import { wikipediaEnrichment } from '../enrichment.ts'; + +export { wikimediaClient } from './client.ts'; + +/** Serves every mining template; shared by all miners of this node. */ +export const mintPool = new MintTemplatePool({ articles: wikipediaArticles, enrichment: wikipediaEnrichment }); + +let cache: WikimediaCache | undefined; + +/** Call once at start-up. Without it the client still works, with in-memory caching only. */ +export function configureWikimedia(options: { dataDir: string }): void { + warnIfNoContact(); + if (cache) return; + try { + cache = new WikimediaCache(resolve(options.dataDir, 'wikimedia-cache.sqlite')); + wikimediaClient.attachCache(cache); + } catch (error) { + console.warn(`[wikimedia] disk cache disabled: ${(error as Error).message}`); + } +} + +export function closeWikimedia(): void { wikimediaClient.attachCache(undefined); cache?.close(); cache = undefined; } + +export function wikimediaStatus() { + return { client: wikimediaClient.status(), cacheEntries: cache?.size() ?? null, mintPool: mintPool.status() }; +} diff --git a/server/wikimedia/pool.ts b/server/wikimedia/pool.ts new file mode 100644 index 0000000..20c23e8 --- /dev/null +++ b/server/wikimedia/pool.ts @@ -0,0 +1,88 @@ +/** + * Mining-template prefetch pool (data review §5, "Mining load"). + * + * Without it every miner costs 3+ Wikimedia requests per template, all in a burst at each new tip. + * The pool keeps a few fresh article + stats snapshots ready, and serves the SAME pair to every + * template built on the same tip (miners race for one block anyway). Wikimedia load is then about + * one pair (3 requests) per block, whatever the number of miners. + * + * Mint data stays transport-only: each pair comes from the mint-path reads (`random()` and + * `getFresh()`, byte-identical parameters, never the disk cache), and the snapshot is stamped when it + * is fetched, so `capturedAt` stays true. Pairs older than `maxAgeMs` are discarded, never served. + * The pool refills only after it is used, so an idle node makes no requests. + */ +import type { Article } from '../../shared/protocol.ts'; +import { snapshotStats, type MintStatsSnapshot, type PageEnrichment } from '../../shared/enrichment.ts'; + +export type MintPair = { article: Article; statsSnapshot: MintStatsSnapshot }; +type Entry = MintPair & { fetchedAt: number }; +type Sources = { + articles: { random(): Promise
}; + enrichment: { getFresh(pageId: number): Promise }; +}; + +export class MintTemplatePool { + private readonly sources: Sources; + private readonly size: number; + private readonly maxAgeMs: number; + private readonly now: () => number; + private readonly ready: Entry[] = []; + private assigned: { tip: string; entry: Entry } | undefined; + private filling: Promise | undefined; + private direct: Promise | undefined; + readonly counters = { served: 0, shared: 0, fetched: 0, expired: 0, failed: 0 }; + + constructor(sources: Sources, options: { size?: number; maxAgeMs?: number; now?: () => number } = {}) { + this.sources = sources; + this.size = options.size ?? 2; + this.maxAgeMs = options.maxAgeMs ?? 5 * 60_000; + this.now = options.now ?? Date.now; + if (!Number.isSafeInteger(this.size) || this.size < 0 || this.size > 16 || + !Number.isFinite(this.maxAgeMs) || this.maxAgeMs <= 0) throw new RangeError('Invalid mint pool options'); + } + + private fresh(entry: Entry): boolean { return this.now() - entry.fetchedAt < this.maxAgeMs; } + + private async fetchOne(): Promise { + const article = await this.sources.articles.random(); + const metrics = await this.sources.enrichment.getFresh(article.pageId); + this.counters.fetched++; + return { article, statsSnapshot: snapshotStats(metrics), fetchedAt: this.now() }; + } + + /** The article and snapshot for a template on `tip`. Rejects when Wikimedia is unavailable. */ + async take(tip: string): Promise { + let entry: Entry | undefined; + if (this.assigned && this.assigned.tip === tip && this.fresh(this.assigned.entry)) { + entry = this.assigned.entry; + this.counters.shared++; + } else { + while (this.ready.length && !this.fresh(this.ready[0])) { this.ready.shift(); this.counters.expired++; } + entry = this.ready.shift(); + if (!entry) { + // Cold: fetch directly, and let concurrent templates share that one fetch. + this.direct ??= this.fetchOne().finally(() => { this.direct = undefined; }); + try { entry = await this.direct; } + catch (error) { this.counters.failed++; throw error; } + if (this.assigned && this.assigned.tip === tip && this.fresh(this.assigned.entry)) entry = this.assigned.entry; + } + this.assigned = { tip, entry }; + } + this.counters.served++; + this.refill(); + return structuredClone({ article: entry.article, statsSnapshot: entry.statsSnapshot }); + } + + /** Top the pool up one pair at a time, in the background. Exposed for tests. */ + refill(): Promise { + this.filling ??= (async () => { + while (this.ready.filter(e => this.fresh(e)).length < this.size) { + try { this.ready.push(await this.fetchOne()); } + catch { this.counters.failed++; break; } // The next take() tries again; no retry loop. + } + })().finally(() => { this.filling = undefined; }); + return this.filling; + } + + status() { return { ready: this.ready.length, size: this.size, maxAgeMs: this.maxAgeMs, ...this.counters }; } +} diff --git a/tests/enrichment.test.ts b/tests/enrichment.test.ts index 72185e3..490ad94 100644 --- a/tests/enrichment.test.ts +++ b/tests/enrichment.test.ts @@ -54,7 +54,7 @@ test('fixed HTTPS sources, encoded title, hidden categories, lower bounds and pl assert.equal(init.redirect, 'error'); assert.ok(init.signal instanceof AbortSignal); assert.equal(new Headers(init.headers).get('accept'), 'application/json'); - assert.match(new Headers(init.headers).get('user-agent')!, /Wikideck/); + assert.match(new Headers(init.headers).get('user-agent')!, /WikiDeck/i); assert.equal(new Headers(init.headers).has('origin'), false); // Server-side requests need no CORS workaround. } assert.equal(result.title, title); diff --git a/tests/testnet-user-agent.test.ts b/tests/testnet-user-agent.test.ts index 818553a..01bf940 100644 --- a/tests/testnet-user-agent.test.ts +++ b/tests/testnet-user-agent.test.ts @@ -2,7 +2,7 @@ import test from 'node:test'; import assert from 'node:assert/strict'; import { WikipediaArticleSource } from '../server/wiki.ts'; import { WikipediaEnrichment } from '../server/enrichment.ts'; -import { ARTICLE_USER_AGENT, ENRICHMENT_USER_AGENT, wikimediaUserAgent } from '../server/wikimedia-user-agent.ts'; +import { PROJECT_URL, defaultUserAgent, wikimediaUserAgent } from '../server/wikimedia-user-agent.ts'; function capture() { const agents: string[] = []; @@ -12,9 +12,10 @@ function capture() { }; return { agents, fetcher }; } -async function agentsFor(env: string | undefined) { - const previous = process.env.WIKIDECK_USER_AGENT; +async function agentsFor(env: string | undefined, contact?: string) { + const previous = process.env.WIKIDECK_USER_AGENT, previousContact = process.env.WIKIDECK_OPERATOR_CONTACT; if (env === undefined) delete process.env.WIKIDECK_USER_AGENT; else process.env.WIKIDECK_USER_AGENT = env; + if (contact === undefined) delete process.env.WIKIDECK_OPERATOR_CONTACT; else process.env.WIKIDECK_OPERATOR_CONTACT = contact; try { const article = capture(), enrichment = capture(); await assert.rejects(new WikipediaArticleSource(article.fetcher).random()); @@ -22,15 +23,22 @@ async function agentsFor(env: string | undefined) { return { article: article.agents, enrichment: enrichment.agents }; } finally { if (previous === undefined) delete process.env.WIKIDECK_USER_AGENT; else process.env.WIKIDECK_USER_AGENT = previous; + if (previousContact === undefined) delete process.env.WIKIDECK_OPERATOR_CONTACT; + else process.env.WIKIDECK_OPERATOR_CONTACT = previousContact; } } -test('default Wikimedia user agents are unchanged when WIKIDECK_USER_AGENT is unset', async () => { - assert.equal(ARTICLE_USER_AGENT, 'Wikideck/0.1 (experimental Wikipedia article cards)'); - assert.equal(ENRICHMENT_USER_AGENT, 'Wikideck/0.1 (Wikipedia off-chain page enrichment)'); - const { article, enrichment } = await agentsFor(undefined); - assert.ok(article.length > 0 && article.every(a => a === ARTICLE_USER_AGENT)); - assert.ok(enrichment.length > 0 && enrichment.every(a => a === ENRICHMENT_USER_AGENT)); +test('default Wikimedia user agent names the project, the operator contact and node', async () => { + const { article, enrichment } = await agentsFor(undefined, 'ops@example.org'); + const expected = `WikiDeck/0.1.0 (+${PROJECT_URL}; ops@example.org) node/${process.versions.node}`; + assert.equal(defaultUserAgent({ WIKIDECK_OPERATOR_CONTACT: 'ops@example.org' }), expected); + assert.ok(article.length > 0 && article.every(a => a === expected)); + assert.ok(enrichment.length > 0 && enrichment.every(a => a === expected)); + assert.equal(defaultUserAgent({ PUBLIC_ORIGIN: 'https://node.example' }), + `WikiDeck/0.1.0 (+${PROJECT_URL}; https://node.example) node/${process.versions.node}`); + assert.equal(defaultUserAgent({}), `WikiDeck/0.1.0 (+${PROJECT_URL}) node/${process.versions.node}`); + // Unsafe contacts are dropped rather than breaking the header's comment syntax. + assert.equal(defaultUserAgent({ WIKIDECK_OPERATOR_CONTACT: 'a); evil(' }), defaultUserAgent({})); }); test('WIKIDECK_USER_AGENT overrides both Wikimedia clients', async () => { diff --git a/tests/wiki.test.ts b/tests/wiki.test.ts index 06e45e2..e1415fc 100755 --- a/tests/wiki.test.ts +++ b/tests/wiki.test.ts @@ -56,13 +56,16 @@ test('random article fetch rejects unsafe response types, redirects, and oversiz () => new Response('{}', { headers: { 'Content-Type': 'text/html' } }), () => new Response(null, { status: 302, headers: { Location: 'https://evil.example' } }), () => new Response(' '.repeat(512 * 1024 + 1), { headers: { 'Content-Type': 'application/json' } }), - () => { throw new Error('offline'); }, ]; for (const fail of failures) { const { fetcher, calls } = mockFetch(() => fail()); await assert.rejects(new WikipediaArticleSource(fetcher).random()); assert.equal(calls.length, 3); } + // A network failure (or throttling) stops at once instead of retrying against an unreachable upstream. + const { fetcher, calls } = mockFetch(() => { throw new Error('offline'); }); + await assert.rejects(new WikipediaArticleSource(fetcher).random(), /Wikimedia unavailable/); + assert.equal(calls.length, 1); }); test('bad page records are skipped while later valid candidates are considered', async () => { diff --git a/tests/wikimedia-client.test.ts b/tests/wikimedia-client.test.ts new file mode 100644 index 0000000..a90520e --- /dev/null +++ b/tests/wikimedia-client.test.ts @@ -0,0 +1,236 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { WikimediaClient, WikimediaUnavailable, configFromEnv, type ClientConfig } from '../server/wikimedia/client.ts'; +import { WikimediaCache, cacheKey } from '../server/wikimedia/cache.ts'; +import { warnIfNoContact } from '../server/wikimedia-user-agent.ts'; +import { FakeClock, flush, json, recorder } from './wikimedia-helpers.ts'; + +const API = 'https://en.wikipedia.org/w/api.php?action=query&format=json&pageids=1'; +const REST = 'https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/x'; + +function setup(handler: Parameters[1] = () => json({ ok: true }), config: Partial = {}, + cache?: WikimediaCache) { + const clock = new FakeClock(); + const rec = recorder(clock, handler); + const client = new WikimediaClient({ fetch: rec.fetcher, clock, config, cache, userAgent: () => 'Test/1 (+x; y)' }); + return { clock, client, ...rec }; +} + +test('token bucket: burst of 3, then 1 request per second, one in flight per host', async () => { + const { clock, client, calls } = setup(); + const jobs = Array.from({ length: 6 }, () => client.request(new URL(API), { lane: 'interactive', timeoutMs: 1000 }) + .then(r => r.response.body?.cancel())); + await clock.advance(0); + assert.equal(calls.length, 3); + await clock.advance(999); + assert.equal(calls.length, 3); + await clock.advance(1); + assert.equal(calls.length, 4); + await clock.advance(2000); + await Promise.all(jobs); + assert.deepEqual(calls.map(c => c.at - calls[0].at), [0, 0, 0, 1000, 2000, 3000]); + // Different hosts have separate buckets. + await client.request(new URL(REST), { lane: 'interactive', timeoutMs: 1000 }); + assert.equal(calls.at(-1)!.url.hostname, 'wikimedia.org'); +}); + +test('only one request is in flight per host, even with tokens to spare', async () => { + const releases: (() => void)[] = []; + const { clock, client, calls } = setup(() => new Promise(resolve => releases.push(() => resolve(json({}))))); + const a = client.request(new URL(API), { lane: 'interactive', timeoutMs: 1000 }); + const b = client.request(new URL(API), { lane: 'interactive', timeoutMs: 1000 }); + await clock.advance(0); + assert.equal(calls.length, 1); + releases.shift()!(); + await a; + await clock.advance(0); + assert.equal(calls.length, 2); + releases.shift()!(); + await b; +}); + +test('priority lanes: mint, then interactive, then background', async () => { + const { clock, client, calls } = setup(url => json({ lane: url.searchParams.get('n') }), { burst: 1 }); + await client.request(new URL(`${API}&n=warm`), { lane: 'background', timeoutMs: 1000 }); // Uses the only token. + const order: string[] = []; + const go = (lane: 'mint' | 'interactive' | 'background', n: string) => + client.request(new URL(`${API}&n=${n}`), { lane, timeoutMs: 1000 }).then(() => order.push(n)); + const jobs = [go('background', 'b1'), go('interactive', 'i1'), go('background', 'b2'), go('mint', 'm1'), go('interactive', 'i2')]; + await clock.advance(5000); + await Promise.all(jobs); + assert.deepEqual(order, ['m1', 'i1', 'i2', 'b1', 'b2']); + assert.equal(calls.length, 6); +}); + +test('maxlag=5 is added to Action API requests only, and a maxlag reply waits for Retry-After', async () => { + let lagged = 1; + const { clock, client, calls } = setup(url => url.pathname === '/w/api.php' && lagged-- > 0 + ? json({ error: { code: 'maxlag' } }, { 'MediaWiki-API-Error': 'maxlag', 'Retry-After': '5' }) + : json({ ok: true })); + const job = client.request(new URL(API), { lane: 'background', timeoutMs: 1000 }); + await clock.advance(0); + assert.equal(calls.length, 1); + assert.equal(calls[0].url.searchParams.get('maxlag'), '5'); + await clock.advance(4999); + assert.equal(calls.length, 1); + await clock.advance(1); + const { response } = await job; + assert.deepEqual(await response.json(), { ok: true }); + assert.equal(calls.length, 2); + await client.request(new URL(REST), { lane: 'background', timeoutMs: 1000 }); + assert.equal(calls[2].url.searchParams.has('maxlag'), false); + assert.equal(calls[0].headers.get('user-agent'), 'Test/1 (+x; y)'); +}); + +test('429 honours Retry-After by pausing the host; three in a row open the circuit for 10 minutes', async () => { + let status = 429; + const { clock, client, calls } = setup(() => status === 200 ? json({}) : new Response('slow down', + { status, headers: { 'Retry-After': '2' } })); + const req = (lane: 'mint' | 'background' = 'background') => client.request(new URL(API), { lane, timeoutMs: 1000 }); + await assert.rejects(req(), (e: unknown) => e instanceof WikimediaUnavailable && e.status === 429); + // The next request waits out Retry-After (2 s) instead of going straight back. + const second = req().catch(e => e); + await clock.advance(1999); + assert.equal(calls.length, 1); + await clock.advance(1); + assert.equal(((await second) as WikimediaUnavailable).status, 429); + const third = req().catch(e => e); + await clock.advance(10_000); + assert.equal(((await third) as WikimediaUnavailable).status, 429); + assert.equal(calls.length, 3); + assert.equal(client.status().hosts['en.wikipedia.org'].circuitOpen, true); + // Open circuit: refused at once without touching Wikimedia, even on the mint lane. + status = 200; + await assert.rejects(req('mint'), /circuit open/); + assert.equal(calls.length, 3); + await clock.advance(10 * 60_000); + await req('mint'); + assert.equal(calls.length, 4); + assert.equal(client.status().counters['breaker.open'], 1); +}); + +test('a success resets the consecutive-429 count', async () => { + const replies = [429, 200, 429, 200, 429]; + const { clock, client } = setup(() => { + const status = replies.shift()!; + return status === 200 ? json({}) : new Response('', { status }); + }); + for (let i = 0; i < 5; i++) { + const job = client.request(new URL(API), { lane: 'background', timeoutMs: 1000 }).catch(() => {}); + await clock.advance(60_000); + await job; + } + assert.equal(client.status().hosts['en.wikipedia.org'].circuitOpen, false); +}); + +test('daily cap: background stops at 80 %, interactive at 100 %, mint keeps a 10 % reserve; resets at UTC midnight', async () => { + const { clock, client } = setup(() => json({}), { dailyCap: 10, requestsPerSecond: 1000, burst: 1000 }); + const req = (lane: 'mint' | 'interactive' | 'background') => client.request(new URL(API), { lane, timeoutMs: 1000 }); + for (let i = 0; i < 8; i++) await req('background'); + await assert.rejects(req('background'), /daily cap/); + await req('interactive'); await req('interactive'); + await assert.rejects(req('interactive'), /daily cap/); + await req('mint'); + await assert.rejects(req('mint'), /daily cap/); + await clock.advance(24 * 3600_000); + await req('background'); +}); + +test('nodes sharing one egress IP split the rate and the daily cap', () => { + const config = configFromEnv({ WIKIDECK_WIKIMEDIA_RPS: '0.9', WIKIDECK_WIKIMEDIA_NODES_PER_IP: '3', + WIKIDECK_WIKIMEDIA_DAILY_CAP: '3000' }); + const client = new WikimediaClient({ fetch: async () => json({}), config }); + assert.ok(Math.abs(client.status().requestsPerSecond - 0.3) < 1e-9); + assert.equal(client.status().dailyCap, 1000); + assert.equal(configFromEnv({ WIKIDECK_WIKIMEDIA_RPS: 'abc' }).requestsPerSecond, 1); +}); + +test('a full queue sheds background work first and a mint request evicts a background waiter', async () => { + const { clock, client } = setup(() => json({}), { queueLimit: 2, burst: 1 }); + await client.request(new URL(API), { lane: 'interactive', timeoutMs: 1000 }); + const b1 = client.request(new URL(API), { lane: 'background', timeoutMs: 1000 }).then(() => 'ok', e => e.message); + const b2 = client.request(new URL(API), { lane: 'background', timeoutMs: 1000 }).then(() => 'ok', e => e.message); + await assert.rejects(client.request(new URL(API), { lane: 'background', timeoutMs: 1000 }), /queue full/); + const mint = client.request(new URL(API), { lane: 'mint', timeoutMs: 1000 }); + await clock.advance(5000); + await mint; + assert.deepEqual([await b1, await b2], ['ok', 'Wikimedia unavailable: queue full']); +}); + +test('waiting longer than the lane allows fails instead of queueing forever', async () => { + const { clock, client } = setup(() => json({}), { burst: 1, requestsPerSecond: 0.01, maxWaitMs: { interactive: 3000 } as ClientConfig['maxWaitMs'] }); + await client.request(new URL(API), { lane: 'interactive', timeoutMs: 1000 }); + const late = client.request(new URL(API), { lane: 'interactive', timeoutMs: 1000 }).catch(e => e); + await clock.advance(3000); + assert.match(String(await late), /queue wait/); +}); + +test('disk cache: ETag revalidation with If-None-Match, 304 refresh, stale-while-revalidate on the background lane', async () => { + const cache = new WikimediaCache(':memory:'); + let body = { v: 1 }; + const { clock, client, calls } = setup((_url, headers) => headers.get('if-none-match') === '"v1"' && body.v === 1 + ? new Response(null, { status: 304 }) : json(body, { ETag: `"v${body.v}"` }), {}, cache); + const policy = { ttlMs: 1000, staleWhileRevalidate: false }; + const read = async (lane: 'interactive' | 'background' = 'interactive', cachePolicy = policy) => + (await (await client.request(new URL(API), { lane, timeoutMs: 1000, cache: cachePolicy })).response.json()); + assert.deepEqual(await read(), { v: 1 }); + assert.deepEqual(await read(), { v: 1 }); + assert.equal(calls.length, 1); // Fresh hit. + await clock.advance(1000); + assert.deepEqual(await read(), { v: 1 }); + assert.equal(calls.length, 2); + assert.equal(calls[1].headers.get('if-none-match'), '"v1"'); // Conditional request, answered 304. + assert.equal(client.status().counters['cache.revalidated'], 1); + // Stale-while-revalidate: the stale copy is served at once; the refresh happens on the background lane. + await clock.advance(1000); + body = { v: 2 }; + assert.deepEqual(await read('interactive', { ttlMs: 1000, staleWhileRevalidate: true }), { v: 1 }); + await clock.advance(0); await flush(); + assert.equal(calls.length, 3); + assert.equal(client.status().counters['lane.background'], 1); + assert.deepEqual(await read(), { v: 2 }); + assert.equal(calls.length, 3); + // The key ignores maxlag and parameter order. + assert.equal(cacheKey(new URL('https://h.org/p?b=2&a=1&maxlag=5')), cacheKey(new URL('https://h.org/p?a=1&b=2'))); +}); + +test('requests without a cache policy (mint) neither read nor write the cache', async () => { + const cache = new WikimediaCache(':memory:'); + const { client, calls } = setup(() => json({ n: calls.length }), {}, cache); + await client.request(new URL(API), { lane: 'interactive', timeoutMs: 1000, cache: { ttlMs: 60_000 } }); + const minted = await client.request(new URL(API), { lane: 'mint', timeoutMs: 1000 }); + assert.equal(minted.fromCache, false); + assert.equal(calls.length, 2); + assert.equal(cache.size(), 1); +}); + +test('a stale copy is served when the upstream refuses, and non-JSON replies are never cached', async () => { + const cache = new WikimediaCache(':memory:'); + let fail = false, html = false; + const { clock, client } = setup(() => fail ? new Response('', { status: 429 }) + : html ? new Response('', { headers: { 'Content-Type': 'text/html' } }) : json({ v: 1 }), {}, cache); + const policy = { ttlMs: 1000 }; + await client.request(new URL(API), { lane: 'interactive', timeoutMs: 1000, cache: policy }); + await clock.advance(2000); + fail = true; + const stale = await client.request(new URL(API), { lane: 'interactive', timeoutMs: 1000, cache: policy }); + assert.equal(stale.fromCache, true); + fail = false; html = true; + await clock.advance(60_000); + await client.request(new URL(`${API}&other=1`), { lane: 'interactive', timeoutMs: 1000, cache: policy }); + assert.equal(cache.size(), 1); +}); + +test('start-up warns when there is no operator contact', () => { + const messages: string[] = []; + assert.equal(warnIfNoContact({}, m => messages.push(m)), true); + assert.match(messages[0], /WIKIDECK_OPERATOR_CONTACT/); + assert.equal(warnIfNoContact({ WIKIDECK_OPERATOR_CONTACT: 'ops@example.org' }, m => messages.push(m)), false); + assert.equal(warnIfNoContact({ WIKIDECK_USER_AGENT: 'X/1 (+https://x; y)' }, m => messages.push(m)), false); + assert.equal(messages.length, 1); +}); + +test('only HTTPS is allowed', async () => { + const { client } = setup(); + await assert.rejects(client.request(new URL('http://en.wikipedia.org/w/api.php'), { lane: 'mint', timeoutMs: 1 }), RangeError); +}); diff --git a/tests/wikimedia-helpers.ts b/tests/wikimedia-helpers.ts new file mode 100644 index 0000000..cec839e --- /dev/null +++ b/tests/wikimedia-helpers.ts @@ -0,0 +1,52 @@ +// Shared fakes for tests/wikimedia-*.test.ts: a manual clock and a recording fetch. Zero network. +import type { Clock } from '../server/wikimedia/client.ts'; + +export class FakeClock implements Clock { + time: number; + private timers: { at: number; fn: () => void; id: number }[] = []; + private nextId = 1; + constructor(start = Date.parse('2026-11-17T12:00:00Z')) { this.time = start; } + now(): number { return this.time; } + setTimeout(fn: () => void, ms: number): unknown { + const id = this.nextId++; + this.timers.push({ at: this.time + Math.max(0, ms), fn, id }); + return id; + } + clearTimeout(handle: unknown): void { this.timers = this.timers.filter(t => t.id !== handle); } + /** Advance time, firing due timers in order and letting promise chains settle between them. */ + async advance(ms: number): Promise { + const target = this.time + ms; + await flush(); + for (;;) { + this.timers.sort((a, b) => a.at - b.at || a.id - b.id); + const next = this.timers[0]; + if (!next || next.at > target) break; + this.timers.shift(); + this.time = Math.max(this.time, next.at); + next.fn(); + await flush(); + } + this.time = target; + await flush(); + } +} + +export async function flush(): Promise { + for (let i = 0; i < 10; i++) await new Promise(resolve => setImmediate(resolve)); +} + +export function json(value: unknown, headers: Record = {}, status = 200): Response { + return new Response(JSON.stringify(value), { status, headers: { 'Content-Type': 'application/json; charset=utf-8', ...headers } }); +} + +export type Call = { url: URL; headers: Headers; at: number }; +export function recorder(clock: { now(): number }, handler: (url: URL, headers: Headers) => Response | Promise) { + const calls: Call[] = []; + const fetcher: typeof fetch = async (input, init) => { + const url = new URL(String(input)); + const headers = new Headers(init?.headers); + calls.push({ url, headers, at: clock.now() }); + return handler(url, headers); + }; + return { calls, fetcher }; +} diff --git a/tests/wikimedia-mint.test.ts b/tests/wikimedia-mint.test.ts new file mode 100644 index 0000000..30444b3 --- /dev/null +++ b/tests/wikimedia-mint.test.ts @@ -0,0 +1,104 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { WikipediaEnrichment } from '../server/enrichment.ts'; +import { WikipediaArticleSource, randomArticlesUrl } from '../server/wiki.ts'; +import { WikimediaClient } from '../server/wikimedia/client.ts'; +import { WikimediaCache } from '../server/wikimedia/cache.ts'; +import { UNTHROTTLED } from '../server/wikimedia/client.ts'; +import { json } from './wikimedia-helpers.ts'; + +const JANUARY = new Date('2025-01-15T23:59:59Z'); +const MONTHS = ['2024-07', '2024-08', '2024-09', '2024-10', '2024-11', '2024-12']; +const metadata = (pageId = 736) => ({ query: { pages: [{ pageid: pageId, ns: 0, title: 'Albert Einstein', lastrevid: 1234, + length: 100_000, pageprops: { wikibase_item: 'Q937' }, categories: [], langlinks: [{ lang: 'de' }] }] } }); +const views = (counts = [10, 20, 30, 40, 50, 100]) => ({ items: MONTHS.map((month, i) => ({ timestamp: `${month.replace('-', '')}0100`, + views: counts[i], project: 'en.wikipedia', access: 'all-access', agent: 'user', granularity: 'monthly' })) }); + +function service(viewsReply: () => Response | Promise, cache?: WikimediaCache) { + const urls: URL[] = []; + const fetcher: typeof fetch = async input => { + const url = new URL(String(input)); + urls.push(url); + return url.hostname === 'en.wikipedia.org' ? json(metadata(Number(url.searchParams.get('pageids')))) : viewsReply(); + }; + const client = new WikimediaClient({ fetch: fetcher, config: UNTHROTTLED, cache }); + return { urls, client, enrichment: new WikipediaEnrichment({ client, now: () => JANUARY }) }; +} + +test('mint fix: a 429, 5xx, timeout or network failure on pageviews fails the mint read instead of pinning attention: null', async () => { + const failures: (() => Response | Promise)[] = [ + () => new Response('', { status: 429, headers: { 'Retry-After': '0' } }), + () => new Response('', { status: 503 }), + () => new Response('', { status: 500 }), + () => new Response('busy', { headers: { 'Content-Type': 'text/html' } }), + () => { throw new TypeError('fetch failed'); }, + () => Promise.reject(new DOMException('The operation timed out.', 'TimeoutError')), + ]; + for (const failure of failures) { + const { enrichment } = service(failure); + await assert.rejects(enrichment.getFresh(736), /unavailable/); + // Display reads keep their old behaviour: metadata with unavailable views. + const shown = await enrichment.get(736); + assert.equal(shown.pageviews.status, 'unavailable'); + assert.equal(shown.stats.attention, null); + } +}); + +test('mint records unavailable views only for a genuine 404 or missing data', async () => { + for (const reply of [() => json({ type: 'not found' }, {}, 404), () => json({ items: [] }), () => json({ items: views().items.slice(1) })]) { + const { enrichment } = service(reply); + const minted = await enrichment.getFresh(736); + assert.equal(minted.pageviews.status, 'unavailable'); + assert.equal(minted.stats.attention, null); + } + const { enrichment } = service(() => json(views())); + assert.equal((await enrichment.getFresh(736)).stats.attention, 33); +}); + +test('mint reads never use the disk cache or join a display read', async () => { + const cache = new WikimediaCache(':memory:'); + let count = 100; + const { enrichment, urls } = service(() => json(views([1, 1, 1, 1, 1, count++])), cache); + const shown = await enrichment.get(736); + assert.equal(cache.size(), 2); + const [display, minted] = await Promise.all([enrichment.get(736), enrichment.getFresh(736)]); + assert.equal(display.pageviews.months[5].views, shown.pageviews.months[5].views); + assert.equal(minted.pageviews.months[5].views, 101); // Fetched again, not the cached 100. + assert.equal(urls.length, 4); + // A later display read (memory cache cleared by a new service) is served from disk: no upstream call. + const second = new WikipediaEnrichment({ client: new WikimediaClient({ fetch: async () => { throw new Error('no network'); }, + config: UNTHROTTLED, cache }), now: () => JANUARY }); + assert.equal((await second.get(736)).pageviews.months[5].views, 100); +}); + +test('golden mint URLs: request parameters are byte-identical apart from the added maxlag', async () => { + assert.equal(randomArticlesUrl().toString(), 'https://en.wikipedia.org/w/api.php?action=query&format=json&formatversion=2' + + '&generator=random&grnnamespace=0&grnlimit=10&prop=extracts%7Cpageimages%7Cinfo%7Ccategories&exintro=1&explaintext=1' + + '&exchars=900&piprop=thumbnail&pithumbsize=800&cllimit=20'); + const urls: string[] = []; + const fetcher: typeof fetch = async input => { + const url = new URL(String(input)); + urls.push(url.toString()); + if (url.searchParams.get('generator') === 'random') return json({ query: { pages: [{ pageid: 9, ns: 0, title: 'T', + extract: 'An extract that is comfortably longer than forty characters.', lastrevid: 2, + thumbnail: { source: 'https://upload.wikimedia.org/a.jpg' } }] } }); + return url.hostname === 'en.wikipedia.org' ? json(metadata(9)) : json(views()); + }; + const client = new WikimediaClient({ fetch: fetcher, config: UNTHROTTLED }); + const article = await new WikipediaArticleSource(client).random(); + await new WikipediaEnrichment({ client, now: () => JANUARY }).getFresh(article.pageId); + assert.deepEqual(urls, [ + `${randomArticlesUrl()}&maxlag=5`, + 'https://en.wikipedia.org/w/api.php?action=query&format=json&formatversion=2&pageids=9' + + '&prop=info%7Ccategories%7Cpageprops%7Cdescription%7Clanglinks&clshow=%21hidden&cllimit=50&lllimit=500' + + '&ppprop=wikibase_item&maxlag=5', + 'https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia.org/all-access/user/Albert_Einstein/monthly/2024070100/2024120100', + ]); +}); + +test('a throttled upstream stops random() at once instead of retrying three times', async () => { + let calls = 0; + const source = new WikipediaArticleSource(async () => { calls++; return new Response('', { status: 429 }); }); + await assert.rejects(source.random(), /HTTP 429/); + assert.equal(calls, 1); +}); diff --git a/tests/wikimedia-pool.test.ts b/tests/wikimedia-pool.test.ts new file mode 100644 index 0000000..41b3be8 --- /dev/null +++ b/tests/wikimedia-pool.test.ts @@ -0,0 +1,74 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { MintTemplatePool } from '../server/wikimedia/pool.ts'; +import type { Article } from '../shared/protocol.ts'; +import type { PageEnrichment } from '../shared/enrichment.ts'; + +function article(pageId: number): Article { + return { pageId, title: `Page ${pageId}`, extract: 'An extract that is comfortably longer than forty characters.', + image: 'https://upload.wikimedia.org/a.jpg', url: `https://en.wikipedia.org/?curid=${pageId}`, revisionId: pageId + 1, + source: 'live', category: 'Culture' }; +} +function enrichment(pageId: number): PageEnrichment { + return { pageId, title: `Page ${pageId}`, description: null, wikidataId: null, revisionId: pageId + 1, pageBytes: 10, + languageCount: 1, languagesTruncated: false, categories: [], categoriesTruncated: false, fetchedAt: '2026-11-17T00:00:00Z', + pageviews: { status: 'available', months: [], latestMonth: '2026-10', total: 1, dailyAverage: 1, trendPercent: null }, + stats: { attention: 1, depth: 1, reach: 1 } }; +} + +function fixture(options: { size?: number; maxAgeMs?: number } = {}) { + let next = 1, now = 0, fail = false; + const log: string[] = []; + const sources = { + articles: { random: async () => { log.push('random'); if (fail) throw new Error('Wikimedia unavailable: HTTP 429'); return article(next++); } }, + enrichment: { getFresh: async (id: number) => { log.push(`fresh:${id}`); return enrichment(id); } }, + }; + const pool = new MintTemplatePool(sources, { now: () => now, ...options }); + return { pool, log, advance: (ms: number) => { now += ms; }, setFail: (value: boolean) => { fail = value; } }; +} + +test('every template on one tip shares one article; a new tip gets a different one', async () => { + const { pool, log } = fixture({ size: 1 }); + const first = await Promise.all(Array.from({ length: 20 }, () => pool.take('tip-a'))); + assert.ok(first.every(pair => pair.article.pageId === first[0].article.pageId)); + await pool.refill(); + const requestsForTwentyMiners = log.length; + assert.equal(requestsForTwentyMiners, 4); // One direct pair, one prefetched pair: 2 calls each. + const second = await pool.take('tip-b'); + assert.notEqual(second.article.pageId, first[0].article.pageId); + assert.equal(second.article.pageId, 2); // Served from the prefetched pair, no wait. + await pool.refill(); + assert.equal(log.length, 6); // ...and that use alone triggers the next prefetch. + // The stats snapshot comes from a mint-path read of the same page. + assert.equal(second.statsSnapshot.revisionId, second.article.pageId + 1); +}); + +test('pairs older than maxAge are dropped, never served', async () => { + const { pool, advance } = fixture({ size: 1, maxAgeMs: 1000 }); + const a = await pool.take('t1'); + await pool.refill(); + advance(1000); + const b = await pool.take('t1'); // Same tip, but its pair expired; the ready pair expired too. + assert.notEqual(b.article.pageId, a.article.pageId); + assert.equal(pool.status().expired, 1); +}); + +test('an idle pool makes no requests, and a failure rejects the template without a retry loop', async () => { + const { pool, log, setFail } = fixture({ size: 2 }); + assert.equal(log.length, 0); + setFail(true); + await assert.rejects(pool.take('t1'), /429/); + await pool.refill(); + assert.equal(log.filter(entry => entry === 'random').length, 2); // The direct attempt and one refill attempt. + setFail(false); + assert.ok((await pool.take('t1')).article.pageId > 0); +}); + +test('callers get copies they cannot mutate', async () => { + const { pool } = fixture(); + const a = await pool.take('t'); + a.article.title = 'changed'; + assert.equal((await pool.take('t')).article.title.startsWith('Page'), true); + assert.throws(() => new MintTemplatePool({ articles: { random: async () => article(1) }, + enrichment: { getFresh: async () => enrichment(1) } }, { size: 99 }), RangeError); +}); -- 2.54.0 From 2ad4d022aa83a069b49a421e6a35f5358af0331f Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 17 Nov 2026 22:20:21 -0500 Subject: [PATCH 2/2] docs: note the deliberate 1 req/s REST rate (review on #73) Co-Authored-By: Claude Opus 5.5 --- docs/night2/WIKIMEDIA.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/night2/WIKIMEDIA.md b/docs/night2/WIKIMEDIA.md index 241f24c..3979092 100644 --- a/docs/night2/WIKIMEDIA.md +++ b/docs/night2/WIKIMEDIA.md @@ -18,6 +18,10 @@ Every server-side Wikimedia request goes through one process-wide `WikimediaClie | Disk cache | `$WIKIDECK_DATA_DIR/wikimedia-cache.sqlite`: ETag / Last-Modified, `If-None-Match`, stale-while-revalidate on the background lane, stale copy served on refusal | | User-Agent | `WikiDeck/ (+https://gitea.eom.dev/WikiDeck/wikideck; ) node/` | +The rate is deliberately 1 req/s for **every** host, including the `wikimedia.org` REST (pageviews) host. +The engineer review suggested 5 req/s there; we are stricter on purpose after the Night 1 rate limiting, because the +budget is per egress IP. Raise it only with a reason and a per-host setting. + Cache lifetimes: article metadata 24 h, monthly pageviews 7 days (the URL names six closed months, which do not change). Random articles and all mint reads are never cached. -- 2.54.0