Compare commits

..
Author SHA1 Message Date
claudeandClaude Opus 5.5 2ad4d022aa docs: note the deliberate 1 req/s REST rate (review on #73)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 22:20:21 -05:00
claudeandClaude Opus 5.5 f544019bfe Polite Wikimedia client, mint-path fix, disk cache and mining prefetch pool (#62)
- server/wikimedia/client.ts: one process-wide client. Per-host token bucket
  (1 req/s, burst 3), 1 in flight per host, mint > interactive > background
  lanes with a bounded queue, maxlag=5 on Action API requests, Retry-After
  and backoff, a circuit breaker after 3 consecutive 429s, a daily cap, and
  a per-egress-IP budget (WIKIDECK_WIKIMEDIA_NODES_PER_IP).
- server/wikimedia/cache.ts: node:sqlite HTTP cache with ETag/If-None-Match,
  stale-while-revalidate on the background lane. Mint reads never use it.
- Mint fix: a 429/5xx/timeout/non-JSON reply on pageviews now fails getFresh()
  (template 503, miner retries) instead of minting attention: null forever.
- server/wikimedia/pool.ts: per-tip article+snapshot prefetch pool for
  /api/mining/template, about 3 requests per block regardless of miner count.
- User-Agent with an operator contact (WIKIDECK_OPERATOR_CONTACT or
  PUBLIC_ORIGIN) and a start-up warning when it is unset.
- Mint request parameters are byte-identical apart from maxlag (golden test).
  No consensus, block or protocol changes. Tests use fake fetch only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 21:51:20 -05:00
22 changed files with 1340 additions and 672 deletions
-67
View File
@@ -1,67 +0,0 @@
# Night 2 private testnet: how to test (testnet-ops, #63)
**Testnet only, LAN only, on poweredge-r350.** It is never mainnet and never the live `wikideck` namespace.
## Nodes
| node | URL | game modes (env from Night 1 #22) |
|---|---|---|
| node-a | http://192.168.1.32:3101 | classic, trending |
| node-b | http://192.168.1.32:3102 | draft |
| node-c | http://192.168.1.32:3103 | classic, draft, trending |
- **Autodeploy:** the nodes follow `night/2026-11-17`. A loop on r350 (`deploy/night/autodeploy.sh`, in a dedicated detached worktree) polls the branch every 3 min.
- On a new head it rebuilds the image and restarts node-a, then b, then c, each after the previous one passes `/healthz`.
- While any match or lobby is live it defers, for up to 15 min.
- Lobbies are in memory, so a deploy that does go through ends games in progress. After night-lead merges, wait about 5 min, or check that all three nodes report the new head, before you start an end-to-end run.
- Ledgers persist in the Docker volumes across deploys.
## Test wallets
Four disposable wallets: **alice**, **bob**, **carol** and **dave**.
| wallet | address | cards (confirmed) |
|---|---|---|
| alice | `ff9273368c225698d8ae34430800cfe30b10a1c3d8c21091d42960493db556df` | see #63 |
| bob | `80f17f10ca782efd16783d10d71ce43b2993aadccda36dbd0364dee1921cbc22` | see #63 |
| carol | `f2d908633c4313bd77e86650c13d11ac3e7e656223ebe881cc34ffba3f3e0524` | leftovers |
| dave | `02bb97f202e1616ab97b45e8938e384dcbf622a0b44a7f4c1589f44236d3ccef` | leftovers |
- Confirmed at height 91: **alice 41, bob 40**, carol 5, dave 5. Block #92 came from the Android emulator (#59).
- alice and bob were consolidated to 40 cards each with on-chain NFT transfers from carol and dave, so **production `deckSize` 40 works on the testnet** and no lower-deckSize test mode is needed.
- **The encrypted browser backups are in a comment on #63**, not in git.
- Backup password: `wikideck-night-testnet`. It is **public and testnet-only**: never reuse it, and never put a real secret next to it.
- Anyone who can read this repo's issues can open these wallets. Keep them on the LAN-only testnet, and never send them anything of value.
- Re-check counts at any time: `GET http://192.168.1.32:3101/api/decks/<address>?offset=0&limit=24` (pages of 24, follow `next`).
### Load a wallet in the browser
**First make the node origins a secure context** (found by night-review on #63).
- `http://192.168.1.32:310x` is plain HTTP on a LAN address, so the browser leaves `crypto.subtle` undefined. Open wallet then fails with "Cannot read properties of undefined (reading 'digest')" or "Wallet storage unavailable".
- **Chrome/Chromium:** open `chrome://flags/#unsafely-treat-insecure-origin-as-secure`, add `http://192.168.1.32:3101,http://192.168.1.32:3102,http://192.168.1.32:3103`, then relaunch.
- **Command line:** `--unsafely-treat-insecure-origin-as-secure=http://192.168.1.32:3101,http://192.168.1.32:3102,http://192.168.1.32:3103 --user-data-dir=<fresh profile dir>`.
- **Playwright:** use the full `chromium` channel. The headless shell ignores this flag.
- **SSH tunnel to `localhost`:** this also works for the wallet, but it breaks the Dex callback URLs, so you can't do the login/link step that way.
- **Firefox** has no per-origin equivalent.
Use a separate browser profile per player, because a browser holds one wallet per origin.
1. Open a node URL, then **Open wallet**.
2. **Recover from a local backup**: choose the backup JSON file (save it from the #63 comment as `alice.backup.json`), and enter the backup password above.
3. Set a new local encryption password (at least 12 characters, anything testnet-only), then confirm it.
4. The wallet dialog should show the address above and its confirmed card count.
### eom.dev login (Dex)
- There is no new Dex config tonight. The existing `wikideck` client already allows the 3101–3103 callbacks.
- Log in with **Log in with eom.dev**, then unlock the wallet and press **Link wallet …** to link it to the account.
- PvP seats are keyed to the account, and ownership is checked on the server against the account's linked wallets. So for a two-player run you need **two eom.dev accounts**, one with alice linked and one with bob. Which accounts to use is night-lead's call (see #53/#63). Never use a real person's account without their OK.
## Mining on the testnet
- Mining isn't needed for tests. Only run a miner if night-lead asks for more blocks, or to confirm transfers.
- `deploy/night/testnet.sh mine N [workers]` runs the native multi-worker miner in a container named `wikideck-night-miner` (at most 4 workers on r350).
- Every template costs Wikipedia requests on the node, so it is paced by `MIN_GAP`.
- **testnet-ops stops any miner by 01:55** (`docker rm -f wikideck-night-miner`). The nodes stay up.
- **Difficulty (for mobile mining, #59):** at block ~90, an accepted block took between about 2 K and 130 K hashes. That is up to ~8 min at the 4-worker r350 rate of ~250 H/s. One light-mode thread on the emulator is far slower, so expect a long wait per block, with high variance.
## Card identity pre-bake (engineer WS E)
- `server/game/identity.ts` is a persistent per-token store (`node:sqlite`). It freezes each card's off-chain identity at first sight and never rewrites it. The identity covers P31/P279 classes, description, categories, countries, years, coordinates and outlink pageIds. Its fields match duel-cards' `CardProfile` (#70), and `toProfile()` returns that view.
- `scripts/night2/prebake-identity.ts --node <url> [--store <file>] [--cards <cached.json> ...]` reads the canonical chain from one node and freezes what the cached card files know. The store defaults to `$WIKIDECK_DATA_DIR/card-identity.sqlite` (`identityStoreFile()`), and hosts should open it at the same path.
- It **never calls Wikimedia itself**.
- Live facts arrive only through the wikimedia team's client on its `background` lane (#62), once that client has merged.
- Cards that are still unknown stay unfrozen and are retried on a later run.
+72
View File
@@ -0,0 +1,72 @@
# Polite Wikimedia client (Night 2, issue #62)
Every server-side Wikimedia request goes through one process-wide `WikimediaClient`
(`server/wikimedia/client.ts`). The design comes from engineer review §4 and data review §5.
## What it does
| Behaviour | Default |
|---|---|
| Rate per upstream host (`en.wikipedia.org`, `wikimedia.org`, `www.wikidata.org`) | token bucket, 1 req/s, burst 3 |
| Concurrency | 1 request in flight per host |
| Priority lanes | `mint` → `interactive` → `background`; the queue holds 64 waiters, and background work is shed first |
| Longest queue wait | mint 10 s, interactive 5 s, background 120 s |
| `maxlag` | `maxlag=5` on every Action API request; a maxlag reply waits for `Retry-After` and retries up to 2 times |
| 429 / 503 | honour `Retry-After` (seconds or HTTP date), else back off 2ⁿ s with jitter, capped at 10 min, by pausing the whole host |
| Circuit breaker | 3 consecutive 429s stop that host for 10 min. Requests are refused at once without touching Wikimedia |
| Daily cap | 20,000 requests a day per egress IP. Background work stops at 80 %, interactive at 100 %; mint has a 10 % reserve |
| Disk cache | `$WIKIDECK_DATA_DIR/wikimedia-cache.sqlite`: ETag / Last-Modified, `If-None-Match`, stale-while-revalidate on the background lane, stale copy served on refusal |
| User-Agent | `WikiDeck/<version> (+https://gitea.eom.dev/WikiDeck/wikideck; <operator contact>) node/<version>` |
The rate is deliberately 1 req/s for **every** host, including the `wikimedia.org` REST (pageviews) host.
The engineer review suggested 5 req/s there; we are stricter on purpose after the Night 1 rate limiting, because the
budget is per egress IP. Raise it only with a reason and a per-host setting.
Cache lifetimes: article metadata 24 h, monthly pageviews 7 days (the URL names six closed
months, which do not change). Random articles and all mint reads are never cached.
## Operator settings
| Variable | Meaning |
|---|---|
| `WIKIDECK_OPERATOR_CONTACT` | **Set this.** An email address or URL, put in the User-Agent. Without it (and without `PUBLIC_ORIGIN`), the node logs a warning at start-up. Wikimedia policy requires a contact, and contact-less clients are the first to be blocked. |
| `PUBLIC_ORIGIN` | Used as the contact when `WIKIDECK_OPERATOR_CONTACT` is unset. |
| `WIKIDECK_USER_AGENT` | Overrides the whole User-Agent string (sanitized; kept from Night 1). |
| `WIKIDECK_WIKIMEDIA_RPS` | Requests per second per host for the **whole egress IP** (default 1). |
| `WIKIDECK_WIKIMEDIA_BURST` | Token-bucket burst (default 3). |
| `WIKIDECK_WIKIMEDIA_CONCURRENCY` | In-flight requests per host (default 1). |
| `WIKIDECK_WIKIMEDIA_DAILY_CAP` | Requests per UTC day for the whole egress IP (default 20000). |
| `WIKIDECK_WIKIMEDIA_NODES_PER_IP` | How many nodes share this egress IP. The rate and the daily cap are divided by it. **The testnet runs several nodes behind one IP: set it to the node count** (e.g. 3 for `deploy/night/compose.lan.yaml`). |
The testnet's `WIKIDECK_USER_AGENT` already names the project URL, which satisfies the policy.
Counters (`lane.*`, `http.429`, `breaker.open`, `cache.hit`, `refused.*`, …), per-host pause state and the
mint pool's state are in the private operations listener's `/status`, under `wikimedia`.
## Mint path: transport only
- `random()` (W1) and `getFresh()` (W2 and W3) keep their request parameters byte-identical. The only
addition is `maxlag=5`, which changes only error behaviour. `tests/wikimedia-mint.test.ts` pins the exact URLs.
- Mint reads use the `mint` lane and **never** read or write the disk cache, and never join a display read.
- **Fixed:** a 429, 5xx, timeout, network error or non-JSON reply on the pageviews call used to mint a card with
`attention: null` forever (`enrichment.ts`). Now `getFresh()` rejects, the template returns 503 and the miner
retries. `unavailable` is minted only for a genuine 404 (no data) or missing months.
- No block, transfer, trade or validation code changed. Validation never calls Wikimedia.
## Mining prefetch pool (`server/wikimedia/pool.ts`)
Each template used to cost 3+ requests, all in a burst at each new tip, so load scaled with the number of miners.
Now every template on the same tip gets the same fresh article+snapshot pair (miners race for one block anyway),
and a new tip takes the next prefetched pair. Load is about one pair, or 3 requests, per block, whatever the number
of miners.
- Pairs come from the mint-path reads above, and the snapshot is stamped when the pair is fetched, so `capturedAt` is true.
- Pairs older than 5 minutes are dropped, never served.
- The pool (2 pairs) refills only after use, so an idle node makes no requests.
## Card faces
`/api/wikipedia/:id` → `get()`: memory LRU, then the disk cache (fresh, or stale with a background
revalidation), and only then the interactive lane. A grid of cards therefore no longer means a burst
of 2 upstream requests per card per render. `TrendCache` can use `getBackground()` to stay off the
interactive lane.
-56
View File
@@ -1,56 +0,0 @@
// testnet-ops (engineer WS E): pre-bake frozen card identities for every card on a chain.
//
// node --experimental-strip-types scripts/night2/prebake-identity.ts --node http://192.168.1.32:3101 \
// --store $WIKIDECK_DATA_DIR/card-identity.sqlite [--cards docs/game/card-sample.json] [--batch 50]
//
// - Reads the canonical chain from ONE node (GET /api/summary, then /api/canonical pages), so every
// card is built from its mint block, never from client JSON.
// - Facts come ONLY from cached card files (`--cards`, repeatable). This script never calls
// Wikipedia or Wikidata itself. Live facts must go through the wikimedia team's process-wide client
// on its `background` lane (server/wikimedia, #62); that IdentitySource plugs into `prebake()` in
// server/game/identity.ts once it has merged. Until then, unknown cards are left unfrozen and are
// picked up by a later run.
// - Idempotent and freeze-at-first-sight: already frozen tokens are skipped and never rewritten.
import { readFileSync } from 'node:fs';
import { parseArgs } from 'node:util';
import { IdentityStore, cachedSource, identityStoreFile, prebake, type ChainArticle } from '../../server/game/identity.ts';
const { values } = parseArgs({ options: {
node: { type: 'string', default: 'http://192.168.1.32:3101' },
store: { type: 'string', default: identityStoreFile() },
cards: { type: 'string', multiple: true, default: ['docs/game/card-sample.json'] },
batch: { type: 'string', default: '50' },
} });
const nodeUrl = new URL(values.node!);
if (!['http:', 'https:'].includes(nodeUrl.protocol)) throw new Error('--node must be an http(s) WikiDeck node URL');
if (/wikipedia\.org|wikidata\.org|wikimedia\.org/i.test(nodeUrl.hostname)) throw new Error('--node must be a WikiDeck node, not Wikimedia');
async function get<T>(path: string): Promise<T> {
const r = await fetch(new URL(path, nodeUrl), { signal: AbortSignal.timeout(30000), redirect: 'error' });
if (!r.ok) throw new Error(`${path}: HTTP ${r.status}`);
return await r.json() as T;
}
type Block = { tokenId: string; article: { pageId: number; title: string; revisionId?: number } };
async function chainArticles(): Promise<ChainArticle[]> {
const { tip } = await get<{ tip: string }>('/api/summary');
const out: ChainArticle[] = [];
for (let offset: number | null = 0; offset !== null;) {
const page: { blocks: Block[]; next: number | null } = await get(`/api/canonical?tip=${tip}&offset=${offset}&limit=16`);
for (const b of page.blocks) if (b.article && b.tokenId)
out.push({ tokenId: b.tokenId, pageId: b.article.pageId, title: b.article.title, revisionId: b.article.revisionId ?? null });
offset = page.next;
}
return out;
}
const cards = values.cards!.flatMap(f => {
const d = JSON.parse(readFileSync(f, 'utf8'));
return Array.isArray(d) ? d : d.cards ?? [];
});
const articles = await chainArticles();
const store = new IdentityStore(values.store!);
try {
const report = await prebake(store, cachedSource(cards), articles, { batchSize: Number(values.batch) });
console.log(JSON.stringify({ node: nodeUrl.origin, chainCards: articles.length, cachedCards: cards.length, ...report, stored: store.count() }));
} finally { store.close(); }
+72 -27
View File
@@ -1,5 +1,5 @@
import { z } from 'zod';
import { ENRICHMENT_USER_AGENT, wikimediaUserAgent } from './wikimedia-user-agent.ts';
import { WikimediaClient, WikimediaUnavailable, wikimediaClient, type CachePolicy, type Lane } from './wikimedia/client.ts';
import { deriveStats, type PageEnrichment } from '../shared/enrichment.ts';
const MAX_BYTES = 1024 * 1024;
@@ -25,7 +25,23 @@ const viewsSchema = z.object({ items: z.array(z.object({
})).max(6) });
type CacheEntry = { expires: number; value: PageEnrichment } | { expires: number; error: Error };
type Options = { fetch?: typeof globalThis.fetch; now?: () => Date; ttlMs?: number; maxEntries?: number };
type Options = { fetch?: typeof globalThis.fetch; client?: WikimediaClient; now?: () => Date; ttlMs?: number; maxEntries?: number };
type Mode = 'mint' | 'display' | 'background';
/** Disk-cache lifetimes (engineer review §4). Mint reads never use the cache. */
const METADATA_CACHE: CachePolicy = { ttlMs: 24 * 3600_000, staleWhileRevalidate: true, maxBytes: MAX_BYTES };
// The URL names its six completed months, which are immutable; a week bounds late corrections.
const PAGEVIEWS_CACHE: CachePolicy = { ttlMs: 7 * 24 * 3600_000, staleWhileRevalidate: true, maxBytes: MAX_BYTES };
const NOT_FOUND = Symbol('not found');
/** The upstream did not answer properly (throttled, 5xx, timeout, network, non-JSON). Not "no data". */
export class UpstreamFailure extends Error {
constructor(message: string) { super(message); this.name = 'UpstreamFailure'; }
}
function transportFailure(error: unknown): boolean {
return error instanceof WikimediaUnavailable || error instanceof UpstreamFailure ||
(error instanceof DOMException && (error.name === 'TimeoutError' || error.name === 'AbortError'));
}
function monthWindow(now: Date): string[] {
return Array.from({ length: 6 }, (_, index) =>
@@ -34,16 +50,17 @@ function monthWindow(now: Date): string[] {
/** Fixed-source read-only enrichment; no consensus, persistence, URL or header inputs. */
export class WikipediaEnrichment {
private readonly fetcher: typeof globalThis.fetch;
private readonly client: WikimediaClient;
private readonly now: () => Date;
private readonly ttlMs: number;
private readonly maxEntries: number;
private readonly cache = new Map<number, CacheEntry>();
private readonly pending = new Map<number, Promise<PageEnrichment>>();
private readonly pending = new Map<string, Promise<PageEnrichment>>();
private active = 0;
private mintActive = 0;
constructor(options: Options = {}) {
this.fetcher = options.fetch ?? globalThis.fetch;
this.client = options.client ?? (options.fetch ? WikimediaClient.unthrottled(options.fetch) : wikimediaClient);
this.now = options.now ?? (() => new Date());
this.ttlMs = options.ttlMs ?? 6 * 60 * 60 * 1000;
this.maxEntries = options.maxEntries ?? 128;
@@ -53,16 +70,27 @@ export class WikipediaEnrichment {
}
}
/** Card faces (`/api/wikipedia/:id`): memory, then the disk cache, then the interactive lane. */
async get(pageId: number): Promise<PageEnrichment> {
return this.loadCached(pageId, false);
return this.loadCached(pageId, 'display');
}
/** Bypass cached values when the result is being pinned into a newly minted block. */
/** Like get(), on the background lane: for trends and other work nobody is waiting on. */
async getBackground(pageId: number): Promise<PageEnrichment> {
return this.loadCached(pageId, 'background');
}
/**
* Mint data: bypasses every cache and uses the mint lane. If pageviews fail in transport
* (429, 5xx, timeout, throttled), this rejects so the template fails and the miner retries;
* `unavailable` is recorded only for a genuine 404 or missing data.
*/
async getFresh(pageId: number): Promise<PageEnrichment> {
return this.loadCached(pageId, true);
return this.loadCached(pageId, 'mint');
}
private async loadCached(pageId: number, fresh: boolean): Promise<PageEnrichment> {
private async loadCached(pageId: number, mode: Mode): Promise<PageEnrichment> {
const fresh = mode === 'mint';
if (!Number.isSafeInteger(pageId) || pageId <= 0) throw new RangeError('Invalid Wikipedia page ID');
const now = this.now();
const cached = this.cache.get(pageId);
@@ -73,23 +101,29 @@ export class WikipediaEnrichment {
if ('error' in cached) throw cached.error;
return structuredClone(cached.value);
}
this.cache.delete(pageId);
const existing = this.pending.get(pageId);
if (!fresh) this.cache.delete(pageId);
// Mint jobs never join a display job: that one may be served from the disk cache.
const pendingKey = `${fresh ? 'mint' : 'cached'}:${pageId}`;
const existing = this.pending.get(pendingKey);
if (existing) return structuredClone(await existing);
// Reject immediately: no unbounded waiting queue. Four active jobs also keep
// the pending-map ceiling of eight conservative (one upstream request/job).
if (this.pending.size >= 8 || this.active >= 4) throw new Error('Wikipedia enrichment busy');
this.active++;
const job = this.load(pageId, now).then(value => {
// Mint jobs have their own small allowance so card faces can never starve a template.
if (fresh ? this.mintActive >= 2 : this.pending.size - this.mintActive >= 8 || this.active >= 4) {
throw new Error('Wikipedia enrichment busy');
}
if (fresh) this.mintActive++; else this.active++;
const job = this.load(pageId, now, mode).then(value => {
const ttl = value.pageviews.status === 'available' ? this.ttlMs : Math.min(this.ttlMs, 5 * 60 * 1000);
this.remember(pageId, { value, expires: this.now().getTime() + ttl });
return value;
}, () => {
const error = new Error('Wikipedia page enrichment unavailable');
this.remember(pageId, { error, expires: this.now().getTime() + 30_000 });
// A failed mint read says nothing new about display data; keep any cached value.
if (!fresh) this.remember(pageId, { error, expires: this.now().getTime() + 30_000 });
throw error;
}).finally(() => { this.active--; this.pending.delete(pageId); });
this.pending.set(pageId, job);
}).finally(() => { if (fresh) this.mintActive--; else this.active--; this.pending.delete(pendingKey); });
this.pending.set(pendingKey, job);
return structuredClone(await job);
}
@@ -99,16 +133,19 @@ export class WikipediaEnrichment {
while (this.cache.size > this.maxEntries) this.cache.delete(this.cache.keys().next().value!);
}
private async json(url: URL): Promise<unknown> {
const signal = AbortSignal.timeout(6000);
const response = await this.fetcher(url, { signal, redirect: 'error',
headers: { Accept: 'application/json', 'User-Agent': wikimediaUserAgent(ENRICHMENT_USER_AGENT) } });
private async json(url: URL, lane: Lane, cache: CachePolicy | undefined, allowNotFound = false): Promise<unknown> {
const { response, signal } = await this.client.request(url, { lane, timeoutMs: 6000, cache });
if (allowNotFound && response.status === 404) { await response.body?.cancel(); return NOT_FOUND; }
if (!response.ok || response.redirected) {
await response.body?.cancel();
throw new UpstreamFailure(`Upstream HTTP ${response.status}`);
}
const length = response.headers.get('content-length');
if (!response.ok || response.redirected ||
if (
!/^application\/json(?:\s*;|\s*$)/i.test(response.headers.get('content-type') ?? '') ||
(length !== null && (!/^\d+$/.test(length) || Number(length) > MAX_BYTES)) || !response.body) {
await response.body?.cancel();
throw new Error('Invalid upstream response');
throw new UpstreamFailure('Invalid upstream response');
}
const reader = response.body.getReader();
const chunks: Uint8Array[] = [];
@@ -130,12 +167,14 @@ export class WikipediaEnrichment {
}
}
private async load(pageId: number, now: Date): Promise<PageEnrichment> {
private async load(pageId: number, now: Date, mode: Mode): Promise<PageEnrichment> {
const lane: Lane = mode === 'mint' ? 'mint' : mode === 'display' ? 'interactive' : 'background';
const fresh = mode === 'mint';
const url = new URL('https://en.wikipedia.org/w/api.php');
url.search = new URLSearchParams({ action: 'query', format: 'json', formatversion: '2',
pageids: String(pageId), prop: 'info|categories|pageprops|description|langlinks',
clshow: '!hidden', cllimit: '50', lllimit: '500', ppprop: 'wikibase_item' }).toString();
const metadata = metadataSchema.parse(await this.json(url));
const metadata = metadataSchema.parse(await this.json(url, lane, fresh ? undefined : METADATA_CACHE));
const page = metadata.query.pages[0];
if (page.pageid !== pageId) throw new Error('Wikipedia page ID mismatch');
const months = monthWindow(now);
@@ -146,7 +185,9 @@ export class WikipediaEnrichment {
const title = encodeURIComponent(page.title.replace(/ /g, '_'));
const stamp = (month: string) => `${month.replace('-', '')}0100`;
const endpoint = new URL(`https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia.org/all-access/user/${title}/monthly/${stamp(months[0])}/${stamp(latestMonth)}`);
const result = viewsSchema.parse(await this.json(endpoint));
const raw = await this.json(endpoint, lane, fresh ? undefined : PAGEVIEWS_CACHE, true);
if (raw === NOT_FOUND) throw new Error('No pageview data');
const result = viewsSchema.parse(raw);
const byMonth = new Map<string, number>();
for (const item of result.items) {
const month = `${item.timestamp.slice(0, 4)}-${item.timestamp.slice(4, 6)}`;
@@ -161,7 +202,11 @@ export class WikipediaEnrichment {
const days = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), 0)).getUTCDate();
pageviews = { status: 'available', months: ordered, latestMonth, total,
dailyAverage: latest / days, trendPercent: prior > 0 ? (latest - prior) / prior * 100 : null };
} catch { /* Metadata remains useful; unknown views must not become fake zeroes. */ }
} catch (error) {
// Mint: a throttled or failed upstream must not pin `attention: null` into a block forever.
if (fresh && transportFailure(error)) throw error;
/* Metadata remains useful; unknown views must not become fake zeroes. */
}
return { pageId, title: page.title, description: page.description ?? null,
wikidataId: page.pageprops?.wikibase_item ?? null, revisionId: page.lastrevid, pageBytes: page.length,
languageCount: page.langlinks?.length ?? 0, languagesTruncated: metadata.continue?.llcontinue !== undefined,
-113
View File
@@ -1,113 +0,0 @@
// Live IdentitySource for server/game/identity.ts (engineer WS E, data §3).
//
// Every request goes through the process-wide polite Wikimedia client on the **background** lane:
// - the page enrichment (description, wikidataId, categories) via `enrichment.getBackground()`,
// which also feeds the disk cache card faces use;
// - one `wbgetentities&props=claims` per batch of at most 50 items;
// - one `wbgetentities&props=labels` per 50 class ids still missing a label.
// That is about 2 Wikidata requests per 50 new cards, plus one cached enrichment per card.
// Outlinks (prop=links, data §4) are not fetched tonight: they cost about 1.2 requests per card,
// and Connected falls back to its other tiers without them.
//
// The profile itself is built by duel-cards' pure `profileFromSources()` (shared/duel/profile.ts, #70),
// so the frozen identity and what derive() reads cannot drift apart. That module, and the client
// (server/wikimedia, #73), are injected structurally, so this file has no import from unmerged PRs.
// Wiring, once both have merged:
// wikimediaIdentitySource({ client: wikimediaClient, enrichment: wikipediaEnrichment,
// profile: await import('../../shared/duel/profile.ts') })
// Any transport failure leaves the affected cards unknown (undefined), so they are retried later,
// never frozen half-empty.
import type { PageEnrichment } from '../../shared/enrichment.ts';
import type { ChainArticle, IdentityFacts, IdentitySource, WikidataClass } from './identity.ts';
/** Structural slices of server/wikimedia/client.ts (#73) and shared/duel/profile.ts (#70). */
export type BackgroundRequest = { lane: 'background'; timeoutMs: number; cache?: { ttlMs: number; maxBytes?: number } };
export type WikidataEntity = { id?: string; claims?: Record<string, readonly {
rank?: string; mainsnak?: { snaktype?: string; datavalue?: { type?: string; value?: unknown } } }[]> };
export type ProfileLike = { wikidataId: string | null; description: string | null; categories: readonly string[];
instanceOf: readonly WikidataClass[]; subclassOf?: readonly WikidataClass[]; countries?: readonly string[];
years?: readonly number[]; died?: boolean; links?: readonly number[]; cites?: readonly string[]; fetchedRevisionId?: number };
export type ProfileBuilder = {
wikidataEntityParams(qids: readonly string[]): Record<string, string>;
wikidataLabelParams(qids: readonly string[]): Record<string, string>;
classIdsOf(entity: WikidataEntity | null | undefined): string[];
profileFromSources(src: { enrichment: PageEnrichment; entity?: WikidataEntity | null; labels?: Readonly<Record<string, string>> }): ProfileLike;
};
export type IdentitySourceDeps = {
enrichment: { getBackground(pageId: number): Promise<PageEnrichment> };
client: { request(url: URL, options: BackgroundRequest): Promise<{ response: Response }> };
profile: ProfileBuilder;
};
const WIKIDATA_API = 'https://www.wikidata.org/w/api.php';
const MAX_BYTES = 4 * 1024 * 1024;
const CACHE = { ttlMs: 7 * 24 * 60 * 60 * 1000, maxBytes: MAX_BYTES };
async function wikidata(client: IdentitySourceDeps['client'], params: Record<string, string>): Promise<Record<string, unknown>> {
const url = new URL(WIKIDATA_API);
for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
const { response } = await client.request(url, { lane: 'background', timeoutMs: 15000, cache: CACHE });
if (!response.ok) { await response.body?.cancel(); throw new Error(`Wikidata HTTP ${response.status}`); }
const text = await response.text();
if (text.length > MAX_BYTES) throw new Error('Wikidata response too large');
const body = JSON.parse(text) as { entities?: Record<string, unknown> | unknown[] };
const entities = body.entities ?? {};
// formatversion=2 returns an object keyed by id; tolerate an array too.
return Array.isArray(entities)
? Object.fromEntries(entities.filter(e => e && typeof (e as { id?: unknown }).id === 'string').map(e => [(e as { id: string }).id, e]))
: entities as Record<string, unknown>;
}
const chunks = <T>(xs: T[], n: number) => Array.from({ length: Math.ceil(xs.length / n) }, (_, i) => xs.slice(i * n, i * n + n));
export function wikimediaIdentitySource(deps: IdentitySourceDeps): IdentitySource {
const { classIdsOf, profileFromSources, wikidataEntityParams, wikidataLabelParams } = deps.profile;
return {
kind: 'wikimedia',
async lookup(articles: ChainArticle[]) {
const out = new Map<number, IdentityFacts | undefined>(articles.map(a => [a.pageId, undefined]));
// 1. Enrichment, one page at a time (the client paces it); failures stay unknown.
const enriched = new Map<number, PageEnrichment>();
for (const a of articles) {
try { enriched.set(a.pageId, await deps.enrichment.getBackground(a.pageId)); } catch { /* retry later */ }
}
// 2. Claims for every item we know, in batches of ≤ 50.
const qids = [...new Set([...enriched.values()].map(e => e.wikidataId).filter((q): q is string => !!q && /^Q\d+$/.test(q)))];
const entities = new Map<string, WikidataEntity>();
let claimsFailed = false;
for (const batch of chunks(qids, 50)) {
try { for (const [id, e] of Object.entries(await wikidata(deps.client, wikidataEntityParams(batch)))) entities.set(id, e as WikidataEntity); }
catch { claimsFailed = true; }
}
// 3. English labels for the P31/P279 classes.
const classIds = [...new Set([...entities.values()].flatMap(e => classIdsOf(e)))];
const labels: Record<string, string> = {};
let labelsFailed = false;
for (const batch of chunks(classIds, 50)) {
try {
for (const [id, e] of Object.entries(await wikidata(deps.client, wikidataLabelParams(batch)))) {
const v = (e as { labels?: { en?: { value?: unknown } } }).labels?.en?.value;
if (typeof v === 'string') labels[id] = v;
}
} catch { labelsFailed = true; }
}
// 4. Freeze only complete profiles: a card whose item couldn't be read stays unknown.
for (const a of articles) {
const e = enriched.get(a.pageId);
if (!e) continue;
const qid = e.wikidataId && /^Q\d+$/.test(e.wikidataId) ? e.wikidataId : null;
const entity = qid ? entities.get(qid) : null;
if (qid && !entity && claimsFailed) continue; // item not read: retry later
if (entity && labelsFailed) continue; // class labels not read: retry later
const p = profileFromSources({ enrichment: e, entity: entity ?? null, labels });
out.set(a.pageId, {
fetchedRevisionId: p.fetchedRevisionId ?? null, wikidataId: p.wikidataId, description: p.description,
categories: [...p.categories], instanceOf: [...p.instanceOf], subclassOf: [...(p.subclassOf ?? [])],
countries: [...(p.countries ?? [])], years: [...(p.years ?? [])], died: p.died === true,
links: [...(p.links ?? [])], cites: [...(p.cites ?? [])],
});
}
return out;
},
};
}
-204
View File
@@ -1,204 +0,0 @@
// Card identity (engineer WS E, data review §3): the off-chain, mutable facts about a card's article
// (P31/P279 classes, a few Wikidata claims, categories, outlinks) frozen ONCE per tokenId, the first time
// the card is seen, so that a later Wikipedia/Wikidata edit can never retype a card someone owns.
//
// - On-chain data (article, statsSnapshot) stays the source of truth for everything it covers.
// - Identities are never fetched at match time. They are pre-baked in the background
// (scripts/night2/prebake-identity.ts), from cached data or through an injected IdentitySource that
// must use the process-wide Wikimedia client on its `background` lane. This file performs no I/O
// besides the sqlite store and never calls fetch itself.
// - First write wins: `freeze` never overwrites an existing record (data §3 "freeze at first sight").
import { DatabaseSync } from 'node:sqlite';
import { createHash } from 'node:crypto';
import { join, resolve } from 'node:path';
export const IDENTITY_VERSION = 1;
/** Where hosts keep the store: next to the ledger, `$WIKIDECK_DATA_DIR/card-identity.sqlite` (default ./data). */
export function identityStoreFile(env: NodeJS.ProcessEnv = process.env): string {
return join(env.WIKIDECK_DATA_DIR ?? resolve('data'), 'card-identity.sqlite');
}
/** Same shape as shared/duel/classify.ts `WikidataClass` (duel-cards, #70). */
export type WikidataClass = { id: string; label: string };
/**
* A frozen card identity. Its profile fields are structurally the `CardProfile` that duel-cards'
* `derive()` reads (shared/duel/classify.ts), so `toProfile(identity)` can be passed straight in.
*/
export type CardIdentity = {
version: typeof IDENTITY_VERSION;
tokenId: string;
pageId: number;
title: string;
/** article.revisionId from the mint block. */
mintRevisionId: number | null;
/** The live revision the off-chain facts were read from (data §3; may differ from the mint revision). */
fetchedRevisionId: number | null;
wikidataId: string | null;
wikidataRev: number | null;
/** Wikipedia short description. */
description: string | null;
/** Non-hidden category titles without the "Category:" prefix, sorted. */
categories: string[];
/** P31 (instance of). */
instanceOf: WikidataClass[];
/** P279 of the item, or its P31 classes' superclasses, nearest first. */
subclassOf: WikidataClass[];
/** P17 / P27 country Q-ids, sorted. */
countries: string[];
/** Years from P569/P570/P571/P580/P585 (negative = BCE), sorted. */
years: number[];
/** P570 (date of death) present. */
died: boolean;
/** P625 or prop=coordinates, [lat, lon]. */
coords: [number, number] | null;
/** ns0 outlinks as pageIds, sorted, deduplicated (Connected, Linked tier). Empty when unknown. */
links: number[];
/** Item Q-ids cited by the frozen claims (P17, P27, P131, P361, P106, ...). */
cites: string[];
/** sha256 of `links`, so match records can log a short fingerprint. */
linksHash: string;
/** Where the facts came from: an offline cache (e.g. card-sample.json) or a live background fetch. */
source: 'cache' | 'wikimedia';
frozenAt: string;
};
/** What a source knows about an article; any field may be missing. */
export type IdentityFacts = Partial<Pick<CardIdentity, 'fetchedRevisionId' | 'wikidataId' | 'wikidataRev' | 'description'
| 'categories' | 'instanceOf' | 'subclassOf' | 'countries' | 'years' | 'died' | 'coords' | 'links' | 'cites'>>;
/** The minimum the chain tells us about a card (built from its mint block, never from client JSON). */
export type ChainArticle = { tokenId: string; pageId: number; title: string; revisionId?: number | null };
/**
* A source of identity facts. A live implementation must route every request through the shared
* Wikimedia client on the background lane (never the mint or interactive lanes) and batch where it
* can. Returning `undefined` for a page means "unknown for now": nothing is frozen and it is retried
* on a later run.
*/
export interface IdentitySource {
readonly kind: CardIdentity['source'];
lookup(articles: ChainArticle[]): Promise<Map<number, IdentityFacts | undefined>>;
}
export function linksHash(links: readonly number[]): string {
return createHash('sha256').update(JSON.stringify(links)).digest('hex');
}
const QID = /^Q\d+$/;
const uniqStrings = (xs: readonly unknown[] | undefined) => [...new Set((xs ?? []).filter((x): x is string => typeof x === 'string'))].sort();
const uniqInts = (xs: readonly unknown[] | undefined) =>
[...new Set((xs ?? []).filter((x): x is number => Number.isSafeInteger(x)))].sort((a, b) => a - b);
const classes = (xs: readonly WikidataClass[] | undefined) =>
(xs ?? []).filter(c => c && QID.test(c.id)).map(c => ({ id: c.id, label: typeof c.label === 'string' ? c.label : '' }));
/** Pure: build a frozen identity from chain data plus whatever facts a source returned. */
export function buildIdentity(article: ChainArticle, facts: IdentityFacts, source: CardIdentity['source'], now: Date): CardIdentity {
const links = uniqInts(facts.links);
const coords = facts.coords && facts.coords.length === 2 && facts.coords.every(Number.isFinite) ? [facts.coords[0], facts.coords[1]] as [number, number] : null;
return {
version: IDENTITY_VERSION,
tokenId: article.tokenId,
pageId: article.pageId,
title: article.title,
mintRevisionId: article.revisionId ?? null,
fetchedRevisionId: facts.fetchedRevisionId ?? null,
wikidataId: facts.wikidataId && QID.test(facts.wikidataId) ? facts.wikidataId : null,
wikidataRev: facts.wikidataRev ?? null,
description: typeof facts.description === 'string' ? facts.description : null,
categories: uniqStrings(facts.categories),
instanceOf: classes(facts.instanceOf),
subclassOf: classes(facts.subclassOf),
countries: uniqStrings(facts.countries),
years: uniqInts(facts.years),
died: facts.died === true,
coords,
links,
cites: uniqStrings(facts.cites).filter(q => QID.test(q)),
linksHash: linksHash(links),
source,
frozenAt: now.toISOString(),
};
}
/** The duel-cards `CardProfile` view of an identity (structural; no import, so no stacked PR). */
export function toProfile(i: CardIdentity) {
return { wikidataId: i.wikidataId, description: i.description, categories: i.categories, instanceOf: i.instanceOf,
subclassOf: i.subclassOf, countries: i.countries, years: i.years, died: i.died, links: i.links, cites: i.cites,
...(i.fetchedRevisionId != null ? { fetchedRevisionId: i.fetchedRevisionId } : {}) };
}
/** Persistent per-token identity store. Records are immutable once written. */
export class IdentityStore {
readonly #db: DatabaseSync;
constructor(file: string) {
this.#db = new DatabaseSync(file);
this.#db.exec(`CREATE TABLE IF NOT EXISTS card_identity (
token_id TEXT PRIMARY KEY, page_id INTEGER NOT NULL, version INTEGER NOT NULL, body TEXT NOT NULL)`);
}
get(tokenId: string): CardIdentity | undefined {
const row = this.#db.prepare('SELECT body FROM card_identity WHERE token_id = ?').get(tokenId) as { body: string } | undefined;
return row ? JSON.parse(row.body) as CardIdentity : undefined;
}
has(tokenId: string): boolean { return this.get(tokenId) !== undefined; }
/** Freeze at first sight: returns the stored record, which is the earlier one if it already existed. */
freeze(identity: CardIdentity): CardIdentity {
this.#db.prepare('INSERT OR IGNORE INTO card_identity (token_id, page_id, version, body) VALUES (?, ?, ?, ?)')
.run(identity.tokenId, identity.pageId, identity.version, JSON.stringify(identity));
return this.get(identity.tokenId)!;
}
/** Freeze several in one transaction (one fsync per batch, not per card). */
freezeMany(identities: CardIdentity[]): void {
this.#db.exec('BEGIN');
try { for (const i of identities) this.freeze(i); this.#db.exec('COMMIT'); }
catch (e) { this.#db.exec('ROLLBACK'); throw e; }
}
count(): number { return (this.#db.prepare('SELECT COUNT(*) AS n FROM card_identity').get() as { n: number }).n; }
all(): CardIdentity[] {
return (this.#db.prepare('SELECT body FROM card_identity ORDER BY token_id').all() as { body: string }[]).map(r => JSON.parse(r.body));
}
close() { this.#db.close(); }
}
export type PrebakeReport = { seen: number; alreadyFrozen: number; frozen: number; unknown: number };
/**
* Freeze identities for every card not yet in the store, asking the source in batches. The caller
* controls pacing (batch size, and the source's client controls request rate).
*/
export async function prebake(store: IdentityStore, source: IdentitySource, articles: ChainArticle[],
opts: { batchSize?: number; now?: () => Date } = {}): Promise<PrebakeReport> {
const batchSize = Math.max(1, Math.min(50, opts.batchSize ?? 50)), now = opts.now ?? (() => new Date());
const seenTokens = new Set<string>(), todo: ChainArticle[] = [];
let alreadyFrozen = 0;
for (const a of articles) {
if (seenTokens.has(a.tokenId)) continue;
seenTokens.add(a.tokenId);
if (store.has(a.tokenId)) alreadyFrozen++; else todo.push(a);
}
let frozen = 0, unknown = 0;
for (let i = 0; i < todo.length; i += batchSize) {
const batch = todo.slice(i, i + batchSize), facts = await source.lookup(batch), ready = [];
for (const a of batch) {
const f = facts.get(a.pageId);
if (!f) { unknown++; continue; }
ready.push(buildIdentity(a, f, source.kind, now()));
}
store.freezeMany(ready); frozen += ready.length;
}
return { seen: seenTokens.size, alreadyFrozen, frozen, unknown };
}
/** Offline source: facts from an already-cached card file such as docs/game/card-sample.json. */
export function cachedSource(cards: readonly { pageId: number; wikidataId?: string | null; revisionId?: number | null;
description?: string | null; instanceOf?: WikidataClass[]; categories?: string[] }[]): IdentitySource {
const byPage = new Map(cards.map(c => [c.pageId, c]));
return {
kind: 'cache',
async lookup(articles) {
return new Map(articles.map(a => {
const c = byPage.get(a.pageId);
return [a.pageId, c ? { fetchedRevisionId: c.revisionId ?? null, wikidataId: c.wikidataId ?? null,
description: c.description ?? null, instanceOf: c.instanceOf ?? [], categories: c.categories ?? [] } : undefined];
}));
},
};
}
+6 -6
View File
@@ -11,8 +11,7 @@ import { RandomX } from './randomx.ts';
import { Ledger } from './ledger.ts';
import { PeerNetwork } from './peers.ts';
import { wikipediaEnrichment } from '../enrichment.ts';
import { snapshotStats } from '../../shared/enrichment.ts';
import { wikipediaArticles } from '../wiki.ts';
import { configureWikimedia, mintPool, wikimediaStatus } from '../wikimedia/index.ts';
import { IS_MAINNET, PROFILE } from '../../shared/network.ts';
import { assertLaunchAllowed, launchManifest } from '../launch.ts';
import { resourceConfig, RpcBudget, RpcError } from '../resources.ts';
@@ -28,6 +27,7 @@ const apiBudget = new RpcBudget(limits);
const peerBudget = new RpcBudget({ ...limits, requestsPerSecond: 40, writesPerSecond: 10, active: 16 });
const dataDir = process.env.WIKIDECK_DATA_DIR ?? resolve(root,'data');
await mkdir(dataDir,{recursive:true});
configureWikimedia({dataDir});
// Never permit two processes to rewrite the same ledger. A stale lock requires operator review.
const lockPath = resolve(dataDir,`${PROFILE}-node.lock`);
const lock = await open(lockPath,'wx',0o600).catch(()=>{throw new Error('Data directory locked. Stop its other node; remove stale testnet-node.lock only after verifying no node is running.');});
@@ -119,9 +119,9 @@ const templateLimit=rateLimit({windowMs:60000,limit:20,standardHeaders:'draft-8'
app.post('/api/mining/template', templateLimit, async (req,res)=>{
const body=z.object({owner:z.string().regex(/^[a-f0-9]{64}$/),publicKey:z.string().max(256)}).strict().parse(req.body);
try {
const article=await wikipediaArticles.random();
const metrics=await wikipediaEnrichment.getFresh(article.pageId);
res.json(ledger.template(body.owner,body.publicKey,article,snapshotStats(metrics)));
// One fresh article+snapshot per tip, shared by every miner (server/wikimedia/pool.ts).
const {article,statsSnapshot}=await mintPool.take(ledger.summary().tip);
res.json(ledger.template(body.owner,body.publicKey,article,statsSnapshot));
} catch {
res.status(503).json({error:'Could not prepare a Wikipedia article and current statistics. Retry shortly.'});
}
@@ -170,7 +170,7 @@ const server=app.listen(port,host,()=>{
const operations=startOperations(opsPort,()=>({network:NETWORK,
ledger:ledger.diagnostics(),native:engine.diagnostics(),
peers:{configured:peers.status().length,connected:peers.status().filter(p=>p.connected).length},
rpc:{api:apiBudget.status(),p2p:peerBudget.status()}}),resolve(dataDir,PROFILE));
rpc:{api:apiBudget.status(),p2p:peerBudget.status()},wikimedia:wikimediaStatus()}),resolve(dataDir,PROFILE));
server.maxConnections=256;
server.headersTimeout=10000;
server.requestTimeout=15000;
+22 -10
View File
@@ -1,6 +1,6 @@
import { z } from 'zod';
import type { Article } from '../shared/protocol.ts';
import { ARTICLE_USER_AGENT, wikimediaUserAgent } from './wikimedia-user-agent.ts';
import { WikimediaClient, WikimediaUnavailable, wikimediaClient } from './wikimedia/client.ts';
const pageSchema = z.object({
pageid: z.number().int().positive().max(Number.MAX_SAFE_INTEGER), ns: z.literal(0),
@@ -58,11 +58,23 @@ function pictureUrl(value: string | undefined): string | undefined {
} catch { return undefined; }
}
/** The exact W1 request. Its parameters decide minted article content: keep them byte-identical. */
export function randomArticlesUrl(): URL {
const url = new URL('https://en.wikipedia.org/w/api.php');
url.search = new URLSearchParams({ action: 'query', format: 'json', formatversion: '2',
generator: 'random', grnnamespace: '0', grnlimit: '10', prop: 'extracts|pageimages|info|categories',
exintro: '1', explaintext: '1', exchars: '900', piprop: 'thumbnail', pithumbsize: '800', cllimit: '20' }).toString();
return url;
}
export class WikipediaArticleSource {
private readonly fetcher: typeof globalThis.fetch;
private readonly client: WikimediaClient;
private active = 0;
constructor(fetcher: typeof globalThis.fetch = globalThis.fetch) { this.fetcher = fetcher; }
/** A bare fetch (tests) gets an unthrottled pass-through client; production uses the shared client. */
constructor(source: typeof globalThis.fetch | WikimediaClient = wikimediaClient) {
this.client = source instanceof WikimediaClient ? source : WikimediaClient.unthrottled(source);
}
async random(): Promise<Article> {
if (this.active >= 4) throw new Error('Wikipedia article service is busy');
@@ -70,12 +82,8 @@ export class WikipediaArticleSource {
try {
for (let attempt = 0; attempt < MAX_ATTEMPTS; attempt++) {
try {
const url = new URL('https://en.wikipedia.org/w/api.php');
url.search = new URLSearchParams({ action: 'query', format: 'json', formatversion: '2',
generator: 'random', grnnamespace: '0', grnlimit: '10', prop: 'extracts|pageimages|info|categories',
exintro: '1', explaintext: '1', exchars: '900', piprop: 'thumbnail', pithumbsize: '800', cllimit: '20' }).toString();
const response = await this.fetcher(url, { signal: AbortSignal.timeout(3500), redirect: 'error',
headers: { 'User-Agent': wikimediaUserAgent(ARTICLE_USER_AGENT), Accept: 'application/json' } });
// Mint lane, never cached: random articles become immutable chain data.
const { response } = await this.client.request(randomArticlesUrl(), { lane: 'mint', timeoutMs: 3500 });
const data = responseSchema.parse(await readJson(response));
for (const rawPage of data.query.pages) {
const parsedPage = pageSchema.safeParse(rawPage);
@@ -88,7 +96,11 @@ export class WikipediaArticleSource {
url: `https://en.wikipedia.org/?curid=${page.pageid}`, revisionId: page.lastrevid,
source: 'live', category: category(page.title, page.categories?.map(c => c.title) ?? []) };
}
} catch { /* Try another random API response, but keep the number of attempts bounded. */ }
} catch (error) {
// Throttled, paused or failing upstream: stop now rather than hammer it with retries.
if (error instanceof WikimediaUnavailable) throw error;
/* Otherwise try another random API response, but keep the number of attempts bounded. */
}
}
throw new Error('Wikipedia did not return an article with a picture after three attempts');
} finally {
+41 -6
View File
@@ -1,13 +1,48 @@
// User-Agent for Wikimedia API requests. Wikimedia asks automated clients to identify
// themselves with contact information; operators can opt in with WIKIDECK_USER_AGENT.
// Unset (or blank), each caller keeps its historical default string unchanged.
export const ARTICLE_USER_AGENT = 'Wikideck/0.1 (experimental Wikipedia article cards)';
export const ENRICHMENT_USER_AGENT = 'Wikideck/0.1 (Wikipedia off-chain page enrichment)';
// User-Agent for Wikimedia API requests. Wikimedia's User-Agent policy asks automated clients for a
// descriptive agent with contact details: `Tool/version (+https://site; contact) lib/version`.
// Operators set WIKIDECK_OPERATOR_CONTACT (an email address or URL); PUBLIC_ORIGIN is the fallback.
// WIKIDECK_USER_AGENT still overrides the whole string.
import { readFileSync } from 'node:fs';
export const PROJECT_URL = 'https://gitea.eom.dev/WikiDeck/wikideck';
const MAX_LENGTH = 256;
const SAFE = /^[\x20-\x7e]+$/;
function packageVersion(): string {
try {
const raw = JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8')) as { version?: unknown };
return typeof raw.version === 'string' && /^[\w.+-]{1,32}$/.test(raw.version) ? raw.version : '0';
} catch { return '0'; }
}
const VERSION = packageVersion();
/** The operator contact, or undefined when none (or only an unsafe value) is configured. */
export function operatorContact(env: Record<string, string | undefined> = process.env): string | undefined {
for (const value of [env.WIKIDECK_OPERATOR_CONTACT, env.PUBLIC_ORIGIN]) {
const trimmed = value?.trim();
if (trimmed && trimmed.length <= 128 && SAFE.test(trimmed) && !/[();]/.test(trimmed)) return trimmed;
}
return undefined;
}
/** `WikiDeck/<version> (+<project URL>; <contact>) node/<version>`. */
export function defaultUserAgent(env: Record<string, string | undefined> = process.env): string {
const contact = operatorContact(env);
return `WikiDeck/${VERSION} (+${PROJECT_URL}${contact ? `; ${contact}` : ''}) node/${process.versions.node}`;
}
export function wikimediaUserAgent(fallback: string, env: Record<string, string | undefined> = process.env): string {
const configured = env.WIKIDECK_USER_AGENT?.trim();
// Reject header-injection/control characters and oversized values; fall back to the default.
if (!configured || configured.length > MAX_LENGTH || !/^[\x20-\x7e]+$/.test(configured)) return fallback;
if (!configured || configured.length > MAX_LENGTH || !SAFE.test(configured)) return fallback;
return configured;
}
/** Start-up check: Wikimedia throttles or blocks clients without contact details first. */
export function warnIfNoContact(env: Record<string, string | undefined> = process.env,
warn: (message: string) => void = console.warn): boolean {
if (wikimediaUserAgent('', env) || operatorContact(env)) return false;
warn('[wikimedia] WARNING: no operator contact in the Wikimedia User-Agent. Set WIKIDECK_OPERATOR_CONTACT '
+ '(an email address or URL) or PUBLIC_ORIGIN; Wikimedia policy requires it and may block this node.');
return true;
}
+74
View File
@@ -0,0 +1,74 @@
/**
* On-disk HTTP cache for off-chain Wikimedia reads (`$WIKIDECK_DATA_DIR/wikimedia-cache.sqlite`).
* It stores the raw body with its ETag / Last-Modified so stale entries can be revalidated with
* conditional requests. Mint-path requests never use it (see client.ts).
*/
import { DatabaseSync } from 'node:sqlite';
export type StoredEntry = {
status: number; body: Buffer; contentType: string; etag: string | null; lastModified: string | null;
expiresAt: number; storedAt: number;
};
export type CacheEntry = StoredEntry & { toResponse(): Response };
/** Canonical key: origin + path + sorted query, without `maxlag` (it changes only error behaviour). */
export function cacheKey(url: URL): string {
const params = [...url.searchParams].filter(([name]) => name !== 'maxlag')
.sort(([a, x], [b, y]) => a < b ? -1 : a > b ? 1 : x < y ? -1 : x > y ? 1 : 0);
return `${url.origin}${url.pathname}?${new URLSearchParams(params).toString()}`;
}
function withResponse(entry: StoredEntry): CacheEntry {
return { ...entry, toResponse: () => new Response(new Uint8Array(entry.body), { status: entry.status,
headers: { 'Content-Type': entry.contentType, 'Content-Length': String(entry.body.byteLength),
...(entry.etag ? { ETag: entry.etag } : {}) } }) };
}
type Row = { status: number; body: Uint8Array; content_type: string; etag: string | null;
last_modified: string | null; expires_at: number; stored_at: number };
export class WikimediaCache {
private readonly db: DatabaseSync;
private readonly maxRows: number;
private writes = 0;
/** `path` may be ':memory:' (tests). */
constructor(path: string, options: { maxRows?: number } = {}) {
this.maxRows = options.maxRows ?? 100_000;
this.db = new DatabaseSync(path);
this.db.exec(`PRAGMA journal_mode=WAL; PRAGMA synchronous=NORMAL;
CREATE TABLE IF NOT EXISTS http_cache (key TEXT PRIMARY KEY, status INTEGER NOT NULL, body BLOB NOT NULL,
content_type TEXT NOT NULL, etag TEXT, last_modified TEXT, expires_at INTEGER NOT NULL, stored_at INTEGER NOT NULL);
CREATE INDEX IF NOT EXISTS http_cache_stored ON http_cache(stored_at);`);
}
get(key: string): CacheEntry | undefined {
const row = this.db.prepare('SELECT * FROM http_cache WHERE key = ?').get(key) as Row | undefined;
if (!row) return undefined;
return withResponse({ status: row.status, body: Buffer.from(row.body), contentType: row.content_type,
etag: row.etag, lastModified: row.last_modified, expiresAt: row.expires_at, storedAt: row.stored_at });
}
put(key: string, entry: StoredEntry): CacheEntry {
this.db.prepare(`INSERT OR REPLACE INTO http_cache (key, status, body, content_type, etag, last_modified, expires_at, stored_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)`).run(key, entry.status, entry.body, entry.contentType, entry.etag,
entry.lastModified, entry.expiresAt, entry.storedAt);
if (++this.writes % 500 === 0) this.prune();
return withResponse(entry);
}
touch(key: string, expiresAt: number): void {
this.db.prepare('UPDATE http_cache SET expires_at = ? WHERE key = ?').run(expiresAt, key);
}
delete(key: string): void { this.db.prepare('DELETE FROM http_cache WHERE key = ?').run(key); }
size(): number { return (this.db.prepare('SELECT COUNT(*) AS n FROM http_cache').get() as { n: number }).n; }
prune(): void {
this.db.prepare(`DELETE FROM http_cache WHERE key IN (SELECT key FROM http_cache ORDER BY stored_at DESC LIMIT -1 OFFSET ?)`)
.run(this.maxRows);
}
close(): void { this.db.close(); }
}
+441
View File
@@ -0,0 +1,441 @@
/**
* One process-wide, polite Wikimedia HTTP client (engineer review §4, data review §5).
*
* Every server-side Wikimedia request goes through `WikimediaClient.request()`:
* - a token bucket per upstream host (default 1 req/s, burst 3) and 1 request in flight per host;
* - priority lanes: `mint` before `interactive` before `background`, with a bounded queue;
* - `maxlag=5` on Action API requests (it changes only error behaviour, never content);
* - `Retry-After` honoured on 429/503/maxlag by pausing the whole host, with exponential backoff;
* - a circuit breaker: 3 consecutive 429s stop all requests to that host for 10 minutes;
* - a daily request cap, and a budget shared by the nodes behind one egress IP;
* - an optional on-disk cache (see cache.ts) with ETag / If-None-Match and stale-while-revalidate.
*
* The client is transport only. It never rewrites request parameters other than adding `maxlag`,
* and requests made with `cache: undefined` (all mint-path requests) never read or write the cache.
* Nothing here is reachable from block validation, so it cannot affect consensus.
*/
import { WikimediaCache, cacheKey } from './cache.ts';
import { wikimediaUserAgent, defaultUserAgent } from '../wikimedia-user-agent.ts';
export type Lane = 'mint' | 'interactive' | 'background';
const LANES: Lane[] = ['mint', 'interactive', 'background'];
export type Clock = {
now(): number;
setTimeout(fn: () => void, ms: number): unknown;
clearTimeout(handle: unknown): void;
};
const realClock: Clock = {
now: () => Date.now(),
setTimeout: (fn, ms) => { const t = setTimeout(fn, ms); t.unref?.(); return t; },
clearTimeout: handle => clearTimeout(handle as ReturnType<typeof setTimeout>),
};
export type ClientConfig = {
/** Sustained requests per second per upstream host, for this whole egress IP. */
requestsPerSecond: number;
burst: number;
concurrencyPerHost: number;
/** Requests per UTC day for this whole egress IP. */
dailyCap: number;
/** Nodes sharing one egress IP; the rate and daily cap are divided between them. */
nodesPerIp: number;
queueLimit: number;
/** Longest wait in the queue before a request fails, per lane. */
maxWaitMs: Record<Lane, number>;
breakerThreshold: number;
breakerMs: number;
maxBackoffMs: number;
maxlag: number;
/** Retries of a single request after a maxlag reply (HTTP 200/503 with MediaWiki-API-Error: maxlag). */
maxlagRetries: number;
};
export const DEFAULT_CONFIG: ClientConfig = {
requestsPerSecond: 1, burst: 3, concurrencyPerHost: 1, dailyCap: 20_000, nodesPerIp: 1, queueLimit: 64,
maxWaitMs: { mint: 10_000, interactive: 5_000, background: 120_000 },
breakerThreshold: 3, breakerMs: 10 * 60_000, maxBackoffMs: 10 * 60_000, maxlag: 5, maxlagRetries: 2,
};
/** Unthrottled settings for tests and tools that inject their own fetch. */
export const UNTHROTTLED: Partial<ClientConfig> = {
requestsPerSecond: 1e9, burst: 1e9, concurrencyPerHost: 1e9, dailyCap: Number.MAX_SAFE_INTEGER, queueLimit: 1e9,
breakerThreshold: Number.MAX_SAFE_INTEGER, maxlagRetries: 0, maxBackoffMs: 0,
};
function positive(value: string | undefined, fallback: number, integer = false): number {
if (value === undefined || value.trim() === '') return fallback;
const n = Number(value);
if (!Number.isFinite(n) || n <= 0 || (integer && !Number.isSafeInteger(n))) {
console.warn(`[wikimedia] ignoring invalid setting ${JSON.stringify(value)}; using ${fallback}`);
return fallback;
}
return n;
}
/** Operator settings. Documented in docs/night2/WIKIMEDIA.md. */
export function configFromEnv(env: Record<string, string | undefined> = process.env): ClientConfig {
return { ...DEFAULT_CONFIG,
requestsPerSecond: positive(env.WIKIDECK_WIKIMEDIA_RPS, DEFAULT_CONFIG.requestsPerSecond),
burst: positive(env.WIKIDECK_WIKIMEDIA_BURST, DEFAULT_CONFIG.burst),
concurrencyPerHost: positive(env.WIKIDECK_WIKIMEDIA_CONCURRENCY, DEFAULT_CONFIG.concurrencyPerHost, true),
dailyCap: positive(env.WIKIDECK_WIKIMEDIA_DAILY_CAP, DEFAULT_CONFIG.dailyCap, true),
nodesPerIp: positive(env.WIKIDECK_WIKIMEDIA_NODES_PER_IP, DEFAULT_CONFIG.nodesPerIp, true),
};
}
/** Thrown for every refusal or transport failure: queue full, cap reached, circuit open, timeout, network. */
export class WikimediaUnavailable extends Error {
readonly reason: string;
readonly status?: number;
constructor(reason: string, status?: number) {
super(`Wikimedia unavailable: ${reason}`);
this.name = 'WikimediaUnavailable';
this.reason = reason; this.status = status;
}
}
export type CachePolicy = {
ttlMs: number;
/** Serve a stale entry at once and revalidate it on the background lane. */
staleWhileRevalidate?: boolean;
/** Largest body stored, in bytes. */
maxBytes?: number;
};
export type RequestOptions = {
lane: Lane;
/** Starts when the request is dispatched, not while it waits in the queue. */
timeoutMs: number;
headers?: Record<string, string>;
/** Omit for mint data: the request then bypasses the cache completely. */
cache?: CachePolicy;
};
export type ClientResponse = { response: Response; signal: AbortSignal; fromCache: boolean };
type Waiter = {
lane: Lane; host: string; enqueuedAt: number;
run: () => void; fail: (error: Error) => void; timer?: unknown;
};
type HostState = {
tokens: number; refilledAt: number; inFlight: number;
pausedUntil: number; breakerUntil: number; consecutive429: number; backoffExponent: number;
};
export type ClientStatus = {
day: string; requestsToday: number; dailyCap: number; requestsPerSecond: number; queued: number;
counters: Record<string, number>;
hosts: Record<string, { inFlight: number; pausedForMs: number; circuitOpen: boolean; consecutive429: number }>;
};
function parseRetryAfter(value: string | null, now: number): number | undefined {
if (!value) return undefined;
if (/^\d+$/.test(value.trim())) return Number(value.trim()) * 1000;
const at = Date.parse(value);
return Number.isNaN(at) ? undefined : Math.max(0, at - now);
}
function isActionApi(url: URL): boolean {
return url.pathname === '/w/api.php' && /(^|\.)(wikipedia|wikidata|wikimedia)\.org$/.test(url.hostname);
}
export class WikimediaClient {
readonly config: ClientConfig;
private readonly fetcher: typeof globalThis.fetch;
private readonly clock: Clock;
private readonly userAgent: () => string;
private cache: WikimediaCache | undefined;
private readonly hosts = new Map<string, HostState>();
private readonly queues = new Map<Lane, Waiter[]>(LANES.map(lane => [lane, []]));
private readonly revalidating = new Set<string>();
private pumpTimer: unknown;
private pumpAt = Infinity;
private day = '';
private requestsToday = 0;
readonly counters: Record<string, number> = {};
constructor(options: { fetch?: typeof globalThis.fetch; config?: Partial<ClientConfig>; clock?: Clock;
cache?: WikimediaCache; userAgent?: () => string } = {}) {
this.fetcher = options.fetch ?? ((...args) => globalThis.fetch(...args));
this.config = { ...DEFAULT_CONFIG, ...options.config,
maxWaitMs: { ...DEFAULT_CONFIG.maxWaitMs, ...options.config?.maxWaitMs } };
this.clock = options.clock ?? realClock;
this.userAgent = options.userAgent ?? (() => wikimediaUserAgent(defaultUserAgent()));
this.cache = options.cache;
}
/** A pass-through client for injected fetchers (tests): no rate limit, no breaker, no cache. */
static unthrottled(fetcher: typeof globalThis.fetch): WikimediaClient {
return new WikimediaClient({ fetch: fetcher, config: UNTHROTTLED });
}
attachCache(cache: WikimediaCache | undefined): void { this.cache = cache; }
private count(name: string): void { this.counters[name] = (this.counters[name] ?? 0) + 1; }
private get rate(): number { return this.config.requestsPerSecond / this.config.nodesPerIp; }
private get cap(): number { return Math.max(1, Math.floor(this.config.dailyCap / this.config.nodesPerIp)); }
private host(name: string): HostState {
let state = this.hosts.get(name);
if (!state) {
state = { tokens: this.config.burst, refilledAt: this.clock.now(), inFlight: 0,
pausedUntil: 0, breakerUntil: 0, consecutive429: 0, backoffExponent: 0 };
this.hosts.set(name, state);
}
return state;
}
private refill(state: HostState, now: number): void {
state.tokens = Math.min(this.config.burst, state.tokens + (now - state.refilledAt) / 1000 * this.rate);
state.refilledAt = now;
}
/** Daily budget by lane: background stops at 80 %, interactive at 100 %, mint has a 10 % reserve. */
private overCap(lane: Lane): boolean {
const today = new Date(this.clock.now()).toISOString().slice(0, 10);
if (today !== this.day) { this.day = today; this.requestsToday = 0; }
const limit = lane === 'background' ? this.cap * 0.8 : lane === 'interactive' ? this.cap : this.cap * 1.1;
return this.requestsToday >= limit;
}
status(): ClientStatus {
const now = this.clock.now();
return { day: this.day, requestsToday: this.requestsToday, dailyCap: this.cap, requestsPerSecond: this.rate,
queued: LANES.reduce((n, lane) => n + this.queues.get(lane)!.length, 0), counters: { ...this.counters },
hosts: Object.fromEntries([...this.hosts].map(([name, s]) => [name,
{ inFlight: s.inFlight, pausedForMs: Math.max(0, s.pausedUntil - now), circuitOpen: s.breakerUntil > now,
consecutive429: s.consecutive429 }])) };
}
async request(input: URL, options: RequestOptions): Promise<ClientResponse> {
const url = new URL(input);
if (url.protocol !== 'https:') throw new RangeError('Wikimedia requests must use HTTPS');
if (isActionApi(url) && !url.searchParams.has('maxlag')) url.searchParams.append('maxlag', String(this.config.maxlag));
const key = options.cache && this.cache ? cacheKey(url) : undefined;
const entry = key ? this.cache?.get(key) : undefined;
const now = this.clock.now();
if (entry && entry.expiresAt > now) {
this.count('cache.hit');
return { response: entry.toResponse(), signal: new AbortController().signal, fromCache: true };
}
if (entry && options.cache?.staleWhileRevalidate) {
this.count('cache.stale');
this.revalidate(url, key!, options);
return { response: entry.toResponse(), signal: new AbortController().signal, fromCache: true };
}
try {
return await this.network(url, options, key);
} catch (error) {
// Prefer a stale copy over an error for anything that is allowed to use the cache.
if (entry && error instanceof WikimediaUnavailable) {
this.count('cache.staleOnError');
return { response: entry.toResponse(), signal: new AbortController().signal, fromCache: true };
}
throw error;
}
}
private revalidate(url: URL, key: string, options: RequestOptions): void {
if (this.revalidating.has(key)) return;
this.revalidating.add(key);
void this.network(url, { ...options, lane: 'background' }, key)
.then(({ response }) => response.body?.cancel(), () => this.count('cache.revalidateFailed'))
.finally(() => this.revalidating.delete(key));
}
private async network(url: URL, options: RequestOptions, key: string | undefined): Promise<ClientResponse> {
for (let attempt = 0; ; attempt++) {
const result = await this.dispatch(url, options, key);
if (result !== 'maxlag') return result;
if (attempt >= this.config.maxlagRetries) throw new WikimediaUnavailable('maxlag');
}
}
/** Wait for a slot, send one request, and classify the reply. */
private async dispatch(url: URL, options: RequestOptions, key: string | undefined): Promise<ClientResponse | 'maxlag'> {
const hostName = url.hostname;
const wait = this.acquire(hostName, options.lane);
if (wait) await wait; // No await on the fast path: an idle host dispatches synchronously.
const state = this.host(hostName);
const headers = new Headers({ Accept: 'application/json', 'Accept-Encoding': 'gzip', ...options.headers,
'User-Agent': this.userAgent() });
const entry = key ? this.cache?.get(key) : undefined;
if (entry?.etag) headers.set('If-None-Match', entry.etag);
if (entry?.lastModified) headers.set('If-Modified-Since', entry.lastModified);
const signal = AbortSignal.timeout(options.timeoutMs);
this.requestsToday++;
this.count(`lane.${options.lane}`);
let response: Response;
try {
response = await this.fetcher(url, { signal, redirect: 'error', headers });
} catch (error) {
this.release(state);
this.count('error.network');
throw new WikimediaUnavailable(signal.aborted ? 'timeout' : `network: ${(error as Error).message}`);
}
const now = this.clock.now();
const retryAfter = parseRetryAfter(response.headers.get('retry-after'), now);
const apiError = response.headers.get('mediawiki-api-error');
if (apiError === 'maxlag') {
await response.body?.cancel();
this.count('maxlag');
this.pause(state, Math.min(this.config.maxBackoffMs, retryAfter ?? 5000), false);
this.release(state);
return 'maxlag';
}
if (response.status === 429 || response.status === 503) {
await response.body?.cancel();
this.count(`http.${response.status}`);
if (response.status === 429) state.consecutive429++;
const backoff = Math.min(this.config.maxBackoffMs, 2 ** state.backoffExponent * 1000 * (1 + Math.random() * 0.25));
state.backoffExponent = Math.min(state.backoffExponent + 1, 20);
if (state.consecutive429 >= this.config.breakerThreshold) {
this.pause(state, this.config.breakerMs, true);
this.count('breaker.open');
} else {
this.pause(state, Math.min(this.config.maxBackoffMs, Math.max(retryAfter ?? 0, backoff)), false);
}
this.release(state);
throw new WikimediaUnavailable(`HTTP ${response.status}`, response.status);
}
state.consecutive429 = 0;
state.backoffExponent = 0;
this.release(state);
if (response.status === 304 && entry && key) {
this.count('cache.revalidated');
this.cache?.touch(key, now + options.cache!.ttlMs);
return { response: entry.toResponse(), signal, fromCache: true };
}
if (key && this.cache && response.status === 200 && !response.redirected && options.cache &&
/^application\/json(?:\s*;|\s*$)/i.test(response.headers.get('content-type') ?? '')) {
// Store the raw body; callers re-validate it exactly as they would a network reply.
const body = await this.readLimited(response, options.cache.maxBytes ?? 1024 * 1024, signal);
const stored = this.cache.put(key, { status: response.status, body,
contentType: response.headers.get('content-type') ?? '', etag: response.headers.get('etag'),
lastModified: response.headers.get('last-modified'), expiresAt: now + options.cache.ttlMs, storedAt: now });
return { response: stored.toResponse(), signal, fromCache: false };
}
if (response.status >= 500) this.count(`http.${response.status}`);
return { response, signal, fromCache: false };
}
private async readLimited(response: Response, maxBytes: number, signal: AbortSignal): Promise<Buffer> {
const length = response.headers.get('content-length');
if (length !== null && (!/^\d+$/.test(length) || Number(length) > maxBytes) || !response.body) {
await response.body?.cancel();
throw new WikimediaUnavailable('oversized or empty response');
}
const reader = response.body.getReader();
const chunks: Uint8Array[] = [];
let size = 0;
try {
for (;;) {
if (signal.aborted) throw new WikimediaUnavailable('timeout');
const { done, value } = await reader.read();
if (done) break;
size += value.byteLength;
if (size > maxBytes) throw new WikimediaUnavailable('oversized response');
chunks.push(value);
}
return Buffer.concat(chunks);
} catch (error) {
throw error instanceof WikimediaUnavailable ? error : new WikimediaUnavailable('read failed');
} finally {
await reader.cancel().catch(() => {});
reader.releaseLock();
}
}
private pause(state: HostState, ms: number, breaker: boolean): void {
const until = this.clock.now() + ms;
if (until > state.pausedUntil) {
if (state.pausedUntil <= this.clock.now()) {
console.warn(`[wikimedia] ${breaker ? 'circuit open' : 'pausing'} for ${Math.ceil(ms / 1000)} s`);
}
state.pausedUntil = until;
}
if (breaker) { state.consecutive429 = 0; state.breakerUntil = until; }
}
private release(state: HostState): void {
state.inFlight--;
this.pump();
}
/** Undefined when a slot was taken at once; otherwise a promise for the slot. Throws on refusal. */
private acquire(hostName: string, lane: Lane): Promise<void> | undefined {
const state = this.host(hostName);
const now = this.clock.now();
if (this.overCap(lane)) { this.count('refused.cap'); throw new WikimediaUnavailable('daily cap reached'); }
// A paused host refuses at once when the pause outlasts this lane's patience.
if (state.pausedUntil - now > this.config.maxWaitMs[lane]) {
this.count('refused.paused');
throw new WikimediaUnavailable(state.breakerUntil > now ? 'circuit open' : 'host paused');
}
this.refill(state, now);
if (state.pausedUntil <= now && state.inFlight < this.config.concurrencyPerHost && state.tokens >= 1 &&
!LANES.some(l => this.queues.get(l)!.some(w => w.host === hostName))) {
state.tokens -= 1;
state.inFlight++;
return undefined;
}
const queued = LANES.reduce((n, l) => n + this.queues.get(l)!.length, 0);
if (queued >= this.config.queueLimit) {
// Background work is shed first; a mint request evicts the newest background waiter.
const background = this.queues.get('background')!;
if (lane === 'background' || background.length === 0) {
this.count('refused.queue');
throw new WikimediaUnavailable('queue full');
}
const evicted = background.pop()!;
this.clock.clearTimeout(evicted.timer);
evicted.fail(new WikimediaUnavailable('queue full'));
}
return new Promise<void>((resolve, reject) => {
const waiter: Waiter = { lane, host: hostName, enqueuedAt: now,
run: () => { this.clock.clearTimeout(waiter.timer); resolve(); },
fail: error => reject(error) };
waiter.timer = this.clock.setTimeout(() => {
const queue = this.queues.get(lane)!;
const index = queue.indexOf(waiter);
if (index >= 0) { queue.splice(index, 1); this.count('refused.wait'); reject(new WikimediaUnavailable('queue wait')); }
}, this.config.maxWaitMs[lane]);
this.queues.get(lane)!.push(waiter);
this.pump();
});
}
/** Start every waiter that may run now, highest lane first; otherwise arm a timer for the next slot. */
private pump(): void {
const now = this.clock.now();
let next = Infinity;
for (const lane of LANES) {
const queue = this.queues.get(lane)!;
for (let i = 0; i < queue.length;) {
const waiter = queue[i];
// A host is served strictly by priority: lower lanes wait while a higher lane waits for the same host.
if (LANES.slice(0, LANES.indexOf(lane)).some(l => this.queues.get(l)!.some(w => w.host === waiter.host))) { i++; continue; }
const state = this.host(waiter.host);
this.refill(state, now);
if (state.pausedUntil > now) { next = Math.min(next, state.pausedUntil); i++; continue; }
if (state.inFlight >= this.config.concurrencyPerHost) { i++; continue; }
if (state.tokens < 1) { next = Math.min(next, now + (1 - state.tokens) / this.rate * 1000); i++; continue; }
state.tokens -= 1;
state.inFlight++;
queue.splice(i, 1);
waiter.run();
}
}
if (next < Infinity && next < this.pumpAt) {
if (this.pumpTimer !== undefined) this.clock.clearTimeout(this.pumpTimer);
this.pumpAt = next;
this.pumpTimer = this.clock.setTimeout(() => { this.pumpTimer = undefined; this.pumpAt = Infinity; this.pump(); },
Math.max(1, Math.ceil(next - now)));
}
}
}
/** The process-wide client. `configureWikimedia()` (server start-up) attaches the disk cache. */
export const wikimediaClient = new WikimediaClient({ config: configFromEnv() });
+36
View File
@@ -0,0 +1,36 @@
/**
* Node start-up wiring for the polite Wikimedia client: the disk cache, the User-Agent warning,
* the mining prefetch pool and an ops status snapshot. See docs/night2/WIKIMEDIA.md.
*/
import { resolve } from 'node:path';
import { WikimediaCache } from './cache.ts';
import { wikimediaClient } from './client.ts';
import { MintTemplatePool } from './pool.ts';
import { warnIfNoContact } from '../wikimedia-user-agent.ts';
import { wikipediaArticles } from '../wiki.ts';
import { wikipediaEnrichment } from '../enrichment.ts';
export { wikimediaClient } from './client.ts';
/** Serves every mining template; shared by all miners of this node. */
export const mintPool = new MintTemplatePool({ articles: wikipediaArticles, enrichment: wikipediaEnrichment });
let cache: WikimediaCache | undefined;
/** Call once at start-up. Without it the client still works, with in-memory caching only. */
export function configureWikimedia(options: { dataDir: string }): void {
warnIfNoContact();
if (cache) return;
try {
cache = new WikimediaCache(resolve(options.dataDir, 'wikimedia-cache.sqlite'));
wikimediaClient.attachCache(cache);
} catch (error) {
console.warn(`[wikimedia] disk cache disabled: ${(error as Error).message}`);
}
}
export function closeWikimedia(): void { wikimediaClient.attachCache(undefined); cache?.close(); cache = undefined; }
export function wikimediaStatus() {
return { client: wikimediaClient.status(), cacheEntries: cache?.size() ?? null, mintPool: mintPool.status() };
}
+88
View File
@@ -0,0 +1,88 @@
/**
* Mining-template prefetch pool (data review §5, "Mining load").
*
* Without it every miner costs 3+ Wikimedia requests per template, all in a burst at each new tip.
* The pool keeps a few fresh article + stats snapshots ready, and serves the SAME pair to every
* template built on the same tip (miners race for one block anyway). Wikimedia load is then about
* one pair (3 requests) per block, whatever the number of miners.
*
* Mint data stays transport-only: each pair comes from the mint-path reads (`random()` and
* `getFresh()`, byte-identical parameters, never the disk cache), and the snapshot is stamped when it
* is fetched, so `capturedAt` stays true. Pairs older than `maxAgeMs` are discarded, never served.
* The pool refills only after it is used, so an idle node makes no requests.
*/
import type { Article } from '../../shared/protocol.ts';
import { snapshotStats, type MintStatsSnapshot, type PageEnrichment } from '../../shared/enrichment.ts';
export type MintPair = { article: Article; statsSnapshot: MintStatsSnapshot };
type Entry = MintPair & { fetchedAt: number };
type Sources = {
articles: { random(): Promise<Article> };
enrichment: { getFresh(pageId: number): Promise<PageEnrichment> };
};
export class MintTemplatePool {
private readonly sources: Sources;
private readonly size: number;
private readonly maxAgeMs: number;
private readonly now: () => number;
private readonly ready: Entry[] = [];
private assigned: { tip: string; entry: Entry } | undefined;
private filling: Promise<void> | undefined;
private direct: Promise<Entry> | undefined;
readonly counters = { served: 0, shared: 0, fetched: 0, expired: 0, failed: 0 };
constructor(sources: Sources, options: { size?: number; maxAgeMs?: number; now?: () => number } = {}) {
this.sources = sources;
this.size = options.size ?? 2;
this.maxAgeMs = options.maxAgeMs ?? 5 * 60_000;
this.now = options.now ?? Date.now;
if (!Number.isSafeInteger(this.size) || this.size < 0 || this.size > 16 ||
!Number.isFinite(this.maxAgeMs) || this.maxAgeMs <= 0) throw new RangeError('Invalid mint pool options');
}
private fresh(entry: Entry): boolean { return this.now() - entry.fetchedAt < this.maxAgeMs; }
private async fetchOne(): Promise<Entry> {
const article = await this.sources.articles.random();
const metrics = await this.sources.enrichment.getFresh(article.pageId);
this.counters.fetched++;
return { article, statsSnapshot: snapshotStats(metrics), fetchedAt: this.now() };
}
/** The article and snapshot for a template on `tip`. Rejects when Wikimedia is unavailable. */
async take(tip: string): Promise<MintPair> {
let entry: Entry | undefined;
if (this.assigned && this.assigned.tip === tip && this.fresh(this.assigned.entry)) {
entry = this.assigned.entry;
this.counters.shared++;
} else {
while (this.ready.length && !this.fresh(this.ready[0])) { this.ready.shift(); this.counters.expired++; }
entry = this.ready.shift();
if (!entry) {
// Cold: fetch directly, and let concurrent templates share that one fetch.
this.direct ??= this.fetchOne().finally(() => { this.direct = undefined; });
try { entry = await this.direct; }
catch (error) { this.counters.failed++; throw error; }
if (this.assigned && this.assigned.tip === tip && this.fresh(this.assigned.entry)) entry = this.assigned.entry;
}
this.assigned = { tip, entry };
}
this.counters.served++;
this.refill();
return structuredClone({ article: entry.article, statsSnapshot: entry.statsSnapshot });
}
/** Top the pool up one pair at a time, in the background. Exposed for tests. */
refill(): Promise<void> {
this.filling ??= (async () => {
while (this.ready.filter(e => this.fresh(e)).length < this.size) {
try { this.ready.push(await this.fetchOne()); }
catch { this.counters.failed++; break; } // The next take() tries again; no retry loop.
}
})().finally(() => { this.filling = undefined; });
return this.filling;
}
status() { return { ready: this.ready.length, size: this.size, maxAgeMs: this.maxAgeMs, ...this.counters }; }
}
+1 -1
View File
@@ -54,7 +54,7 @@ test('fixed HTTPS sources, encoded title, hidden categories, lower bounds and pl
assert.equal(init.redirect, 'error');
assert.ok(init.signal instanceof AbortSignal);
assert.equal(new Headers(init.headers).get('accept'), 'application/json');
assert.match(new Headers(init.headers).get('user-agent')!, /Wikideck/);
assert.match(new Headers(init.headers).get('user-agent')!, /WikiDeck/i);
assert.equal(new Headers(init.headers).has('origin'), false); // Server-side requests need no CORS workaround.
}
assert.equal(result.title, title);
-82
View File
@@ -1,82 +0,0 @@
// testnet-ops: live identity source over a FAKE Wikimedia client. Zero network calls.
import test from 'node:test';
import assert from 'node:assert/strict';
import { mkdtempSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { wikimediaIdentitySource } from '../server/game/identity-source.ts';
import { IdentityStore, prebake, toProfile } from '../server/game/identity.ts';
import type { BackgroundRequest, ProfileBuilder } from '../server/game/identity-source.ts';
// A minimal stand-in for shared/duel/profile.ts (#70), so this test needs no unmerged module.
const ids = (e: any, p: string): string[] => (e?.claims?.[p] ?? []).map((s: any) => s.mainsnak?.datavalue?.value?.id).filter(Boolean);
const yrs = (e: any, p: string): number[] => (e?.claims?.[p] ?? []).map((s: any) => Number(/^[+-]0*(\d+)-/.exec(s.mainsnak?.datavalue?.value?.time ?? '')?.[1])).filter(Boolean);
const profile: ProfileBuilder = {
wikidataEntityParams: q => ({ action: 'wbgetentities', ids: [...q].sort().join('|'), props: 'claims', format: 'json', formatversion: '2' }),
wikidataLabelParams: q => ({ action: 'wbgetentities', ids: [...q].sort().join('|'), props: 'labels', languages: 'en', format: 'json', formatversion: '2' }),
classIdsOf: e => [...new Set([...ids(e, 'P31'), ...ids(e, 'P279')])].sort(),
profileFromSources: ({ enrichment, entity, labels = {} }) => ({
wikidataId: enrichment.wikidataId, description: enrichment.description, categories: enrichment.categories,
instanceOf: ids(entity, 'P31').map(id => ({ id, label: labels[id] ?? id })), countries: ids(entity, 'P27'),
years: [...yrs(entity, 'P569'), ...yrs(entity, 'P570')], died: ids(entity, 'P570').length > 0 || yrs(entity, 'P570').length > 0,
fetchedRevisionId: enrichment.revisionId }),
};
const enrichmentFor = (pageId: number, wikidataId: string | null) => ({
pageId, title: `Page ${pageId}`, description: `desc ${pageId}`, wikidataId, revisionId: 1000 + pageId, pageBytes: 1,
languageCount: 1, languagesTruncated: false, categories: ['Living people', '1948 births'], categoriesTruncated: false,
fetchedAt: '2026-11-17T00:00:00Z',
pageviews: { status: 'unavailable', months: [], latestMonth: '2026-10', total: null, dailyAverage: null, trendPercent: null },
}) as any;
const claim = (id: string) => ({ rank: 'normal', mainsnak: { snaktype: 'value', datavalue: { type: 'wikibase-entityid', value: { id } } } });
const time = (t: string) => ({ rank: 'normal', mainsnak: { snaktype: 'value', datavalue: { type: 'time', value: { time: t } } } });
function fakeClient(opts: { failClaims?: boolean } = {}) {
const calls: { url: URL; options: BackgroundRequest }[] = [];
return { calls, async request(url: URL, options: BackgroundRequest) {
calls.push({ url, options });
const props = url.searchParams.get('props'), ids = url.searchParams.get('ids')!.split('|');
if (props === 'claims' && opts.failClaims) return { response: new Response('busy', { status: 503 }), signal: new AbortController().signal, fromCache: false };
const entities = Object.fromEntries(ids.map(id => [id, props === 'claims'
? { id, claims: { P31: [claim('Q5')], P27: [claim('Q30')], P569: [time('+1948-01-02T00:00:00Z')], P570: [time('+2001-01-01T00:00:00Z')] } }
: { id, labels: { en: { value: id === 'Q5' ? 'human' : `label ${id}` } } }]));
return { response: Response.json({ entities }), signal: new AbortController().signal, fromCache: false };
} };
}
test('every request is on the background lane, batched, and the frozen identity is derive-ready', async () => {
const realFetch = globalThis.fetch;
globalThis.fetch = (() => { throw new Error('network call in test'); }) as typeof fetch;
const dir = mkdtempSync(join(tmpdir(), 'identity-src-'));
try {
const client = fakeClient();
const source = wikimediaIdentitySource({ client, profile, enrichment: { async getBackground(id) { return enrichmentFor(id, `Q${id}`); } } });
const store = new IdentityStore(join(dir, 'id.sqlite'));
const articles = [1, 2, 3].map(i => ({ tokenId: String(i).repeat(64), pageId: i, title: `Page ${i}`, revisionId: i }));
const r = await prebake(store, source, articles);
assert.deepEqual(r, { seen: 3, alreadyFrozen: 0, frozen: 3, unknown: 0 });
assert.ok(client.calls.every(c => c.options.lane === 'background'));
assert.ok(client.calls.every(c => c.url.hostname === 'www.wikidata.org'));
assert.deepEqual(client.calls.map(c => c.url.searchParams.get('props')), ['claims', 'labels']);
assert.equal(client.calls[0].url.searchParams.get('ids'), 'Q1|Q2|Q3');
const one = store.get('1'.repeat(64))!;
assert.equal(one.source, 'wikimedia');
assert.deepEqual(one.instanceOf, [{ id: 'Q5', label: 'human' }]);
assert.deepEqual(one.countries, ['Q30']);
assert.deepEqual(one.years, [1948, 2001]);
assert.equal(one.died, true);
assert.equal(one.fetchedRevisionId, 1001);
assert.equal(toProfile(one).description, 'desc 1');
store.close();
} finally { globalThis.fetch = realFetch; rmSync(dir, { recursive: true, force: true }); }
});
test('a failed claims request leaves cards unknown so they are retried, never frozen half-empty', async () => {
const source = wikimediaIdentitySource({ client: fakeClient({ failClaims: true }), profile,
enrichment: { async getBackground(id) { if (id === 9) throw new Error('throttled'); return enrichmentFor(id, id === 8 ? null : `Q${id}`); } } });
const got = await source.lookup([7, 8, 9].map(i => ({ tokenId: String(i).repeat(64), pageId: i, title: `P${i}` })));
assert.equal(got.get(7), undefined); // has an item, claims failed → retry
assert.equal(got.get(9), undefined); // enrichment failed → retry
assert.equal(got.get(8)?.wikidataId, null); // no Wikidata item at all → enrichment-only profile is final
assert.deepEqual(got.get(8)?.instanceOf, []);
});
-90
View File
@@ -1,90 +0,0 @@
// testnet-ops: card identity freeze-at-first-sight and pre-bake. Zero network calls.
import test from 'node:test';
import assert from 'node:assert/strict';
import { mkdtempSync, readFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { identityStoreFile, IdentityStore, buildIdentity, cachedSource, linksHash, prebake, toProfile,
type ChainArticle, type IdentitySource } from '../server/game/identity.ts';
const sample = JSON.parse(readFileSync(new URL('../docs/game/card-sample.json', import.meta.url), 'utf8')).cards as any[];
const now = () => new Date('2026-11-17T22:00:00Z');
function tmpStore() {
const dir = mkdtempSync(join(tmpdir(), 'identity-'));
return { store: new IdentityStore(join(dir, 'identity.sqlite')), done: () => rmSync(dir, { recursive: true, force: true }) };
}
const art = (c: any, i: number): ChainArticle => ({ tokenId: i.toString(16).padStart(64, '0'), pageId: c.pageId, title: c.title, revisionId: c.revisionId });
test('no fetch: pre-bake from the cached sample freezes every sample card', async () => {
const realFetch = globalThis.fetch;
globalThis.fetch = (() => { throw new Error('network call in test'); }) as typeof fetch;
const { store, done } = tmpStore();
try {
const articles = sample.map(art);
const r = await prebake(store, cachedSource(sample), articles, { now });
assert.deepEqual(r, { seen: sample.length, alreadyFrozen: 0, frozen: sample.length, unknown: 0 });
const salah = store.get(articles[sample.findIndex(c => c.title === 'Salahuddin of Aceh')].tokenId)!;
assert.equal(salah.wikidataId, 'Q4264920');
assert.deepEqual(salah.instanceOf, [{ id: 'Q5', label: 'human' }]);
assert.equal(salah.source, 'cache');
assert.ok(salah.categories.includes('Sultans of Aceh'));
assert.equal(salah.description, 'Sultan of Aceh (1530–1537/9)');
const profile = toProfile(salah);
assert.deepEqual(profile.instanceOf, [{ id: 'Q5', label: 'human' }]);
assert.equal(profile.fetchedRevisionId, 1358732737);
} finally { globalThis.fetch = realFetch; store.close(); done(); }
});
test('freeze at first sight: a later, different profile never overwrites the stored one', async () => {
const { store, done } = tmpStore();
try {
const a = art(sample[0], 1);
const first = store.freeze(buildIdentity(a, { instanceOf: [{ id: 'Q5', label: 'human' }] }, 'cache', now()));
const second = store.freeze(buildIdentity(a, { instanceOf: [{ id: 'Q515', label: 'city' }] }, 'wikimedia', new Date()));
assert.deepEqual(second, first);
assert.equal(store.get(a.tokenId)!.instanceOf[0].id, 'Q5');
const again = await prebake(store, cachedSource(sample), [a, a], { now });
assert.deepEqual(again, { seen: 1, alreadyFrozen: 1, frozen: 0, unknown: 0 });
} finally { store.close(); done(); }
});
test('unknown pages are not frozen, so a later run can retry them', async () => {
const { store, done } = tmpStore();
try {
const stranger: ChainArticle = { tokenId: 'f'.repeat(64), pageId: 999999999, title: 'Not in sample', revisionId: 1 };
const r = await prebake(store, cachedSource(sample), [stranger], { now });
assert.deepEqual(r, { seen: 1, alreadyFrozen: 0, frozen: 0, unknown: 1 });
assert.equal(store.has(stranger.tokenId), false);
} finally { store.close(); done(); }
});
test('sources are asked in bounded batches and identities persist across reopen', async () => {
const dir = mkdtempSync(join(tmpdir(), 'identity-'));
try {
const calls: number[] = [];
const source: IdentitySource = { kind: 'wikimedia', async lookup(batch) {
calls.push(batch.length);
return new Map(batch.map(a => [a.pageId, { links: [7, 3, 3, 1.5], coords: [1, 2] as [number, number], cites: ['Q30', 'bad'], years: [1948, -44], died: true }]));
} };
let store = new IdentityStore(join(dir, 'id.sqlite'));
await prebake(store, source, sample.slice(0, 12).map(art), { batchSize: 5, now });
assert.deepEqual(calls, [5, 5, 2]);
store.close();
store = new IdentityStore(join(dir, 'id.sqlite'));
const one = store.get(art(sample[3], 3).tokenId)!;
assert.deepEqual(one.links, [3, 7]);
assert.equal(one.linksHash, linksHash([3, 7]));
assert.deepEqual(one.cites, ['Q30']);
assert.deepEqual(one.years, [-44, 1948]);
assert.equal(one.died, true);
assert.deepEqual(one.coords, [1, 2]);
assert.equal(one.mintRevisionId, sample[3].revisionId);
assert.equal(store.count(), 12);
store.close();
} finally { rmSync(dir, { recursive: true, force: true }); }
});
test('the store file lives under WIKIDECK_DATA_DIR, next to the ledger', () => {
assert.equal(identityStoreFile({ WIKIDECK_DATA_DIR: '/srv/wd' }), join('/srv/wd', 'card-identity.sqlite'));
assert.ok(identityStoreFile({}).endsWith(join('data', 'card-identity.sqlite')));
});
+17 -9
View File
@@ -2,7 +2,7 @@ import test from 'node:test';
import assert from 'node:assert/strict';
import { WikipediaArticleSource } from '../server/wiki.ts';
import { WikipediaEnrichment } from '../server/enrichment.ts';
import { ARTICLE_USER_AGENT, ENRICHMENT_USER_AGENT, wikimediaUserAgent } from '../server/wikimedia-user-agent.ts';
import { PROJECT_URL, defaultUserAgent, wikimediaUserAgent } from '../server/wikimedia-user-agent.ts';
function capture() {
const agents: string[] = [];
@@ -12,9 +12,10 @@ function capture() {
};
return { agents, fetcher };
}
async function agentsFor(env: string | undefined) {
const previous = process.env.WIKIDECK_USER_AGENT;
async function agentsFor(env: string | undefined, contact?: string) {
const previous = process.env.WIKIDECK_USER_AGENT, previousContact = process.env.WIKIDECK_OPERATOR_CONTACT;
if (env === undefined) delete process.env.WIKIDECK_USER_AGENT; else process.env.WIKIDECK_USER_AGENT = env;
if (contact === undefined) delete process.env.WIKIDECK_OPERATOR_CONTACT; else process.env.WIKIDECK_OPERATOR_CONTACT = contact;
try {
const article = capture(), enrichment = capture();
await assert.rejects(new WikipediaArticleSource(article.fetcher).random());
@@ -22,15 +23,22 @@ async function agentsFor(env: string | undefined) {
return { article: article.agents, enrichment: enrichment.agents };
} finally {
if (previous === undefined) delete process.env.WIKIDECK_USER_AGENT; else process.env.WIKIDECK_USER_AGENT = previous;
if (previousContact === undefined) delete process.env.WIKIDECK_OPERATOR_CONTACT;
else process.env.WIKIDECK_OPERATOR_CONTACT = previousContact;
}
}
test('default Wikimedia user agents are unchanged when WIKIDECK_USER_AGENT is unset', async () => {
assert.equal(ARTICLE_USER_AGENT, 'Wikideck/0.1 (experimental Wikipedia article cards)');
assert.equal(ENRICHMENT_USER_AGENT, 'Wikideck/0.1 (Wikipedia off-chain page enrichment)');
const { article, enrichment } = await agentsFor(undefined);
assert.ok(article.length > 0 && article.every(a => a === ARTICLE_USER_AGENT));
assert.ok(enrichment.length > 0 && enrichment.every(a => a === ENRICHMENT_USER_AGENT));
test('default Wikimedia user agent names the project, the operator contact and node', async () => {
const { article, enrichment } = await agentsFor(undefined, 'ops@example.org');
const expected = `WikiDeck/0.1.0 (+${PROJECT_URL}; ops@example.org) node/${process.versions.node}`;
assert.equal(defaultUserAgent({ WIKIDECK_OPERATOR_CONTACT: 'ops@example.org' }), expected);
assert.ok(article.length > 0 && article.every(a => a === expected));
assert.ok(enrichment.length > 0 && enrichment.every(a => a === expected));
assert.equal(defaultUserAgent({ PUBLIC_ORIGIN: 'https://node.example' }),
`WikiDeck/0.1.0 (+${PROJECT_URL}; https://node.example) node/${process.versions.node}`);
assert.equal(defaultUserAgent({}), `WikiDeck/0.1.0 (+${PROJECT_URL}) node/${process.versions.node}`);
// Unsafe contacts are dropped rather than breaking the header's comment syntax.
assert.equal(defaultUserAgent({ WIKIDECK_OPERATOR_CONTACT: 'a); evil(' }), defaultUserAgent({}));
});
test('WIKIDECK_USER_AGENT overrides both Wikimedia clients', async () => {
+4 -1
View File
@@ -56,13 +56,16 @@ test('random article fetch rejects unsafe response types, redirects, and oversiz
() => new Response('{}', { headers: { 'Content-Type': 'text/html' } }),
() => new Response(null, { status: 302, headers: { Location: 'https://evil.example' } }),
() => new Response(' '.repeat(512 * 1024 + 1), { headers: { 'Content-Type': 'application/json' } }),
() => { throw new Error('offline'); },
];
for (const fail of failures) {
const { fetcher, calls } = mockFetch(() => fail());
await assert.rejects(new WikipediaArticleSource(fetcher).random());
assert.equal(calls.length, 3);
}
// A network failure (or throttling) stops at once instead of retrying against an unreachable upstream.
const { fetcher, calls } = mockFetch(() => { throw new Error('offline'); });
await assert.rejects(new WikipediaArticleSource(fetcher).random(), /Wikimedia unavailable/);
assert.equal(calls.length, 1);
});
test('bad page records are skipped while later valid candidates are considered', async () => {
+236
View File
@@ -0,0 +1,236 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { WikimediaClient, WikimediaUnavailable, configFromEnv, type ClientConfig } from '../server/wikimedia/client.ts';
import { WikimediaCache, cacheKey } from '../server/wikimedia/cache.ts';
import { warnIfNoContact } from '../server/wikimedia-user-agent.ts';
import { FakeClock, flush, json, recorder } from './wikimedia-helpers.ts';
const API = 'https://en.wikipedia.org/w/api.php?action=query&format=json&pageids=1';
const REST = 'https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/x';
function setup(handler: Parameters<typeof recorder>[1] = () => json({ ok: true }), config: Partial<ClientConfig> = {},
cache?: WikimediaCache) {
const clock = new FakeClock();
const rec = recorder(clock, handler);
const client = new WikimediaClient({ fetch: rec.fetcher, clock, config, cache, userAgent: () => 'Test/1 (+x; y)' });
return { clock, client, ...rec };
}
test('token bucket: burst of 3, then 1 request per second, one in flight per host', async () => {
const { clock, client, calls } = setup();
const jobs = Array.from({ length: 6 }, () => client.request(new URL(API), { lane: 'interactive', timeoutMs: 1000 })
.then(r => r.response.body?.cancel()));
await clock.advance(0);
assert.equal(calls.length, 3);
await clock.advance(999);
assert.equal(calls.length, 3);
await clock.advance(1);
assert.equal(calls.length, 4);
await clock.advance(2000);
await Promise.all(jobs);
assert.deepEqual(calls.map(c => c.at - calls[0].at), [0, 0, 0, 1000, 2000, 3000]);
// Different hosts have separate buckets.
await client.request(new URL(REST), { lane: 'interactive', timeoutMs: 1000 });
assert.equal(calls.at(-1)!.url.hostname, 'wikimedia.org');
});
test('only one request is in flight per host, even with tokens to spare', async () => {
const releases: (() => void)[] = [];
const { clock, client, calls } = setup(() => new Promise(resolve => releases.push(() => resolve(json({})))));
const a = client.request(new URL(API), { lane: 'interactive', timeoutMs: 1000 });
const b = client.request(new URL(API), { lane: 'interactive', timeoutMs: 1000 });
await clock.advance(0);
assert.equal(calls.length, 1);
releases.shift()!();
await a;
await clock.advance(0);
assert.equal(calls.length, 2);
releases.shift()!();
await b;
});
test('priority lanes: mint, then interactive, then background', async () => {
const { clock, client, calls } = setup(url => json({ lane: url.searchParams.get('n') }), { burst: 1 });
await client.request(new URL(`${API}&n=warm`), { lane: 'background', timeoutMs: 1000 }); // Uses the only token.
const order: string[] = [];
const go = (lane: 'mint' | 'interactive' | 'background', n: string) =>
client.request(new URL(`${API}&n=${n}`), { lane, timeoutMs: 1000 }).then(() => order.push(n));
const jobs = [go('background', 'b1'), go('interactive', 'i1'), go('background', 'b2'), go('mint', 'm1'), go('interactive', 'i2')];
await clock.advance(5000);
await Promise.all(jobs);
assert.deepEqual(order, ['m1', 'i1', 'i2', 'b1', 'b2']);
assert.equal(calls.length, 6);
});
test('maxlag=5 is added to Action API requests only, and a maxlag reply waits for Retry-After', async () => {
let lagged = 1;
const { clock, client, calls } = setup(url => url.pathname === '/w/api.php' && lagged-- > 0
? json({ error: { code: 'maxlag' } }, { 'MediaWiki-API-Error': 'maxlag', 'Retry-After': '5' })
: json({ ok: true }));
const job = client.request(new URL(API), { lane: 'background', timeoutMs: 1000 });
await clock.advance(0);
assert.equal(calls.length, 1);
assert.equal(calls[0].url.searchParams.get('maxlag'), '5');
await clock.advance(4999);
assert.equal(calls.length, 1);
await clock.advance(1);
const { response } = await job;
assert.deepEqual(await response.json(), { ok: true });
assert.equal(calls.length, 2);
await client.request(new URL(REST), { lane: 'background', timeoutMs: 1000 });
assert.equal(calls[2].url.searchParams.has('maxlag'), false);
assert.equal(calls[0].headers.get('user-agent'), 'Test/1 (+x; y)');
});
test('429 honours Retry-After by pausing the host; three in a row open the circuit for 10 minutes', async () => {
let status = 429;
const { clock, client, calls } = setup(() => status === 200 ? json({}) : new Response('slow down',
{ status, headers: { 'Retry-After': '2' } }));
const req = (lane: 'mint' | 'background' = 'background') => client.request(new URL(API), { lane, timeoutMs: 1000 });
await assert.rejects(req(), (e: unknown) => e instanceof WikimediaUnavailable && e.status === 429);
// The next request waits out Retry-After (2 s) instead of going straight back.
const second = req().catch(e => e);
await clock.advance(1999);
assert.equal(calls.length, 1);
await clock.advance(1);
assert.equal(((await second) as WikimediaUnavailable).status, 429);
const third = req().catch(e => e);
await clock.advance(10_000);
assert.equal(((await third) as WikimediaUnavailable).status, 429);
assert.equal(calls.length, 3);
assert.equal(client.status().hosts['en.wikipedia.org'].circuitOpen, true);
// Open circuit: refused at once without touching Wikimedia, even on the mint lane.
status = 200;
await assert.rejects(req('mint'), /circuit open/);
assert.equal(calls.length, 3);
await clock.advance(10 * 60_000);
await req('mint');
assert.equal(calls.length, 4);
assert.equal(client.status().counters['breaker.open'], 1);
});
test('a success resets the consecutive-429 count', async () => {
const replies = [429, 200, 429, 200, 429];
const { clock, client } = setup(() => {
const status = replies.shift()!;
return status === 200 ? json({}) : new Response('', { status });
});
for (let i = 0; i < 5; i++) {
const job = client.request(new URL(API), { lane: 'background', timeoutMs: 1000 }).catch(() => {});
await clock.advance(60_000);
await job;
}
assert.equal(client.status().hosts['en.wikipedia.org'].circuitOpen, false);
});
test('daily cap: background stops at 80 %, interactive at 100 %, mint keeps a 10 % reserve; resets at UTC midnight', async () => {
const { clock, client } = setup(() => json({}), { dailyCap: 10, requestsPerSecond: 1000, burst: 1000 });
const req = (lane: 'mint' | 'interactive' | 'background') => client.request(new URL(API), { lane, timeoutMs: 1000 });
for (let i = 0; i < 8; i++) await req('background');
await assert.rejects(req('background'), /daily cap/);
await req('interactive'); await req('interactive');
await assert.rejects(req('interactive'), /daily cap/);
await req('mint');
await assert.rejects(req('mint'), /daily cap/);
await clock.advance(24 * 3600_000);
await req('background');
});
test('nodes sharing one egress IP split the rate and the daily cap', () => {
const config = configFromEnv({ WIKIDECK_WIKIMEDIA_RPS: '0.9', WIKIDECK_WIKIMEDIA_NODES_PER_IP: '3',
WIKIDECK_WIKIMEDIA_DAILY_CAP: '3000' });
const client = new WikimediaClient({ fetch: async () => json({}), config });
assert.ok(Math.abs(client.status().requestsPerSecond - 0.3) < 1e-9);
assert.equal(client.status().dailyCap, 1000);
assert.equal(configFromEnv({ WIKIDECK_WIKIMEDIA_RPS: 'abc' }).requestsPerSecond, 1);
});
test('a full queue sheds background work first and a mint request evicts a background waiter', async () => {
const { clock, client } = setup(() => json({}), { queueLimit: 2, burst: 1 });
await client.request(new URL(API), { lane: 'interactive', timeoutMs: 1000 });
const b1 = client.request(new URL(API), { lane: 'background', timeoutMs: 1000 }).then(() => 'ok', e => e.message);
const b2 = client.request(new URL(API), { lane: 'background', timeoutMs: 1000 }).then(() => 'ok', e => e.message);
await assert.rejects(client.request(new URL(API), { lane: 'background', timeoutMs: 1000 }), /queue full/);
const mint = client.request(new URL(API), { lane: 'mint', timeoutMs: 1000 });
await clock.advance(5000);
await mint;
assert.deepEqual([await b1, await b2], ['ok', 'Wikimedia unavailable: queue full']);
});
test('waiting longer than the lane allows fails instead of queueing forever', async () => {
const { clock, client } = setup(() => json({}), { burst: 1, requestsPerSecond: 0.01, maxWaitMs: { interactive: 3000 } as ClientConfig['maxWaitMs'] });
await client.request(new URL(API), { lane: 'interactive', timeoutMs: 1000 });
const late = client.request(new URL(API), { lane: 'interactive', timeoutMs: 1000 }).catch(e => e);
await clock.advance(3000);
assert.match(String(await late), /queue wait/);
});
test('disk cache: ETag revalidation with If-None-Match, 304 refresh, stale-while-revalidate on the background lane', async () => {
const cache = new WikimediaCache(':memory:');
let body = { v: 1 };
const { clock, client, calls } = setup((_url, headers) => headers.get('if-none-match') === '"v1"' && body.v === 1
? new Response(null, { status: 304 }) : json(body, { ETag: `"v${body.v}"` }), {}, cache);
const policy = { ttlMs: 1000, staleWhileRevalidate: false };
const read = async (lane: 'interactive' | 'background' = 'interactive', cachePolicy = policy) =>
(await (await client.request(new URL(API), { lane, timeoutMs: 1000, cache: cachePolicy })).response.json());
assert.deepEqual(await read(), { v: 1 });
assert.deepEqual(await read(), { v: 1 });
assert.equal(calls.length, 1); // Fresh hit.
await clock.advance(1000);
assert.deepEqual(await read(), { v: 1 });
assert.equal(calls.length, 2);
assert.equal(calls[1].headers.get('if-none-match'), '"v1"'); // Conditional request, answered 304.
assert.equal(client.status().counters['cache.revalidated'], 1);
// Stale-while-revalidate: the stale copy is served at once; the refresh happens on the background lane.
await clock.advance(1000);
body = { v: 2 };
assert.deepEqual(await read('interactive', { ttlMs: 1000, staleWhileRevalidate: true }), { v: 1 });
await clock.advance(0); await flush();
assert.equal(calls.length, 3);
assert.equal(client.status().counters['lane.background'], 1);
assert.deepEqual(await read(), { v: 2 });
assert.equal(calls.length, 3);
// The key ignores maxlag and parameter order.
assert.equal(cacheKey(new URL('https://h.org/p?b=2&a=1&maxlag=5')), cacheKey(new URL('https://h.org/p?a=1&b=2')));
});
test('requests without a cache policy (mint) neither read nor write the cache', async () => {
const cache = new WikimediaCache(':memory:');
const { client, calls } = setup(() => json({ n: calls.length }), {}, cache);
await client.request(new URL(API), { lane: 'interactive', timeoutMs: 1000, cache: { ttlMs: 60_000 } });
const minted = await client.request(new URL(API), { lane: 'mint', timeoutMs: 1000 });
assert.equal(minted.fromCache, false);
assert.equal(calls.length, 2);
assert.equal(cache.size(), 1);
});
test('a stale copy is served when the upstream refuses, and non-JSON replies are never cached', async () => {
const cache = new WikimediaCache(':memory:');
let fail = false, html = false;
const { clock, client } = setup(() => fail ? new Response('', { status: 429 })
: html ? new Response('<html>', { headers: { 'Content-Type': 'text/html' } }) : json({ v: 1 }), {}, cache);
const policy = { ttlMs: 1000 };
await client.request(new URL(API), { lane: 'interactive', timeoutMs: 1000, cache: policy });
await clock.advance(2000);
fail = true;
const stale = await client.request(new URL(API), { lane: 'interactive', timeoutMs: 1000, cache: policy });
assert.equal(stale.fromCache, true);
fail = false; html = true;
await clock.advance(60_000);
await client.request(new URL(`${API}&other=1`), { lane: 'interactive', timeoutMs: 1000, cache: policy });
assert.equal(cache.size(), 1);
});
test('start-up warns when there is no operator contact', () => {
const messages: string[] = [];
assert.equal(warnIfNoContact({}, m => messages.push(m)), true);
assert.match(messages[0], /WIKIDECK_OPERATOR_CONTACT/);
assert.equal(warnIfNoContact({ WIKIDECK_OPERATOR_CONTACT: 'ops@example.org' }, m => messages.push(m)), false);
assert.equal(warnIfNoContact({ WIKIDECK_USER_AGENT: 'X/1 (+https://x; y)' }, m => messages.push(m)), false);
assert.equal(messages.length, 1);
});
test('only HTTPS is allowed', async () => {
const { client } = setup();
await assert.rejects(client.request(new URL('http://en.wikipedia.org/w/api.php'), { lane: 'mint', timeoutMs: 1 }), RangeError);
});
+52
View File
@@ -0,0 +1,52 @@
// Shared fakes for tests/wikimedia-*.test.ts: a manual clock and a recording fetch. Zero network.
import type { Clock } from '../server/wikimedia/client.ts';
export class FakeClock implements Clock {
time: number;
private timers: { at: number; fn: () => void; id: number }[] = [];
private nextId = 1;
constructor(start = Date.parse('2026-11-17T12:00:00Z')) { this.time = start; }
now(): number { return this.time; }
setTimeout(fn: () => void, ms: number): unknown {
const id = this.nextId++;
this.timers.push({ at: this.time + Math.max(0, ms), fn, id });
return id;
}
clearTimeout(handle: unknown): void { this.timers = this.timers.filter(t => t.id !== handle); }
/** Advance time, firing due timers in order and letting promise chains settle between them. */
async advance(ms: number): Promise<void> {
const target = this.time + ms;
await flush();
for (;;) {
this.timers.sort((a, b) => a.at - b.at || a.id - b.id);
const next = this.timers[0];
if (!next || next.at > target) break;
this.timers.shift();
this.time = Math.max(this.time, next.at);
next.fn();
await flush();
}
this.time = target;
await flush();
}
}
export async function flush(): Promise<void> {
for (let i = 0; i < 10; i++) await new Promise(resolve => setImmediate(resolve));
}
export function json(value: unknown, headers: Record<string, string> = {}, status = 200): Response {
return new Response(JSON.stringify(value), { status, headers: { 'Content-Type': 'application/json; charset=utf-8', ...headers } });
}
export type Call = { url: URL; headers: Headers; at: number };
export function recorder(clock: { now(): number }, handler: (url: URL, headers: Headers) => Response | Promise<Response>) {
const calls: Call[] = [];
const fetcher: typeof fetch = async (input, init) => {
const url = new URL(String(input));
const headers = new Headers(init?.headers);
calls.push({ url, headers, at: clock.now() });
return handler(url, headers);
};
return { calls, fetcher };
}
+104
View File
@@ -0,0 +1,104 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { WikipediaEnrichment } from '../server/enrichment.ts';
import { WikipediaArticleSource, randomArticlesUrl } from '../server/wiki.ts';
import { WikimediaClient } from '../server/wikimedia/client.ts';
import { WikimediaCache } from '../server/wikimedia/cache.ts';
import { UNTHROTTLED } from '../server/wikimedia/client.ts';
import { json } from './wikimedia-helpers.ts';
const JANUARY = new Date('2025-01-15T23:59:59Z');
const MONTHS = ['2024-07', '2024-08', '2024-09', '2024-10', '2024-11', '2024-12'];
const metadata = (pageId = 736) => ({ query: { pages: [{ pageid: pageId, ns: 0, title: 'Albert Einstein', lastrevid: 1234,
length: 100_000, pageprops: { wikibase_item: 'Q937' }, categories: [], langlinks: [{ lang: 'de' }] }] } });
const views = (counts = [10, 20, 30, 40, 50, 100]) => ({ items: MONTHS.map((month, i) => ({ timestamp: `${month.replace('-', '')}0100`,
views: counts[i], project: 'en.wikipedia', access: 'all-access', agent: 'user', granularity: 'monthly' })) });
function service(viewsReply: () => Response | Promise<Response>, cache?: WikimediaCache) {
const urls: URL[] = [];
const fetcher: typeof fetch = async input => {
const url = new URL(String(input));
urls.push(url);
return url.hostname === 'en.wikipedia.org' ? json(metadata(Number(url.searchParams.get('pageids')))) : viewsReply();
};
const client = new WikimediaClient({ fetch: fetcher, config: UNTHROTTLED, cache });
return { urls, client, enrichment: new WikipediaEnrichment({ client, now: () => JANUARY }) };
}
test('mint fix: a 429, 5xx, timeout or network failure on pageviews fails the mint read instead of pinning attention: null', async () => {
const failures: (() => Response | Promise<Response>)[] = [
() => new Response('', { status: 429, headers: { 'Retry-After': '0' } }),
() => new Response('', { status: 503 }),
() => new Response('', { status: 500 }),
() => new Response('<html>busy</html>', { headers: { 'Content-Type': 'text/html' } }),
() => { throw new TypeError('fetch failed'); },
() => Promise.reject(new DOMException('The operation timed out.', 'TimeoutError')),
];
for (const failure of failures) {
const { enrichment } = service(failure);
await assert.rejects(enrichment.getFresh(736), /unavailable/);
// Display reads keep their old behaviour: metadata with unavailable views.
const shown = await enrichment.get(736);
assert.equal(shown.pageviews.status, 'unavailable');
assert.equal(shown.stats.attention, null);
}
});
test('mint records unavailable views only for a genuine 404 or missing data', async () => {
for (const reply of [() => json({ type: 'not found' }, {}, 404), () => json({ items: [] }), () => json({ items: views().items.slice(1) })]) {
const { enrichment } = service(reply);
const minted = await enrichment.getFresh(736);
assert.equal(minted.pageviews.status, 'unavailable');
assert.equal(minted.stats.attention, null);
}
const { enrichment } = service(() => json(views()));
assert.equal((await enrichment.getFresh(736)).stats.attention, 33);
});
test('mint reads never use the disk cache or join a display read', async () => {
const cache = new WikimediaCache(':memory:');
let count = 100;
const { enrichment, urls } = service(() => json(views([1, 1, 1, 1, 1, count++])), cache);
const shown = await enrichment.get(736);
assert.equal(cache.size(), 2);
const [display, minted] = await Promise.all([enrichment.get(736), enrichment.getFresh(736)]);
assert.equal(display.pageviews.months[5].views, shown.pageviews.months[5].views);
assert.equal(minted.pageviews.months[5].views, 101); // Fetched again, not the cached 100.
assert.equal(urls.length, 4);
// A later display read (memory cache cleared by a new service) is served from disk: no upstream call.
const second = new WikipediaEnrichment({ client: new WikimediaClient({ fetch: async () => { throw new Error('no network'); },
config: UNTHROTTLED, cache }), now: () => JANUARY });
assert.equal((await second.get(736)).pageviews.months[5].views, 100);
});
test('golden mint URLs: request parameters are byte-identical apart from the added maxlag', async () => {
assert.equal(randomArticlesUrl().toString(), 'https://en.wikipedia.org/w/api.php?action=query&format=json&formatversion=2'
+ '&generator=random&grnnamespace=0&grnlimit=10&prop=extracts%7Cpageimages%7Cinfo%7Ccategories&exintro=1&explaintext=1'
+ '&exchars=900&piprop=thumbnail&pithumbsize=800&cllimit=20');
const urls: string[] = [];
const fetcher: typeof fetch = async input => {
const url = new URL(String(input));
urls.push(url.toString());
if (url.searchParams.get('generator') === 'random') return json({ query: { pages: [{ pageid: 9, ns: 0, title: 'T',
extract: 'An extract that is comfortably longer than forty characters.', lastrevid: 2,
thumbnail: { source: 'https://upload.wikimedia.org/a.jpg' } }] } });
return url.hostname === 'en.wikipedia.org' ? json(metadata(9)) : json(views());
};
const client = new WikimediaClient({ fetch: fetcher, config: UNTHROTTLED });
const article = await new WikipediaArticleSource(client).random();
await new WikipediaEnrichment({ client, now: () => JANUARY }).getFresh(article.pageId);
assert.deepEqual(urls, [
`${randomArticlesUrl()}&maxlag=5`,
'https://en.wikipedia.org/w/api.php?action=query&format=json&formatversion=2&pageids=9'
+ '&prop=info%7Ccategories%7Cpageprops%7Cdescription%7Clanglinks&clshow=%21hidden&cllimit=50&lllimit=500'
+ '&ppprop=wikibase_item&maxlag=5',
'https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia.org/all-access/user/Albert_Einstein/monthly/2024070100/2024120100',
]);
});
test('a throttled upstream stops random() at once instead of retrying three times', async () => {
let calls = 0;
const source = new WikipediaArticleSource(async () => { calls++; return new Response('', { status: 429 }); });
await assert.rejects(source.random(), /HTTP 429/);
assert.equal(calls, 1);
});
+74
View File
@@ -0,0 +1,74 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { MintTemplatePool } from '../server/wikimedia/pool.ts';
import type { Article } from '../shared/protocol.ts';
import type { PageEnrichment } from '../shared/enrichment.ts';
function article(pageId: number): Article {
return { pageId, title: `Page ${pageId}`, extract: 'An extract that is comfortably longer than forty characters.',
image: 'https://upload.wikimedia.org/a.jpg', url: `https://en.wikipedia.org/?curid=${pageId}`, revisionId: pageId + 1,
source: 'live', category: 'Culture' };
}
function enrichment(pageId: number): PageEnrichment {
return { pageId, title: `Page ${pageId}`, description: null, wikidataId: null, revisionId: pageId + 1, pageBytes: 10,
languageCount: 1, languagesTruncated: false, categories: [], categoriesTruncated: false, fetchedAt: '2026-11-17T00:00:00Z',
pageviews: { status: 'available', months: [], latestMonth: '2026-10', total: 1, dailyAverage: 1, trendPercent: null },
stats: { attention: 1, depth: 1, reach: 1 } };
}
function fixture(options: { size?: number; maxAgeMs?: number } = {}) {
let next = 1, now = 0, fail = false;
const log: string[] = [];
const sources = {
articles: { random: async () => { log.push('random'); if (fail) throw new Error('Wikimedia unavailable: HTTP 429'); return article(next++); } },
enrichment: { getFresh: async (id: number) => { log.push(`fresh:${id}`); return enrichment(id); } },
};
const pool = new MintTemplatePool(sources, { now: () => now, ...options });
return { pool, log, advance: (ms: number) => { now += ms; }, setFail: (value: boolean) => { fail = value; } };
}
test('every template on one tip shares one article; a new tip gets a different one', async () => {
const { pool, log } = fixture({ size: 1 });
const first = await Promise.all(Array.from({ length: 20 }, () => pool.take('tip-a')));
assert.ok(first.every(pair => pair.article.pageId === first[0].article.pageId));
await pool.refill();
const requestsForTwentyMiners = log.length;
assert.equal(requestsForTwentyMiners, 4); // One direct pair, one prefetched pair: 2 calls each.
const second = await pool.take('tip-b');
assert.notEqual(second.article.pageId, first[0].article.pageId);
assert.equal(second.article.pageId, 2); // Served from the prefetched pair, no wait.
await pool.refill();
assert.equal(log.length, 6); // ...and that use alone triggers the next prefetch.
// The stats snapshot comes from a mint-path read of the same page.
assert.equal(second.statsSnapshot.revisionId, second.article.pageId + 1);
});
test('pairs older than maxAge are dropped, never served', async () => {
const { pool, advance } = fixture({ size: 1, maxAgeMs: 1000 });
const a = await pool.take('t1');
await pool.refill();
advance(1000);
const b = await pool.take('t1'); // Same tip, but its pair expired; the ready pair expired too.
assert.notEqual(b.article.pageId, a.article.pageId);
assert.equal(pool.status().expired, 1);
});
test('an idle pool makes no requests, and a failure rejects the template without a retry loop', async () => {
const { pool, log, setFail } = fixture({ size: 2 });
assert.equal(log.length, 0);
setFail(true);
await assert.rejects(pool.take('t1'), /429/);
await pool.refill();
assert.equal(log.filter(entry => entry === 'random').length, 2); // The direct attempt and one refill attempt.
setFail(false);
assert.ok((await pool.take('t1')).article.pageId > 0);
});
test('callers get copies they cannot mutate', async () => {
const { pool } = fixture();
const a = await pool.take('t');
a.article.title = 'changed';
assert.equal((await pool.take('t')).article.title.startsWith('Page'), true);
assert.throws(() => new MintTemplatePool({ articles: { random: async () => article(1) },
enrichment: { getFresh: async () => enrichment(1) } }, { size: 99 }), RangeError);
});