forked from WikiDeck/wikideck
Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
9943de3b53 | ||
|
|
fcdd05f2b1 |
@@ -0,0 +1,73 @@
|
||||
# Ownership and collection API (Night 2, #57)
|
||||
|
||||
Server-side checks that a player owns the cards they play. Off-chain only: nothing here moves, burns or locks a card, and nothing changes consensus.
|
||||
|
||||
## The rules (ruleset data)
|
||||
|
||||
| Key | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `deckSize` | 40 | A constructed deck has exactly this many cards. A testnet mode file may lower it. |
|
||||
| `maxCopiesPerPage` | 4 | The MTG-style copy limit. Cards minted from the same Wikipedia `pageId` are separate cards, but at most this many go in one deck. |
|
||||
| `minConfirmations` | 1 | The confirmations each card's mint block needs. Rated play uses `RATED_MIN_CONFIRMATIONS` = 3. |
|
||||
|
||||
## `server/game/ownership.ts` (used by the PvP host)
|
||||
|
||||
```ts
|
||||
verifyAccount(deps, idToken: unknown): Promise<Account>
|
||||
resolveDeck(deps, account, tokenIds: unknown, opts?: Partial<OwnershipRules>): Promise<ChainCard[]>
|
||||
collection(deps, account, opts?: { minConfirmations?: number }): Promise<ChainCard[]>
|
||||
```
|
||||
|
||||
- **`deps`** is `{ ledger, accounts, verifyIdToken }`. In production, `server/testnet/http.ts` builds it from the node's `Ledger`, the shared `AccountStore`, and `accountVerifier()` from `server/account/router.ts`. It passes the result to `gameRouter({ ownership })`.
|
||||
- **`resolveDeck`** accepts token ids only, never card JSON. For each card it checks:
|
||||
- the card is on the canonical chain (`ledger.nft()`);
|
||||
- its current owner (after transfers) is one of the account's linked wallets;
|
||||
- it has enough confirmations;
|
||||
- the deck size and copy limit.
|
||||
|
||||
Each card is rebuilt from its own block (`article` and `statsSnapshot`). The cards come back in the order given.
|
||||
- **Errors** are thrown as `DeckError { code, status, tokenIds, pageId? }`. `tokenIds` lists every offending card of that class.
|
||||
|
||||
| `code` | `status` |
|
||||
|---|---|
|
||||
| `auth` | 401 |
|
||||
| `size` | 400 |
|
||||
| `duplicate-token` | 400 |
|
||||
| `unknown-token` (not on chain, or malformed) | 400 |
|
||||
| `not-owned` (includes "no linked wallet") | 403 |
|
||||
| `unconfirmed` | 400 |
|
||||
| `copy-limit` | 400 |
|
||||
|
||||
`OwnershipUnavailable` (503) means Dex is unreachable or the chain tip kept moving during a read.
|
||||
- **`collection`** pages `ledger.deck()` (24 per page) across every linked wallet at one canonical tip, restarting if the tip moves. There is no 128-block snapshot cap.
|
||||
|
||||
## `GET /api/game/collection`
|
||||
|
||||
`Authorization: Bearer <Dex ID token>` (the same token as `/api/account/me`). Optional: `?minConfirmations=N`.
|
||||
|
||||
```ts
|
||||
{
|
||||
tip, wallets: string[],
|
||||
deckSize: 40, copyLimit: 4, maxCopiesPerPage: 4, minConfirmations: 1,
|
||||
owned: number, // confirmed cards across all linked wallets
|
||||
playable: number, // Σ min(copies, copyLimit) per pageId
|
||||
needed: number, // max(0, deckSize - playable)
|
||||
eligible: boolean, // playable >= deckSize
|
||||
cards: ChainCard[] // tokenId, owner, confirmations, pageId, title, revisionId, url, extract, imageUrl?,
|
||||
// height, blockHash, createdAt, powHash, article, statsSnapshot
|
||||
}
|
||||
```
|
||||
|
||||
| Case | Response |
|
||||
|---|---|
|
||||
| No linked wallets | 200 with `cards: []` and `needed: 40` |
|
||||
| Missing or invalid token | 401 |
|
||||
| Dex unreachable | 503 |
|
||||
| Bad query | 400 |
|
||||
|
||||
The UI shows `playable / deckSize`, for example "23 / 40 cards", and how to mine.
|
||||
|
||||
## Not covered (known gaps)
|
||||
|
||||
- **Peer hosts.** Account stores are per node, and a Dex token sent to a peer could be replayed elsewhere (engineer review §2, G6). This API is for the self-hosted node only. The stretch fix is a wallet-signed deck claim.
|
||||
- **Plausibility of minted stats** (G5): `pageId` → `title` and `revisionId` → `size` are not cross-checked yet.
|
||||
@@ -1,72 +0,0 @@
|
||||
# Polite Wikimedia client (Night 2, issue #62)
|
||||
|
||||
Every server-side Wikimedia request goes through one process-wide `WikimediaClient`
|
||||
(`server/wikimedia/client.ts`). The design comes from engineer review §4 and data review §5.
|
||||
|
||||
## What it does
|
||||
|
||||
| Behaviour | Default |
|
||||
|---|---|
|
||||
| Rate per upstream host (`en.wikipedia.org`, `wikimedia.org`, `www.wikidata.org`) | token bucket, 1 req/s, burst 3 |
|
||||
| Concurrency | 1 request in flight per host |
|
||||
| Priority lanes | `mint` → `interactive` → `background`; the queue holds 64 waiters, and background work is shed first |
|
||||
| Longest queue wait | mint 10 s, interactive 5 s, background 120 s |
|
||||
| `maxlag` | `maxlag=5` on every Action API request; a maxlag reply waits for `Retry-After` and retries up to 2 times |
|
||||
| 429 / 503 | honour `Retry-After` (seconds or HTTP date), else back off 2ⁿ s with jitter, capped at 10 min, by pausing the whole host |
|
||||
| Circuit breaker | 3 consecutive 429s stop that host for 10 min. Requests are refused at once without touching Wikimedia |
|
||||
| Daily cap | 20,000 requests a day per egress IP. Background work stops at 80 %, interactive at 100 %; mint has a 10 % reserve |
|
||||
| Disk cache | `$WIKIDECK_DATA_DIR/wikimedia-cache.sqlite`: ETag / Last-Modified, `If-None-Match`, stale-while-revalidate on the background lane, stale copy served on refusal |
|
||||
| User-Agent | `WikiDeck/<version> (+https://gitea.eom.dev/WikiDeck/wikideck; <operator contact>) node/<version>` |
|
||||
|
||||
The rate is deliberately 1 req/s for **every** host, including the `wikimedia.org` REST (pageviews) host.
|
||||
The engineer review suggested 5 req/s there; we are stricter on purpose after the Night 1 rate limiting, because the
|
||||
budget is per egress IP. Raise it only with a reason and a per-host setting.
|
||||
|
||||
Cache lifetimes: article metadata 24 h, monthly pageviews 7 days (the URL names six closed
|
||||
months, which do not change). Random articles and all mint reads are never cached.
|
||||
|
||||
## Operator settings
|
||||
|
||||
| Variable | Meaning |
|
||||
|---|---|
|
||||
| `WIKIDECK_OPERATOR_CONTACT` | **Set this.** An email address or URL, put in the User-Agent. Without it (and without `PUBLIC_ORIGIN`), the node logs a warning at start-up. Wikimedia policy requires a contact, and contact-less clients are the first to be blocked. |
|
||||
| `PUBLIC_ORIGIN` | Used as the contact when `WIKIDECK_OPERATOR_CONTACT` is unset. |
|
||||
| `WIKIDECK_USER_AGENT` | Overrides the whole User-Agent string (sanitized; kept from Night 1). |
|
||||
| `WIKIDECK_WIKIMEDIA_RPS` | Requests per second per host for the **whole egress IP** (default 1). |
|
||||
| `WIKIDECK_WIKIMEDIA_BURST` | Token-bucket burst (default 3). |
|
||||
| `WIKIDECK_WIKIMEDIA_CONCURRENCY` | In-flight requests per host (default 1). |
|
||||
| `WIKIDECK_WIKIMEDIA_DAILY_CAP` | Requests per UTC day for the whole egress IP (default 20000). |
|
||||
| `WIKIDECK_WIKIMEDIA_NODES_PER_IP` | How many nodes share this egress IP. The rate and the daily cap are divided by it. **The testnet runs several nodes behind one IP: set it to the node count** (e.g. 3 for `deploy/night/compose.lan.yaml`). |
|
||||
|
||||
The testnet's `WIKIDECK_USER_AGENT` already names the project URL, which satisfies the policy.
|
||||
|
||||
Counters (`lane.*`, `http.429`, `breaker.open`, `cache.hit`, `refused.*`, …), per-host pause state and the
|
||||
mint pool's state are in the private operations listener's `/status`, under `wikimedia`.
|
||||
|
||||
## Mint path: transport only
|
||||
|
||||
- `random()` (W1) and `getFresh()` (W2 and W3) keep their request parameters byte-identical. The only
|
||||
addition is `maxlag=5`, which changes only error behaviour. `tests/wikimedia-mint.test.ts` pins the exact URLs.
|
||||
- Mint reads use the `mint` lane and **never** read or write the disk cache, and never join a display read.
|
||||
- **Fixed:** a 429, 5xx, timeout, network error or non-JSON reply on the pageviews call used to mint a card with
|
||||
`attention: null` forever (`enrichment.ts`). Now `getFresh()` rejects, the template returns 503 and the miner
|
||||
retries. `unavailable` is minted only for a genuine 404 (no data) or missing months.
|
||||
- No block, transfer, trade or validation code changed. Validation never calls Wikimedia.
|
||||
|
||||
## Mining prefetch pool (`server/wikimedia/pool.ts`)
|
||||
|
||||
Each template used to cost 3+ requests, all in a burst at each new tip, so load scaled with the number of miners.
|
||||
Now every template on the same tip gets the same fresh article+snapshot pair (miners race for one block anyway),
|
||||
and a new tip takes the next prefetched pair. Load is about one pair, or 3 requests, per block, whatever the number
|
||||
of miners.
|
||||
|
||||
- Pairs come from the mint-path reads above, and the snapshot is stamped when the pair is fetched, so `capturedAt` is true.
|
||||
- Pairs older than 5 minutes are dropped, never served.
|
||||
- The pool (2 pairs) refills only after use, so an idle node makes no requests.
|
||||
|
||||
## Card faces
|
||||
|
||||
`/api/wikipedia/:id` → `get()`: memory LRU, then the disk cache (fresh, or stale with a background
|
||||
revalidation), and only then the interactive lane. A grid of cards therefore no longer means a burst
|
||||
of 2 upstream requests per card per render. `TrendCache` can use `getBackground()` to stay off the
|
||||
interactive lane.
|
||||
@@ -12,12 +12,28 @@ export type AccountDeps = { config?: OidcConfig; keys?: DexKeys; store?: Account
|
||||
type Authed = Request & { account?: Account };
|
||||
const hex64 = z.string().regex(/^[a-f0-9]{64}$/);
|
||||
|
||||
/**
|
||||
* Verifies a Dex ID token and records the account. Shared with the game routes
|
||||
* (server/game/collection.ts, the PvP host) so every route uses the same store and keys.
|
||||
* Throws TokenError (status 401) for a bad token; other errors mean Dex is unreachable.
|
||||
*/
|
||||
export function accountVerifier(config: OidcConfig, keys: Pick<DexKeys, 'key'>, store: Pick<AccountStore, 'upsert'>,
|
||||
now: () => number = Date.now): (idToken: string) => Promise<Account> {
|
||||
return async idToken => {
|
||||
const claims = await verifyIdToken(idToken, keys, config, { now: now() });
|
||||
const account: Account = { sub: claims.sub, issuer: claims.iss, name: displayName(claims), username: claims.preferred_username ?? '' };
|
||||
store.upsert(account);
|
||||
return account;
|
||||
};
|
||||
}
|
||||
|
||||
export function createAccountRouter(deps: AccountDeps = {}): Router {
|
||||
const config = deps.config ?? oidcConfig();
|
||||
const now = deps.now ?? Date.now;
|
||||
const keys = deps.keys ?? new DexKeys(config, deps.fetcher, now);
|
||||
const store = deps.store ?? new AccountStore(resolve(deps.dataDir ?? 'data', 'accounts.sqlite'), now);
|
||||
const router = Router();
|
||||
const verify = accountVerifier(config, keys, store, now);
|
||||
|
||||
const toAccount = (claims: Awaited<ReturnType<typeof verifyIdToken>>): Account =>
|
||||
({ sub: claims.sub, issuer: claims.iss, name: displayName(claims), username: claims.preferred_username ?? '' });
|
||||
@@ -26,8 +42,7 @@ export function createAccountRouter(deps: AccountDeps = {}): Router {
|
||||
try {
|
||||
const m = /^Bearer ([A-Za-z0-9_.-]+)$/.exec(req.get('authorization') ?? '');
|
||||
if (!m) throw new TokenError('Log in with your eom.dev account');
|
||||
req.account = toAccount(await verifyIdToken(m[1], keys, config, { now: now() }));
|
||||
store.upsert(req.account);
|
||||
req.account = await verify(m[1]);
|
||||
next();
|
||||
} catch (e) { next(e); }
|
||||
};
|
||||
|
||||
+27
-72
@@ -1,5 +1,5 @@
|
||||
import { z } from 'zod';
|
||||
import { WikimediaClient, WikimediaUnavailable, wikimediaClient, type CachePolicy, type Lane } from './wikimedia/client.ts';
|
||||
import { ENRICHMENT_USER_AGENT, wikimediaUserAgent } from './wikimedia-user-agent.ts';
|
||||
import { deriveStats, type PageEnrichment } from '../shared/enrichment.ts';
|
||||
|
||||
const MAX_BYTES = 1024 * 1024;
|
||||
@@ -25,23 +25,7 @@ const viewsSchema = z.object({ items: z.array(z.object({
|
||||
})).max(6) });
|
||||
|
||||
type CacheEntry = { expires: number; value: PageEnrichment } | { expires: number; error: Error };
|
||||
type Options = { fetch?: typeof globalThis.fetch; client?: WikimediaClient; now?: () => Date; ttlMs?: number; maxEntries?: number };
|
||||
type Mode = 'mint' | 'display' | 'background';
|
||||
|
||||
/** Disk-cache lifetimes (engineer review §4). Mint reads never use the cache. */
|
||||
const METADATA_CACHE: CachePolicy = { ttlMs: 24 * 3600_000, staleWhileRevalidate: true, maxBytes: MAX_BYTES };
|
||||
// The URL names its six completed months, which are immutable; a week bounds late corrections.
|
||||
const PAGEVIEWS_CACHE: CachePolicy = { ttlMs: 7 * 24 * 3600_000, staleWhileRevalidate: true, maxBytes: MAX_BYTES };
|
||||
const NOT_FOUND = Symbol('not found');
|
||||
|
||||
/** The upstream did not answer properly (throttled, 5xx, timeout, network, non-JSON). Not "no data". */
|
||||
export class UpstreamFailure extends Error {
|
||||
constructor(message: string) { super(message); this.name = 'UpstreamFailure'; }
|
||||
}
|
||||
function transportFailure(error: unknown): boolean {
|
||||
return error instanceof WikimediaUnavailable || error instanceof UpstreamFailure ||
|
||||
(error instanceof DOMException && (error.name === 'TimeoutError' || error.name === 'AbortError'));
|
||||
}
|
||||
type Options = { fetch?: typeof globalThis.fetch; now?: () => Date; ttlMs?: number; maxEntries?: number };
|
||||
|
||||
function monthWindow(now: Date): string[] {
|
||||
return Array.from({ length: 6 }, (_, index) =>
|
||||
@@ -50,17 +34,16 @@ function monthWindow(now: Date): string[] {
|
||||
|
||||
/** Fixed-source read-only enrichment; no consensus, persistence, URL or header inputs. */
|
||||
export class WikipediaEnrichment {
|
||||
private readonly client: WikimediaClient;
|
||||
private readonly fetcher: typeof globalThis.fetch;
|
||||
private readonly now: () => Date;
|
||||
private readonly ttlMs: number;
|
||||
private readonly maxEntries: number;
|
||||
private readonly cache = new Map<number, CacheEntry>();
|
||||
private readonly pending = new Map<string, Promise<PageEnrichment>>();
|
||||
private readonly pending = new Map<number, Promise<PageEnrichment>>();
|
||||
private active = 0;
|
||||
private mintActive = 0;
|
||||
|
||||
constructor(options: Options = {}) {
|
||||
this.client = options.client ?? (options.fetch ? WikimediaClient.unthrottled(options.fetch) : wikimediaClient);
|
||||
this.fetcher = options.fetch ?? globalThis.fetch;
|
||||
this.now = options.now ?? (() => new Date());
|
||||
this.ttlMs = options.ttlMs ?? 6 * 60 * 60 * 1000;
|
||||
this.maxEntries = options.maxEntries ?? 128;
|
||||
@@ -70,27 +53,16 @@ export class WikipediaEnrichment {
|
||||
}
|
||||
}
|
||||
|
||||
/** Card faces (`/api/wikipedia/:id`): memory, then the disk cache, then the interactive lane. */
|
||||
async get(pageId: number): Promise<PageEnrichment> {
|
||||
return this.loadCached(pageId, 'display');
|
||||
return this.loadCached(pageId, false);
|
||||
}
|
||||
|
||||
/** Like get(), on the background lane: for trends and other work nobody is waiting on. */
|
||||
async getBackground(pageId: number): Promise<PageEnrichment> {
|
||||
return this.loadCached(pageId, 'background');
|
||||
}
|
||||
|
||||
/**
|
||||
* Mint data: bypasses every cache and uses the mint lane. If pageviews fail in transport
|
||||
* (429, 5xx, timeout, throttled), this rejects so the template fails and the miner retries;
|
||||
* `unavailable` is recorded only for a genuine 404 or missing data.
|
||||
*/
|
||||
/** Bypass cached values when the result is being pinned into a newly minted block. */
|
||||
async getFresh(pageId: number): Promise<PageEnrichment> {
|
||||
return this.loadCached(pageId, 'mint');
|
||||
return this.loadCached(pageId, true);
|
||||
}
|
||||
|
||||
private async loadCached(pageId: number, mode: Mode): Promise<PageEnrichment> {
|
||||
const fresh = mode === 'mint';
|
||||
private async loadCached(pageId: number, fresh: boolean): Promise<PageEnrichment> {
|
||||
if (!Number.isSafeInteger(pageId) || pageId <= 0) throw new RangeError('Invalid Wikipedia page ID');
|
||||
const now = this.now();
|
||||
const cached = this.cache.get(pageId);
|
||||
@@ -101,29 +73,23 @@ export class WikipediaEnrichment {
|
||||
if ('error' in cached) throw cached.error;
|
||||
return structuredClone(cached.value);
|
||||
}
|
||||
if (!fresh) this.cache.delete(pageId);
|
||||
// Mint jobs never join a display job: that one may be served from the disk cache.
|
||||
const pendingKey = `${fresh ? 'mint' : 'cached'}:${pageId}`;
|
||||
const existing = this.pending.get(pendingKey);
|
||||
this.cache.delete(pageId);
|
||||
const existing = this.pending.get(pageId);
|
||||
if (existing) return structuredClone(await existing);
|
||||
// Reject immediately: no unbounded waiting queue. Four active jobs also keep
|
||||
// the pending-map ceiling of eight conservative (one upstream request/job).
|
||||
// Mint jobs have their own small allowance so card faces can never starve a template.
|
||||
if (fresh ? this.mintActive >= 2 : this.pending.size - this.mintActive >= 8 || this.active >= 4) {
|
||||
throw new Error('Wikipedia enrichment busy');
|
||||
}
|
||||
if (fresh) this.mintActive++; else this.active++;
|
||||
const job = this.load(pageId, now, mode).then(value => {
|
||||
if (this.pending.size >= 8 || this.active >= 4) throw new Error('Wikipedia enrichment busy');
|
||||
this.active++;
|
||||
const job = this.load(pageId, now).then(value => {
|
||||
const ttl = value.pageviews.status === 'available' ? this.ttlMs : Math.min(this.ttlMs, 5 * 60 * 1000);
|
||||
this.remember(pageId, { value, expires: this.now().getTime() + ttl });
|
||||
return value;
|
||||
}, () => {
|
||||
const error = new Error('Wikipedia page enrichment unavailable');
|
||||
// A failed mint read says nothing new about display data; keep any cached value.
|
||||
if (!fresh) this.remember(pageId, { error, expires: this.now().getTime() + 30_000 });
|
||||
this.remember(pageId, { error, expires: this.now().getTime() + 30_000 });
|
||||
throw error;
|
||||
}).finally(() => { if (fresh) this.mintActive--; else this.active--; this.pending.delete(pendingKey); });
|
||||
this.pending.set(pendingKey, job);
|
||||
}).finally(() => { this.active--; this.pending.delete(pageId); });
|
||||
this.pending.set(pageId, job);
|
||||
return structuredClone(await job);
|
||||
}
|
||||
|
||||
@@ -133,19 +99,16 @@ export class WikipediaEnrichment {
|
||||
while (this.cache.size > this.maxEntries) this.cache.delete(this.cache.keys().next().value!);
|
||||
}
|
||||
|
||||
private async json(url: URL, lane: Lane, cache: CachePolicy | undefined, allowNotFound = false): Promise<unknown> {
|
||||
const { response, signal } = await this.client.request(url, { lane, timeoutMs: 6000, cache });
|
||||
if (allowNotFound && response.status === 404) { await response.body?.cancel(); return NOT_FOUND; }
|
||||
if (!response.ok || response.redirected) {
|
||||
await response.body?.cancel();
|
||||
throw new UpstreamFailure(`Upstream HTTP ${response.status}`);
|
||||
}
|
||||
private async json(url: URL): Promise<unknown> {
|
||||
const signal = AbortSignal.timeout(6000);
|
||||
const response = await this.fetcher(url, { signal, redirect: 'error',
|
||||
headers: { Accept: 'application/json', 'User-Agent': wikimediaUserAgent(ENRICHMENT_USER_AGENT) } });
|
||||
const length = response.headers.get('content-length');
|
||||
if (
|
||||
if (!response.ok || response.redirected ||
|
||||
!/^application\/json(?:\s*;|\s*$)/i.test(response.headers.get('content-type') ?? '') ||
|
||||
(length !== null && (!/^\d+$/.test(length) || Number(length) > MAX_BYTES)) || !response.body) {
|
||||
await response.body?.cancel();
|
||||
throw new UpstreamFailure('Invalid upstream response');
|
||||
throw new Error('Invalid upstream response');
|
||||
}
|
||||
const reader = response.body.getReader();
|
||||
const chunks: Uint8Array[] = [];
|
||||
@@ -167,14 +130,12 @@ export class WikipediaEnrichment {
|
||||
}
|
||||
}
|
||||
|
||||
private async load(pageId: number, now: Date, mode: Mode): Promise<PageEnrichment> {
|
||||
const lane: Lane = mode === 'mint' ? 'mint' : mode === 'display' ? 'interactive' : 'background';
|
||||
const fresh = mode === 'mint';
|
||||
private async load(pageId: number, now: Date): Promise<PageEnrichment> {
|
||||
const url = new URL('https://en.wikipedia.org/w/api.php');
|
||||
url.search = new URLSearchParams({ action: 'query', format: 'json', formatversion: '2',
|
||||
pageids: String(pageId), prop: 'info|categories|pageprops|description|langlinks',
|
||||
clshow: '!hidden', cllimit: '50', lllimit: '500', ppprop: 'wikibase_item' }).toString();
|
||||
const metadata = metadataSchema.parse(await this.json(url, lane, fresh ? undefined : METADATA_CACHE));
|
||||
const metadata = metadataSchema.parse(await this.json(url));
|
||||
const page = metadata.query.pages[0];
|
||||
if (page.pageid !== pageId) throw new Error('Wikipedia page ID mismatch');
|
||||
const months = monthWindow(now);
|
||||
@@ -185,9 +146,7 @@ export class WikipediaEnrichment {
|
||||
const title = encodeURIComponent(page.title.replace(/ /g, '_'));
|
||||
const stamp = (month: string) => `${month.replace('-', '')}0100`;
|
||||
const endpoint = new URL(`https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia.org/all-access/user/${title}/monthly/${stamp(months[0])}/${stamp(latestMonth)}`);
|
||||
const raw = await this.json(endpoint, lane, fresh ? undefined : PAGEVIEWS_CACHE, true);
|
||||
if (raw === NOT_FOUND) throw new Error('No pageview data');
|
||||
const result = viewsSchema.parse(raw);
|
||||
const result = viewsSchema.parse(await this.json(endpoint));
|
||||
const byMonth = new Map<string, number>();
|
||||
for (const item of result.items) {
|
||||
const month = `${item.timestamp.slice(0, 4)}-${item.timestamp.slice(4, 6)}`;
|
||||
@@ -202,11 +161,7 @@ export class WikipediaEnrichment {
|
||||
const days = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), 0)).getUTCDate();
|
||||
pageviews = { status: 'available', months: ordered, latestMonth, total,
|
||||
dailyAverage: latest / days, trendPercent: prior > 0 ? (latest - prior) / prior * 100 : null };
|
||||
} catch (error) {
|
||||
// Mint: a throttled or failed upstream must not pin `attention: null` into a block forever.
|
||||
if (fresh && transportFailure(error)) throw error;
|
||||
/* Metadata remains useful; unknown views must not become fake zeroes. */
|
||||
}
|
||||
} catch { /* Metadata remains useful; unknown views must not become fake zeroes. */ }
|
||||
return { pageId, title: page.title, description: page.description ?? null,
|
||||
wikidataId: page.pageprops?.wikibase_item ?? null, revisionId: page.lastrevid, pageBytes: page.length,
|
||||
languageCount: page.langlinks?.length ?? 0, languagesTruncated: metadata.continue?.llcontinue !== undefined,
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
/**
|
||||
* GET /api/game/collection Authorization: Bearer <Dex ID token> [?minConfirmations=N]
|
||||
*
|
||||
* Every card the account's linked wallets own (no 128-block cap) and whether that is enough to
|
||||
* build a deck. Eligibility counts the copy limit: 40 copies of one article is 4 playable cards.
|
||||
*/
|
||||
import { Router, type Request, type Response } from 'express';
|
||||
import { DeckError, collectionAt, ownershipRules, playableCount, verifyAccount,
|
||||
type ChainCard, type OwnershipDeps, type OwnershipRules } from './ownership.ts';
|
||||
|
||||
export type CollectionResponse = {
|
||||
tip: string;
|
||||
wallets: string[];
|
||||
deckSize: number;
|
||||
/** Max copies per pageId. `maxCopiesPerPage` is the same value under the ruleset's name. */
|
||||
copyLimit: number;
|
||||
maxCopiesPerPage: number;
|
||||
minConfirmations: number;
|
||||
/** Confirmed cards across all linked wallets. */
|
||||
owned: number;
|
||||
/** Σ min(copies, copyLimit) per pageId: the most cards a legal deck can draw from. */
|
||||
playable: number;
|
||||
/** max(0, deckSize - playable). */
|
||||
needed: number;
|
||||
/** playable >= deckSize. */
|
||||
eligible: boolean;
|
||||
/** `duel` is present when the server has a frozen profile for the card (see collectionRouter's `duelCard`). */
|
||||
cards: (ChainCard & { duel?: unknown })[];
|
||||
};
|
||||
|
||||
export function collectionResponse(deps: Pick<OwnershipDeps, 'ledger' | 'accounts'>, sub: string,
|
||||
rules?: Partial<OwnershipRules>): CollectionResponse {
|
||||
const r = ownershipRules(rules);
|
||||
const { tip, wallets, cards } = collectionAt(deps, { sub }, { minConfirmations: r.minConfirmations });
|
||||
const playable = playableCount(cards, r.maxCopiesPerPage);
|
||||
return { tip, wallets, deckSize: r.deckSize, copyLimit: r.maxCopiesPerPage, maxCopiesPerPage: r.maxCopiesPerPage,
|
||||
minConfirmations: r.minConfirmations, owned: cards.length, playable, needed: Math.max(0, r.deckSize - playable),
|
||||
eligible: playable >= r.deckSize, cards };
|
||||
}
|
||||
|
||||
export type CollectionRouterDeps = OwnershipDeps & {
|
||||
rules?: Partial<OwnershipRules>;
|
||||
/**
|
||||
* Optional: the same chain card → duel card function the PvP host uses, so the browser shows and
|
||||
* plays exactly what PvP would. Return undefined while a card's profile is pending; it must not fetch.
|
||||
*/
|
||||
duelCard?: (card: ChainCard) => unknown;
|
||||
};
|
||||
|
||||
export function collectionRouter(deps: CollectionRouterDeps): Router {
|
||||
const router = Router();
|
||||
router.get('/', async (req: Request, res: Response) => {
|
||||
res.setHeader('Cache-Control', 'no-store');
|
||||
const q = req.query.minConfirmations;
|
||||
if (q !== undefined && (typeof q !== 'string' || !/^[1-9]\d{0,3}$/.test(q))) { res.status(400).json({ error: 'Invalid minConfirmations' }); return; }
|
||||
const m = /^Bearer ([A-Za-z0-9_.-]{1,8192})$/.exec(req.get('authorization') ?? '');
|
||||
try {
|
||||
const account = await verifyAccount(deps, m?.[1]);
|
||||
const body = collectionResponse(deps, account.sub, { ...deps.rules, ...(q ? { minConfirmations: Number(q) } : {}) });
|
||||
if (deps.duelCard) {
|
||||
const toDuel = deps.duelCard;
|
||||
body.cards = body.cards.map(c => {
|
||||
let duel: unknown;
|
||||
try { duel = toDuel(c); } catch { duel = undefined; }
|
||||
return duel === undefined ? c : { ...c, duel };
|
||||
});
|
||||
}
|
||||
res.json(body);
|
||||
} catch (e) {
|
||||
const err = e as Error & { status?: number };
|
||||
const status = e instanceof DeckError ? err.status! : err.status === 503 ? 503 : 500;
|
||||
res.status(status).json({ error: status === 500 ? 'Collection unavailable' : err.message });
|
||||
}
|
||||
});
|
||||
return router;
|
||||
}
|
||||
@@ -12,11 +12,14 @@ import { createGameRouter } from './router.ts';
|
||||
import { MatchStore } from './store.ts';
|
||||
import type { EnginePort } from './engine-port.ts';
|
||||
import { TrendCache } from './trends.ts';
|
||||
import type { OwnershipDeps } from './ownership.ts';
|
||||
|
||||
type Enrichment = { get(pageId: number): Promise<{ pageviews: { trendPercent: number | null } }> };
|
||||
|
||||
export async function gameRouter(options: { dataDir: string | null; enrichment?: Enrichment;
|
||||
engine?: EnginePort<any, any, any, any>; env?: NodeJS.ProcessEnv; warmTrends?: boolean; handleSignals?: boolean }): Promise<express.Router> {
|
||||
engine?: EnginePort<any, any, any, any>; env?: NodeJS.ProcessEnv; warmTrends?: boolean; handleSignals?: boolean;
|
||||
/** Server-side ownership (#57) for ./ownership.ts verifyAccount/resolveDeck/collection; absent in Night 1 tests. */
|
||||
ownership?: OwnershipDeps }): Promise<express.Router> {
|
||||
const env = options.env ?? process.env;
|
||||
const engine = options.engine ?? (await import('./engine-binding.ts')).engineBinding;
|
||||
const modes = loadModes(env);
|
||||
|
||||
@@ -0,0 +1,211 @@
|
||||
/**
|
||||
* Server-side ownership for constructed play (Night 2, #57). Signature agreed with host-pvp (#56).
|
||||
*
|
||||
* The host never trusts card data from a client. A deck arrives as token ids only; each one
|
||||
* must be on the canonical chain, owned by one of the account's linked wallets, and confirmed
|
||||
* enough. The card is then rebuilt from its own block (article + mint stats snapshot).
|
||||
*
|
||||
* Off-chain only: nothing here moves, burns or locks a card, and nothing touches consensus.
|
||||
* No Wikipedia calls; the only I/O is the injected ID-token verifier (Dex JWKS, cached).
|
||||
*/
|
||||
import type { Account } from '../account/store.ts';
|
||||
import type { Article } from '../../shared/protocol.ts';
|
||||
import type { MintStatsSnapshot } from '../../shared/enrichment.ts';
|
||||
import type { TestnetBlock } from '../../shared/testnet.ts';
|
||||
|
||||
export type OwnershipRules = {
|
||||
/** Exact deck size for constructed play. Production is 40; a testnet mode file may lower it. */
|
||||
deckSize: number;
|
||||
/** MTG-style copy limit: cards minted from the same Wikipedia pageId. */
|
||||
maxCopiesPerPage: number;
|
||||
/** Minimum confirmations of each card's mint block (1 casual, 3 rated). */
|
||||
minConfirmations: number;
|
||||
};
|
||||
export const DEFAULT_OWNERSHIP_RULES: Readonly<OwnershipRules> = Object.freeze({ deckSize: 40, maxCopiesPerPage: 4, minConfirmations: 1 });
|
||||
export const RATED_MIN_CONFIRMATIONS = 3;
|
||||
/** Upper bound on any deck request we will even look at (a deck is 40; this only guards the loop). */
|
||||
export const MAX_DECK_TOKENS = 200;
|
||||
/** Guard against an unbounded collection. Far above anything the chain can mint for one player soon. */
|
||||
export const MAX_COLLECTION = 20_000;
|
||||
/** server/testnet/ledger.ts MAX_DECK_PAGE. */
|
||||
const PAGE = 24;
|
||||
/** A tip change mid-scan restarts the scan; give up after this many rather than loop forever. */
|
||||
const MAX_RESTARTS = 4;
|
||||
|
||||
// Structural: server/testnet/ledger.ts `Ledger` and server/account/store.ts `AccountStore`
|
||||
// satisfy these, and tests pass in-memory fixtures.
|
||||
export type OwnershipLedger = {
|
||||
nft(tokenId: string): { block: TestnetBlock; owner: string; sequence: number; confirmations: number } | undefined;
|
||||
deck(address: string, offset: number, limit: number): {
|
||||
tip: string; total: number; next: number | null;
|
||||
nfts: { block: TestnetBlock; sequence: number; confirmations: number }[];
|
||||
};
|
||||
summary(): { tip: string; height: number };
|
||||
};
|
||||
export type OwnershipAccounts = { wallets(sub: string): { address: string }[]; upsert?(a: Account): void };
|
||||
export type OwnershipDeps = {
|
||||
ledger: OwnershipLedger;
|
||||
accounts: OwnershipAccounts;
|
||||
/** server/account/router.ts accountVerifier(); throws an error with status 401 for a bad token. */
|
||||
verifyIdToken(idToken: string): Promise<Account>;
|
||||
};
|
||||
|
||||
/** One owned NFT as built from its canonical block (never from client JSON). */
|
||||
export type ChainCard = {
|
||||
tokenId: string;
|
||||
/** Current owner (after transfers), one of the account's linked wallets. */
|
||||
owner: string;
|
||||
confirmations: number;
|
||||
pageId: number;
|
||||
title: string;
|
||||
revisionId: number;
|
||||
url: string;
|
||||
extract: string;
|
||||
imageUrl?: string;
|
||||
height: number;
|
||||
blockHash: string;
|
||||
createdAt: number;
|
||||
powHash: string;
|
||||
article: Article;
|
||||
statsSnapshot: MintStatsSnapshot;
|
||||
};
|
||||
|
||||
export type DeckErrorCode = 'auth' | 'size' | 'duplicate-token' | 'unknown-token' | 'not-owned' | 'unconfirmed' | 'copy-limit';
|
||||
const STATUS: Record<DeckErrorCode, 400 | 401 | 403> = {
|
||||
auth: 401, size: 400, 'duplicate-token': 400, 'unknown-token': 400, 'not-owned': 403, unconfirmed: 400, 'copy-limit': 400,
|
||||
};
|
||||
export class DeckError extends Error {
|
||||
readonly code: DeckErrorCode;
|
||||
/** Every offending token of this class, so the UI can highlight them. */
|
||||
readonly tokenIds: string[];
|
||||
readonly status: 400 | 401 | 403;
|
||||
readonly pageId?: number;
|
||||
constructor(code: DeckErrorCode, message: string, tokenIds: string[] = [], pageId?: number) {
|
||||
super(message);
|
||||
this.name = 'DeckError';
|
||||
this.code = code; this.tokenIds = tokenIds; this.status = STATUS[code];
|
||||
if (pageId !== undefined) this.pageId = pageId;
|
||||
}
|
||||
}
|
||||
/** Thrown when Dex or the ledger cannot answer right now (not the player's fault). */
|
||||
export class OwnershipUnavailable extends Error { readonly status = 503; }
|
||||
|
||||
const TOKEN = /^[a-f0-9]{64}$/;
|
||||
const isInt = (n: unknown, min: number, max: number): n is number => Number.isSafeInteger(n) && (n as number) >= min && (n as number) <= max;
|
||||
|
||||
/** Merges partial rules over the defaults and rejects nonsense (mode files are data, so validate). */
|
||||
export function ownershipRules(rules: Partial<OwnershipRules> = {}): OwnershipRules {
|
||||
const defined = Object.fromEntries(Object.entries(rules).filter(([, v]) => v !== undefined)) as Partial<OwnershipRules>;
|
||||
const r = { ...DEFAULT_OWNERSHIP_RULES, ...defined };
|
||||
if (!isInt(r.deckSize, 1, MAX_DECK_TOKENS)) throw new RangeError(`deckSize must be 1..${MAX_DECK_TOKENS}`);
|
||||
if (!isInt(r.maxCopiesPerPage, 1, MAX_DECK_TOKENS)) throw new RangeError(`maxCopiesPerPage must be 1..${MAX_DECK_TOKENS}`);
|
||||
if (!isInt(r.minConfirmations, 1, 1_000_000)) throw new RangeError('minConfirmations must be >= 1');
|
||||
return r;
|
||||
}
|
||||
|
||||
/** Builds the card from the canonical block. `owner` is the current owner from the index, not the miner. */
|
||||
export function cardFromChain(nft: { block: TestnetBlock; confirmations: number }, owner: string): ChainCard {
|
||||
const b = structuredClone(nft.block);
|
||||
const card: ChainCard = { tokenId: b.tokenId, owner, confirmations: nft.confirmations,
|
||||
pageId: b.article.pageId, title: b.article.title, revisionId: b.article.revisionId, url: b.article.url, extract: b.article.extract,
|
||||
height: b.height, blockHash: b.hash, createdAt: b.createdAt, powHash: b.powHash, article: b.article, statsSnapshot: b.statsSnapshot };
|
||||
if (b.article.image) card.imageUrl = b.article.image;
|
||||
return card;
|
||||
}
|
||||
|
||||
/** Verifies a Dex ID token from a request body. Throws DeckError('auth') or OwnershipUnavailable. */
|
||||
export async function verifyAccount(deps: Pick<OwnershipDeps, 'verifyIdToken'>, idToken: unknown): Promise<Account> {
|
||||
if (typeof idToken !== 'string' || !idToken || idToken.length > 8192) throw new DeckError('auth', 'Log in with your eom.dev account to play with your cards');
|
||||
try { return await deps.verifyIdToken(idToken); }
|
||||
catch (e) {
|
||||
if ((e as { status?: number }).status === 401) throw new DeckError('auth', (e as Error).message || 'Log in again with your eom.dev account');
|
||||
throw new OwnershipUnavailable('eom.dev login is temporarily unavailable');
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Verifies a constructed deck for an already-verified account. Resolves to the cards in the
|
||||
* order given, or throws the first class of DeckError found (listing every offending token).
|
||||
*/
|
||||
export async function resolveDeck(deps: Pick<OwnershipDeps, 'ledger' | 'accounts'>, account: Pick<Account, 'sub'>, tokenIds: unknown,
|
||||
opts?: Partial<OwnershipRules>): Promise<ChainCard[]> {
|
||||
const r = ownershipRules(opts);
|
||||
if (!Array.isArray(tokenIds) || tokenIds.length > MAX_DECK_TOKENS) throw new DeckError('size', `A deck is exactly ${r.deckSize} cards`);
|
||||
const invalid = tokenIds.filter(t => typeof t !== 'string' || !TOKEN.test(t)).map(t => String(t).slice(0, 80)).slice(0, 10);
|
||||
if (invalid.length) throw new DeckError('unknown-token', 'Invalid card identifier', invalid);
|
||||
const ids = tokenIds as string[];
|
||||
if (ids.length !== r.deckSize) throw new DeckError('size', `A deck is exactly ${r.deckSize} cards (got ${ids.length})`);
|
||||
const seen = new Set<string>(), dupes = new Set<string>();
|
||||
for (const id of ids) (seen.has(id) ? dupes : seen).add(id);
|
||||
if (dupes.size) throw new DeckError('duplicate-token', 'The same card appears more than once', [...dupes]);
|
||||
|
||||
const wallets = new Set(deps.accounts.wallets(account.sub).map(w => w.address));
|
||||
if (!wallets.size) throw new DeckError('not-owned', 'Link a wallet to your eom.dev account to play with your cards', ids);
|
||||
|
||||
const cards: ChainCard[] = [], missing: string[] = [], foreign: string[] = [], young: string[] = [];
|
||||
for (const id of ids) {
|
||||
const nft = deps.ledger.nft(id);
|
||||
if (!nft || nft.block.tokenId !== id) { missing.push(id); continue; }
|
||||
if (!wallets.has(nft.owner)) { foreign.push(id); continue; }
|
||||
if (nft.confirmations < r.minConfirmations) { young.push(id); continue; }
|
||||
cards.push(cardFromChain(nft, nft.owner));
|
||||
}
|
||||
if (missing.length) throw new DeckError('unknown-token', 'Card not found on the canonical chain', missing);
|
||||
if (foreign.length) throw new DeckError('not-owned', 'Card is not owned by a wallet linked to this account', foreign);
|
||||
if (young.length) throw new DeckError('unconfirmed', `Card needs ${r.minConfirmations} confirmation${r.minConfirmations === 1 ? '' : 's'}`, young);
|
||||
|
||||
const byPage = new Map<number, string[]>();
|
||||
for (const c of cards) byPage.set(c.pageId, [...(byPage.get(c.pageId) ?? []), c.tokenId]);
|
||||
for (const [pageId, tokens] of byPage)
|
||||
if (tokens.length > r.maxCopiesPerPage) throw new DeckError('copy-limit', `At most ${r.maxCopiesPerPage} copies of the same article`, tokens, pageId);
|
||||
return cards;
|
||||
}
|
||||
|
||||
/**
|
||||
* Every card the account owns across its linked wallets, read at one canonical tip (restarting
|
||||
* if the tip moves), with no snapshot cap. Only cards with `minConfirmations` (default 1).
|
||||
*/
|
||||
export async function collection(deps: Pick<OwnershipDeps, 'ledger' | 'accounts'>, account: Pick<Account, 'sub'>,
|
||||
opts: { minConfirmations?: number } = {}): Promise<ChainCard[]> {
|
||||
return collectionAt(deps, account, opts).cards;
|
||||
}
|
||||
|
||||
/** collection() plus the tip and wallets it was read at. Synchronous. */
|
||||
export function collectionAt(deps: Pick<OwnershipDeps, 'ledger' | 'accounts'>, account: Pick<Account, 'sub'>,
|
||||
opts: { minConfirmations?: number } = {}): { tip: string; wallets: string[]; cards: ChainCard[] } {
|
||||
const minConfirmations = ownershipRules({ minConfirmations: opts.minConfirmations }).minConfirmations;
|
||||
const wallets = [...new Set(deps.accounts.wallets(account.sub).map(w => w.address))];
|
||||
for (let attempt = 0; attempt <= MAX_RESTARTS; attempt++) {
|
||||
const tip = deps.ledger.summary().tip;
|
||||
const cards = scan(deps.ledger, wallets, tip, minConfirmations);
|
||||
if (cards) return { tip, wallets, cards };
|
||||
}
|
||||
throw new OwnershipUnavailable('The chain is moving too fast to read a consistent collection; try again');
|
||||
}
|
||||
|
||||
/** Returns null if the tip moved during the scan. */
|
||||
function scan(ledger: OwnershipLedger, wallets: string[], tip: string, minConfirmations: number): ChainCard[] | null {
|
||||
const cards: ChainCard[] = [];
|
||||
for (const address of wallets) {
|
||||
let offset: number | null = 0;
|
||||
while (offset !== null) {
|
||||
const page = ledger.deck(address, offset, PAGE);
|
||||
if (page.tip !== tip) return null;
|
||||
for (const nft of page.nfts) {
|
||||
if (nft.confirmations >= minConfirmations) cards.push(cardFromChain(nft, address));
|
||||
if (cards.length > MAX_COLLECTION) throw new OwnershipUnavailable('Collection too large to list');
|
||||
}
|
||||
// Defensive: a page that makes no progress would loop forever.
|
||||
offset = page.next !== null && page.next > offset ? page.next : null;
|
||||
}
|
||||
}
|
||||
return cards;
|
||||
}
|
||||
|
||||
/** How many of these cards a legal deck could use under the copy limit: Σ min(copies, limit) per pageId. */
|
||||
export function playableCount(cards: Pick<ChainCard, 'pageId'>[], maxCopiesPerPage: number): number {
|
||||
const perPage = new Map<number, number>();
|
||||
for (const c of cards) perPage.set(c.pageId, (perPage.get(c.pageId) ?? 0) + 1);
|
||||
let n = 0; for (const count of perPage.values()) n += Math.min(count, maxCopiesPerPage);
|
||||
return n;
|
||||
}
|
||||
+16
-9
@@ -11,12 +11,16 @@ import { RandomX } from './randomx.ts';
|
||||
import { Ledger } from './ledger.ts';
|
||||
import { PeerNetwork } from './peers.ts';
|
||||
import { wikipediaEnrichment } from '../enrichment.ts';
|
||||
import { configureWikimedia, mintPool, wikimediaStatus } from '../wikimedia/index.ts';
|
||||
import { snapshotStats } from '../../shared/enrichment.ts';
|
||||
import { wikipediaArticles } from '../wiki.ts';
|
||||
import { IS_MAINNET, PROFILE } from '../../shared/network.ts';
|
||||
import { assertLaunchAllowed, launchManifest } from '../launch.ts';
|
||||
import { resourceConfig, RpcBudget, RpcError } from '../resources.ts';
|
||||
import { trustedProxies, operationsPort, startOperations } from '../operations.ts';
|
||||
import { createAccountRouter } from '../account/router.ts';
|
||||
import { accountVerifier, createAccountRouter } from '../account/router.ts';
|
||||
import { AccountStore } from '../account/store.ts';
|
||||
import { DexKeys, oidcConfig } from '../account/oidc.ts';
|
||||
import { collectionRouter } from '../game/collection.ts';
|
||||
|
||||
const root = fileURLToPath(new URL('../../', import.meta.url));
|
||||
assertLaunchAllowed();
|
||||
@@ -27,7 +31,6 @@ const apiBudget = new RpcBudget(limits);
|
||||
const peerBudget = new RpcBudget({ ...limits, requestsPerSecond: 40, writesPerSecond: 10, active: 16 });
|
||||
const dataDir = process.env.WIKIDECK_DATA_DIR ?? resolve(root,'data');
|
||||
await mkdir(dataDir,{recursive:true});
|
||||
configureWikimedia({dataDir});
|
||||
// Never permit two processes to rewrite the same ledger. A stale lock requires operator review.
|
||||
const lockPath = resolve(dataDir,`${PROFILE}-node.lock`);
|
||||
const lock = await open(lockPath,'wx',0o600).catch(()=>{throw new Error('Data directory locked. Stop its other node; remove stale testnet-node.lock only after verifying no node is running.');});
|
||||
@@ -119,9 +122,9 @@ const templateLimit=rateLimit({windowMs:60000,limit:20,standardHeaders:'draft-8'
|
||||
app.post('/api/mining/template', templateLimit, async (req,res)=>{
|
||||
const body=z.object({owner:z.string().regex(/^[a-f0-9]{64}$/),publicKey:z.string().max(256)}).strict().parse(req.body);
|
||||
try {
|
||||
// One fresh article+snapshot per tip, shared by every miner (server/wikimedia/pool.ts).
|
||||
const {article,statsSnapshot}=await mintPool.take(ledger.summary().tip);
|
||||
res.json(ledger.template(body.owner,body.publicKey,article,statsSnapshot));
|
||||
const article=await wikipediaArticles.random();
|
||||
const metrics=await wikipediaEnrichment.getFresh(article.pageId);
|
||||
res.json(ledger.template(body.owner,body.publicKey,article,snapshotStats(metrics)));
|
||||
} catch {
|
||||
res.status(503).json({error:'Could not prepare a Wikipedia article and current statistics. Retry shortly.'});
|
||||
}
|
||||
@@ -150,8 +153,12 @@ app.post('/p2p/transaction',workLimit,async(req,res)=>{
|
||||
const id=await ledger.submitTransaction(transaction);
|
||||
res.status(202).json({id,status:'pending'});
|
||||
});
|
||||
app.use('/api/account',createAccountRouter({dataDir}));
|
||||
app.use('/api/game',await (await import('../game/index.ts')).gameRouter({dataDir,enrichment:wikipediaEnrichment}));
|
||||
// One account store and one Dex key cache, shared by the account and game routes (#57).
|
||||
const oidc=oidcConfig(), dexKeys=new DexKeys(oidc), accounts=new AccountStore(resolve(dataDir,'accounts.sqlite'));
|
||||
const ownership={ledger,accounts,verifyIdToken:accountVerifier(oidc,dexKeys,accounts)};
|
||||
app.use('/api/account',createAccountRouter({config:oidc,keys:dexKeys,store:accounts}));
|
||||
app.use('/api/game/collection',collectionRouter(ownership));
|
||||
app.use('/api/game',await (await import('../game/index.ts')).gameRouter({dataDir,enrichment:wikipediaEnrichment,ownership}));
|
||||
app.use(['/api','/p2p'],(_req,res)=>{res.status(404).json({error:'Unknown testnet endpoint'});});
|
||||
app.use(express.static(resolve(root,'dist')));
|
||||
app.get('/{*path}',(_req,res)=>res.sendFile(resolve(root,'dist/index.html')));
|
||||
@@ -170,7 +177,7 @@ const server=app.listen(port,host,()=>{
|
||||
const operations=startOperations(opsPort,()=>({network:NETWORK,
|
||||
ledger:ledger.diagnostics(),native:engine.diagnostics(),
|
||||
peers:{configured:peers.status().length,connected:peers.status().filter(p=>p.connected).length},
|
||||
rpc:{api:apiBudget.status(),p2p:peerBudget.status()},wikimedia:wikimediaStatus()}),resolve(dataDir,PROFILE));
|
||||
rpc:{api:apiBudget.status(),p2p:peerBudget.status()}}),resolve(dataDir,PROFILE));
|
||||
server.maxConnections=256;
|
||||
server.headersTimeout=10000;
|
||||
server.requestTimeout=15000;
|
||||
|
||||
+10
-22
@@ -1,6 +1,6 @@
|
||||
import { z } from 'zod';
|
||||
import type { Article } from '../shared/protocol.ts';
|
||||
import { WikimediaClient, WikimediaUnavailable, wikimediaClient } from './wikimedia/client.ts';
|
||||
import { ARTICLE_USER_AGENT, wikimediaUserAgent } from './wikimedia-user-agent.ts';
|
||||
|
||||
const pageSchema = z.object({
|
||||
pageid: z.number().int().positive().max(Number.MAX_SAFE_INTEGER), ns: z.literal(0),
|
||||
@@ -58,23 +58,11 @@ function pictureUrl(value: string | undefined): string | undefined {
|
||||
} catch { return undefined; }
|
||||
}
|
||||
|
||||
/** The exact W1 request. Its parameters decide minted article content: keep them byte-identical. */
|
||||
export function randomArticlesUrl(): URL {
|
||||
const url = new URL('https://en.wikipedia.org/w/api.php');
|
||||
url.search = new URLSearchParams({ action: 'query', format: 'json', formatversion: '2',
|
||||
generator: 'random', grnnamespace: '0', grnlimit: '10', prop: 'extracts|pageimages|info|categories',
|
||||
exintro: '1', explaintext: '1', exchars: '900', piprop: 'thumbnail', pithumbsize: '800', cllimit: '20' }).toString();
|
||||
return url;
|
||||
}
|
||||
|
||||
export class WikipediaArticleSource {
|
||||
private readonly client: WikimediaClient;
|
||||
private readonly fetcher: typeof globalThis.fetch;
|
||||
private active = 0;
|
||||
|
||||
/** A bare fetch (tests) gets an unthrottled pass-through client; production uses the shared client. */
|
||||
constructor(source: typeof globalThis.fetch | WikimediaClient = wikimediaClient) {
|
||||
this.client = source instanceof WikimediaClient ? source : WikimediaClient.unthrottled(source);
|
||||
}
|
||||
constructor(fetcher: typeof globalThis.fetch = globalThis.fetch) { this.fetcher = fetcher; }
|
||||
|
||||
async random(): Promise<Article> {
|
||||
if (this.active >= 4) throw new Error('Wikipedia article service is busy');
|
||||
@@ -82,8 +70,12 @@ export class WikipediaArticleSource {
|
||||
try {
|
||||
for (let attempt = 0; attempt < MAX_ATTEMPTS; attempt++) {
|
||||
try {
|
||||
// Mint lane, never cached: random articles become immutable chain data.
|
||||
const { response } = await this.client.request(randomArticlesUrl(), { lane: 'mint', timeoutMs: 3500 });
|
||||
const url = new URL('https://en.wikipedia.org/w/api.php');
|
||||
url.search = new URLSearchParams({ action: 'query', format: 'json', formatversion: '2',
|
||||
generator: 'random', grnnamespace: '0', grnlimit: '10', prop: 'extracts|pageimages|info|categories',
|
||||
exintro: '1', explaintext: '1', exchars: '900', piprop: 'thumbnail', pithumbsize: '800', cllimit: '20' }).toString();
|
||||
const response = await this.fetcher(url, { signal: AbortSignal.timeout(3500), redirect: 'error',
|
||||
headers: { 'User-Agent': wikimediaUserAgent(ARTICLE_USER_AGENT), Accept: 'application/json' } });
|
||||
const data = responseSchema.parse(await readJson(response));
|
||||
for (const rawPage of data.query.pages) {
|
||||
const parsedPage = pageSchema.safeParse(rawPage);
|
||||
@@ -96,11 +88,7 @@ export class WikipediaArticleSource {
|
||||
url: `https://en.wikipedia.org/?curid=${page.pageid}`, revisionId: page.lastrevid,
|
||||
source: 'live', category: category(page.title, page.categories?.map(c => c.title) ?? []) };
|
||||
}
|
||||
} catch (error) {
|
||||
// Throttled, paused or failing upstream: stop now rather than hammer it with retries.
|
||||
if (error instanceof WikimediaUnavailable) throw error;
|
||||
/* Otherwise try another random API response, but keep the number of attempts bounded. */
|
||||
}
|
||||
} catch { /* Try another random API response, but keep the number of attempts bounded. */ }
|
||||
}
|
||||
throw new Error('Wikipedia did not return an article with a picture after three attempts');
|
||||
} finally {
|
||||
|
||||
@@ -1,48 +1,13 @@
|
||||
// User-Agent for Wikimedia API requests. Wikimedia's User-Agent policy asks automated clients for a
|
||||
// descriptive agent with contact details: `Tool/version (+https://site; contact) lib/version`.
|
||||
// Operators set WIKIDECK_OPERATOR_CONTACT (an email address or URL); PUBLIC_ORIGIN is the fallback.
|
||||
// WIKIDECK_USER_AGENT still overrides the whole string.
|
||||
import { readFileSync } from 'node:fs';
|
||||
|
||||
export const PROJECT_URL = 'https://gitea.eom.dev/WikiDeck/wikideck';
|
||||
// User-Agent for Wikimedia API requests. Wikimedia asks automated clients to identify
|
||||
// themselves with contact information; operators can opt in with WIKIDECK_USER_AGENT.
|
||||
// Unset (or blank), each caller keeps its historical default string unchanged.
|
||||
export const ARTICLE_USER_AGENT = 'Wikideck/0.1 (experimental Wikipedia article cards)';
|
||||
export const ENRICHMENT_USER_AGENT = 'Wikideck/0.1 (Wikipedia off-chain page enrichment)';
|
||||
const MAX_LENGTH = 256;
|
||||
const SAFE = /^[\x20-\x7e]+$/;
|
||||
|
||||
function packageVersion(): string {
|
||||
try {
|
||||
const raw = JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8')) as { version?: unknown };
|
||||
return typeof raw.version === 'string' && /^[\w.+-]{1,32}$/.test(raw.version) ? raw.version : '0';
|
||||
} catch { return '0'; }
|
||||
}
|
||||
const VERSION = packageVersion();
|
||||
|
||||
/** The operator contact, or undefined when none (or only an unsafe value) is configured. */
|
||||
export function operatorContact(env: Record<string, string | undefined> = process.env): string | undefined {
|
||||
for (const value of [env.WIKIDECK_OPERATOR_CONTACT, env.PUBLIC_ORIGIN]) {
|
||||
const trimmed = value?.trim();
|
||||
if (trimmed && trimmed.length <= 128 && SAFE.test(trimmed) && !/[();]/.test(trimmed)) return trimmed;
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
/** `WikiDeck/<version> (+<project URL>; <contact>) node/<version>`. */
|
||||
export function defaultUserAgent(env: Record<string, string | undefined> = process.env): string {
|
||||
const contact = operatorContact(env);
|
||||
return `WikiDeck/${VERSION} (+${PROJECT_URL}${contact ? `; ${contact}` : ''}) node/${process.versions.node}`;
|
||||
}
|
||||
|
||||
export function wikimediaUserAgent(fallback: string, env: Record<string, string | undefined> = process.env): string {
|
||||
const configured = env.WIKIDECK_USER_AGENT?.trim();
|
||||
// Reject header-injection/control characters and oversized values; fall back to the default.
|
||||
if (!configured || configured.length > MAX_LENGTH || !SAFE.test(configured)) return fallback;
|
||||
if (!configured || configured.length > MAX_LENGTH || !/^[\x20-\x7e]+$/.test(configured)) return fallback;
|
||||
return configured;
|
||||
}
|
||||
|
||||
/** Start-up check: Wikimedia throttles or blocks clients without contact details first. */
|
||||
export function warnIfNoContact(env: Record<string, string | undefined> = process.env,
|
||||
warn: (message: string) => void = console.warn): boolean {
|
||||
if (wikimediaUserAgent('', env) || operatorContact(env)) return false;
|
||||
warn('[wikimedia] WARNING: no operator contact in the Wikimedia User-Agent. Set WIKIDECK_OPERATOR_CONTACT '
|
||||
+ '(an email address or URL) or PUBLIC_ORIGIN; Wikimedia policy requires it and may block this node.');
|
||||
return true;
|
||||
}
|
||||
|
||||
@@ -1,74 +0,0 @@
|
||||
/**
|
||||
* On-disk HTTP cache for off-chain Wikimedia reads (`$WIKIDECK_DATA_DIR/wikimedia-cache.sqlite`).
|
||||
* It stores the raw body with its ETag / Last-Modified so stale entries can be revalidated with
|
||||
* conditional requests. Mint-path requests never use it (see client.ts).
|
||||
*/
|
||||
import { DatabaseSync } from 'node:sqlite';
|
||||
|
||||
export type StoredEntry = {
|
||||
status: number; body: Buffer; contentType: string; etag: string | null; lastModified: string | null;
|
||||
expiresAt: number; storedAt: number;
|
||||
};
|
||||
export type CacheEntry = StoredEntry & { toResponse(): Response };
|
||||
|
||||
/** Canonical key: origin + path + sorted query, without `maxlag` (it changes only error behaviour). */
|
||||
export function cacheKey(url: URL): string {
|
||||
const params = [...url.searchParams].filter(([name]) => name !== 'maxlag')
|
||||
.sort(([a, x], [b, y]) => a < b ? -1 : a > b ? 1 : x < y ? -1 : x > y ? 1 : 0);
|
||||
return `${url.origin}${url.pathname}?${new URLSearchParams(params).toString()}`;
|
||||
}
|
||||
|
||||
function withResponse(entry: StoredEntry): CacheEntry {
|
||||
return { ...entry, toResponse: () => new Response(new Uint8Array(entry.body), { status: entry.status,
|
||||
headers: { 'Content-Type': entry.contentType, 'Content-Length': String(entry.body.byteLength),
|
||||
...(entry.etag ? { ETag: entry.etag } : {}) } }) };
|
||||
}
|
||||
|
||||
type Row = { status: number; body: Uint8Array; content_type: string; etag: string | null;
|
||||
last_modified: string | null; expires_at: number; stored_at: number };
|
||||
|
||||
export class WikimediaCache {
|
||||
private readonly db: DatabaseSync;
|
||||
private readonly maxRows: number;
|
||||
private writes = 0;
|
||||
|
||||
/** `path` may be ':memory:' (tests). */
|
||||
constructor(path: string, options: { maxRows?: number } = {}) {
|
||||
this.maxRows = options.maxRows ?? 100_000;
|
||||
this.db = new DatabaseSync(path);
|
||||
this.db.exec(`PRAGMA journal_mode=WAL; PRAGMA synchronous=NORMAL;
|
||||
CREATE TABLE IF NOT EXISTS http_cache (key TEXT PRIMARY KEY, status INTEGER NOT NULL, body BLOB NOT NULL,
|
||||
content_type TEXT NOT NULL, etag TEXT, last_modified TEXT, expires_at INTEGER NOT NULL, stored_at INTEGER NOT NULL);
|
||||
CREATE INDEX IF NOT EXISTS http_cache_stored ON http_cache(stored_at);`);
|
||||
}
|
||||
|
||||
get(key: string): CacheEntry | undefined {
|
||||
const row = this.db.prepare('SELECT * FROM http_cache WHERE key = ?').get(key) as Row | undefined;
|
||||
if (!row) return undefined;
|
||||
return withResponse({ status: row.status, body: Buffer.from(row.body), contentType: row.content_type,
|
||||
etag: row.etag, lastModified: row.last_modified, expiresAt: row.expires_at, storedAt: row.stored_at });
|
||||
}
|
||||
|
||||
put(key: string, entry: StoredEntry): CacheEntry {
|
||||
this.db.prepare(`INSERT OR REPLACE INTO http_cache (key, status, body, content_type, etag, last_modified, expires_at, stored_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?)`).run(key, entry.status, entry.body, entry.contentType, entry.etag,
|
||||
entry.lastModified, entry.expiresAt, entry.storedAt);
|
||||
if (++this.writes % 500 === 0) this.prune();
|
||||
return withResponse(entry);
|
||||
}
|
||||
|
||||
touch(key: string, expiresAt: number): void {
|
||||
this.db.prepare('UPDATE http_cache SET expires_at = ? WHERE key = ?').run(expiresAt, key);
|
||||
}
|
||||
|
||||
delete(key: string): void { this.db.prepare('DELETE FROM http_cache WHERE key = ?').run(key); }
|
||||
|
||||
size(): number { return (this.db.prepare('SELECT COUNT(*) AS n FROM http_cache').get() as { n: number }).n; }
|
||||
|
||||
prune(): void {
|
||||
this.db.prepare(`DELETE FROM http_cache WHERE key IN (SELECT key FROM http_cache ORDER BY stored_at DESC LIMIT -1 OFFSET ?)`)
|
||||
.run(this.maxRows);
|
||||
}
|
||||
|
||||
close(): void { this.db.close(); }
|
||||
}
|
||||
@@ -1,441 +0,0 @@
|
||||
/**
|
||||
* One process-wide, polite Wikimedia HTTP client (engineer review §4, data review §5).
|
||||
*
|
||||
* Every server-side Wikimedia request goes through `WikimediaClient.request()`:
|
||||
* - a token bucket per upstream host (default 1 req/s, burst 3) and 1 request in flight per host;
|
||||
* - priority lanes: `mint` before `interactive` before `background`, with a bounded queue;
|
||||
* - `maxlag=5` on Action API requests (it changes only error behaviour, never content);
|
||||
* - `Retry-After` honoured on 429/503/maxlag by pausing the whole host, with exponential backoff;
|
||||
* - a circuit breaker: 3 consecutive 429s stop all requests to that host for 10 minutes;
|
||||
* - a daily request cap, and a budget shared by the nodes behind one egress IP;
|
||||
* - an optional on-disk cache (see cache.ts) with ETag / If-None-Match and stale-while-revalidate.
|
||||
*
|
||||
* The client is transport only. It never rewrites request parameters other than adding `maxlag`,
|
||||
* and requests made with `cache: undefined` (all mint-path requests) never read or write the cache.
|
||||
* Nothing here is reachable from block validation, so it cannot affect consensus.
|
||||
*/
|
||||
import { WikimediaCache, cacheKey } from './cache.ts';
|
||||
import { wikimediaUserAgent, defaultUserAgent } from '../wikimedia-user-agent.ts';
|
||||
|
||||
export type Lane = 'mint' | 'interactive' | 'background';
|
||||
const LANES: Lane[] = ['mint', 'interactive', 'background'];
|
||||
|
||||
export type Clock = {
|
||||
now(): number;
|
||||
setTimeout(fn: () => void, ms: number): unknown;
|
||||
clearTimeout(handle: unknown): void;
|
||||
};
|
||||
const realClock: Clock = {
|
||||
now: () => Date.now(),
|
||||
setTimeout: (fn, ms) => { const t = setTimeout(fn, ms); t.unref?.(); return t; },
|
||||
clearTimeout: handle => clearTimeout(handle as ReturnType<typeof setTimeout>),
|
||||
};
|
||||
|
||||
export type ClientConfig = {
|
||||
/** Sustained requests per second per upstream host, for this whole egress IP. */
|
||||
requestsPerSecond: number;
|
||||
burst: number;
|
||||
concurrencyPerHost: number;
|
||||
/** Requests per UTC day for this whole egress IP. */
|
||||
dailyCap: number;
|
||||
/** Nodes sharing one egress IP; the rate and daily cap are divided between them. */
|
||||
nodesPerIp: number;
|
||||
queueLimit: number;
|
||||
/** Longest wait in the queue before a request fails, per lane. */
|
||||
maxWaitMs: Record<Lane, number>;
|
||||
breakerThreshold: number;
|
||||
breakerMs: number;
|
||||
maxBackoffMs: number;
|
||||
maxlag: number;
|
||||
/** Retries of a single request after a maxlag reply (HTTP 200/503 with MediaWiki-API-Error: maxlag). */
|
||||
maxlagRetries: number;
|
||||
};
|
||||
|
||||
export const DEFAULT_CONFIG: ClientConfig = {
|
||||
requestsPerSecond: 1, burst: 3, concurrencyPerHost: 1, dailyCap: 20_000, nodesPerIp: 1, queueLimit: 64,
|
||||
maxWaitMs: { mint: 10_000, interactive: 5_000, background: 120_000 },
|
||||
breakerThreshold: 3, breakerMs: 10 * 60_000, maxBackoffMs: 10 * 60_000, maxlag: 5, maxlagRetries: 2,
|
||||
};
|
||||
|
||||
/** Unthrottled settings for tests and tools that inject their own fetch. */
|
||||
export const UNTHROTTLED: Partial<ClientConfig> = {
|
||||
requestsPerSecond: 1e9, burst: 1e9, concurrencyPerHost: 1e9, dailyCap: Number.MAX_SAFE_INTEGER, queueLimit: 1e9,
|
||||
breakerThreshold: Number.MAX_SAFE_INTEGER, maxlagRetries: 0, maxBackoffMs: 0,
|
||||
};
|
||||
|
||||
function positive(value: string | undefined, fallback: number, integer = false): number {
|
||||
if (value === undefined || value.trim() === '') return fallback;
|
||||
const n = Number(value);
|
||||
if (!Number.isFinite(n) || n <= 0 || (integer && !Number.isSafeInteger(n))) {
|
||||
console.warn(`[wikimedia] ignoring invalid setting ${JSON.stringify(value)}; using ${fallback}`);
|
||||
return fallback;
|
||||
}
|
||||
return n;
|
||||
}
|
||||
|
||||
/** Operator settings. Documented in docs/night2/WIKIMEDIA.md. */
|
||||
export function configFromEnv(env: Record<string, string | undefined> = process.env): ClientConfig {
|
||||
return { ...DEFAULT_CONFIG,
|
||||
requestsPerSecond: positive(env.WIKIDECK_WIKIMEDIA_RPS, DEFAULT_CONFIG.requestsPerSecond),
|
||||
burst: positive(env.WIKIDECK_WIKIMEDIA_BURST, DEFAULT_CONFIG.burst),
|
||||
concurrencyPerHost: positive(env.WIKIDECK_WIKIMEDIA_CONCURRENCY, DEFAULT_CONFIG.concurrencyPerHost, true),
|
||||
dailyCap: positive(env.WIKIDECK_WIKIMEDIA_DAILY_CAP, DEFAULT_CONFIG.dailyCap, true),
|
||||
nodesPerIp: positive(env.WIKIDECK_WIKIMEDIA_NODES_PER_IP, DEFAULT_CONFIG.nodesPerIp, true),
|
||||
};
|
||||
}
|
||||
|
||||
/** Thrown for every refusal or transport failure: queue full, cap reached, circuit open, timeout, network. */
|
||||
export class WikimediaUnavailable extends Error {
|
||||
readonly reason: string;
|
||||
readonly status?: number;
|
||||
constructor(reason: string, status?: number) {
|
||||
super(`Wikimedia unavailable: ${reason}`);
|
||||
this.name = 'WikimediaUnavailable';
|
||||
this.reason = reason; this.status = status;
|
||||
}
|
||||
}
|
||||
|
||||
export type CachePolicy = {
|
||||
ttlMs: number;
|
||||
/** Serve a stale entry at once and revalidate it on the background lane. */
|
||||
staleWhileRevalidate?: boolean;
|
||||
/** Largest body stored, in bytes. */
|
||||
maxBytes?: number;
|
||||
};
|
||||
|
||||
export type RequestOptions = {
|
||||
lane: Lane;
|
||||
/** Starts when the request is dispatched, not while it waits in the queue. */
|
||||
timeoutMs: number;
|
||||
headers?: Record<string, string>;
|
||||
/** Omit for mint data: the request then bypasses the cache completely. */
|
||||
cache?: CachePolicy;
|
||||
};
|
||||
|
||||
export type ClientResponse = { response: Response; signal: AbortSignal; fromCache: boolean };
|
||||
|
||||
type Waiter = {
|
||||
lane: Lane; host: string; enqueuedAt: number;
|
||||
run: () => void; fail: (error: Error) => void; timer?: unknown;
|
||||
};
|
||||
type HostState = {
|
||||
tokens: number; refilledAt: number; inFlight: number;
|
||||
pausedUntil: number; breakerUntil: number; consecutive429: number; backoffExponent: number;
|
||||
};
|
||||
|
||||
export type ClientStatus = {
|
||||
day: string; requestsToday: number; dailyCap: number; requestsPerSecond: number; queued: number;
|
||||
counters: Record<string, number>;
|
||||
hosts: Record<string, { inFlight: number; pausedForMs: number; circuitOpen: boolean; consecutive429: number }>;
|
||||
};
|
||||
|
||||
function parseRetryAfter(value: string | null, now: number): number | undefined {
|
||||
if (!value) return undefined;
|
||||
if (/^\d+$/.test(value.trim())) return Number(value.trim()) * 1000;
|
||||
const at = Date.parse(value);
|
||||
return Number.isNaN(at) ? undefined : Math.max(0, at - now);
|
||||
}
|
||||
|
||||
function isActionApi(url: URL): boolean {
|
||||
return url.pathname === '/w/api.php' && /(^|\.)(wikipedia|wikidata|wikimedia)\.org$/.test(url.hostname);
|
||||
}
|
||||
|
||||
export class WikimediaClient {
|
||||
readonly config: ClientConfig;
|
||||
private readonly fetcher: typeof globalThis.fetch;
|
||||
private readonly clock: Clock;
|
||||
private readonly userAgent: () => string;
|
||||
private cache: WikimediaCache | undefined;
|
||||
private readonly hosts = new Map<string, HostState>();
|
||||
private readonly queues = new Map<Lane, Waiter[]>(LANES.map(lane => [lane, []]));
|
||||
private readonly revalidating = new Set<string>();
|
||||
private pumpTimer: unknown;
|
||||
private pumpAt = Infinity;
|
||||
private day = '';
|
||||
private requestsToday = 0;
|
||||
readonly counters: Record<string, number> = {};
|
||||
|
||||
constructor(options: { fetch?: typeof globalThis.fetch; config?: Partial<ClientConfig>; clock?: Clock;
|
||||
cache?: WikimediaCache; userAgent?: () => string } = {}) {
|
||||
this.fetcher = options.fetch ?? ((...args) => globalThis.fetch(...args));
|
||||
this.config = { ...DEFAULT_CONFIG, ...options.config,
|
||||
maxWaitMs: { ...DEFAULT_CONFIG.maxWaitMs, ...options.config?.maxWaitMs } };
|
||||
this.clock = options.clock ?? realClock;
|
||||
this.userAgent = options.userAgent ?? (() => wikimediaUserAgent(defaultUserAgent()));
|
||||
this.cache = options.cache;
|
||||
}
|
||||
|
||||
/** A pass-through client for injected fetchers (tests): no rate limit, no breaker, no cache. */
|
||||
static unthrottled(fetcher: typeof globalThis.fetch): WikimediaClient {
|
||||
return new WikimediaClient({ fetch: fetcher, config: UNTHROTTLED });
|
||||
}
|
||||
|
||||
attachCache(cache: WikimediaCache | undefined): void { this.cache = cache; }
|
||||
|
||||
private count(name: string): void { this.counters[name] = (this.counters[name] ?? 0) + 1; }
|
||||
|
||||
private get rate(): number { return this.config.requestsPerSecond / this.config.nodesPerIp; }
|
||||
private get cap(): number { return Math.max(1, Math.floor(this.config.dailyCap / this.config.nodesPerIp)); }
|
||||
|
||||
private host(name: string): HostState {
|
||||
let state = this.hosts.get(name);
|
||||
if (!state) {
|
||||
state = { tokens: this.config.burst, refilledAt: this.clock.now(), inFlight: 0,
|
||||
pausedUntil: 0, breakerUntil: 0, consecutive429: 0, backoffExponent: 0 };
|
||||
this.hosts.set(name, state);
|
||||
}
|
||||
return state;
|
||||
}
|
||||
|
||||
private refill(state: HostState, now: number): void {
|
||||
state.tokens = Math.min(this.config.burst, state.tokens + (now - state.refilledAt) / 1000 * this.rate);
|
||||
state.refilledAt = now;
|
||||
}
|
||||
|
||||
/** Daily budget by lane: background stops at 80 %, interactive at 100 %, mint has a 10 % reserve. */
|
||||
private overCap(lane: Lane): boolean {
|
||||
const today = new Date(this.clock.now()).toISOString().slice(0, 10);
|
||||
if (today !== this.day) { this.day = today; this.requestsToday = 0; }
|
||||
const limit = lane === 'background' ? this.cap * 0.8 : lane === 'interactive' ? this.cap : this.cap * 1.1;
|
||||
return this.requestsToday >= limit;
|
||||
}
|
||||
|
||||
status(): ClientStatus {
|
||||
const now = this.clock.now();
|
||||
return { day: this.day, requestsToday: this.requestsToday, dailyCap: this.cap, requestsPerSecond: this.rate,
|
||||
queued: LANES.reduce((n, lane) => n + this.queues.get(lane)!.length, 0), counters: { ...this.counters },
|
||||
hosts: Object.fromEntries([...this.hosts].map(([name, s]) => [name,
|
||||
{ inFlight: s.inFlight, pausedForMs: Math.max(0, s.pausedUntil - now), circuitOpen: s.breakerUntil > now,
|
||||
consecutive429: s.consecutive429 }])) };
|
||||
}
|
||||
|
||||
async request(input: URL, options: RequestOptions): Promise<ClientResponse> {
|
||||
const url = new URL(input);
|
||||
if (url.protocol !== 'https:') throw new RangeError('Wikimedia requests must use HTTPS');
|
||||
if (isActionApi(url) && !url.searchParams.has('maxlag')) url.searchParams.append('maxlag', String(this.config.maxlag));
|
||||
const key = options.cache && this.cache ? cacheKey(url) : undefined;
|
||||
const entry = key ? this.cache?.get(key) : undefined;
|
||||
const now = this.clock.now();
|
||||
if (entry && entry.expiresAt > now) {
|
||||
this.count('cache.hit');
|
||||
return { response: entry.toResponse(), signal: new AbortController().signal, fromCache: true };
|
||||
}
|
||||
if (entry && options.cache?.staleWhileRevalidate) {
|
||||
this.count('cache.stale');
|
||||
this.revalidate(url, key!, options);
|
||||
return { response: entry.toResponse(), signal: new AbortController().signal, fromCache: true };
|
||||
}
|
||||
try {
|
||||
return await this.network(url, options, key);
|
||||
} catch (error) {
|
||||
// Prefer a stale copy over an error for anything that is allowed to use the cache.
|
||||
if (entry && error instanceof WikimediaUnavailable) {
|
||||
this.count('cache.staleOnError');
|
||||
return { response: entry.toResponse(), signal: new AbortController().signal, fromCache: true };
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
private revalidate(url: URL, key: string, options: RequestOptions): void {
|
||||
if (this.revalidating.has(key)) return;
|
||||
this.revalidating.add(key);
|
||||
void this.network(url, { ...options, lane: 'background' }, key)
|
||||
.then(({ response }) => response.body?.cancel(), () => this.count('cache.revalidateFailed'))
|
||||
.finally(() => this.revalidating.delete(key));
|
||||
}
|
||||
|
||||
private async network(url: URL, options: RequestOptions, key: string | undefined): Promise<ClientResponse> {
|
||||
for (let attempt = 0; ; attempt++) {
|
||||
const result = await this.dispatch(url, options, key);
|
||||
if (result !== 'maxlag') return result;
|
||||
if (attempt >= this.config.maxlagRetries) throw new WikimediaUnavailable('maxlag');
|
||||
}
|
||||
}
|
||||
|
||||
/** Wait for a slot, send one request, and classify the reply. */
|
||||
private async dispatch(url: URL, options: RequestOptions, key: string | undefined): Promise<ClientResponse | 'maxlag'> {
|
||||
const hostName = url.hostname;
|
||||
const wait = this.acquire(hostName, options.lane);
|
||||
if (wait) await wait; // No await on the fast path: an idle host dispatches synchronously.
|
||||
const state = this.host(hostName);
|
||||
const headers = new Headers({ Accept: 'application/json', 'Accept-Encoding': 'gzip', ...options.headers,
|
||||
'User-Agent': this.userAgent() });
|
||||
const entry = key ? this.cache?.get(key) : undefined;
|
||||
if (entry?.etag) headers.set('If-None-Match', entry.etag);
|
||||
if (entry?.lastModified) headers.set('If-Modified-Since', entry.lastModified);
|
||||
const signal = AbortSignal.timeout(options.timeoutMs);
|
||||
this.requestsToday++;
|
||||
this.count(`lane.${options.lane}`);
|
||||
let response: Response;
|
||||
try {
|
||||
response = await this.fetcher(url, { signal, redirect: 'error', headers });
|
||||
} catch (error) {
|
||||
this.release(state);
|
||||
this.count('error.network');
|
||||
throw new WikimediaUnavailable(signal.aborted ? 'timeout' : `network: ${(error as Error).message}`);
|
||||
}
|
||||
const now = this.clock.now();
|
||||
const retryAfter = parseRetryAfter(response.headers.get('retry-after'), now);
|
||||
const apiError = response.headers.get('mediawiki-api-error');
|
||||
if (apiError === 'maxlag') {
|
||||
await response.body?.cancel();
|
||||
this.count('maxlag');
|
||||
this.pause(state, Math.min(this.config.maxBackoffMs, retryAfter ?? 5000), false);
|
||||
this.release(state);
|
||||
return 'maxlag';
|
||||
}
|
||||
if (response.status === 429 || response.status === 503) {
|
||||
await response.body?.cancel();
|
||||
this.count(`http.${response.status}`);
|
||||
if (response.status === 429) state.consecutive429++;
|
||||
const backoff = Math.min(this.config.maxBackoffMs, 2 ** state.backoffExponent * 1000 * (1 + Math.random() * 0.25));
|
||||
state.backoffExponent = Math.min(state.backoffExponent + 1, 20);
|
||||
if (state.consecutive429 >= this.config.breakerThreshold) {
|
||||
this.pause(state, this.config.breakerMs, true);
|
||||
this.count('breaker.open');
|
||||
} else {
|
||||
this.pause(state, Math.min(this.config.maxBackoffMs, Math.max(retryAfter ?? 0, backoff)), false);
|
||||
}
|
||||
this.release(state);
|
||||
throw new WikimediaUnavailable(`HTTP ${response.status}`, response.status);
|
||||
}
|
||||
state.consecutive429 = 0;
|
||||
state.backoffExponent = 0;
|
||||
this.release(state);
|
||||
if (response.status === 304 && entry && key) {
|
||||
this.count('cache.revalidated');
|
||||
this.cache?.touch(key, now + options.cache!.ttlMs);
|
||||
return { response: entry.toResponse(), signal, fromCache: true };
|
||||
}
|
||||
if (key && this.cache && response.status === 200 && !response.redirected && options.cache &&
|
||||
/^application\/json(?:\s*;|\s*$)/i.test(response.headers.get('content-type') ?? '')) {
|
||||
// Store the raw body; callers re-validate it exactly as they would a network reply.
|
||||
const body = await this.readLimited(response, options.cache.maxBytes ?? 1024 * 1024, signal);
|
||||
const stored = this.cache.put(key, { status: response.status, body,
|
||||
contentType: response.headers.get('content-type') ?? '', etag: response.headers.get('etag'),
|
||||
lastModified: response.headers.get('last-modified'), expiresAt: now + options.cache.ttlMs, storedAt: now });
|
||||
return { response: stored.toResponse(), signal, fromCache: false };
|
||||
}
|
||||
if (response.status >= 500) this.count(`http.${response.status}`);
|
||||
return { response, signal, fromCache: false };
|
||||
}
|
||||
|
||||
private async readLimited(response: Response, maxBytes: number, signal: AbortSignal): Promise<Buffer> {
|
||||
const length = response.headers.get('content-length');
|
||||
if (length !== null && (!/^\d+$/.test(length) || Number(length) > maxBytes) || !response.body) {
|
||||
await response.body?.cancel();
|
||||
throw new WikimediaUnavailable('oversized or empty response');
|
||||
}
|
||||
const reader = response.body.getReader();
|
||||
const chunks: Uint8Array[] = [];
|
||||
let size = 0;
|
||||
try {
|
||||
for (;;) {
|
||||
if (signal.aborted) throw new WikimediaUnavailable('timeout');
|
||||
const { done, value } = await reader.read();
|
||||
if (done) break;
|
||||
size += value.byteLength;
|
||||
if (size > maxBytes) throw new WikimediaUnavailable('oversized response');
|
||||
chunks.push(value);
|
||||
}
|
||||
return Buffer.concat(chunks);
|
||||
} catch (error) {
|
||||
throw error instanceof WikimediaUnavailable ? error : new WikimediaUnavailable('read failed');
|
||||
} finally {
|
||||
await reader.cancel().catch(() => {});
|
||||
reader.releaseLock();
|
||||
}
|
||||
}
|
||||
|
||||
private pause(state: HostState, ms: number, breaker: boolean): void {
|
||||
const until = this.clock.now() + ms;
|
||||
if (until > state.pausedUntil) {
|
||||
if (state.pausedUntil <= this.clock.now()) {
|
||||
console.warn(`[wikimedia] ${breaker ? 'circuit open' : 'pausing'} for ${Math.ceil(ms / 1000)} s`);
|
||||
}
|
||||
state.pausedUntil = until;
|
||||
}
|
||||
if (breaker) { state.consecutive429 = 0; state.breakerUntil = until; }
|
||||
}
|
||||
|
||||
private release(state: HostState): void {
|
||||
state.inFlight--;
|
||||
this.pump();
|
||||
}
|
||||
|
||||
/** Undefined when a slot was taken at once; otherwise a promise for the slot. Throws on refusal. */
|
||||
private acquire(hostName: string, lane: Lane): Promise<void> | undefined {
|
||||
const state = this.host(hostName);
|
||||
const now = this.clock.now();
|
||||
if (this.overCap(lane)) { this.count('refused.cap'); throw new WikimediaUnavailable('daily cap reached'); }
|
||||
// A paused host refuses at once when the pause outlasts this lane's patience.
|
||||
if (state.pausedUntil - now > this.config.maxWaitMs[lane]) {
|
||||
this.count('refused.paused');
|
||||
throw new WikimediaUnavailable(state.breakerUntil > now ? 'circuit open' : 'host paused');
|
||||
}
|
||||
this.refill(state, now);
|
||||
if (state.pausedUntil <= now && state.inFlight < this.config.concurrencyPerHost && state.tokens >= 1 &&
|
||||
!LANES.some(l => this.queues.get(l)!.some(w => w.host === hostName))) {
|
||||
state.tokens -= 1;
|
||||
state.inFlight++;
|
||||
return undefined;
|
||||
}
|
||||
const queued = LANES.reduce((n, l) => n + this.queues.get(l)!.length, 0);
|
||||
if (queued >= this.config.queueLimit) {
|
||||
// Background work is shed first; a mint request evicts the newest background waiter.
|
||||
const background = this.queues.get('background')!;
|
||||
if (lane === 'background' || background.length === 0) {
|
||||
this.count('refused.queue');
|
||||
throw new WikimediaUnavailable('queue full');
|
||||
}
|
||||
const evicted = background.pop()!;
|
||||
this.clock.clearTimeout(evicted.timer);
|
||||
evicted.fail(new WikimediaUnavailable('queue full'));
|
||||
}
|
||||
return new Promise<void>((resolve, reject) => {
|
||||
const waiter: Waiter = { lane, host: hostName, enqueuedAt: now,
|
||||
run: () => { this.clock.clearTimeout(waiter.timer); resolve(); },
|
||||
fail: error => reject(error) };
|
||||
waiter.timer = this.clock.setTimeout(() => {
|
||||
const queue = this.queues.get(lane)!;
|
||||
const index = queue.indexOf(waiter);
|
||||
if (index >= 0) { queue.splice(index, 1); this.count('refused.wait'); reject(new WikimediaUnavailable('queue wait')); }
|
||||
}, this.config.maxWaitMs[lane]);
|
||||
this.queues.get(lane)!.push(waiter);
|
||||
this.pump();
|
||||
});
|
||||
}
|
||||
|
||||
/** Start every waiter that may run now, highest lane first; otherwise arm a timer for the next slot. */
|
||||
private pump(): void {
|
||||
const now = this.clock.now();
|
||||
let next = Infinity;
|
||||
for (const lane of LANES) {
|
||||
const queue = this.queues.get(lane)!;
|
||||
for (let i = 0; i < queue.length;) {
|
||||
const waiter = queue[i];
|
||||
// A host is served strictly by priority: lower lanes wait while a higher lane waits for the same host.
|
||||
if (LANES.slice(0, LANES.indexOf(lane)).some(l => this.queues.get(l)!.some(w => w.host === waiter.host))) { i++; continue; }
|
||||
const state = this.host(waiter.host);
|
||||
this.refill(state, now);
|
||||
if (state.pausedUntil > now) { next = Math.min(next, state.pausedUntil); i++; continue; }
|
||||
if (state.inFlight >= this.config.concurrencyPerHost) { i++; continue; }
|
||||
if (state.tokens < 1) { next = Math.min(next, now + (1 - state.tokens) / this.rate * 1000); i++; continue; }
|
||||
state.tokens -= 1;
|
||||
state.inFlight++;
|
||||
queue.splice(i, 1);
|
||||
waiter.run();
|
||||
}
|
||||
}
|
||||
if (next < Infinity && next < this.pumpAt) {
|
||||
if (this.pumpTimer !== undefined) this.clock.clearTimeout(this.pumpTimer);
|
||||
this.pumpAt = next;
|
||||
this.pumpTimer = this.clock.setTimeout(() => { this.pumpTimer = undefined; this.pumpAt = Infinity; this.pump(); },
|
||||
Math.max(1, Math.ceil(next - now)));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** The process-wide client. `configureWikimedia()` (server start-up) attaches the disk cache. */
|
||||
export const wikimediaClient = new WikimediaClient({ config: configFromEnv() });
|
||||
@@ -1,36 +0,0 @@
|
||||
/**
|
||||
* Node start-up wiring for the polite Wikimedia client: the disk cache, the User-Agent warning,
|
||||
* the mining prefetch pool and an ops status snapshot. See docs/night2/WIKIMEDIA.md.
|
||||
*/
|
||||
import { resolve } from 'node:path';
|
||||
import { WikimediaCache } from './cache.ts';
|
||||
import { wikimediaClient } from './client.ts';
|
||||
import { MintTemplatePool } from './pool.ts';
|
||||
import { warnIfNoContact } from '../wikimedia-user-agent.ts';
|
||||
import { wikipediaArticles } from '../wiki.ts';
|
||||
import { wikipediaEnrichment } from '../enrichment.ts';
|
||||
|
||||
export { wikimediaClient } from './client.ts';
|
||||
|
||||
/** Serves every mining template; shared by all miners of this node. */
|
||||
export const mintPool = new MintTemplatePool({ articles: wikipediaArticles, enrichment: wikipediaEnrichment });
|
||||
|
||||
let cache: WikimediaCache | undefined;
|
||||
|
||||
/** Call once at start-up. Without it the client still works, with in-memory caching only. */
|
||||
export function configureWikimedia(options: { dataDir: string }): void {
|
||||
warnIfNoContact();
|
||||
if (cache) return;
|
||||
try {
|
||||
cache = new WikimediaCache(resolve(options.dataDir, 'wikimedia-cache.sqlite'));
|
||||
wikimediaClient.attachCache(cache);
|
||||
} catch (error) {
|
||||
console.warn(`[wikimedia] disk cache disabled: ${(error as Error).message}`);
|
||||
}
|
||||
}
|
||||
|
||||
export function closeWikimedia(): void { wikimediaClient.attachCache(undefined); cache?.close(); cache = undefined; }
|
||||
|
||||
export function wikimediaStatus() {
|
||||
return { client: wikimediaClient.status(), cacheEntries: cache?.size() ?? null, mintPool: mintPool.status() };
|
||||
}
|
||||
@@ -1,88 +0,0 @@
|
||||
/**
|
||||
* Mining-template prefetch pool (data review §5, "Mining load").
|
||||
*
|
||||
* Without it every miner costs 3+ Wikimedia requests per template, all in a burst at each new tip.
|
||||
* The pool keeps a few fresh article + stats snapshots ready, and serves the SAME pair to every
|
||||
* template built on the same tip (miners race for one block anyway). Wikimedia load is then about
|
||||
* one pair (3 requests) per block, whatever the number of miners.
|
||||
*
|
||||
* Mint data stays transport-only: each pair comes from the mint-path reads (`random()` and
|
||||
* `getFresh()`, byte-identical parameters, never the disk cache), and the snapshot is stamped when it
|
||||
* is fetched, so `capturedAt` stays true. Pairs older than `maxAgeMs` are discarded, never served.
|
||||
* The pool refills only after it is used, so an idle node makes no requests.
|
||||
*/
|
||||
import type { Article } from '../../shared/protocol.ts';
|
||||
import { snapshotStats, type MintStatsSnapshot, type PageEnrichment } from '../../shared/enrichment.ts';
|
||||
|
||||
export type MintPair = { article: Article; statsSnapshot: MintStatsSnapshot };
|
||||
type Entry = MintPair & { fetchedAt: number };
|
||||
type Sources = {
|
||||
articles: { random(): Promise<Article> };
|
||||
enrichment: { getFresh(pageId: number): Promise<PageEnrichment> };
|
||||
};
|
||||
|
||||
export class MintTemplatePool {
|
||||
private readonly sources: Sources;
|
||||
private readonly size: number;
|
||||
private readonly maxAgeMs: number;
|
||||
private readonly now: () => number;
|
||||
private readonly ready: Entry[] = [];
|
||||
private assigned: { tip: string; entry: Entry } | undefined;
|
||||
private filling: Promise<void> | undefined;
|
||||
private direct: Promise<Entry> | undefined;
|
||||
readonly counters = { served: 0, shared: 0, fetched: 0, expired: 0, failed: 0 };
|
||||
|
||||
constructor(sources: Sources, options: { size?: number; maxAgeMs?: number; now?: () => number } = {}) {
|
||||
this.sources = sources;
|
||||
this.size = options.size ?? 2;
|
||||
this.maxAgeMs = options.maxAgeMs ?? 5 * 60_000;
|
||||
this.now = options.now ?? Date.now;
|
||||
if (!Number.isSafeInteger(this.size) || this.size < 0 || this.size > 16 ||
|
||||
!Number.isFinite(this.maxAgeMs) || this.maxAgeMs <= 0) throw new RangeError('Invalid mint pool options');
|
||||
}
|
||||
|
||||
private fresh(entry: Entry): boolean { return this.now() - entry.fetchedAt < this.maxAgeMs; }
|
||||
|
||||
private async fetchOne(): Promise<Entry> {
|
||||
const article = await this.sources.articles.random();
|
||||
const metrics = await this.sources.enrichment.getFresh(article.pageId);
|
||||
this.counters.fetched++;
|
||||
return { article, statsSnapshot: snapshotStats(metrics), fetchedAt: this.now() };
|
||||
}
|
||||
|
||||
/** The article and snapshot for a template on `tip`. Rejects when Wikimedia is unavailable. */
|
||||
async take(tip: string): Promise<MintPair> {
|
||||
let entry: Entry | undefined;
|
||||
if (this.assigned && this.assigned.tip === tip && this.fresh(this.assigned.entry)) {
|
||||
entry = this.assigned.entry;
|
||||
this.counters.shared++;
|
||||
} else {
|
||||
while (this.ready.length && !this.fresh(this.ready[0])) { this.ready.shift(); this.counters.expired++; }
|
||||
entry = this.ready.shift();
|
||||
if (!entry) {
|
||||
// Cold: fetch directly, and let concurrent templates share that one fetch.
|
||||
this.direct ??= this.fetchOne().finally(() => { this.direct = undefined; });
|
||||
try { entry = await this.direct; }
|
||||
catch (error) { this.counters.failed++; throw error; }
|
||||
if (this.assigned && this.assigned.tip === tip && this.fresh(this.assigned.entry)) entry = this.assigned.entry;
|
||||
}
|
||||
this.assigned = { tip, entry };
|
||||
}
|
||||
this.counters.served++;
|
||||
this.refill();
|
||||
return structuredClone({ article: entry.article, statsSnapshot: entry.statsSnapshot });
|
||||
}
|
||||
|
||||
/** Top the pool up one pair at a time, in the background. Exposed for tests. */
|
||||
refill(): Promise<void> {
|
||||
this.filling ??= (async () => {
|
||||
while (this.ready.filter(e => this.fresh(e)).length < this.size) {
|
||||
try { this.ready.push(await this.fetchOne()); }
|
||||
catch { this.counters.failed++; break; } // The next take() tries again; no retry loop.
|
||||
}
|
||||
})().finally(() => { this.filling = undefined; });
|
||||
return this.filling;
|
||||
}
|
||||
|
||||
status() { return { ready: this.ready.length, size: this.size, maxAgeMs: this.maxAgeMs, ...this.counters }; }
|
||||
}
|
||||
+6
-2
@@ -10,7 +10,7 @@ import { Card, samples, imageCredit, type CardData } from './Card.tsx';
|
||||
import { WalletPanel } from './WalletPanel.tsx';
|
||||
import { useModalFocus } from './useModalFocus.ts';
|
||||
import { ThemePicker } from './ThemePicker.tsx';
|
||||
import { LoginButton } from './auth/LoginButton.tsx';
|
||||
import { LoginButton, useAccount } from './auth/LoginButton.tsx';
|
||||
import { TradeDesk } from './TradeDesk.tsx';
|
||||
import { IS_MAINNET, NETWORK_GENESIS_TIME, NETWORK_LABEL } from '../shared/network.ts';
|
||||
import { StateClient } from '../shared/rpc-client.ts';
|
||||
@@ -48,6 +48,7 @@ function transferProblem(state: TestnetSnapshot, card: CardData, address: string
|
||||
|
||||
export default function App() {
|
||||
const [view, setView] = useState<View>('Discover');
|
||||
const linkedAccount = useAccount();
|
||||
const [wallet, setWallet] = useState<Wallet>();
|
||||
const [snapshot, setSnapshot] = useState<TestnetSnapshot>();
|
||||
const [rpcAvailable, setRpcAvailable] = useState(false);
|
||||
@@ -127,7 +128,10 @@ export default function App() {
|
||||
} catch (e) { setTransferError(message(e)); }
|
||||
finally { transferInFlight.current = false; setTransferBusy(false); }
|
||||
};
|
||||
const owned = snapshot?.blocks.filter(b => wallet && testnetOwnerOf(snapshot, b.tokenId) === wallet.address) ?? [];
|
||||
// Every canonical card owned by this browser's wallet or any wallet linked to the eom.dev account (#57).
|
||||
// The full canonical chain is paged in by StateClient, so nothing is capped at 128 blocks.
|
||||
const mine = new Set([wallet?.address, ...(linkedAccount?.wallets ?? []).map(w => w.address)].filter(Boolean));
|
||||
const owned = snapshot?.blocks.filter(b => mine.has(testnetOwnerOf(snapshot, b.tokenId) ?? '')) ?? [];
|
||||
const visibleCards = view === 'My collection' ? owned.map(asCard)
|
||||
: snapshot?.blocks.length ? [...snapshot.blocks].reverse().slice(0, 12).map(asCard) : samples;
|
||||
const filtered = visibleCards.filter(c => (category === 'All cards' || c.article.category === category) &&
|
||||
|
||||
@@ -54,7 +54,7 @@ test('fixed HTTPS sources, encoded title, hidden categories, lower bounds and pl
|
||||
assert.equal(init.redirect, 'error');
|
||||
assert.ok(init.signal instanceof AbortSignal);
|
||||
assert.equal(new Headers(init.headers).get('accept'), 'application/json');
|
||||
assert.match(new Headers(init.headers).get('user-agent')!, /WikiDeck/i);
|
||||
assert.match(new Headers(init.headers).get('user-agent')!, /Wikideck/);
|
||||
assert.equal(new Headers(init.headers).has('origin'), false); // Server-side requests need no CORS workaround.
|
||||
}
|
||||
assert.equal(result.title, title);
|
||||
|
||||
@@ -0,0 +1,291 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { createHash } from 'node:crypto';
|
||||
import express from 'express';
|
||||
import type { AddressInfo } from 'node:net';
|
||||
import { resolveDeck, collection, verifyAccount, playableCount, ownershipRules, DEFAULT_OWNERSHIP_RULES,
|
||||
DeckError, OwnershipUnavailable, type OwnershipDeps, type OwnershipLedger } from '../server/game/ownership.ts';
|
||||
import { collectionRouter, collectionResponse } from '../server/game/collection.ts';
|
||||
import type { Account } from '../server/account/store.ts';
|
||||
import type { TestnetBlock } from '../shared/testnet.ts';
|
||||
|
||||
// In-memory fixture: the same shape as Ledger.nft/deck/summary and AccountStore.wallets. No network.
|
||||
const hex = (s: string) => createHash('sha256').update(s).digest('hex');
|
||||
const ALICE = hex('alice'), ALICE2 = hex('alice-2'), BOB = hex('bob');
|
||||
|
||||
function block(n: number, pageId: number, miner: string): TestnetBlock {
|
||||
return {
|
||||
version: 6, network: 'testnet', chainId: 'test', height: n, previousHash: hex(`b${n - 1}`), createdAt: 1_800_000_000 + n,
|
||||
article: { pageId, title: `Article ${pageId}`, extract: `Extract ${pageId}.`, url: `https://en.wikipedia.org/?curid=${pageId}`,
|
||||
revisionId: 1000 + pageId, source: 'live', category: 'History' },
|
||||
statsSnapshot: { capturedAt: '2026-11-17T00:00:00Z', revisionId: 1000 + pageId, pageBytes: 5000 + pageId, languageCount: 3,
|
||||
languagesTruncated: false, pageviews: null } as unknown as TestnetBlock['statsSnapshot'],
|
||||
owner: miner, publicKey: 'pk', nonce: 0, target: 'f'.repeat(64), key: '0'.repeat(64), transactionsRoot: '0'.repeat(64),
|
||||
stateRoot: '0'.repeat(64), hash: hex(`b${n}`), tokenId: hex(`t${n}`), powHash: hex(`p${n}`), signature: 'sig', transactions: [],
|
||||
} as unknown as TestnetBlock;
|
||||
}
|
||||
|
||||
class FakeLedger implements OwnershipLedger {
|
||||
tip = 'tip-0';
|
||||
height = 0;
|
||||
cards: { block: TestnetBlock; owner: string }[] = [];
|
||||
deckCalls = 0;
|
||||
onDeck?: () => void;
|
||||
mint(owner: string, pageId: number): string {
|
||||
const b = block(++this.height, pageId, owner);
|
||||
this.cards.push({ block: b, owner });
|
||||
this.tip = `tip-${this.height}`;
|
||||
return b.tokenId;
|
||||
}
|
||||
mineEmpty(n = 1) { this.height += n; this.tip = `tip-${this.height}`; }
|
||||
transfer(token: string, to: string) { this.cards.find(c => c.block.tokenId === token)!.owner = to; }
|
||||
private conf(b: TestnetBlock) { return this.height - b.height + 1; }
|
||||
nft(token: string) {
|
||||
const c = this.cards.find(x => x.block.tokenId === token);
|
||||
return c ? { block: structuredClone(c.block), owner: c.owner, sequence: 0, confirmations: this.conf(c.block) } : undefined;
|
||||
}
|
||||
deck(address: string, offset: number, limit: number) {
|
||||
this.deckCalls++;
|
||||
assert.ok(limit <= 24, 'ledger pages are at most 24');
|
||||
const mine = this.cards.filter(c => c.owner === address);
|
||||
const nfts = mine.slice(offset, offset + limit).map(c => ({ block: structuredClone(c.block), sequence: 0, confirmations: this.conf(c.block) }));
|
||||
const result = { tip: this.tip, total: mine.length, next: offset + nfts.length < mine.length ? offset + nfts.length : null, nfts };
|
||||
this.onDeck?.();
|
||||
return result;
|
||||
}
|
||||
summary() { return { tip: this.tip, height: this.height, network: 'testnet', chainId: 'test' }; }
|
||||
}
|
||||
|
||||
function fixture(wallets: Record<string, string[]> = { alice: [ALICE, ALICE2], bob: [BOB] }) {
|
||||
const ledger = new FakeLedger();
|
||||
const deps: OwnershipDeps = { ledger, accounts: { wallets: sub => (wallets[sub] ?? []).map(address => ({ address })) },
|
||||
verifyIdToken: async (t: string): Promise<Account> => {
|
||||
const m = /^good\.token\.(\w+)$/.exec(t);
|
||||
if (m) return { sub: m[1], issuer: 'https://dex.example/', name: m[1], username: m[1] };
|
||||
if (t === 'dex.is.down') throw new Error('fetch failed');
|
||||
throw Object.assign(new Error('Token expired'), { status: 401 });
|
||||
} };
|
||||
return { ledger, deps };
|
||||
}
|
||||
/** n cards for `owner`, distinct pageIds starting at `firstPage`. */
|
||||
const mintMany = (l: FakeLedger, owner: string, n: number, firstPage = 1) => Array.from({ length: n }, (_, i) => l.mint(owner, firstPage + i));
|
||||
|
||||
const ALICE_ACCT = { sub: 'alice' }, BOB_ACCT = { sub: 'bob' };
|
||||
async function rejects(p: Promise<unknown>, code: string, tokenIds?: string[]) {
|
||||
await assert.rejects(p, (e: unknown) => {
|
||||
assert.ok(e instanceof DeckError, String(e));
|
||||
assert.equal(e.code, code);
|
||||
if (tokenIds) assert.deepEqual(e.tokenIds, tokenIds);
|
||||
return true;
|
||||
});
|
||||
}
|
||||
|
||||
test('defaults are a 40-card deck, 4 copies per article, 1 confirmation', () => {
|
||||
assert.deepEqual({ ...DEFAULT_OWNERSHIP_RULES }, { deckSize: 40, maxCopiesPerPage: 4, minConfirmations: 1 });
|
||||
assert.equal(ownershipRules({ deckSize: 10, minConfirmations: undefined }).minConfirmations, 1);
|
||||
assert.throws(() => ownershipRules({ deckSize: 0 }), RangeError);
|
||||
assert.throws(() => ownershipRules({ maxCopiesPerPage: 1.5 }), RangeError);
|
||||
assert.throws(() => ownershipRules({ minConfirmations: 0 }), RangeError);
|
||||
});
|
||||
|
||||
test('verifyAccount maps token failures to DeckError(auth) and Dex outages to OwnershipUnavailable', async () => {
|
||||
const { deps } = fixture();
|
||||
assert.equal((await verifyAccount(deps, 'good.token.alice')).sub, 'alice');
|
||||
for (const t of [undefined, '', 42, 'x'.repeat(9000)]) await rejects(verifyAccount(deps, t), 'auth');
|
||||
await assert.rejects(verifyAccount(deps, 'bad'), (e: unknown) => e instanceof DeckError && e.code === 'auth' && e.status === 401 && e.message === 'Token expired');
|
||||
await assert.rejects(verifyAccount(deps, 'dex.is.down'), (e: unknown) => e instanceof OwnershipUnavailable && e.status === 503);
|
||||
});
|
||||
|
||||
test('resolveDeck accepts 40 owned cards across linked wallets and builds them from the chain, in order', async () => {
|
||||
const { ledger, deps } = fixture();
|
||||
const ids = [...mintMany(ledger, ALICE, 25, 1), ...mintMany(ledger, ALICE2, 15, 100)].reverse();
|
||||
const cards = await resolveDeck(deps, ALICE_ACCT, ids);
|
||||
assert.equal(cards.length, 40);
|
||||
assert.deepEqual(cards.map(c => c.tokenId), ids);
|
||||
const c = cards.find(x => x.pageId === 100)!;
|
||||
const chain = ledger.nft(c.tokenId)!.block;
|
||||
assert.equal(c.owner, ALICE2);
|
||||
assert.equal(c.title, 'Article 100');
|
||||
assert.equal(c.revisionId, 1100);
|
||||
assert.equal(c.article.title, 'Article 100');
|
||||
assert.equal(c.statsSnapshot.pageBytes, 5100);
|
||||
assert.equal(c.blockHash, chain.hash);
|
||||
assert.equal(c.powHash, chain.powHash);
|
||||
assert.equal(c.height, chain.height);
|
||||
});
|
||||
|
||||
test('resolveDeck takes token ids only: card data is never taken from the client', async () => {
|
||||
const { ledger, deps } = fixture();
|
||||
const ids = mintMany(ledger, ALICE, 40);
|
||||
const forged = ids.map(tokenId => ({ tokenId, article: { title: 'Barack Obama' }, statsSnapshot: { pageBytes: 9e9 } }));
|
||||
await rejects(resolveDeck(deps, ALICE_ACCT, forged), 'unknown-token');
|
||||
});
|
||||
|
||||
test('resolveDeck enforces the exact deck size from ruleset data', async () => {
|
||||
const { ledger, deps } = fixture();
|
||||
const ids = mintMany(ledger, ALICE, 41);
|
||||
for (const deck of [ids.slice(0, 39), ids, [], 'nope', null, Array(10_000).fill(ids[0])]) await rejects(resolveDeck(deps, ALICE_ACCT, deck), 'size');
|
||||
assert.equal((await resolveDeck(deps, ALICE_ACCT, ids.slice(0, 10), { deckSize: 10 })).length, 10, 'a testnet mode may lower deckSize');
|
||||
});
|
||||
|
||||
test('resolveDeck rejects malformed and duplicated token ids', async () => {
|
||||
const { ledger, deps } = fixture();
|
||||
const ids = mintMany(ledger, ALICE, 40);
|
||||
await rejects(resolveDeck(deps, ALICE_ACCT, [...ids.slice(0, 39), 'ABC']), 'unknown-token', ['ABC']);
|
||||
await rejects(resolveDeck(deps, ALICE_ACCT, [...ids.slice(0, 39), ids[0]]), 'duplicate-token', [ids[0]]);
|
||||
});
|
||||
|
||||
test('resolveDeck requires a linked wallet', async () => {
|
||||
const { ledger, deps } = fixture();
|
||||
const ids = mintMany(ledger, ALICE, 40);
|
||||
await assert.rejects(resolveDeck(deps, { sub: 'stranger' }, ids), (e: unknown) => e instanceof DeckError && e.code === 'not-owned' && e.status === 403 && /Link a wallet/.test(e.message));
|
||||
});
|
||||
|
||||
test('resolveDeck rejects unknown cards and cards owned by someone else, listing every one', async () => {
|
||||
const { ledger, deps } = fixture();
|
||||
const ids = mintMany(ledger, ALICE, 38);
|
||||
const bobs = mintMany(ledger, BOB, 2, 500);
|
||||
await rejects(resolveDeck(deps, ALICE_ACCT, [...ids, ...bobs]), 'not-owned', bobs);
|
||||
await rejects(resolveDeck(deps, ALICE_ACCT, [...ids, hex('nope'), bobs[0]]), 'unknown-token', [hex('nope')]);
|
||||
// Bob cannot field Alice's cards either, even though they exist.
|
||||
await rejects(resolveDeck(deps, BOB_ACCT, [...ids, ...bobs]), 'not-owned', ids);
|
||||
});
|
||||
|
||||
test('ownership follows transfers: the current owner, not the miner, may play the card', async () => {
|
||||
const { ledger, deps } = fixture();
|
||||
const ids = mintMany(ledger, ALICE, 40);
|
||||
ledger.transfer(ids[7], BOB);
|
||||
await rejects(resolveDeck(deps, ALICE_ACCT, ids), 'not-owned', [ids[7]]);
|
||||
const cards = await resolveDeck(deps, BOB_ACCT, [ids[7], ...mintMany(ledger, BOB, 39, 300)]);
|
||||
assert.equal(cards[0].owner, BOB);
|
||||
});
|
||||
|
||||
test('resolveDeck requires minConfirmations (3 for rated play)', async () => {
|
||||
const { ledger, deps } = fixture();
|
||||
const ids = mintMany(ledger, ALICE, 40);
|
||||
// The last card has 1 confirmation, the one before it 2.
|
||||
await rejects(resolveDeck(deps, ALICE_ACCT, ids, { minConfirmations: 3 }), 'unconfirmed', ids.slice(38));
|
||||
ledger.mineEmpty(2);
|
||||
assert.equal((await resolveDeck(deps, ALICE_ACCT, ids, { minConfirmations: 3 })).length, 40);
|
||||
});
|
||||
|
||||
test('duplicates of an article count as separate cards, up to 4 copies per pageId', async () => {
|
||||
const { ledger, deps } = fixture();
|
||||
const base = mintMany(ledger, ALICE, 35, 1);
|
||||
const copies = Array.from({ length: 5 }, () => ledger.mint(ALICE, 777));
|
||||
await assert.rejects(resolveDeck(deps, ALICE_ACCT, [...base, ...copies]),
|
||||
(e: unknown) => e instanceof DeckError && e.code === 'copy-limit' && e.pageId === 777 && e.tokenIds.length === 5);
|
||||
const cards = await resolveDeck(deps, ALICE_ACCT, [...base, ...copies.slice(0, 4), ledger.mint(ALICE, 900)]);
|
||||
assert.equal(cards.filter(c => c.pageId === 777).length, 4);
|
||||
assert.equal((await resolveDeck(deps, ALICE_ACCT, [...base, ...copies], { maxCopiesPerPage: 5 })).length, 40, 'the copy limit is ruleset data');
|
||||
});
|
||||
|
||||
test('playableCount applies the copy limit', () => {
|
||||
const cards = [1, 1, 1, 1, 1, 1, 2, 3].map(pageId => ({ pageId }));
|
||||
assert.equal(playableCount(cards, 4), 6);
|
||||
assert.equal(playableCount(cards, 1), 3);
|
||||
});
|
||||
|
||||
test('collection lists every owned card across wallets with no 128-block cap', async () => {
|
||||
const { ledger, deps } = fixture();
|
||||
mintMany(ledger, ALICE, 150, 1);
|
||||
mintMany(ledger, BOB, 10, 1000);
|
||||
mintMany(ledger, ALICE2, 3, 2000);
|
||||
const cards = await collection(deps, ALICE_ACCT);
|
||||
assert.equal(cards.length, 153);
|
||||
assert.equal(new Set(cards.map(x => x.tokenId)).size, 153);
|
||||
assert.ok(cards.every(x => x.owner === ALICE || x.owner === ALICE2));
|
||||
const r = collectionResponse(deps, 'alice');
|
||||
assert.equal(r.owned, 153); assert.equal(r.playable, 153); assert.equal(r.needed, 0); assert.equal(r.eligible, true);
|
||||
assert.deepEqual(r.wallets, [ALICE, ALICE2]);
|
||||
assert.equal(r.tip, ledger.tip);
|
||||
assert.deepEqual(await collection(deps, { sub: 'stranger' }), []);
|
||||
});
|
||||
|
||||
test('collection shows progress toward 40 and applies the copy limit to eligibility', () => {
|
||||
const { ledger, deps } = fixture();
|
||||
mintMany(ledger, ALICE, 20, 1);
|
||||
for (let i = 0; i < 10; i++) ledger.mint(ALICE, 4242);
|
||||
const c = collectionResponse(deps, 'alice');
|
||||
assert.equal(c.owned, 30); assert.equal(c.playable, 24); assert.equal(c.needed, 16); assert.equal(c.eligible, false);
|
||||
assert.equal(c.deckSize, 40); assert.equal(c.copyLimit, 4); assert.equal(c.maxCopiesPerPage, 4);
|
||||
assert.equal(collectionResponse(deps, 'alice', { deckSize: 20 }).eligible, true);
|
||||
const none = collectionResponse(deps, 'stranger');
|
||||
assert.equal(none.owned, 0); assert.equal(none.needed, 40); assert.deepEqual(none.cards, []);
|
||||
});
|
||||
|
||||
test('collection only counts cards with enough confirmations', async () => {
|
||||
const { ledger, deps } = fixture();
|
||||
mintMany(ledger, ALICE, 5);
|
||||
assert.equal((await collection(deps, ALICE_ACCT, { minConfirmations: 3 })).length, 3);
|
||||
});
|
||||
|
||||
test('collection restarts when the tip moves mid-scan and gives up if it never settles', async () => {
|
||||
const { ledger, deps } = fixture();
|
||||
mintMany(ledger, ALICE, 60);
|
||||
let moves = 1;
|
||||
ledger.onDeck = () => { if (moves-- > 0) ledger.mint(ALICE, 9999); };
|
||||
const c = collectionResponse(deps, 'alice');
|
||||
assert.equal(c.owned, 61);
|
||||
assert.equal(c.tip, ledger.tip);
|
||||
ledger.onDeck = () => { ledger.mineEmpty(); };
|
||||
await assert.rejects(collection(deps, ALICE_ACCT), OwnershipUnavailable);
|
||||
});
|
||||
|
||||
async function serve(router: express.Router) {
|
||||
const app = express();
|
||||
app.use('/api/game/collection', router);
|
||||
const server = app.listen(0, '127.0.0.1');
|
||||
await new Promise(r => server.once('listening', r));
|
||||
const base = `http://127.0.0.1:${(server.address() as AddressInfo).port}/api/game/collection`;
|
||||
return { base, close: () => new Promise(r => server.close(r)) };
|
||||
}
|
||||
|
||||
test('GET /api/game/collection requires a verified eom.dev token (local loopback only, no upstream)', async () => {
|
||||
const { ledger, deps } = fixture();
|
||||
mintMany(ledger, ALICE, 23);
|
||||
const { base, close } = await serve(collectionRouter(deps));
|
||||
const as = (t: string) => ({ headers: { authorization: `Bearer ${t}` } });
|
||||
try {
|
||||
let res = await fetch(base);
|
||||
assert.equal(res.status, 401);
|
||||
res = await fetch(base, as('bad.token.x!'));
|
||||
assert.equal(res.status, 401);
|
||||
res = await fetch(base, as('expired.token.x'));
|
||||
assert.equal(res.status, 401);
|
||||
assert.equal((await res.json()).error, 'Token expired');
|
||||
res = await fetch(base, as('dex.is.down'));
|
||||
assert.equal(res.status, 503);
|
||||
res = await fetch(`${base}?minConfirmations=abc`, as('good.token.alice'));
|
||||
assert.equal(res.status, 400);
|
||||
res = await fetch(base, as('good.token.alice'));
|
||||
assert.equal(res.status, 200);
|
||||
assert.equal(res.headers.get('cache-control'), 'no-store');
|
||||
const body = await res.json();
|
||||
assert.equal(body.owned, 23); assert.equal(body.playable, 23); assert.equal(body.needed, 17);
|
||||
assert.equal(body.eligible, false); assert.equal(body.deckSize, 40); assert.equal(body.copyLimit, 4);
|
||||
assert.equal(body.cards.length, 23);
|
||||
assert.equal(body.cards[0].title, 'Article 1');
|
||||
assert.equal(body.cards[0].pageId, 1);
|
||||
res = await fetch(`${base}?minConfirmations=3`, as('good.token.alice'));
|
||||
assert.equal((await res.json()).owned, 21);
|
||||
res = await fetch(base, as('good.token.bob'));
|
||||
assert.equal((await res.json()).owned, 0);
|
||||
} finally { await close(); }
|
||||
});
|
||||
|
||||
test('GET /api/game/collection attaches the injected duel card when one is ready, and omits it otherwise', async () => {
|
||||
const { ledger, deps } = fixture();
|
||||
mintMany(ledger, ALICE, 3);
|
||||
const duelCard = (c: { pageId: number }) => {
|
||||
if (c.pageId === 2) return undefined; // profile pending
|
||||
if (c.pageId === 3) throw new Error('boom'); // never breaks the collection
|
||||
return { kind: 'Figure', pageId: c.pageId };
|
||||
};
|
||||
const { base, close } = await serve(collectionRouter({ ...deps, duelCard }));
|
||||
try {
|
||||
const body = await (await fetch(base, { headers: { authorization: 'Bearer good.token.alice' } })).json();
|
||||
assert.deepEqual(body.cards.map((c: { duel?: unknown }) => c.duel), [{ kind: 'Figure', pageId: 1 }, undefined, undefined]);
|
||||
assert.ok(!('duel' in body.cards[1]));
|
||||
} finally { await close(); }
|
||||
});
|
||||
@@ -2,7 +2,7 @@ import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { WikipediaArticleSource } from '../server/wiki.ts';
|
||||
import { WikipediaEnrichment } from '../server/enrichment.ts';
|
||||
import { PROJECT_URL, defaultUserAgent, wikimediaUserAgent } from '../server/wikimedia-user-agent.ts';
|
||||
import { ARTICLE_USER_AGENT, ENRICHMENT_USER_AGENT, wikimediaUserAgent } from '../server/wikimedia-user-agent.ts';
|
||||
|
||||
function capture() {
|
||||
const agents: string[] = [];
|
||||
@@ -12,10 +12,9 @@ function capture() {
|
||||
};
|
||||
return { agents, fetcher };
|
||||
}
|
||||
async function agentsFor(env: string | undefined, contact?: string) {
|
||||
const previous = process.env.WIKIDECK_USER_AGENT, previousContact = process.env.WIKIDECK_OPERATOR_CONTACT;
|
||||
async function agentsFor(env: string | undefined) {
|
||||
const previous = process.env.WIKIDECK_USER_AGENT;
|
||||
if (env === undefined) delete process.env.WIKIDECK_USER_AGENT; else process.env.WIKIDECK_USER_AGENT = env;
|
||||
if (contact === undefined) delete process.env.WIKIDECK_OPERATOR_CONTACT; else process.env.WIKIDECK_OPERATOR_CONTACT = contact;
|
||||
try {
|
||||
const article = capture(), enrichment = capture();
|
||||
await assert.rejects(new WikipediaArticleSource(article.fetcher).random());
|
||||
@@ -23,22 +22,15 @@ async function agentsFor(env: string | undefined, contact?: string) {
|
||||
return { article: article.agents, enrichment: enrichment.agents };
|
||||
} finally {
|
||||
if (previous === undefined) delete process.env.WIKIDECK_USER_AGENT; else process.env.WIKIDECK_USER_AGENT = previous;
|
||||
if (previousContact === undefined) delete process.env.WIKIDECK_OPERATOR_CONTACT;
|
||||
else process.env.WIKIDECK_OPERATOR_CONTACT = previousContact;
|
||||
}
|
||||
}
|
||||
|
||||
test('default Wikimedia user agent names the project, the operator contact and node', async () => {
|
||||
const { article, enrichment } = await agentsFor(undefined, 'ops@example.org');
|
||||
const expected = `WikiDeck/0.1.0 (+${PROJECT_URL}; ops@example.org) node/${process.versions.node}`;
|
||||
assert.equal(defaultUserAgent({ WIKIDECK_OPERATOR_CONTACT: 'ops@example.org' }), expected);
|
||||
assert.ok(article.length > 0 && article.every(a => a === expected));
|
||||
assert.ok(enrichment.length > 0 && enrichment.every(a => a === expected));
|
||||
assert.equal(defaultUserAgent({ PUBLIC_ORIGIN: 'https://node.example' }),
|
||||
`WikiDeck/0.1.0 (+${PROJECT_URL}; https://node.example) node/${process.versions.node}`);
|
||||
assert.equal(defaultUserAgent({}), `WikiDeck/0.1.0 (+${PROJECT_URL}) node/${process.versions.node}`);
|
||||
// Unsafe contacts are dropped rather than breaking the header's comment syntax.
|
||||
assert.equal(defaultUserAgent({ WIKIDECK_OPERATOR_CONTACT: 'a); evil(' }), defaultUserAgent({}));
|
||||
test('default Wikimedia user agents are unchanged when WIKIDECK_USER_AGENT is unset', async () => {
|
||||
assert.equal(ARTICLE_USER_AGENT, 'Wikideck/0.1 (experimental Wikipedia article cards)');
|
||||
assert.equal(ENRICHMENT_USER_AGENT, 'Wikideck/0.1 (Wikipedia off-chain page enrichment)');
|
||||
const { article, enrichment } = await agentsFor(undefined);
|
||||
assert.ok(article.length > 0 && article.every(a => a === ARTICLE_USER_AGENT));
|
||||
assert.ok(enrichment.length > 0 && enrichment.every(a => a === ENRICHMENT_USER_AGENT));
|
||||
});
|
||||
|
||||
test('WIKIDECK_USER_AGENT overrides both Wikimedia clients', async () => {
|
||||
|
||||
+1
-4
@@ -56,16 +56,13 @@ test('random article fetch rejects unsafe response types, redirects, and oversiz
|
||||
() => new Response('{}', { headers: { 'Content-Type': 'text/html' } }),
|
||||
() => new Response(null, { status: 302, headers: { Location: 'https://evil.example' } }),
|
||||
() => new Response(' '.repeat(512 * 1024 + 1), { headers: { 'Content-Type': 'application/json' } }),
|
||||
() => { throw new Error('offline'); },
|
||||
];
|
||||
for (const fail of failures) {
|
||||
const { fetcher, calls } = mockFetch(() => fail());
|
||||
await assert.rejects(new WikipediaArticleSource(fetcher).random());
|
||||
assert.equal(calls.length, 3);
|
||||
}
|
||||
// A network failure (or throttling) stops at once instead of retrying against an unreachable upstream.
|
||||
const { fetcher, calls } = mockFetch(() => { throw new Error('offline'); });
|
||||
await assert.rejects(new WikipediaArticleSource(fetcher).random(), /Wikimedia unavailable/);
|
||||
assert.equal(calls.length, 1);
|
||||
});
|
||||
|
||||
test('bad page records are skipped while later valid candidates are considered', async () => {
|
||||
|
||||
@@ -1,236 +0,0 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { WikimediaClient, WikimediaUnavailable, configFromEnv, type ClientConfig } from '../server/wikimedia/client.ts';
|
||||
import { WikimediaCache, cacheKey } from '../server/wikimedia/cache.ts';
|
||||
import { warnIfNoContact } from '../server/wikimedia-user-agent.ts';
|
||||
import { FakeClock, flush, json, recorder } from './wikimedia-helpers.ts';
|
||||
|
||||
const API = 'https://en.wikipedia.org/w/api.php?action=query&format=json&pageids=1';
|
||||
const REST = 'https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/x';
|
||||
|
||||
function setup(handler: Parameters<typeof recorder>[1] = () => json({ ok: true }), config: Partial<ClientConfig> = {},
|
||||
cache?: WikimediaCache) {
|
||||
const clock = new FakeClock();
|
||||
const rec = recorder(clock, handler);
|
||||
const client = new WikimediaClient({ fetch: rec.fetcher, clock, config, cache, userAgent: () => 'Test/1 (+x; y)' });
|
||||
return { clock, client, ...rec };
|
||||
}
|
||||
|
||||
test('token bucket: burst of 3, then 1 request per second, one in flight per host', async () => {
|
||||
const { clock, client, calls } = setup();
|
||||
const jobs = Array.from({ length: 6 }, () => client.request(new URL(API), { lane: 'interactive', timeoutMs: 1000 })
|
||||
.then(r => r.response.body?.cancel()));
|
||||
await clock.advance(0);
|
||||
assert.equal(calls.length, 3);
|
||||
await clock.advance(999);
|
||||
assert.equal(calls.length, 3);
|
||||
await clock.advance(1);
|
||||
assert.equal(calls.length, 4);
|
||||
await clock.advance(2000);
|
||||
await Promise.all(jobs);
|
||||
assert.deepEqual(calls.map(c => c.at - calls[0].at), [0, 0, 0, 1000, 2000, 3000]);
|
||||
// Different hosts have separate buckets.
|
||||
await client.request(new URL(REST), { lane: 'interactive', timeoutMs: 1000 });
|
||||
assert.equal(calls.at(-1)!.url.hostname, 'wikimedia.org');
|
||||
});
|
||||
|
||||
test('only one request is in flight per host, even with tokens to spare', async () => {
|
||||
const releases: (() => void)[] = [];
|
||||
const { clock, client, calls } = setup(() => new Promise(resolve => releases.push(() => resolve(json({})))));
|
||||
const a = client.request(new URL(API), { lane: 'interactive', timeoutMs: 1000 });
|
||||
const b = client.request(new URL(API), { lane: 'interactive', timeoutMs: 1000 });
|
||||
await clock.advance(0);
|
||||
assert.equal(calls.length, 1);
|
||||
releases.shift()!();
|
||||
await a;
|
||||
await clock.advance(0);
|
||||
assert.equal(calls.length, 2);
|
||||
releases.shift()!();
|
||||
await b;
|
||||
});
|
||||
|
||||
test('priority lanes: mint, then interactive, then background', async () => {
|
||||
const { clock, client, calls } = setup(url => json({ lane: url.searchParams.get('n') }), { burst: 1 });
|
||||
await client.request(new URL(`${API}&n=warm`), { lane: 'background', timeoutMs: 1000 }); // Uses the only token.
|
||||
const order: string[] = [];
|
||||
const go = (lane: 'mint' | 'interactive' | 'background', n: string) =>
|
||||
client.request(new URL(`${API}&n=${n}`), { lane, timeoutMs: 1000 }).then(() => order.push(n));
|
||||
const jobs = [go('background', 'b1'), go('interactive', 'i1'), go('background', 'b2'), go('mint', 'm1'), go('interactive', 'i2')];
|
||||
await clock.advance(5000);
|
||||
await Promise.all(jobs);
|
||||
assert.deepEqual(order, ['m1', 'i1', 'i2', 'b1', 'b2']);
|
||||
assert.equal(calls.length, 6);
|
||||
});
|
||||
|
||||
test('maxlag=5 is added to Action API requests only, and a maxlag reply waits for Retry-After', async () => {
|
||||
let lagged = 1;
|
||||
const { clock, client, calls } = setup(url => url.pathname === '/w/api.php' && lagged-- > 0
|
||||
? json({ error: { code: 'maxlag' } }, { 'MediaWiki-API-Error': 'maxlag', 'Retry-After': '5' })
|
||||
: json({ ok: true }));
|
||||
const job = client.request(new URL(API), { lane: 'background', timeoutMs: 1000 });
|
||||
await clock.advance(0);
|
||||
assert.equal(calls.length, 1);
|
||||
assert.equal(calls[0].url.searchParams.get('maxlag'), '5');
|
||||
await clock.advance(4999);
|
||||
assert.equal(calls.length, 1);
|
||||
await clock.advance(1);
|
||||
const { response } = await job;
|
||||
assert.deepEqual(await response.json(), { ok: true });
|
||||
assert.equal(calls.length, 2);
|
||||
await client.request(new URL(REST), { lane: 'background', timeoutMs: 1000 });
|
||||
assert.equal(calls[2].url.searchParams.has('maxlag'), false);
|
||||
assert.equal(calls[0].headers.get('user-agent'), 'Test/1 (+x; y)');
|
||||
});
|
||||
|
||||
test('429 honours Retry-After by pausing the host; three in a row open the circuit for 10 minutes', async () => {
|
||||
let status = 429;
|
||||
const { clock, client, calls } = setup(() => status === 200 ? json({}) : new Response('slow down',
|
||||
{ status, headers: { 'Retry-After': '2' } }));
|
||||
const req = (lane: 'mint' | 'background' = 'background') => client.request(new URL(API), { lane, timeoutMs: 1000 });
|
||||
await assert.rejects(req(), (e: unknown) => e instanceof WikimediaUnavailable && e.status === 429);
|
||||
// The next request waits out Retry-After (2 s) instead of going straight back.
|
||||
const second = req().catch(e => e);
|
||||
await clock.advance(1999);
|
||||
assert.equal(calls.length, 1);
|
||||
await clock.advance(1);
|
||||
assert.equal(((await second) as WikimediaUnavailable).status, 429);
|
||||
const third = req().catch(e => e);
|
||||
await clock.advance(10_000);
|
||||
assert.equal(((await third) as WikimediaUnavailable).status, 429);
|
||||
assert.equal(calls.length, 3);
|
||||
assert.equal(client.status().hosts['en.wikipedia.org'].circuitOpen, true);
|
||||
// Open circuit: refused at once without touching Wikimedia, even on the mint lane.
|
||||
status = 200;
|
||||
await assert.rejects(req('mint'), /circuit open/);
|
||||
assert.equal(calls.length, 3);
|
||||
await clock.advance(10 * 60_000);
|
||||
await req('mint');
|
||||
assert.equal(calls.length, 4);
|
||||
assert.equal(client.status().counters['breaker.open'], 1);
|
||||
});
|
||||
|
||||
test('a success resets the consecutive-429 count', async () => {
|
||||
const replies = [429, 200, 429, 200, 429];
|
||||
const { clock, client } = setup(() => {
|
||||
const status = replies.shift()!;
|
||||
return status === 200 ? json({}) : new Response('', { status });
|
||||
});
|
||||
for (let i = 0; i < 5; i++) {
|
||||
const job = client.request(new URL(API), { lane: 'background', timeoutMs: 1000 }).catch(() => {});
|
||||
await clock.advance(60_000);
|
||||
await job;
|
||||
}
|
||||
assert.equal(client.status().hosts['en.wikipedia.org'].circuitOpen, false);
|
||||
});
|
||||
|
||||
test('daily cap: background stops at 80 %, interactive at 100 %, mint keeps a 10 % reserve; resets at UTC midnight', async () => {
|
||||
const { clock, client } = setup(() => json({}), { dailyCap: 10, requestsPerSecond: 1000, burst: 1000 });
|
||||
const req = (lane: 'mint' | 'interactive' | 'background') => client.request(new URL(API), { lane, timeoutMs: 1000 });
|
||||
for (let i = 0; i < 8; i++) await req('background');
|
||||
await assert.rejects(req('background'), /daily cap/);
|
||||
await req('interactive'); await req('interactive');
|
||||
await assert.rejects(req('interactive'), /daily cap/);
|
||||
await req('mint');
|
||||
await assert.rejects(req('mint'), /daily cap/);
|
||||
await clock.advance(24 * 3600_000);
|
||||
await req('background');
|
||||
});
|
||||
|
||||
test('nodes sharing one egress IP split the rate and the daily cap', () => {
|
||||
const config = configFromEnv({ WIKIDECK_WIKIMEDIA_RPS: '0.9', WIKIDECK_WIKIMEDIA_NODES_PER_IP: '3',
|
||||
WIKIDECK_WIKIMEDIA_DAILY_CAP: '3000' });
|
||||
const client = new WikimediaClient({ fetch: async () => json({}), config });
|
||||
assert.ok(Math.abs(client.status().requestsPerSecond - 0.3) < 1e-9);
|
||||
assert.equal(client.status().dailyCap, 1000);
|
||||
assert.equal(configFromEnv({ WIKIDECK_WIKIMEDIA_RPS: 'abc' }).requestsPerSecond, 1);
|
||||
});
|
||||
|
||||
test('a full queue sheds background work first and a mint request evicts a background waiter', async () => {
|
||||
const { clock, client } = setup(() => json({}), { queueLimit: 2, burst: 1 });
|
||||
await client.request(new URL(API), { lane: 'interactive', timeoutMs: 1000 });
|
||||
const b1 = client.request(new URL(API), { lane: 'background', timeoutMs: 1000 }).then(() => 'ok', e => e.message);
|
||||
const b2 = client.request(new URL(API), { lane: 'background', timeoutMs: 1000 }).then(() => 'ok', e => e.message);
|
||||
await assert.rejects(client.request(new URL(API), { lane: 'background', timeoutMs: 1000 }), /queue full/);
|
||||
const mint = client.request(new URL(API), { lane: 'mint', timeoutMs: 1000 });
|
||||
await clock.advance(5000);
|
||||
await mint;
|
||||
assert.deepEqual([await b1, await b2], ['ok', 'Wikimedia unavailable: queue full']);
|
||||
});
|
||||
|
||||
test('waiting longer than the lane allows fails instead of queueing forever', async () => {
|
||||
const { clock, client } = setup(() => json({}), { burst: 1, requestsPerSecond: 0.01, maxWaitMs: { interactive: 3000 } as ClientConfig['maxWaitMs'] });
|
||||
await client.request(new URL(API), { lane: 'interactive', timeoutMs: 1000 });
|
||||
const late = client.request(new URL(API), { lane: 'interactive', timeoutMs: 1000 }).catch(e => e);
|
||||
await clock.advance(3000);
|
||||
assert.match(String(await late), /queue wait/);
|
||||
});
|
||||
|
||||
test('disk cache: ETag revalidation with If-None-Match, 304 refresh, stale-while-revalidate on the background lane', async () => {
|
||||
const cache = new WikimediaCache(':memory:');
|
||||
let body = { v: 1 };
|
||||
const { clock, client, calls } = setup((_url, headers) => headers.get('if-none-match') === '"v1"' && body.v === 1
|
||||
? new Response(null, { status: 304 }) : json(body, { ETag: `"v${body.v}"` }), {}, cache);
|
||||
const policy = { ttlMs: 1000, staleWhileRevalidate: false };
|
||||
const read = async (lane: 'interactive' | 'background' = 'interactive', cachePolicy = policy) =>
|
||||
(await (await client.request(new URL(API), { lane, timeoutMs: 1000, cache: cachePolicy })).response.json());
|
||||
assert.deepEqual(await read(), { v: 1 });
|
||||
assert.deepEqual(await read(), { v: 1 });
|
||||
assert.equal(calls.length, 1); // Fresh hit.
|
||||
await clock.advance(1000);
|
||||
assert.deepEqual(await read(), { v: 1 });
|
||||
assert.equal(calls.length, 2);
|
||||
assert.equal(calls[1].headers.get('if-none-match'), '"v1"'); // Conditional request, answered 304.
|
||||
assert.equal(client.status().counters['cache.revalidated'], 1);
|
||||
// Stale-while-revalidate: the stale copy is served at once; the refresh happens on the background lane.
|
||||
await clock.advance(1000);
|
||||
body = { v: 2 };
|
||||
assert.deepEqual(await read('interactive', { ttlMs: 1000, staleWhileRevalidate: true }), { v: 1 });
|
||||
await clock.advance(0); await flush();
|
||||
assert.equal(calls.length, 3);
|
||||
assert.equal(client.status().counters['lane.background'], 1);
|
||||
assert.deepEqual(await read(), { v: 2 });
|
||||
assert.equal(calls.length, 3);
|
||||
// The key ignores maxlag and parameter order.
|
||||
assert.equal(cacheKey(new URL('https://h.org/p?b=2&a=1&maxlag=5')), cacheKey(new URL('https://h.org/p?a=1&b=2')));
|
||||
});
|
||||
|
||||
test('requests without a cache policy (mint) neither read nor write the cache', async () => {
|
||||
const cache = new WikimediaCache(':memory:');
|
||||
const { client, calls } = setup(() => json({ n: calls.length }), {}, cache);
|
||||
await client.request(new URL(API), { lane: 'interactive', timeoutMs: 1000, cache: { ttlMs: 60_000 } });
|
||||
const minted = await client.request(new URL(API), { lane: 'mint', timeoutMs: 1000 });
|
||||
assert.equal(minted.fromCache, false);
|
||||
assert.equal(calls.length, 2);
|
||||
assert.equal(cache.size(), 1);
|
||||
});
|
||||
|
||||
test('a stale copy is served when the upstream refuses, and non-JSON replies are never cached', async () => {
|
||||
const cache = new WikimediaCache(':memory:');
|
||||
let fail = false, html = false;
|
||||
const { clock, client } = setup(() => fail ? new Response('', { status: 429 })
|
||||
: html ? new Response('<html>', { headers: { 'Content-Type': 'text/html' } }) : json({ v: 1 }), {}, cache);
|
||||
const policy = { ttlMs: 1000 };
|
||||
await client.request(new URL(API), { lane: 'interactive', timeoutMs: 1000, cache: policy });
|
||||
await clock.advance(2000);
|
||||
fail = true;
|
||||
const stale = await client.request(new URL(API), { lane: 'interactive', timeoutMs: 1000, cache: policy });
|
||||
assert.equal(stale.fromCache, true);
|
||||
fail = false; html = true;
|
||||
await clock.advance(60_000);
|
||||
await client.request(new URL(`${API}&other=1`), { lane: 'interactive', timeoutMs: 1000, cache: policy });
|
||||
assert.equal(cache.size(), 1);
|
||||
});
|
||||
|
||||
test('start-up warns when there is no operator contact', () => {
|
||||
const messages: string[] = [];
|
||||
assert.equal(warnIfNoContact({}, m => messages.push(m)), true);
|
||||
assert.match(messages[0], /WIKIDECK_OPERATOR_CONTACT/);
|
||||
assert.equal(warnIfNoContact({ WIKIDECK_OPERATOR_CONTACT: 'ops@example.org' }, m => messages.push(m)), false);
|
||||
assert.equal(warnIfNoContact({ WIKIDECK_USER_AGENT: 'X/1 (+https://x; y)' }, m => messages.push(m)), false);
|
||||
assert.equal(messages.length, 1);
|
||||
});
|
||||
|
||||
test('only HTTPS is allowed', async () => {
|
||||
const { client } = setup();
|
||||
await assert.rejects(client.request(new URL('http://en.wikipedia.org/w/api.php'), { lane: 'mint', timeoutMs: 1 }), RangeError);
|
||||
});
|
||||
@@ -1,52 +0,0 @@
|
||||
// Shared fakes for tests/wikimedia-*.test.ts: a manual clock and a recording fetch. Zero network.
|
||||
import type { Clock } from '../server/wikimedia/client.ts';
|
||||
|
||||
export class FakeClock implements Clock {
|
||||
time: number;
|
||||
private timers: { at: number; fn: () => void; id: number }[] = [];
|
||||
private nextId = 1;
|
||||
constructor(start = Date.parse('2026-11-17T12:00:00Z')) { this.time = start; }
|
||||
now(): number { return this.time; }
|
||||
setTimeout(fn: () => void, ms: number): unknown {
|
||||
const id = this.nextId++;
|
||||
this.timers.push({ at: this.time + Math.max(0, ms), fn, id });
|
||||
return id;
|
||||
}
|
||||
clearTimeout(handle: unknown): void { this.timers = this.timers.filter(t => t.id !== handle); }
|
||||
/** Advance time, firing due timers in order and letting promise chains settle between them. */
|
||||
async advance(ms: number): Promise<void> {
|
||||
const target = this.time + ms;
|
||||
await flush();
|
||||
for (;;) {
|
||||
this.timers.sort((a, b) => a.at - b.at || a.id - b.id);
|
||||
const next = this.timers[0];
|
||||
if (!next || next.at > target) break;
|
||||
this.timers.shift();
|
||||
this.time = Math.max(this.time, next.at);
|
||||
next.fn();
|
||||
await flush();
|
||||
}
|
||||
this.time = target;
|
||||
await flush();
|
||||
}
|
||||
}
|
||||
|
||||
export async function flush(): Promise<void> {
|
||||
for (let i = 0; i < 10; i++) await new Promise(resolve => setImmediate(resolve));
|
||||
}
|
||||
|
||||
export function json(value: unknown, headers: Record<string, string> = {}, status = 200): Response {
|
||||
return new Response(JSON.stringify(value), { status, headers: { 'Content-Type': 'application/json; charset=utf-8', ...headers } });
|
||||
}
|
||||
|
||||
export type Call = { url: URL; headers: Headers; at: number };
|
||||
export function recorder(clock: { now(): number }, handler: (url: URL, headers: Headers) => Response | Promise<Response>) {
|
||||
const calls: Call[] = [];
|
||||
const fetcher: typeof fetch = async (input, init) => {
|
||||
const url = new URL(String(input));
|
||||
const headers = new Headers(init?.headers);
|
||||
calls.push({ url, headers, at: clock.now() });
|
||||
return handler(url, headers);
|
||||
};
|
||||
return { calls, fetcher };
|
||||
}
|
||||
@@ -1,104 +0,0 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { WikipediaEnrichment } from '../server/enrichment.ts';
|
||||
import { WikipediaArticleSource, randomArticlesUrl } from '../server/wiki.ts';
|
||||
import { WikimediaClient } from '../server/wikimedia/client.ts';
|
||||
import { WikimediaCache } from '../server/wikimedia/cache.ts';
|
||||
import { UNTHROTTLED } from '../server/wikimedia/client.ts';
|
||||
import { json } from './wikimedia-helpers.ts';
|
||||
|
||||
const JANUARY = new Date('2025-01-15T23:59:59Z');
|
||||
const MONTHS = ['2024-07', '2024-08', '2024-09', '2024-10', '2024-11', '2024-12'];
|
||||
const metadata = (pageId = 736) => ({ query: { pages: [{ pageid: pageId, ns: 0, title: 'Albert Einstein', lastrevid: 1234,
|
||||
length: 100_000, pageprops: { wikibase_item: 'Q937' }, categories: [], langlinks: [{ lang: 'de' }] }] } });
|
||||
const views = (counts = [10, 20, 30, 40, 50, 100]) => ({ items: MONTHS.map((month, i) => ({ timestamp: `${month.replace('-', '')}0100`,
|
||||
views: counts[i], project: 'en.wikipedia', access: 'all-access', agent: 'user', granularity: 'monthly' })) });
|
||||
|
||||
function service(viewsReply: () => Response | Promise<Response>, cache?: WikimediaCache) {
|
||||
const urls: URL[] = [];
|
||||
const fetcher: typeof fetch = async input => {
|
||||
const url = new URL(String(input));
|
||||
urls.push(url);
|
||||
return url.hostname === 'en.wikipedia.org' ? json(metadata(Number(url.searchParams.get('pageids')))) : viewsReply();
|
||||
};
|
||||
const client = new WikimediaClient({ fetch: fetcher, config: UNTHROTTLED, cache });
|
||||
return { urls, client, enrichment: new WikipediaEnrichment({ client, now: () => JANUARY }) };
|
||||
}
|
||||
|
||||
test('mint fix: a 429, 5xx, timeout or network failure on pageviews fails the mint read instead of pinning attention: null', async () => {
|
||||
const failures: (() => Response | Promise<Response>)[] = [
|
||||
() => new Response('', { status: 429, headers: { 'Retry-After': '0' } }),
|
||||
() => new Response('', { status: 503 }),
|
||||
() => new Response('', { status: 500 }),
|
||||
() => new Response('<html>busy</html>', { headers: { 'Content-Type': 'text/html' } }),
|
||||
() => { throw new TypeError('fetch failed'); },
|
||||
() => Promise.reject(new DOMException('The operation timed out.', 'TimeoutError')),
|
||||
];
|
||||
for (const failure of failures) {
|
||||
const { enrichment } = service(failure);
|
||||
await assert.rejects(enrichment.getFresh(736), /unavailable/);
|
||||
// Display reads keep their old behaviour: metadata with unavailable views.
|
||||
const shown = await enrichment.get(736);
|
||||
assert.equal(shown.pageviews.status, 'unavailable');
|
||||
assert.equal(shown.stats.attention, null);
|
||||
}
|
||||
});
|
||||
|
||||
test('mint records unavailable views only for a genuine 404 or missing data', async () => {
|
||||
for (const reply of [() => json({ type: 'not found' }, {}, 404), () => json({ items: [] }), () => json({ items: views().items.slice(1) })]) {
|
||||
const { enrichment } = service(reply);
|
||||
const minted = await enrichment.getFresh(736);
|
||||
assert.equal(minted.pageviews.status, 'unavailable');
|
||||
assert.equal(minted.stats.attention, null);
|
||||
}
|
||||
const { enrichment } = service(() => json(views()));
|
||||
assert.equal((await enrichment.getFresh(736)).stats.attention, 33);
|
||||
});
|
||||
|
||||
test('mint reads never use the disk cache or join a display read', async () => {
|
||||
const cache = new WikimediaCache(':memory:');
|
||||
let count = 100;
|
||||
const { enrichment, urls } = service(() => json(views([1, 1, 1, 1, 1, count++])), cache);
|
||||
const shown = await enrichment.get(736);
|
||||
assert.equal(cache.size(), 2);
|
||||
const [display, minted] = await Promise.all([enrichment.get(736), enrichment.getFresh(736)]);
|
||||
assert.equal(display.pageviews.months[5].views, shown.pageviews.months[5].views);
|
||||
assert.equal(minted.pageviews.months[5].views, 101); // Fetched again, not the cached 100.
|
||||
assert.equal(urls.length, 4);
|
||||
// A later display read (memory cache cleared by a new service) is served from disk: no upstream call.
|
||||
const second = new WikipediaEnrichment({ client: new WikimediaClient({ fetch: async () => { throw new Error('no network'); },
|
||||
config: UNTHROTTLED, cache }), now: () => JANUARY });
|
||||
assert.equal((await second.get(736)).pageviews.months[5].views, 100);
|
||||
});
|
||||
|
||||
test('golden mint URLs: request parameters are byte-identical apart from the added maxlag', async () => {
|
||||
assert.equal(randomArticlesUrl().toString(), 'https://en.wikipedia.org/w/api.php?action=query&format=json&formatversion=2'
|
||||
+ '&generator=random&grnnamespace=0&grnlimit=10&prop=extracts%7Cpageimages%7Cinfo%7Ccategories&exintro=1&explaintext=1'
|
||||
+ '&exchars=900&piprop=thumbnail&pithumbsize=800&cllimit=20');
|
||||
const urls: string[] = [];
|
||||
const fetcher: typeof fetch = async input => {
|
||||
const url = new URL(String(input));
|
||||
urls.push(url.toString());
|
||||
if (url.searchParams.get('generator') === 'random') return json({ query: { pages: [{ pageid: 9, ns: 0, title: 'T',
|
||||
extract: 'An extract that is comfortably longer than forty characters.', lastrevid: 2,
|
||||
thumbnail: { source: 'https://upload.wikimedia.org/a.jpg' } }] } });
|
||||
return url.hostname === 'en.wikipedia.org' ? json(metadata(9)) : json(views());
|
||||
};
|
||||
const client = new WikimediaClient({ fetch: fetcher, config: UNTHROTTLED });
|
||||
const article = await new WikipediaArticleSource(client).random();
|
||||
await new WikipediaEnrichment({ client, now: () => JANUARY }).getFresh(article.pageId);
|
||||
assert.deepEqual(urls, [
|
||||
`${randomArticlesUrl()}&maxlag=5`,
|
||||
'https://en.wikipedia.org/w/api.php?action=query&format=json&formatversion=2&pageids=9'
|
||||
+ '&prop=info%7Ccategories%7Cpageprops%7Cdescription%7Clanglinks&clshow=%21hidden&cllimit=50&lllimit=500'
|
||||
+ '&ppprop=wikibase_item&maxlag=5',
|
||||
'https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia.org/all-access/user/Albert_Einstein/monthly/2024070100/2024120100',
|
||||
]);
|
||||
});
|
||||
|
||||
test('a throttled upstream stops random() at once instead of retrying three times', async () => {
|
||||
let calls = 0;
|
||||
const source = new WikipediaArticleSource(async () => { calls++; return new Response('', { status: 429 }); });
|
||||
await assert.rejects(source.random(), /HTTP 429/);
|
||||
assert.equal(calls, 1);
|
||||
});
|
||||
@@ -1,74 +0,0 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { MintTemplatePool } from '../server/wikimedia/pool.ts';
|
||||
import type { Article } from '../shared/protocol.ts';
|
||||
import type { PageEnrichment } from '../shared/enrichment.ts';
|
||||
|
||||
function article(pageId: number): Article {
|
||||
return { pageId, title: `Page ${pageId}`, extract: 'An extract that is comfortably longer than forty characters.',
|
||||
image: 'https://upload.wikimedia.org/a.jpg', url: `https://en.wikipedia.org/?curid=${pageId}`, revisionId: pageId + 1,
|
||||
source: 'live', category: 'Culture' };
|
||||
}
|
||||
function enrichment(pageId: number): PageEnrichment {
|
||||
return { pageId, title: `Page ${pageId}`, description: null, wikidataId: null, revisionId: pageId + 1, pageBytes: 10,
|
||||
languageCount: 1, languagesTruncated: false, categories: [], categoriesTruncated: false, fetchedAt: '2026-11-17T00:00:00Z',
|
||||
pageviews: { status: 'available', months: [], latestMonth: '2026-10', total: 1, dailyAverage: 1, trendPercent: null },
|
||||
stats: { attention: 1, depth: 1, reach: 1 } };
|
||||
}
|
||||
|
||||
function fixture(options: { size?: number; maxAgeMs?: number } = {}) {
|
||||
let next = 1, now = 0, fail = false;
|
||||
const log: string[] = [];
|
||||
const sources = {
|
||||
articles: { random: async () => { log.push('random'); if (fail) throw new Error('Wikimedia unavailable: HTTP 429'); return article(next++); } },
|
||||
enrichment: { getFresh: async (id: number) => { log.push(`fresh:${id}`); return enrichment(id); } },
|
||||
};
|
||||
const pool = new MintTemplatePool(sources, { now: () => now, ...options });
|
||||
return { pool, log, advance: (ms: number) => { now += ms; }, setFail: (value: boolean) => { fail = value; } };
|
||||
}
|
||||
|
||||
test('every template on one tip shares one article; a new tip gets a different one', async () => {
|
||||
const { pool, log } = fixture({ size: 1 });
|
||||
const first = await Promise.all(Array.from({ length: 20 }, () => pool.take('tip-a')));
|
||||
assert.ok(first.every(pair => pair.article.pageId === first[0].article.pageId));
|
||||
await pool.refill();
|
||||
const requestsForTwentyMiners = log.length;
|
||||
assert.equal(requestsForTwentyMiners, 4); // One direct pair, one prefetched pair: 2 calls each.
|
||||
const second = await pool.take('tip-b');
|
||||
assert.notEqual(second.article.pageId, first[0].article.pageId);
|
||||
assert.equal(second.article.pageId, 2); // Served from the prefetched pair, no wait.
|
||||
await pool.refill();
|
||||
assert.equal(log.length, 6); // ...and that use alone triggers the next prefetch.
|
||||
// The stats snapshot comes from a mint-path read of the same page.
|
||||
assert.equal(second.statsSnapshot.revisionId, second.article.pageId + 1);
|
||||
});
|
||||
|
||||
test('pairs older than maxAge are dropped, never served', async () => {
|
||||
const { pool, advance } = fixture({ size: 1, maxAgeMs: 1000 });
|
||||
const a = await pool.take('t1');
|
||||
await pool.refill();
|
||||
advance(1000);
|
||||
const b = await pool.take('t1'); // Same tip, but its pair expired; the ready pair expired too.
|
||||
assert.notEqual(b.article.pageId, a.article.pageId);
|
||||
assert.equal(pool.status().expired, 1);
|
||||
});
|
||||
|
||||
test('an idle pool makes no requests, and a failure rejects the template without a retry loop', async () => {
|
||||
const { pool, log, setFail } = fixture({ size: 2 });
|
||||
assert.equal(log.length, 0);
|
||||
setFail(true);
|
||||
await assert.rejects(pool.take('t1'), /429/);
|
||||
await pool.refill();
|
||||
assert.equal(log.filter(entry => entry === 'random').length, 2); // The direct attempt and one refill attempt.
|
||||
setFail(false);
|
||||
assert.ok((await pool.take('t1')).article.pageId > 0);
|
||||
});
|
||||
|
||||
test('callers get copies they cannot mutate', async () => {
|
||||
const { pool } = fixture();
|
||||
const a = await pool.take('t');
|
||||
a.article.title = 'changed';
|
||||
assert.equal((await pool.take('t')).article.title.startsWith('Page'), true);
|
||||
assert.throws(() => new MintTemplatePool({ articles: { random: async () => article(1) },
|
||||
enrichment: { getFresh: async () => enrichment(1) } }, { size: 99 }), RangeError);
|
||||
});
|
||||
Reference in New Issue
Block a user