Files
wikideck/docs/night2/WIKIMEDIA.md
claudeanderic dbb89ce9c5 Night 2: 40-card Magic-style duel, PvP with owned decks, polite Wikimedia client, Android mining (#23)
Night 2 (2026-11-17), from the `claude/wikideck` fork's `night/2026-11-17` branch. It follows Night 1 (#22).

## What's in it
- **Duel engine** (fork #72, #80): 40-card decks of cards you own, Notability/Citations, keywords, effects T1–T12, Connected, a hidden-information-safe `viewFor`/`logFor`, and replay. Balance v1 tuning (#84) gives the first player about 51%.
- **Card profiles** (#70, #82): Figure, Location, Event, Institution or Doctrine, plus stats, keywords and rules text, all derived from real article data. Templates only.
- **Design and story** (#66, #69, #83): DUEL_DESIGN.md, plus the match log and the encyclopedic "Battle of X (2026)" recap.
- **PvP** (#74, #77, #81, #76, #85):
  - The server checks ownership, and 40-card eligibility is enforced.
  - Modes: Classic (ranked), Draft and Trending.
  - Invites, quick match, Elo, a ladder and recap pages. Online matches render on the duel board.
- **Duel board UI** (#79): MTG-style card faces and combat declaration, built for 360 px first.
- **Polite Wikimedia client** (#73, #75): throttled, cached and identified requests, the mint-path fix, and a prefetch pool.
- **Android mining** (#78): native RandomX, as a sideload build. The emulator mined a testnet block.
- **Testnet ops** (#67, #68).

PR numbers refer to `claude/wikideck`.

## Status
- **Tests:** 423, of which 422 pass, 0 fail and 1 is skipped. The build is OK (`node:24-alpine`, `--network none`).
- **Testnet:** running this commit (036489c) on r350.
- **Untouched:** no changes to `server/testnet/consensus.ts`, `randomx.ts`, `native/` or the mainnet compose.
- **Review:** night-review reviewed fork #66–#79. #80–#85 came from the agents' unfinished branches and were not reviewed.
- **Known gaps:**
  - The PvP end-to-end test hasn't been run (it needs two eom.dev accounts).
  - Bot matches aren't reachable from Play.
  - Card identity pre-bake hasn't been done.
  - The RandomX verifier latch (`server/testnet/randomx.ts:140`) is still open.

This PR is for Eric to merge.

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: WikiDeck/wikideck#23

Co-authored-by: Claude <claude@eom.dev>
2026-11-18 18:12:24 +00:00

5.0 KiB

Polite Wikimedia client (Night 2, issue #62)

Every server-side Wikimedia request goes through one process-wide WikimediaClient (server/wikimedia/client.ts). The design comes from engineer review §4 and data review §5.

What it does

Behaviour Default
Rate per upstream host (en.wikipedia.org, wikimedia.org, www.wikidata.org) token bucket, 1 req/s, burst 3
Concurrency 1 request in flight per host
Priority lanes mint → interactive → background; the queue holds 64 waiters, and background work is shed first
Longest queue wait mint 10 s, interactive 5 s, background 120 s
maxlag maxlag=5 on every Action API request; a maxlag reply waits for Retry-After and retries up to 2 times
429 / 503 honour Retry-After (seconds or HTTP date), else back off 2ⁿ s with jitter, capped at 10 min, by pausing the whole host
Circuit breaker 3 consecutive 429s stop that host for 10 min. Requests are refused at once without touching Wikimedia
Daily cap 20,000 requests a day per egress IP. Background work stops at 80 %, interactive at 100 %; mint has a 10 % reserve
Disk cache $WIKIDECK_DATA_DIR/wikimedia-cache.sqlite: ETag / Last-Modified, If-None-Match, stale-while-revalidate on the background lane, stale copy served on refusal
User-Agent WikiDeck/<version> (+https://gitea.eom.dev/WikiDeck/wikideck; <operator contact>) node/<version>

The rate is deliberately 1 req/s for every host, including the wikimedia.org REST (pageviews) host. The engineer review suggested 5 req/s there; we are stricter on purpose after the Night 1 rate limiting, because the budget is per egress IP. Raise it only with a reason and a per-host setting.

Cache lifetimes: article metadata 24 h, monthly pageviews 7 days (the URL names six closed months, which do not change). Random articles and all mint reads are never cached.

Operator settings

Variable Meaning
WIKIDECK_OPERATOR_CONTACT Set this. An email address or URL, put in the User-Agent. Without it (and without PUBLIC_ORIGIN), the node logs a warning at start-up. Wikimedia policy requires a contact, and contact-less clients are the first to be blocked.
PUBLIC_ORIGIN Used as the contact when WIKIDECK_OPERATOR_CONTACT is unset.
WIKIDECK_USER_AGENT Overrides the whole User-Agent string (sanitized; kept from Night 1).
WIKIDECK_WIKIMEDIA_RPS Requests per second per host for the whole egress IP (default 1).
WIKIDECK_WIKIMEDIA_BURST Token-bucket burst (default 3).
WIKIDECK_WIKIMEDIA_CONCURRENCY In-flight requests per host (default 1).
WIKIDECK_WIKIMEDIA_DAILY_CAP Requests per UTC day for the whole egress IP (default 20000).
WIKIDECK_WIKIMEDIA_NODES_PER_IP How many nodes share this egress IP. The rate and the daily cap are divided by it. The testnet runs several nodes behind one IP: set it to the node count (e.g. 3 for deploy/night/compose.lan.yaml).

The testnet's WIKIDECK_USER_AGENT already names the project URL, which satisfies the policy.

Counters (lane.*, http.429, breaker.open, cache.hit, refused.*, …), per-host pause state and the mint pool's state are in the private operations listener's /status, under wikimedia.

Mint path: transport only

  • random() (W1) and getFresh() (W2 and W3) keep their request parameters byte-identical. The only addition is maxlag=5, which changes only error behaviour. tests/wikimedia-mint.test.ts pins the exact URLs.
  • Mint reads use the mint lane and never read or write the disk cache, and never join a display read.
  • Fixed: a 429, 5xx, timeout, network error or non-JSON reply on the pageviews call used to mint a card with attention: null forever (enrichment.ts). Now getFresh() rejects, the template returns 503 and the miner retries. unavailable is minted only for a genuine 404 (no data) or missing months.
  • No block, transfer, trade or validation code changed. Validation never calls Wikimedia.

Mining prefetch pool (server/wikimedia/pool.ts)

Each template used to cost 3+ requests, all in a burst at each new tip, so load scaled with the number of miners. Now every template on the same tip gets the same fresh article+snapshot pair (miners race for one block anyway), and a new tip takes the next prefetched pair. Load is about one pair, or 3 requests, per block, whatever the number of miners.

  • Pairs come from the mint-path reads above, and the snapshot is stamped when the pair is fetched, so capturedAt is true.
  • Pairs older than 5 minutes are dropped, never served.
  • The pool (2 pairs) refills only after use, so an idle node makes no requests.

Card faces

/api/wikipedia/:id → get(): memory LRU, then the disk cache (fresh, or stale with a background revalidation), and only then the interactive lane. A grid of cards therefore no longer means a burst of 2 upstream requests per card per render. TrendCache can use getBackground() to stay off the interactive lane.