Files
wikideck/docs/night2/WIKIMEDIA.md
claudeanderic dbb89ce9c5 Night 2: 40-card Magic-style duel, PvP with owned decks, polite Wikimedia client, Android mining (#23)
Night 2 (2026-11-17), from the `claude/wikideck` fork's `night/2026-11-17` branch. It follows Night 1 (#22).

## What's in it
- **Duel engine** (fork #72, #80): 40-card decks of cards you own, Notability/Citations, keywords, effects T1–T12, Connected, a hidden-information-safe `viewFor`/`logFor`, and replay. Balance v1 tuning (#84) gives the first player about 51%.
- **Card profiles** (#70, #82): Figure, Location, Event, Institution or Doctrine, plus stats, keywords and rules text, all derived from real article data. Templates only.
- **Design and story** (#66, #69, #83): DUEL_DESIGN.md, plus the match log and the encyclopedic "Battle of X (2026)" recap.
- **PvP** (#74, #77, #81, #76, #85):
  - The server checks ownership, and 40-card eligibility is enforced.
  - Modes: Classic (ranked), Draft and Trending.
  - Invites, quick match, Elo, a ladder and recap pages. Online matches render on the duel board.
- **Duel board UI** (#79): MTG-style card faces and combat declaration, built for 360 px first.
- **Polite Wikimedia client** (#73, #75): throttled, cached and identified requests, the mint-path fix, and a prefetch pool.
- **Android mining** (#78): native RandomX, as a sideload build. The emulator mined a testnet block.
- **Testnet ops** (#67, #68).

PR numbers refer to `claude/wikideck`.

## Status
- **Tests:** 423, of which 422 pass, 0 fail and 1 is skipped. The build is OK (`node:24-alpine`, `--network none`).
- **Testnet:** running this commit (036489c) on r350.
- **Untouched:** no changes to `server/testnet/consensus.ts`, `randomx.ts`, `native/` or the mainnet compose.
- **Review:** night-review reviewed fork #66–#79. #80–#85 came from the agents' unfinished branches and were not reviewed.
- **Known gaps:**
  - The PvP end-to-end test hasn't been run (it needs two eom.dev accounts).
  - Bot matches aren't reachable from Play.
  - Card identity pre-bake hasn't been done.
  - The RandomX verifier latch (`server/testnet/randomx.ts:140`) is still open.

This PR is for Eric to merge.

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: WikiDeck/wikideck#23

Co-authored-by: Claude <claude@eom.dev>
2026-11-18 18:12:24 +00:00

73 lines
5.0 KiB
Markdown

# Polite Wikimedia client (Night 2, issue #62)
Every server-side Wikimedia request goes through one process-wide `WikimediaClient`
(`server/wikimedia/client.ts`). The design comes from engineer review §4 and data review §5.
## What it does
| Behaviour | Default |
|---|---|
| Rate per upstream host (`en.wikipedia.org`, `wikimedia.org`, `www.wikidata.org`) | token bucket, 1 req/s, burst 3 |
| Concurrency | 1 request in flight per host |
| Priority lanes | `mint` → `interactive` → `background`; the queue holds 64 waiters, and background work is shed first |
| Longest queue wait | mint 10 s, interactive 5 s, background 120 s |
| `maxlag` | `maxlag=5` on every Action API request; a maxlag reply waits for `Retry-After` and retries up to 2 times |
| 429 / 503 | honour `Retry-After` (seconds or HTTP date), else back off 2ⁿ s with jitter, capped at 10 min, by pausing the whole host |
| Circuit breaker | 3 consecutive 429s stop that host for 10 min. Requests are refused at once without touching Wikimedia |
| Daily cap | 20,000 requests a day per egress IP. Background work stops at 80 %, interactive at 100 %; mint has a 10 % reserve |
| Disk cache | `$WIKIDECK_DATA_DIR/wikimedia-cache.sqlite`: ETag / Last-Modified, `If-None-Match`, stale-while-revalidate on the background lane, stale copy served on refusal |
| User-Agent | `WikiDeck/<version> (+https://gitea.eom.dev/WikiDeck/wikideck; <operator contact>) node/<version>` |
The rate is deliberately 1 req/s for **every** host, including the `wikimedia.org` REST (pageviews) host.
The engineer review suggested 5 req/s there; we are stricter on purpose after the Night 1 rate limiting, because the
budget is per egress IP. Raise it only with a reason and a per-host setting.
Cache lifetimes: article metadata 24 h, monthly pageviews 7 days (the URL names six closed
months, which do not change). Random articles and all mint reads are never cached.
## Operator settings
| Variable | Meaning |
|---|---|
| `WIKIDECK_OPERATOR_CONTACT` | **Set this.** An email address or URL, put in the User-Agent. Without it (and without `PUBLIC_ORIGIN`), the node logs a warning at start-up. Wikimedia policy requires a contact, and contact-less clients are the first to be blocked. |
| `PUBLIC_ORIGIN` | Used as the contact when `WIKIDECK_OPERATOR_CONTACT` is unset. |
| `WIKIDECK_USER_AGENT` | Overrides the whole User-Agent string (sanitized; kept from Night 1). |
| `WIKIDECK_WIKIMEDIA_RPS` | Requests per second per host for the **whole egress IP** (default 1). |
| `WIKIDECK_WIKIMEDIA_BURST` | Token-bucket burst (default 3). |
| `WIKIDECK_WIKIMEDIA_CONCURRENCY` | In-flight requests per host (default 1). |
| `WIKIDECK_WIKIMEDIA_DAILY_CAP` | Requests per UTC day for the whole egress IP (default 20000). |
| `WIKIDECK_WIKIMEDIA_NODES_PER_IP` | How many nodes share this egress IP. The rate and the daily cap are divided by it. **The testnet runs several nodes behind one IP: set it to the node count** (e.g. 3 for `deploy/night/compose.lan.yaml`). |
The testnet's `WIKIDECK_USER_AGENT` already names the project URL, which satisfies the policy.
Counters (`lane.*`, `http.429`, `breaker.open`, `cache.hit`, `refused.*`, …), per-host pause state and the
mint pool's state are in the private operations listener's `/status`, under `wikimedia`.
## Mint path: transport only
- `random()` (W1) and `getFresh()` (W2 and W3) keep their request parameters byte-identical. The only
addition is `maxlag=5`, which changes only error behaviour. `tests/wikimedia-mint.test.ts` pins the exact URLs.
- Mint reads use the `mint` lane and **never** read or write the disk cache, and never join a display read.
- **Fixed:** a 429, 5xx, timeout, network error or non-JSON reply on the pageviews call used to mint a card with
`attention: null` forever (`enrichment.ts`). Now `getFresh()` rejects, the template returns 503 and the miner
retries. `unavailable` is minted only for a genuine 404 (no data) or missing months.
- No block, transfer, trade or validation code changed. Validation never calls Wikimedia.
## Mining prefetch pool (`server/wikimedia/pool.ts`)
Each template used to cost 3+ requests, all in a burst at each new tip, so load scaled with the number of miners.
Now every template on the same tip gets the same fresh article+snapshot pair (miners race for one block anyway),
and a new tip takes the next prefetched pair. Load is about one pair, or 3 requests, per block, whatever the number
of miners.
- Pairs come from the mint-path reads above, and the snapshot is stamped when the pair is fetched, so `capturedAt` is true.
- Pairs older than 5 minutes are dropped, never served.
- The pool (2 pairs) refills only after use, so an idle node makes no requests.
## Card faces
`/api/wikipedia/:id` → `get()`: memory LRU, then the disk cache (fresh, or stale with a background
revalidation), and only then the interactive lane. A grid of cards therefore no longer means a burst
of 2 upstream requests per card per render. `TrendCache` can use `getBackground()` to stay off the
interactive lane.