forked from WikiDeck/wikideck
Night 2 (2026-11-17), from the `claude/wikideck` fork's `night/2026-11-17` branch. It follows Night 1 (#22).
## What's in it
- **Duel engine** (fork #72, #80): 40-card decks of cards you own, Notability/Citations, keywords, effects T1–T12, Connected, a hidden-information-safe `viewFor`/`logFor`, and replay. Balance v1 tuning (#84) gives the first player about 51%.
- **Card profiles** (#70, #82): Figure, Location, Event, Institution or Doctrine, plus stats, keywords and rules text, all derived from real article data. Templates only.
- **Design and story** (#66, #69, #83): DUEL_DESIGN.md, plus the match log and the encyclopedic "Battle of X (2026)" recap.
- **PvP** (#74, #77, #81, #76, #85):
- The server checks ownership, and 40-card eligibility is enforced.
- Modes: Classic (ranked), Draft and Trending.
- Invites, quick match, Elo, a ladder and recap pages. Online matches render on the duel board.
- **Duel board UI** (#79): MTG-style card faces and combat declaration, built for 360 px first.
- **Polite Wikimedia client** (#73, #75): throttled, cached and identified requests, the mint-path fix, and a prefetch pool.
- **Android mining** (#78): native RandomX, as a sideload build. The emulator mined a testnet block.
- **Testnet ops** (#67, #68).
PR numbers refer to `claude/wikideck`.
## Status
- **Tests:** 423, of which 422 pass, 0 fail and 1 is skipped. The build is OK (`node:24-alpine`, `--network none`).
- **Testnet:** running this commit (036489c) on r350.
- **Untouched:** no changes to `server/testnet/consensus.ts`, `randomx.ts`, `native/` or the mainnet compose.
- **Review:** night-review reviewed fork #66–#79. #80–#85 came from the agents' unfinished branches and were not reviewed.
- **Known gaps:**
- The PvP end-to-end test hasn't been run (it needs two eom.dev accounts).
- Bot matches aren't reachable from Play.
- Card identity pre-bake hasn't been done.
- The RandomX verifier latch (`server/testnet/randomx.ts:140`) is still open.
This PR is for Eric to merge.
🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: WikiDeck/wikideck#23
Co-authored-by: Claude <claude@eom.dev>
73 lines
5.0 KiB
Markdown
73 lines
5.0 KiB
Markdown
# Polite Wikimedia client (Night 2, issue #62)
|
|
|
|
Every server-side Wikimedia request goes through one process-wide `WikimediaClient`
|
|
(`server/wikimedia/client.ts`). The design comes from engineer review §4 and data review §5.
|
|
|
|
## What it does
|
|
|
|
| Behaviour | Default |
|
|
|---|---|
|
|
| Rate per upstream host (`en.wikipedia.org`, `wikimedia.org`, `www.wikidata.org`) | token bucket, 1 req/s, burst 3 |
|
|
| Concurrency | 1 request in flight per host |
|
|
| Priority lanes | `mint` → `interactive` → `background`; the queue holds 64 waiters, and background work is shed first |
|
|
| Longest queue wait | mint 10 s, interactive 5 s, background 120 s |
|
|
| `maxlag` | `maxlag=5` on every Action API request; a maxlag reply waits for `Retry-After` and retries up to 2 times |
|
|
| 429 / 503 | honour `Retry-After` (seconds or HTTP date), else back off 2ⁿ s with jitter, capped at 10 min, by pausing the whole host |
|
|
| Circuit breaker | 3 consecutive 429s stop that host for 10 min. Requests are refused at once without touching Wikimedia |
|
|
| Daily cap | 20,000 requests a day per egress IP. Background work stops at 80 %, interactive at 100 %; mint has a 10 % reserve |
|
|
| Disk cache | `$WIKIDECK_DATA_DIR/wikimedia-cache.sqlite`: ETag / Last-Modified, `If-None-Match`, stale-while-revalidate on the background lane, stale copy served on refusal |
|
|
| User-Agent | `WikiDeck/<version> (+https://gitea.eom.dev/WikiDeck/wikideck; <operator contact>) node/<version>` |
|
|
|
|
The rate is deliberately 1 req/s for **every** host, including the `wikimedia.org` REST (pageviews) host.
|
|
The engineer review suggested 5 req/s there; we are stricter on purpose after the Night 1 rate limiting, because the
|
|
budget is per egress IP. Raise it only with a reason and a per-host setting.
|
|
|
|
Cache lifetimes: article metadata 24 h, monthly pageviews 7 days (the URL names six closed
|
|
months, which do not change). Random articles and all mint reads are never cached.
|
|
|
|
## Operator settings
|
|
|
|
| Variable | Meaning |
|
|
|---|---|
|
|
| `WIKIDECK_OPERATOR_CONTACT` | **Set this.** An email address or URL, put in the User-Agent. Without it (and without `PUBLIC_ORIGIN`), the node logs a warning at start-up. Wikimedia policy requires a contact, and contact-less clients are the first to be blocked. |
|
|
| `PUBLIC_ORIGIN` | Used as the contact when `WIKIDECK_OPERATOR_CONTACT` is unset. |
|
|
| `WIKIDECK_USER_AGENT` | Overrides the whole User-Agent string (sanitized; kept from Night 1). |
|
|
| `WIKIDECK_WIKIMEDIA_RPS` | Requests per second per host for the **whole egress IP** (default 1). |
|
|
| `WIKIDECK_WIKIMEDIA_BURST` | Token-bucket burst (default 3). |
|
|
| `WIKIDECK_WIKIMEDIA_CONCURRENCY` | In-flight requests per host (default 1). |
|
|
| `WIKIDECK_WIKIMEDIA_DAILY_CAP` | Requests per UTC day for the whole egress IP (default 20000). |
|
|
| `WIKIDECK_WIKIMEDIA_NODES_PER_IP` | How many nodes share this egress IP. The rate and the daily cap are divided by it. **The testnet runs several nodes behind one IP: set it to the node count** (e.g. 3 for `deploy/night/compose.lan.yaml`). |
|
|
|
|
The testnet's `WIKIDECK_USER_AGENT` already names the project URL, which satisfies the policy.
|
|
|
|
Counters (`lane.*`, `http.429`, `breaker.open`, `cache.hit`, `refused.*`, …), per-host pause state and the
|
|
mint pool's state are in the private operations listener's `/status`, under `wikimedia`.
|
|
|
|
## Mint path: transport only
|
|
|
|
- `random()` (W1) and `getFresh()` (W2 and W3) keep their request parameters byte-identical. The only
|
|
addition is `maxlag=5`, which changes only error behaviour. `tests/wikimedia-mint.test.ts` pins the exact URLs.
|
|
- Mint reads use the `mint` lane and **never** read or write the disk cache, and never join a display read.
|
|
- **Fixed:** a 429, 5xx, timeout, network error or non-JSON reply on the pageviews call used to mint a card with
|
|
`attention: null` forever (`enrichment.ts`). Now `getFresh()` rejects, the template returns 503 and the miner
|
|
retries. `unavailable` is minted only for a genuine 404 (no data) or missing months.
|
|
- No block, transfer, trade or validation code changed. Validation never calls Wikimedia.
|
|
|
|
## Mining prefetch pool (`server/wikimedia/pool.ts`)
|
|
|
|
Each template used to cost 3+ requests, all in a burst at each new tip, so load scaled with the number of miners.
|
|
Now every template on the same tip gets the same fresh article+snapshot pair (miners race for one block anyway),
|
|
and a new tip takes the next prefetched pair. Load is about one pair, or 3 requests, per block, whatever the number
|
|
of miners.
|
|
|
|
- Pairs come from the mint-path reads above, and the snapshot is stamped when the pair is fetched, so `capturedAt` is true.
|
|
- Pairs older than 5 minutes are dropped, never served.
|
|
- The pool (2 pairs) refills only after use, so an idle node makes no requests.
|
|
|
|
## Card faces
|
|
|
|
`/api/wikipedia/:id` → `get()`: memory LRU, then the disk cache (fresh, or stale with a background
|
|
revalidation), and only then the interactive lane. A grid of cards therefore no longer means a burst
|
|
of 2 upstream requests per card per render. `TrendCache` can use `getBackground()` to stay off the
|
|
interactive lane.
|