From 36cbaccd42dc048c33b25ac6a1f2b9982c2b2543 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 17 Nov 2026 21:42:23 -0500 Subject: [PATCH 1/6] testnet-ops: card identity store, pre-bake script, testnet guide server/game/identity.ts freezes each card's off-chain identity (P31, categories, country, dates, coords, outlinks hash) once per tokenId in node:sqlite and never rewrites it. scripts/night2/prebake-identity.ts reads the canonical chain from one node and freezes from cached card files only; live facts plug in via an IdentitySource on the wikimedia client's background lane (#62). Tests make zero network calls. docs/night2/TESTNET.md: nodes, wallets, login, mining, autodeploy. Co-Authored-By: Claude Opus 5.5 --- docs/night2/TESTNET.md | 57 ++++++++++ scripts/night2/prebake-identity.ts | 56 ++++++++++ server/game/identity.ts | 168 +++++++++++++++++++++++++++++ tests/testnet-ops-identity.test.ts | 79 ++++++++++++++ 4 files changed, 360 insertions(+) create mode 100644 docs/night2/TESTNET.md create mode 100644 scripts/night2/prebake-identity.ts create mode 100644 server/game/identity.ts create mode 100644 tests/testnet-ops-identity.test.ts diff --git a/docs/night2/TESTNET.md b/docs/night2/TESTNET.md new file mode 100644 index 0000000..c7d312d --- /dev/null +++ b/docs/night2/TESTNET.md @@ -0,0 +1,57 @@ +# Night 2 private testnet: how to test (testnet-ops, #63) + +**Testnet only, LAN only, on poweredge-r350.** It is never mainnet and never the live `wikideck` namespace. + +## Nodes +| node | URL | game modes (env from Night 1 #22) | +|---|---|---| +| node-a | http://192.168.1.32:3101 | classic, trending | +| node-b | http://192.168.1.32:3102 | draft | +| node-c | http://192.168.1.32:3103 | classic, draft, trending | + +- **Autodeploy:** the nodes follow `night/2026-11-17`. A loop on r350 (`deploy/night/autodeploy.sh`, in a dedicated detached worktree) polls the branch every 3 min. + - On a new head it rebuilds the image and restarts node-a, then b, then c, each after the previous one passes `/healthz`. + - While any match or lobby is live it defers, for up to 15 min. + - Lobbies are in memory, so a deploy that does go through ends games in progress. After night-lead merges, wait about 5 min, or check that all three nodes report the new head, before you start an end-to-end run. +- Ledgers persist in the Docker volumes across deploys. + +## Test wallets +Four disposable wallets: **alice**, **bob**, **carol** and **dave**. + +| wallet | address | cards (confirmed) | +|---|---|---| +| alice | `ff9273368c225698d8ae34430800cfe30b10a1c3d8c21091d42960493db556df` | see #63 | +| bob | `80f17f10ca782efd16783d10d71ce43b2993aadccda36dbd0364dee1921cbc22` | see #63 | +| carol | `f2d908633c4313bd77e86650c13d11ac3e7e656223ebe881cc34ffba3f3e0524` | leftovers | +| dave | `02bb97f202e1616ab97b45e8938e384dcbf622a0b44a7f4c1589f44236d3ccef` | leftovers | + +- alice and bob were consolidated to 40 cards each with on-chain NFT transfers from carol and dave, so **production `deckSize` 40 works on the testnet** and no lower-deckSize test mode is needed. +- **The encrypted browser backups are in a comment on #63**, not in git. + - Backup password: `wikideck-night-testnet`. It is **public and testnet-only**: never reuse it, and never put a real secret next to it. +- Re-check counts at any time: `GET http://192.168.1.32:3101/api/decks/
?offset=0&limit=24` (pages of 24, follow `next`). + +### Load a wallet in the browser +Use a separate browser profile, or a private window, per player, because a browser holds one wallet per origin. +1. Open a node URL, then **Open wallet**. +2. **Recover from a local backup**: choose the backup JSON file (save it from the #63 comment as `alice.backup.json`), and enter the backup password above. +3. Set a new local encryption password (at least 12 characters, anything testnet-only), then confirm it. +4. The wallet dialog should show the address above and its confirmed card count. + +### eom.dev login (Dex) +- There is no new Dex config tonight. The existing `wikideck` client already allows the 3101–3103 callbacks. +- Log in with **Log in with eom.dev**, then unlock the wallet and press **Link wallet …** to link it to the account. +- PvP seats are keyed to the account, and ownership is checked on the server against the account's linked wallets. So for a two-player run you need **two eom.dev accounts**, one with alice linked and one with bob. Which accounts to use is night-lead's call (see #53/#63). Never use a real person's account without their OK. + +## Mining on the testnet +- Mining isn't needed for tests. Only run a miner if night-lead asks for more blocks, or to confirm transfers. +- `deploy/night/testnet.sh mine N [workers]` runs the native multi-worker miner in a container named `wikideck-night-miner` (at most 4 workers on r350). + - Every template costs Wikipedia requests on the node, so it is paced by `MIN_GAP`. + - **testnet-ops stops any miner by 01:55** (`docker rm -f wikideck-night-miner`). The nodes stay up. +- **Difficulty (for mobile mining, #59):** at block ~90, an accepted block took between about 2 K and 130 K hashes. That is up to ~8 min at the 4-worker r350 rate of ~250 H/s. One light-mode thread on the emulator is far slower, so expect a long wait per block, with high variance. + +## Card identity pre-bake (engineer WS E) +- `server/game/identity.ts` is a persistent per-token store (`node:sqlite`). It freezes each card's off-chain identity at first sight and never rewrites it. The identity covers P31 classes, categories, country, dates, coordinates and an outlinks hash. +- `scripts/night2/prebake-identity.ts --node --store [--cards ...]` reads the canonical chain from one node and freezes what the cached card files know. + - It **never calls Wikimedia itself**. + - Live facts arrive only through the wikimedia team's client on its `background` lane (#62), once that client has merged. + - Cards that are still unknown stay unfrozen and are retried on a later run. diff --git a/scripts/night2/prebake-identity.ts b/scripts/night2/prebake-identity.ts new file mode 100644 index 0000000..f7ab0c8 --- /dev/null +++ b/scripts/night2/prebake-identity.ts @@ -0,0 +1,56 @@ +// testnet-ops (engineer WS E): pre-bake frozen card identities for every card on a chain. +// +// node --experimental-strip-types scripts/night2/prebake-identity.ts --node http://192.168.1.32:3101 \ +// --store ./identity.sqlite [--cards docs/game/card-sample.json] [--batch 50] +// +// - Reads the canonical chain from ONE node (GET /api/summary, then /api/canonical pages), so every +// card is built from its mint block, never from client JSON. +// - Facts come ONLY from cached card files (`--cards`, repeatable). This script never calls +// Wikipedia or Wikidata itself. Live facts must go through the wikimedia team's process-wide client +// on its `background` lane (server/wikimedia, #62); that IdentitySource plugs into `prebake()` in +// server/game/identity.ts once it has merged. Until then, unknown cards are left unfrozen and are +// picked up by a later run. +// - Idempotent and freeze-at-first-sight: already frozen tokens are skipped and never rewritten. +import { readFileSync } from 'node:fs'; +import { parseArgs } from 'node:util'; +import { IdentityStore, cachedSource, prebake, type ChainArticle } from '../../server/game/identity.ts'; + +const { values } = parseArgs({ options: { + node: { type: 'string', default: 'http://192.168.1.32:3101' }, + store: { type: 'string', default: 'identity.sqlite' }, + cards: { type: 'string', multiple: true, default: ['docs/game/card-sample.json'] }, + batch: { type: 'string', default: '50' }, +} }); +const nodeUrl = new URL(values.node!); +if (!['http:', 'https:'].includes(nodeUrl.protocol)) throw new Error('--node must be an http(s) WikiDeck node URL'); +if (/wikipedia\.org|wikidata\.org|wikimedia\.org/i.test(nodeUrl.hostname)) throw new Error('--node must be a WikiDeck node, not Wikimedia'); + +async function get(path: string): Promise { + const r = await fetch(new URL(path, nodeUrl), { signal: AbortSignal.timeout(30000), redirect: 'error' }); + if (!r.ok) throw new Error(`${path}: HTTP ${r.status}`); + return await r.json() as T; +} + +type Block = { tokenId: string; article: { pageId: number; title: string; revisionId?: number } }; +async function chainArticles(): Promise { + const { tip } = await get<{ tip: string }>('/api/summary'); + const out: ChainArticle[] = []; + for (let offset: number | null = 0; offset !== null;) { + const page: { blocks: Block[]; next: number | null } = await get(`/api/canonical?tip=${tip}&offset=${offset}&limit=16`); + for (const b of page.blocks) if (b.article && b.tokenId) + out.push({ tokenId: b.tokenId, pageId: b.article.pageId, title: b.article.title, revisionId: b.article.revisionId ?? null }); + offset = page.next; + } + return out; +} + +const cards = values.cards!.flatMap(f => { + const d = JSON.parse(readFileSync(f, 'utf8')); + return Array.isArray(d) ? d : d.cards ?? []; +}); +const articles = await chainArticles(); +const store = new IdentityStore(values.store!); +try { + const report = await prebake(store, cachedSource(cards), articles, { batchSize: Number(values.batch) }); + console.log(JSON.stringify({ node: nodeUrl.origin, chainCards: articles.length, cachedCards: cards.length, ...report, stored: store.count() })); +} finally { store.close(); } diff --git a/server/game/identity.ts b/server/game/identity.ts new file mode 100644 index 0000000..3d099fb --- /dev/null +++ b/server/game/identity.ts @@ -0,0 +1,168 @@ +// Card identity (engineer WS E, data review §3): the off-chain, mutable facts about a card's article +// (P31 classes, a few Wikidata claims, categories, outlinks) frozen ONCE per tokenId, the first time +// the card is seen, so that a later Wikipedia/Wikidata edit can never retype a card someone owns. +// +// - On-chain data (article, statsSnapshot) stays the source of truth for everything it covers. +// - Identities are never fetched at match time. They are pre-baked in the background +// (scripts/night2/prebake-identity.ts), from cached data or through an injected IdentitySource that +// must use the process-wide Wikimedia client on its `background` lane. This file performs no I/O +// besides the sqlite store and never calls fetch itself. +// - First write wins: `freeze` never overwrites an existing record (data §3 "freeze at first sight"). +import { DatabaseSync } from 'node:sqlite'; +import { createHash } from 'node:crypto'; + +export const IDENTITY_VERSION = 1; + +export type WikidataClass = { id: string; label?: string }; +export type IdentityDates = { birth?: string; death?: string; inception?: string; pointInTime?: string; start?: string }; +export type CardIdentity = { + version: typeof IDENTITY_VERSION; + tokenId: string; + pageId: number; + title: string; + /** article.revisionId from the mint block. */ + mintRevisionId: number | null; + /** The revision the off-chain facts were read from (may differ from the mint revision). */ + fetchedRevisionId: number | null; + wikidataId: string | null; + wikidataRev: number | null; + instanceOf: WikidataClass[]; + categories: string[]; + /** P17 (country) or P27 (citizenship) Q-ids. */ + country: string[]; + dates: IdentityDates; + /** P625 or prop=coordinates, [lat, lon]. */ + coords: [number, number] | null; + /** Main-namespace outlink titles, sorted, deduplicated. Empty when unknown. */ + outlinks: string[]; + /** sha256 of the sorted outlinks, so match records can log a short fingerprint. */ + outlinksHash: string; + /** Where the facts came from: an offline cache (e.g. card-sample.json) or a live background fetch. */ + source: 'cache' | 'wikimedia'; + frozenAt: string; +}; + +/** What a source knows about an article; any field may be missing. */ +export type IdentityFacts = Partial>; +/** The minimum the chain tells us about a card (built from its mint block, never from client JSON). */ +export type ChainArticle = { tokenId: string; pageId: number; title: string; revisionId?: number | null }; + +/** + * A source of identity facts. A live implementation must route every request through the shared + * Wikimedia client on the background lane (never the mint or interactive lanes) and batch where it + * can. Returning `undefined` for a page means "unknown for now": nothing is frozen and it is retried + * on a later run. + */ +export interface IdentitySource { + readonly kind: CardIdentity['source']; + lookup(articles: ChainArticle[]): Promise>; +} + +export function outlinksHash(outlinks: readonly string[]): string { + return createHash('sha256').update(JSON.stringify(outlinks)).digest('hex'); +} + +const uniqSorted = (xs: readonly string[] | undefined) => [...new Set((xs ?? []).filter(x => typeof x === 'string'))].sort(); + +/** Pure: build a frozen identity from chain data plus whatever facts a source returned. */ +export function buildIdentity(article: ChainArticle, facts: IdentityFacts, source: CardIdentity['source'], now: Date): CardIdentity { + const outlinks = uniqSorted(facts.outlinks); + const coords = facts.coords && facts.coords.length === 2 && facts.coords.every(Number.isFinite) ? [facts.coords[0], facts.coords[1]] as [number, number] : null; + return { + version: IDENTITY_VERSION, + tokenId: article.tokenId, + pageId: article.pageId, + title: article.title, + mintRevisionId: article.revisionId ?? null, + fetchedRevisionId: facts.fetchedRevisionId ?? null, + wikidataId: facts.wikidataId && /^Q\d+$/.test(facts.wikidataId) ? facts.wikidataId : null, + wikidataRev: facts.wikidataRev ?? null, + instanceOf: (facts.instanceOf ?? []).filter(c => /^Q\d+$/.test(c.id)).map(c => c.label ? { id: c.id, label: c.label } : { id: c.id }), + categories: uniqSorted(facts.categories), + country: uniqSorted(facts.country).filter(q => /^Q\d+$/.test(q)), + dates: { ...(facts.dates ?? {}) }, + coords, + outlinks, + outlinksHash: outlinksHash(outlinks), + source, + frozenAt: now.toISOString(), + }; +} + +/** Persistent per-token identity store. Records are immutable once written. */ +export class IdentityStore { + readonly #db: DatabaseSync; + constructor(file: string) { + this.#db = new DatabaseSync(file); + this.#db.exec(`CREATE TABLE IF NOT EXISTS card_identity ( + token_id TEXT PRIMARY KEY, page_id INTEGER NOT NULL, version INTEGER NOT NULL, body TEXT NOT NULL)`); + } + get(tokenId: string): CardIdentity | undefined { + const row = this.#db.prepare('SELECT body FROM card_identity WHERE token_id = ?').get(tokenId) as { body: string } | undefined; + return row ? JSON.parse(row.body) as CardIdentity : undefined; + } + has(tokenId: string): boolean { return this.get(tokenId) !== undefined; } + /** Freeze at first sight: returns the stored record, which is the earlier one if it already existed. */ + freeze(identity: CardIdentity): CardIdentity { + this.#db.prepare('INSERT OR IGNORE INTO card_identity (token_id, page_id, version, body) VALUES (?, ?, ?, ?)') + .run(identity.tokenId, identity.pageId, identity.version, JSON.stringify(identity)); + return this.get(identity.tokenId)!; + } + /** Freeze several in one transaction (one fsync per batch, not per card). */ + freezeMany(identities: CardIdentity[]): void { + this.#db.exec('BEGIN'); + try { for (const i of identities) this.freeze(i); this.#db.exec('COMMIT'); } + catch (e) { this.#db.exec('ROLLBACK'); throw e; } + } + count(): number { return (this.#db.prepare('SELECT COUNT(*) AS n FROM card_identity').get() as { n: number }).n; } + all(): CardIdentity[] { + return (this.#db.prepare('SELECT body FROM card_identity ORDER BY token_id').all() as { body: string }[]).map(r => JSON.parse(r.body)); + } + close() { this.#db.close(); } +} + +export type PrebakeReport = { seen: number; alreadyFrozen: number; frozen: number; unknown: number }; + +/** + * Freeze identities for every card not yet in the store, asking the source in batches. The caller + * controls pacing (batch size, and the source's client controls request rate). + */ +export async function prebake(store: IdentityStore, source: IdentitySource, articles: ChainArticle[], + opts: { batchSize?: number; now?: () => Date } = {}): Promise { + const batchSize = Math.max(1, Math.min(50, opts.batchSize ?? 50)), now = opts.now ?? (() => new Date()); + const seenTokens = new Set(), todo: ChainArticle[] = []; + let alreadyFrozen = 0; + for (const a of articles) { + if (seenTokens.has(a.tokenId)) continue; + seenTokens.add(a.tokenId); + if (store.has(a.tokenId)) alreadyFrozen++; else todo.push(a); + } + let frozen = 0, unknown = 0; + for (let i = 0; i < todo.length; i += batchSize) { + const batch = todo.slice(i, i + batchSize), facts = await source.lookup(batch), ready = []; + for (const a of batch) { + const f = facts.get(a.pageId); + if (!f) { unknown++; continue; } + ready.push(buildIdentity(a, f, source.kind, now())); + } + store.freezeMany(ready); frozen += ready.length; + } + return { seen: seenTokens.size, alreadyFrozen, frozen, unknown }; +} + +/** Offline source: facts from an already-cached card file such as docs/game/card-sample.json. */ +export function cachedSource(cards: readonly { pageId: number; wikidataId?: string | null; revisionId?: number | null; + instanceOf?: WikidataClass[]; categories?: string[] }[]): IdentitySource { + const byPage = new Map(cards.map(c => [c.pageId, c])); + return { + kind: 'cache', + async lookup(articles) { + return new Map(articles.map(a => { + const c = byPage.get(a.pageId); + return [a.pageId, c ? { fetchedRevisionId: c.revisionId ?? null, wikidataId: c.wikidataId ?? null, + instanceOf: c.instanceOf ?? [], categories: c.categories ?? [] } : undefined]; + })); + }, + }; +} diff --git a/tests/testnet-ops-identity.test.ts b/tests/testnet-ops-identity.test.ts new file mode 100644 index 0000000..dd7ae8f --- /dev/null +++ b/tests/testnet-ops-identity.test.ts @@ -0,0 +1,79 @@ +// testnet-ops: card identity freeze-at-first-sight and pre-bake. Zero network calls. +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { mkdtempSync, readFileSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { IdentityStore, buildIdentity, cachedSource, outlinksHash, prebake, + type ChainArticle, type IdentitySource } from '../server/game/identity.ts'; + +const sample = JSON.parse(readFileSync(new URL('../docs/game/card-sample.json', import.meta.url), 'utf8')).cards as any[]; +const now = () => new Date('2026-11-17T22:00:00Z'); +function tmpStore() { + const dir = mkdtempSync(join(tmpdir(), 'identity-')); + return { store: new IdentityStore(join(dir, 'identity.sqlite')), done: () => rmSync(dir, { recursive: true, force: true }) }; +} +const art = (c: any, i: number): ChainArticle => ({ tokenId: i.toString(16).padStart(64, '0'), pageId: c.pageId, title: c.title, revisionId: c.revisionId }); + +test('no fetch: pre-bake from the cached sample freezes every sample card', async () => { + const realFetch = globalThis.fetch; + globalThis.fetch = (() => { throw new Error('network call in test'); }) as typeof fetch; + const { store, done } = tmpStore(); + try { + const articles = sample.map(art); + const r = await prebake(store, cachedSource(sample), articles, { now }); + assert.deepEqual(r, { seen: sample.length, alreadyFrozen: 0, frozen: sample.length, unknown: 0 }); + const salah = store.get(articles[sample.findIndex(c => c.title === 'Salahuddin of Aceh')].tokenId)!; + assert.equal(salah.wikidataId, 'Q4264920'); + assert.deepEqual(salah.instanceOf, [{ id: 'Q5', label: 'human' }]); + assert.equal(salah.source, 'cache'); + assert.ok(salah.categories.includes('Sultans of Aceh')); + } finally { globalThis.fetch = realFetch; store.close(); done(); } +}); + +test('freeze at first sight: a later, different profile never overwrites the stored one', async () => { + const { store, done } = tmpStore(); + try { + const a = art(sample[0], 1); + const first = store.freeze(buildIdentity(a, { instanceOf: [{ id: 'Q5' }] }, 'cache', now())); + const second = store.freeze(buildIdentity(a, { instanceOf: [{ id: 'Q515' }] }, 'wikimedia', new Date())); + assert.deepEqual(second, first); + assert.equal(store.get(a.tokenId)!.instanceOf[0].id, 'Q5'); + const again = await prebake(store, cachedSource(sample), [a, a], { now }); + assert.deepEqual(again, { seen: 1, alreadyFrozen: 1, frozen: 0, unknown: 0 }); + } finally { store.close(); done(); } +}); + +test('unknown pages are not frozen, so a later run can retry them', async () => { + const { store, done } = tmpStore(); + try { + const stranger: ChainArticle = { tokenId: 'f'.repeat(64), pageId: 999999999, title: 'Not in sample', revisionId: 1 }; + const r = await prebake(store, cachedSource(sample), [stranger], { now }); + assert.deepEqual(r, { seen: 1, alreadyFrozen: 0, frozen: 0, unknown: 1 }); + assert.equal(store.has(stranger.tokenId), false); + } finally { store.close(); done(); } +}); + +test('sources are asked in bounded batches and identities persist across reopen', async () => { + const dir = mkdtempSync(join(tmpdir(), 'identity-')); + try { + const calls: number[] = []; + const source: IdentitySource = { kind: 'wikimedia', async lookup(batch) { + calls.push(batch.length); + return new Map(batch.map(a => [a.pageId, { outlinks: ['B', 'A', 'A'], coords: [1, 2] as [number, number], country: ['Q30', 'bad'] }])); + } }; + let store = new IdentityStore(join(dir, 'id.sqlite')); + await prebake(store, source, sample.slice(0, 12).map(art), { batchSize: 5, now }); + assert.deepEqual(calls, [5, 5, 2]); + store.close(); + store = new IdentityStore(join(dir, 'id.sqlite')); + const one = store.get(art(sample[3], 3).tokenId)!; + assert.deepEqual(one.outlinks, ['A', 'B']); + assert.equal(one.outlinksHash, outlinksHash(['A', 'B'])); + assert.deepEqual(one.country, ['Q30']); + assert.deepEqual(one.coords, [1, 2]); + assert.equal(one.mintRevisionId, sample[3].revisionId); + assert.equal(store.count(), 12); + store.close(); + } finally { rmSync(dir, { recursive: true, force: true }); } +}); -- 2.54.0 From 47cab9f6196f38a8b647b9898b0e297079af9042 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 17 Nov 2026 21:46:09 -0500 Subject: [PATCH 2/6] testnet-ops: identity fields match duel-cards CardProfile (links as pageIds, years, cites, subclassOf, description); toProfile() Co-Authored-By: Claude Opus 5.5 --- docs/night2/TESTNET.md | 2 +- server/game/identity.ts | 85 ++++++++++++++++++++---------- tests/testnet-ops-identity.test.ts | 20 ++++--- 3 files changed, 72 insertions(+), 35 deletions(-) diff --git a/docs/night2/TESTNET.md b/docs/night2/TESTNET.md index c7d312d..5b6696c 100644 --- a/docs/night2/TESTNET.md +++ b/docs/night2/TESTNET.md @@ -50,7 +50,7 @@ Use a separate browser profile, or a private window, per player, because a brows - **Difficulty (for mobile mining, #59):** at block ~90, an accepted block took between about 2 K and 130 K hashes. That is up to ~8 min at the 4-worker r350 rate of ~250 H/s. One light-mode thread on the emulator is far slower, so expect a long wait per block, with high variance. ## Card identity pre-bake (engineer WS E) -- `server/game/identity.ts` is a persistent per-token store (`node:sqlite`). It freezes each card's off-chain identity at first sight and never rewrites it. The identity covers P31 classes, categories, country, dates, coordinates and an outlinks hash. +- `server/game/identity.ts` is a persistent per-token store (`node:sqlite`). It freezes each card's off-chain identity at first sight and never rewrites it. The identity covers P31/P279 classes, description, categories, countries, years, coordinates and outlink pageIds. Its fields match duel-cards' `CardProfile` (#70), and `toProfile()` returns that view. - `scripts/night2/prebake-identity.ts --node --store [--cards ...]` reads the canonical chain from one node and freezes what the cached card files know. - It **never calls Wikimedia itself**. - Live facts arrive only through the wikimedia team's client on its `background` lane (#62), once that client has merged. diff --git a/server/game/identity.ts b/server/game/identity.ts index 3d099fb..9578826 100644 --- a/server/game/identity.ts +++ b/server/game/identity.ts @@ -1,5 +1,5 @@ // Card identity (engineer WS E, data review §3): the off-chain, mutable facts about a card's article -// (P31 classes, a few Wikidata claims, categories, outlinks) frozen ONCE per tokenId, the first time +// (P31/P279 classes, a few Wikidata claims, categories, outlinks) frozen ONCE per tokenId, the first time // the card is seen, so that a later Wikipedia/Wikidata edit can never retype a card someone owns. // // - On-chain data (article, statsSnapshot) stays the source of truth for everything it covers. @@ -13,8 +13,12 @@ import { createHash } from 'node:crypto'; export const IDENTITY_VERSION = 1; -export type WikidataClass = { id: string; label?: string }; -export type IdentityDates = { birth?: string; death?: string; inception?: string; pointInTime?: string; start?: string }; +/** Same shape as shared/duel/classify.ts `WikidataClass` (duel-cards, #70). */ +export type WikidataClass = { id: string; label: string }; +/** + * A frozen card identity. Its profile fields are structurally the `CardProfile` that duel-cards' + * `derive()` reads (shared/duel/classify.ts), so `toProfile(identity)` can be passed straight in. + */ export type CardIdentity = { version: typeof IDENTITY_VERSION; tokenId: string; @@ -22,29 +26,40 @@ export type CardIdentity = { title: string; /** article.revisionId from the mint block. */ mintRevisionId: number | null; - /** The revision the off-chain facts were read from (may differ from the mint revision). */ + /** The live revision the off-chain facts were read from (data §3; may differ from the mint revision). */ fetchedRevisionId: number | null; wikidataId: string | null; wikidataRev: number | null; - instanceOf: WikidataClass[]; + /** Wikipedia short description. */ + description: string | null; + /** Non-hidden category titles without the "Category:" prefix, sorted. */ categories: string[]; - /** P17 (country) or P27 (citizenship) Q-ids. */ - country: string[]; - dates: IdentityDates; + /** P31 (instance of). */ + instanceOf: WikidataClass[]; + /** P279 of the item, or its P31 classes' superclasses, nearest first. */ + subclassOf: WikidataClass[]; + /** P17 / P27 country Q-ids, sorted. */ + countries: string[]; + /** Years from P569/P570/P571/P580/P585 (negative = BCE), sorted. */ + years: number[]; + /** P570 (date of death) present. */ + died: boolean; /** P625 or prop=coordinates, [lat, lon]. */ coords: [number, number] | null; - /** Main-namespace outlink titles, sorted, deduplicated. Empty when unknown. */ - outlinks: string[]; - /** sha256 of the sorted outlinks, so match records can log a short fingerprint. */ - outlinksHash: string; + /** ns0 outlinks as pageIds, sorted, deduplicated (Connected, Linked tier). Empty when unknown. */ + links: number[]; + /** Item Q-ids cited by the frozen claims (P17, P27, P131, P361, P106, ...). */ + cites: string[]; + /** sha256 of `links`, so match records can log a short fingerprint. */ + linksHash: string; /** Where the facts came from: an offline cache (e.g. card-sample.json) or a live background fetch. */ source: 'cache' | 'wikimedia'; frozenAt: string; }; /** What a source knows about an article; any field may be missing. */ -export type IdentityFacts = Partial>; +export type IdentityFacts = Partial>; /** The minimum the chain tells us about a card (built from its mint block, never from client JSON). */ export type ChainArticle = { tokenId: string; pageId: number; title: string; revisionId?: number | null }; @@ -59,15 +74,20 @@ export interface IdentitySource { lookup(articles: ChainArticle[]): Promise>; } -export function outlinksHash(outlinks: readonly string[]): string { - return createHash('sha256').update(JSON.stringify(outlinks)).digest('hex'); +export function linksHash(links: readonly number[]): string { + return createHash('sha256').update(JSON.stringify(links)).digest('hex'); } -const uniqSorted = (xs: readonly string[] | undefined) => [...new Set((xs ?? []).filter(x => typeof x === 'string'))].sort(); +const QID = /^Q\d+$/; +const uniqStrings = (xs: readonly unknown[] | undefined) => [...new Set((xs ?? []).filter((x): x is string => typeof x === 'string'))].sort(); +const uniqInts = (xs: readonly unknown[] | undefined) => + [...new Set((xs ?? []).filter((x): x is number => Number.isSafeInteger(x)))].sort((a, b) => a - b); +const classes = (xs: readonly WikidataClass[] | undefined) => + (xs ?? []).filter(c => c && QID.test(c.id)).map(c => ({ id: c.id, label: typeof c.label === 'string' ? c.label : '' })); /** Pure: build a frozen identity from chain data plus whatever facts a source returned. */ export function buildIdentity(article: ChainArticle, facts: IdentityFacts, source: CardIdentity['source'], now: Date): CardIdentity { - const outlinks = uniqSorted(facts.outlinks); + const links = uniqInts(facts.links); const coords = facts.coords && facts.coords.length === 2 && facts.coords.every(Number.isFinite) ? [facts.coords[0], facts.coords[1]] as [number, number] : null; return { version: IDENTITY_VERSION, @@ -76,20 +96,31 @@ export function buildIdentity(article: ChainArticle, facts: IdentityFacts, sourc title: article.title, mintRevisionId: article.revisionId ?? null, fetchedRevisionId: facts.fetchedRevisionId ?? null, - wikidataId: facts.wikidataId && /^Q\d+$/.test(facts.wikidataId) ? facts.wikidataId : null, + wikidataId: facts.wikidataId && QID.test(facts.wikidataId) ? facts.wikidataId : null, wikidataRev: facts.wikidataRev ?? null, - instanceOf: (facts.instanceOf ?? []).filter(c => /^Q\d+$/.test(c.id)).map(c => c.label ? { id: c.id, label: c.label } : { id: c.id }), - categories: uniqSorted(facts.categories), - country: uniqSorted(facts.country).filter(q => /^Q\d+$/.test(q)), - dates: { ...(facts.dates ?? {}) }, + description: typeof facts.description === 'string' ? facts.description : null, + categories: uniqStrings(facts.categories), + instanceOf: classes(facts.instanceOf), + subclassOf: classes(facts.subclassOf), + countries: uniqStrings(facts.countries), + years: uniqInts(facts.years), + died: facts.died === true, coords, - outlinks, - outlinksHash: outlinksHash(outlinks), + links, + cites: uniqStrings(facts.cites).filter(q => QID.test(q)), + linksHash: linksHash(links), source, frozenAt: now.toISOString(), }; } +/** The duel-cards `CardProfile` view of an identity (structural; no import, so no stacked PR). */ +export function toProfile(i: CardIdentity) { + return { wikidataId: i.wikidataId, description: i.description, categories: i.categories, instanceOf: i.instanceOf, + subclassOf: i.subclassOf, countries: i.countries, years: i.years, died: i.died, links: i.links, cites: i.cites, + ...(i.fetchedRevisionId != null ? { fetchedRevisionId: i.fetchedRevisionId } : {}) }; +} + /** Persistent per-token identity store. Records are immutable once written. */ export class IdentityStore { readonly #db: DatabaseSync; @@ -153,7 +184,7 @@ export async function prebake(store: IdentityStore, source: IdentitySource, arti /** Offline source: facts from an already-cached card file such as docs/game/card-sample.json. */ export function cachedSource(cards: readonly { pageId: number; wikidataId?: string | null; revisionId?: number | null; - instanceOf?: WikidataClass[]; categories?: string[] }[]): IdentitySource { + description?: string | null; instanceOf?: WikidataClass[]; categories?: string[] }[]): IdentitySource { const byPage = new Map(cards.map(c => [c.pageId, c])); return { kind: 'cache', @@ -161,7 +192,7 @@ export function cachedSource(cards: readonly { pageId: number; wikidataId?: stri return new Map(articles.map(a => { const c = byPage.get(a.pageId); return [a.pageId, c ? { fetchedRevisionId: c.revisionId ?? null, wikidataId: c.wikidataId ?? null, - instanceOf: c.instanceOf ?? [], categories: c.categories ?? [] } : undefined]; + description: c.description ?? null, instanceOf: c.instanceOf ?? [], categories: c.categories ?? [] } : undefined]; })); }, }; diff --git a/tests/testnet-ops-identity.test.ts b/tests/testnet-ops-identity.test.ts index dd7ae8f..4ac71c1 100644 --- a/tests/testnet-ops-identity.test.ts +++ b/tests/testnet-ops-identity.test.ts @@ -4,7 +4,7 @@ import assert from 'node:assert/strict'; import { mkdtempSync, readFileSync, rmSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; -import { IdentityStore, buildIdentity, cachedSource, outlinksHash, prebake, +import { IdentityStore, buildIdentity, cachedSource, linksHash, prebake, toProfile, type ChainArticle, type IdentitySource } from '../server/game/identity.ts'; const sample = JSON.parse(readFileSync(new URL('../docs/game/card-sample.json', import.meta.url), 'utf8')).cards as any[]; @@ -28,6 +28,10 @@ test('no fetch: pre-bake from the cached sample freezes every sample card', asyn assert.deepEqual(salah.instanceOf, [{ id: 'Q5', label: 'human' }]); assert.equal(salah.source, 'cache'); assert.ok(salah.categories.includes('Sultans of Aceh')); + assert.equal(salah.description, 'Sultan of Aceh (1530–1537/9)'); + const profile = toProfile(salah); + assert.deepEqual(profile.instanceOf, [{ id: 'Q5', label: 'human' }]); + assert.equal(profile.fetchedRevisionId, 1358732737); } finally { globalThis.fetch = realFetch; store.close(); done(); } }); @@ -35,8 +39,8 @@ test('freeze at first sight: a later, different profile never overwrites the sto const { store, done } = tmpStore(); try { const a = art(sample[0], 1); - const first = store.freeze(buildIdentity(a, { instanceOf: [{ id: 'Q5' }] }, 'cache', now())); - const second = store.freeze(buildIdentity(a, { instanceOf: [{ id: 'Q515' }] }, 'wikimedia', new Date())); + const first = store.freeze(buildIdentity(a, { instanceOf: [{ id: 'Q5', label: 'human' }] }, 'cache', now())); + const second = store.freeze(buildIdentity(a, { instanceOf: [{ id: 'Q515', label: 'city' }] }, 'wikimedia', new Date())); assert.deepEqual(second, first); assert.equal(store.get(a.tokenId)!.instanceOf[0].id, 'Q5'); const again = await prebake(store, cachedSource(sample), [a, a], { now }); @@ -60,7 +64,7 @@ test('sources are asked in bounded batches and identities persist across reopen' const calls: number[] = []; const source: IdentitySource = { kind: 'wikimedia', async lookup(batch) { calls.push(batch.length); - return new Map(batch.map(a => [a.pageId, { outlinks: ['B', 'A', 'A'], coords: [1, 2] as [number, number], country: ['Q30', 'bad'] }])); + return new Map(batch.map(a => [a.pageId, { links: [7, 3, 3, 1.5], coords: [1, 2] as [number, number], cites: ['Q30', 'bad'], years: [1948, -44], died: true }])); } }; let store = new IdentityStore(join(dir, 'id.sqlite')); await prebake(store, source, sample.slice(0, 12).map(art), { batchSize: 5, now }); @@ -68,9 +72,11 @@ test('sources are asked in bounded batches and identities persist across reopen' store.close(); store = new IdentityStore(join(dir, 'id.sqlite')); const one = store.get(art(sample[3], 3).tokenId)!; - assert.deepEqual(one.outlinks, ['A', 'B']); - assert.equal(one.outlinksHash, outlinksHash(['A', 'B'])); - assert.deepEqual(one.country, ['Q30']); + assert.deepEqual(one.links, [3, 7]); + assert.equal(one.linksHash, linksHash([3, 7])); + assert.deepEqual(one.cites, ['Q30']); + assert.deepEqual(one.years, [-44, 1948]); + assert.equal(one.died, true); assert.deepEqual(one.coords, [1, 2]); assert.equal(one.mintRevisionId, sample[3].revisionId); assert.equal(store.count(), 12); -- 2.54.0 From c35db68c2049f4583a62b5dd638fcb4561821119 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 17 Nov 2026 22:40:59 -0500 Subject: [PATCH 3/6] testnet-ops: TESTNET.md secure-context workaround and counts; identity store under WIKIDECK_DATA_DIR Co-Authored-By: Claude Opus 5.5 --- docs/night2/TESTNET.md | 13 +++++++++++-- scripts/night2/prebake-identity.ts | 6 +++--- server/game/identity.ts | 5 +++++ tests/testnet-ops-identity.test.ts | 7 ++++++- 4 files changed, 25 insertions(+), 6 deletions(-) diff --git a/docs/night2/TESTNET.md b/docs/night2/TESTNET.md index 5b6696c..6a8ec7a 100644 --- a/docs/night2/TESTNET.md +++ b/docs/night2/TESTNET.md @@ -25,13 +25,22 @@ Four disposable wallets: **alice**, **bob**, **carol** and **dave**. | carol | `f2d908633c4313bd77e86650c13d11ac3e7e656223ebe881cc34ffba3f3e0524` | leftovers | | dave | `02bb97f202e1616ab97b45e8938e384dcbf622a0b44a7f4c1589f44236d3ccef` | leftovers | +- Confirmed at height 91: **alice 41, bob 40**, carol 5, dave 5. Block #92 came from the Android emulator (#59). - alice and bob were consolidated to 40 cards each with on-chain NFT transfers from carol and dave, so **production `deckSize` 40 works on the testnet** and no lower-deckSize test mode is needed. - **The encrypted browser backups are in a comment on #63**, not in git. - Backup password: `wikideck-night-testnet`. It is **public and testnet-only**: never reuse it, and never put a real secret next to it. - Re-check counts at any time: `GET http://192.168.1.32:3101/api/decks/
?offset=0&limit=24` (pages of 24, follow `next`). ### Load a wallet in the browser -Use a separate browser profile, or a private window, per player, because a browser holds one wallet per origin. +**First make the node origins a secure context** (found by night-review on #63). +- `http://192.168.1.32:310x` is plain HTTP on a LAN address, so the browser leaves `crypto.subtle` undefined. Open wallet then fails with "Cannot read properties of undefined (reading 'digest')" or "Wallet storage unavailable". +- **Chrome/Chromium:** open `chrome://flags/#unsafely-treat-insecure-origin-as-secure`, add `http://192.168.1.32:3101,http://192.168.1.32:3102,http://192.168.1.32:3103`, then relaunch. +- **Command line:** `--unsafely-treat-insecure-origin-as-secure=http://192.168.1.32:3101,http://192.168.1.32:3102,http://192.168.1.32:3103 --user-data-dir=`. +- **Playwright:** use the full `chromium` channel. The headless shell ignores this flag. +- **SSH tunnel to `localhost`:** this also works for the wallet, but it breaks the Dex callback URLs, so you can't do the login/link step that way. +- **Firefox** has no per-origin equivalent. + +Use a separate browser profile per player, because a browser holds one wallet per origin. 1. Open a node URL, then **Open wallet**. 2. **Recover from a local backup**: choose the backup JSON file (save it from the #63 comment as `alice.backup.json`), and enter the backup password above. 3. Set a new local encryption password (at least 12 characters, anything testnet-only), then confirm it. @@ -51,7 +60,7 @@ Use a separate browser profile, or a private window, per player, because a brows ## Card identity pre-bake (engineer WS E) - `server/game/identity.ts` is a persistent per-token store (`node:sqlite`). It freezes each card's off-chain identity at first sight and never rewrites it. The identity covers P31/P279 classes, description, categories, countries, years, coordinates and outlink pageIds. Its fields match duel-cards' `CardProfile` (#70), and `toProfile()` returns that view. -- `scripts/night2/prebake-identity.ts --node --store [--cards ...]` reads the canonical chain from one node and freezes what the cached card files know. +- `scripts/night2/prebake-identity.ts --node [--store ] [--cards ...]` reads the canonical chain from one node and freezes what the cached card files know. The store defaults to `$WIKIDECK_DATA_DIR/card-identity.sqlite` (`identityStoreFile()`), and hosts should open it at the same path. - It **never calls Wikimedia itself**. - Live facts arrive only through the wikimedia team's client on its `background` lane (#62), once that client has merged. - Cards that are still unknown stay unfrozen and are retried on a later run. diff --git a/scripts/night2/prebake-identity.ts b/scripts/night2/prebake-identity.ts index f7ab0c8..31c88f0 100644 --- a/scripts/night2/prebake-identity.ts +++ b/scripts/night2/prebake-identity.ts @@ -1,7 +1,7 @@ // testnet-ops (engineer WS E): pre-bake frozen card identities for every card on a chain. // // node --experimental-strip-types scripts/night2/prebake-identity.ts --node http://192.168.1.32:3101 \ -// --store ./identity.sqlite [--cards docs/game/card-sample.json] [--batch 50] +// --store $WIKIDECK_DATA_DIR/card-identity.sqlite [--cards docs/game/card-sample.json] [--batch 50] // // - Reads the canonical chain from ONE node (GET /api/summary, then /api/canonical pages), so every // card is built from its mint block, never from client JSON. @@ -13,11 +13,11 @@ // - Idempotent and freeze-at-first-sight: already frozen tokens are skipped and never rewritten. import { readFileSync } from 'node:fs'; import { parseArgs } from 'node:util'; -import { IdentityStore, cachedSource, prebake, type ChainArticle } from '../../server/game/identity.ts'; +import { IdentityStore, cachedSource, identityStoreFile, prebake, type ChainArticle } from '../../server/game/identity.ts'; const { values } = parseArgs({ options: { node: { type: 'string', default: 'http://192.168.1.32:3101' }, - store: { type: 'string', default: 'identity.sqlite' }, + store: { type: 'string', default: identityStoreFile() }, cards: { type: 'string', multiple: true, default: ['docs/game/card-sample.json'] }, batch: { type: 'string', default: '50' }, } }); diff --git a/server/game/identity.ts b/server/game/identity.ts index 9578826..1e5f48b 100644 --- a/server/game/identity.ts +++ b/server/game/identity.ts @@ -10,8 +10,13 @@ // - First write wins: `freeze` never overwrites an existing record (data §3 "freeze at first sight"). import { DatabaseSync } from 'node:sqlite'; import { createHash } from 'node:crypto'; +import { join, resolve } from 'node:path'; export const IDENTITY_VERSION = 1; +/** Where hosts keep the store: next to the ledger, `$WIKIDECK_DATA_DIR/card-identity.sqlite` (default ./data). */ +export function identityStoreFile(env: NodeJS.ProcessEnv = process.env): string { + return join(env.WIKIDECK_DATA_DIR ?? resolve('data'), 'card-identity.sqlite'); +} /** Same shape as shared/duel/classify.ts `WikidataClass` (duel-cards, #70). */ export type WikidataClass = { id: string; label: string }; diff --git a/tests/testnet-ops-identity.test.ts b/tests/testnet-ops-identity.test.ts index 4ac71c1..9b9cf69 100644 --- a/tests/testnet-ops-identity.test.ts +++ b/tests/testnet-ops-identity.test.ts @@ -4,7 +4,7 @@ import assert from 'node:assert/strict'; import { mkdtempSync, readFileSync, rmSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; -import { IdentityStore, buildIdentity, cachedSource, linksHash, prebake, toProfile, +import { identityStoreFile, IdentityStore, buildIdentity, cachedSource, linksHash, prebake, toProfile, type ChainArticle, type IdentitySource } from '../server/game/identity.ts'; const sample = JSON.parse(readFileSync(new URL('../docs/game/card-sample.json', import.meta.url), 'utf8')).cards as any[]; @@ -83,3 +83,8 @@ test('sources are asked in bounded batches and identities persist across reopen' store.close(); } finally { rmSync(dir, { recursive: true, force: true }); } }); + +test('the store file lives under WIKIDECK_DATA_DIR, next to the ledger', () => { + assert.equal(identityStoreFile({ WIKIDECK_DATA_DIR: '/srv/wd' }), join('/srv/wd', 'card-identity.sqlite')); + assert.ok(identityStoreFile({}).endsWith(join('data', 'card-identity.sqlite'))); +}); -- 2.54.0 From 4f9cbfeaaedb4c786571b2dfb1bea8825e7d2363 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 17 Nov 2026 22:41:21 -0500 Subject: [PATCH 4/6] TESTNET.md: test wallets are readable by anyone with issue access Co-Authored-By: Claude Opus 5.5 --- docs/night2/TESTNET.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/night2/TESTNET.md b/docs/night2/TESTNET.md index 6a8ec7a..57cb211 100644 --- a/docs/night2/TESTNET.md +++ b/docs/night2/TESTNET.md @@ -29,6 +29,7 @@ Four disposable wallets: **alice**, **bob**, **carol** and **dave**. - alice and bob were consolidated to 40 cards each with on-chain NFT transfers from carol and dave, so **production `deckSize` 40 works on the testnet** and no lower-deckSize test mode is needed. - **The encrypted browser backups are in a comment on #63**, not in git. - Backup password: `wikideck-night-testnet`. It is **public and testnet-only**: never reuse it, and never put a real secret next to it. + - Anyone who can read this repo's issues can open these wallets. Keep them on the LAN-only testnet, and never send them anything of value. - Re-check counts at any time: `GET http://192.168.1.32:3101/api/decks/
?offset=0&limit=24` (pages of 24, follow `next`). ### Load a wallet in the browser -- 2.54.0 From c9197e1a9f68ef59746ed76d2440ce0bc0c9dff2 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 17 Nov 2026 22:59:19 -0500 Subject: [PATCH 5/6] testnet-ops: live identity source (background lane only), deps injected structurally wikimediaIdentitySource() takes the wikimedia client (#73), the page enrichment and duel-cards' profile builder (#70) as structural deps, so it imports nothing unmerged. About 2 Wikidata requests per 50 new cards; transport failures leave cards unknown. Co-Authored-By: Claude Opus 5.5 --- server/game/identity-source.ts | 112 ++++++++++++++++++++++ tests/testnet-ops-identity-source.test.ts | 82 ++++++++++++++++ 2 files changed, 194 insertions(+) create mode 100644 server/game/identity-source.ts create mode 100644 tests/testnet-ops-identity-source.test.ts diff --git a/server/game/identity-source.ts b/server/game/identity-source.ts new file mode 100644 index 0000000..a996ed0 --- /dev/null +++ b/server/game/identity-source.ts @@ -0,0 +1,112 @@ +// Live IdentitySource for server/game/identity.ts (engineer WS E, data §3). +// +// Every request goes through the process-wide polite Wikimedia client on the **background** lane: +// - the page enrichment (description, wikidataId, categories) via `enrichment.getBackground()`, +// which also feeds the disk cache card faces use; +// - one `wbgetentities&props=claims` per batch of at most 50 items; +// - one `wbgetentities&props=labels` per 50 class ids still missing a label. +// That is about 2 Wikidata requests per 50 new cards, plus one cached enrichment per card. +// Outlinks (prop=links, data §4) are not fetched tonight: they cost about 1.2 requests per card, +// and Connected falls back to its other tiers without them. +// +// The profile itself is built by duel-cards' pure `profileFromSources()` (shared/duel/profile.ts, #70), +// so the frozen identity and what derive() reads cannot drift apart. That module, and the client +// (server/wikimedia, #73), are injected structurally, so this file has no import from unmerged PRs. +// Wiring, once both have merged: +// wikimediaIdentitySource({ client: wikimediaClient, enrichment: wikipediaEnrichment, +// profile: await import('../../shared/duel/profile.ts') }) +// Any transport failure leaves the affected cards unknown (undefined), so they are retried later, +// never frozen half-empty. +import type { PageEnrichment } from '../../shared/enrichment.ts'; +import type { ChainArticle, IdentityFacts, IdentitySource, WikidataClass } from './identity.ts'; + +/** Structural slices of server/wikimedia/client.ts (#73) and shared/duel/profile.ts (#70). */ +export type BackgroundRequest = { lane: 'background'; timeoutMs: number; cache?: { ttlMs: number; maxBytes?: number } }; +export type WikidataEntity = { id?: string; claims?: Record }; +export type ProfileLike = { wikidataId: string | null; description: string | null; categories: readonly string[]; + instanceOf: readonly WikidataClass[]; subclassOf?: readonly WikidataClass[]; countries?: readonly string[]; + years?: readonly number[]; died?: boolean; links?: readonly number[]; cites?: readonly string[]; fetchedRevisionId?: number }; +export type ProfileBuilder = { + wikidataEntityParams(qids: readonly string[]): Record; + wikidataLabelParams(qids: readonly string[]): Record; + classIdsOf(entity: WikidataEntity | null | undefined): string[]; + profileFromSources(src: { enrichment: PageEnrichment; entity?: WikidataEntity | null; labels?: Readonly> }): ProfileLike; +}; +export type IdentitySourceDeps = { + enrichment: { getBackground(pageId: number): Promise }; + client: { request(url: URL, options: BackgroundRequest): Promise<{ response: Response }> }; + profile: ProfileBuilder; +}; + +const WIKIDATA_API = 'https://www.wikidata.org/w/api.php'; +const MAX_BYTES = 4 * 1024 * 1024; +const CACHE = { ttlMs: 7 * 24 * 60 * 60 * 1000, maxBytes: MAX_BYTES }; + +async function wikidata(client: IdentitySourceDeps['client'], params: Record): Promise> { + const url = new URL(WIKIDATA_API); + for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v); + const { response } = await client.request(url, { lane: 'background', timeoutMs: 15000, cache: CACHE }); + if (!response.ok) { await response.body?.cancel(); throw new Error(`Wikidata HTTP ${response.status}`); } + const text = await response.text(); + if (text.length > MAX_BYTES) throw new Error('Wikidata response too large'); + const body = JSON.parse(text) as { entities?: Record | unknown[] }; + const entities = body.entities ?? {}; + // formatversion=2 returns an object keyed by id; tolerate an array too. + return Array.isArray(entities) + ? Object.fromEntries(entities.filter(e => e && typeof (e as { id?: unknown }).id === 'string').map(e => [(e as { id: string }).id, e])) + : entities as Record; +} + +const chunks = (xs: T[], n: number) => Array.from({ length: Math.ceil(xs.length / n) }, (_, i) => xs.slice(i * n, i * n + n)); + +export function wikimediaIdentitySource(deps: IdentitySourceDeps): IdentitySource { + const { classIdsOf, profileFromSources, wikidataEntityParams, wikidataLabelParams } = deps.profile; + return { + kind: 'wikimedia', + async lookup(articles: ChainArticle[]) { + const out = new Map(articles.map(a => [a.pageId, undefined])); + // 1. Enrichment, one page at a time (the client paces it); failures stay unknown. + const enriched = new Map(); + for (const a of articles) { + try { enriched.set(a.pageId, await deps.enrichment.getBackground(a.pageId)); } catch { /* retry later */ } + } + // 2. Claims for every item we know, in batches of ≤ 50. + const qids = [...new Set([...enriched.values()].map(e => e.wikidataId).filter((q): q is string => !!q && /^Q\d+$/.test(q)))]; + const entities = new Map(); + let claimsFailed = false; + for (const batch of chunks(qids, 50)) { + try { for (const [id, e] of Object.entries(await wikidata(deps.client, wikidataEntityParams(batch)))) entities.set(id, e as WikidataEntity); } + catch { claimsFailed = true; } + } + // 3. English labels for the P31/P279 classes. + const classIds = [...new Set([...entities.values()].flatMap(e => classIdsOf(e)))]; + const labels: Record = {}; + let labelsFailed = false; + for (const batch of chunks(classIds, 50)) { + try { + for (const [id, e] of Object.entries(await wikidata(deps.client, wikidataLabelParams(batch)))) { + const v = (e as { labels?: { en?: { value?: unknown } } }).labels?.en?.value; + if (typeof v === 'string') labels[id] = v; + } + } catch { labelsFailed = true; } + } + // 4. Freeze only complete profiles: a card whose item couldn't be read stays unknown. + for (const a of articles) { + const e = enriched.get(a.pageId); + if (!e) continue; + const qid = e.wikidataId && /^Q\d+$/.test(e.wikidataId) ? e.wikidataId : null; + const entity = qid ? entities.get(qid) : null; + if (qid && !entity && claimsFailed) continue; // item not read: retry later + if (entity && labelsFailed) continue; // class labels not read: retry later + const p = profileFromSources({ enrichment: e, entity: entity ?? null, labels }); + out.set(a.pageId, { + fetchedRevisionId: p.fetchedRevisionId ?? null, wikidataId: p.wikidataId, description: p.description, + categories: [...p.categories], instanceOf: [...p.instanceOf], subclassOf: [...(p.subclassOf ?? [])], + countries: [...(p.countries ?? [])], years: [...(p.years ?? [])], died: p.died === true, + links: [...(p.links ?? [])], cites: [...(p.cites ?? [])], + }); + } + return out; + }, + }; +} diff --git a/tests/testnet-ops-identity-source.test.ts b/tests/testnet-ops-identity-source.test.ts new file mode 100644 index 0000000..0344ba1 --- /dev/null +++ b/tests/testnet-ops-identity-source.test.ts @@ -0,0 +1,82 @@ +// testnet-ops: live identity source over a FAKE Wikimedia client. Zero network calls. +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { mkdtempSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { wikimediaIdentitySource } from '../server/game/identity-source.ts'; +import { IdentityStore, prebake, toProfile } from '../server/game/identity.ts'; +import type { BackgroundRequest, ProfileBuilder } from '../server/game/identity-source.ts'; + +// A minimal stand-in for shared/duel/profile.ts (#70), so this test needs no unmerged module. +const ids = (e: any, p: string): string[] => (e?.claims?.[p] ?? []).map((s: any) => s.mainsnak?.datavalue?.value?.id).filter(Boolean); +const yrs = (e: any, p: string): number[] => (e?.claims?.[p] ?? []).map((s: any) => Number(/^[+-]0*(\d+)-/.exec(s.mainsnak?.datavalue?.value?.time ?? '')?.[1])).filter(Boolean); +const profile: ProfileBuilder = { + wikidataEntityParams: q => ({ action: 'wbgetentities', ids: [...q].sort().join('|'), props: 'claims', format: 'json', formatversion: '2' }), + wikidataLabelParams: q => ({ action: 'wbgetentities', ids: [...q].sort().join('|'), props: 'labels', languages: 'en', format: 'json', formatversion: '2' }), + classIdsOf: e => [...new Set([...ids(e, 'P31'), ...ids(e, 'P279')])].sort(), + profileFromSources: ({ enrichment, entity, labels = {} }) => ({ + wikidataId: enrichment.wikidataId, description: enrichment.description, categories: enrichment.categories, + instanceOf: ids(entity, 'P31').map(id => ({ id, label: labels[id] ?? id })), countries: ids(entity, 'P27'), + years: [...yrs(entity, 'P569'), ...yrs(entity, 'P570')], died: ids(entity, 'P570').length > 0 || yrs(entity, 'P570').length > 0, + fetchedRevisionId: enrichment.revisionId }), +}; + +const enrichmentFor = (pageId: number, wikidataId: string | null) => ({ + pageId, title: `Page ${pageId}`, description: `desc ${pageId}`, wikidataId, revisionId: 1000 + pageId, pageBytes: 1, + languageCount: 1, languagesTruncated: false, categories: ['Living people', '1948 births'], categoriesTruncated: false, + fetchedAt: '2026-11-17T00:00:00Z', + pageviews: { status: 'unavailable', months: [], latestMonth: '2026-10', total: null, dailyAverage: null, trendPercent: null }, +}) as any; +const claim = (id: string) => ({ rank: 'normal', mainsnak: { snaktype: 'value', datavalue: { type: 'wikibase-entityid', value: { id } } } }); +const time = (t: string) => ({ rank: 'normal', mainsnak: { snaktype: 'value', datavalue: { type: 'time', value: { time: t } } } }); + +function fakeClient(opts: { failClaims?: boolean } = {}) { + const calls: { url: URL; options: BackgroundRequest }[] = []; + return { calls, async request(url: URL, options: BackgroundRequest) { + calls.push({ url, options }); + const props = url.searchParams.get('props'), ids = url.searchParams.get('ids')!.split('|'); + if (props === 'claims' && opts.failClaims) return { response: new Response('busy', { status: 503 }), signal: new AbortController().signal, fromCache: false }; + const entities = Object.fromEntries(ids.map(id => [id, props === 'claims' + ? { id, claims: { P31: [claim('Q5')], P27: [claim('Q30')], P569: [time('+1948-01-02T00:00:00Z')], P570: [time('+2001-01-01T00:00:00Z')] } } + : { id, labels: { en: { value: id === 'Q5' ? 'human' : `label ${id}` } } }])); + return { response: Response.json({ entities }), signal: new AbortController().signal, fromCache: false }; + } }; +} + +test('every request is on the background lane, batched, and the frozen identity is derive-ready', async () => { + const realFetch = globalThis.fetch; + globalThis.fetch = (() => { throw new Error('network call in test'); }) as typeof fetch; + const dir = mkdtempSync(join(tmpdir(), 'identity-src-')); + try { + const client = fakeClient(); + const source = wikimediaIdentitySource({ client, profile, enrichment: { async getBackground(id) { return enrichmentFor(id, `Q${id}`); } } }); + const store = new IdentityStore(join(dir, 'id.sqlite')); + const articles = [1, 2, 3].map(i => ({ tokenId: String(i).repeat(64), pageId: i, title: `Page ${i}`, revisionId: i })); + const r = await prebake(store, source, articles); + assert.deepEqual(r, { seen: 3, alreadyFrozen: 0, frozen: 3, unknown: 0 }); + assert.ok(client.calls.every(c => c.options.lane === 'background')); + assert.ok(client.calls.every(c => c.url.hostname === 'www.wikidata.org')); + assert.deepEqual(client.calls.map(c => c.url.searchParams.get('props')), ['claims', 'labels']); + assert.equal(client.calls[0].url.searchParams.get('ids'), 'Q1|Q2|Q3'); + const one = store.get('1'.repeat(64))!; + assert.equal(one.source, 'wikimedia'); + assert.deepEqual(one.instanceOf, [{ id: 'Q5', label: 'human' }]); + assert.deepEqual(one.countries, ['Q30']); + assert.deepEqual(one.years, [1948, 2001]); + assert.equal(one.died, true); + assert.equal(one.fetchedRevisionId, 1001); + assert.equal(toProfile(one).description, 'desc 1'); + store.close(); + } finally { globalThis.fetch = realFetch; rmSync(dir, { recursive: true, force: true }); } +}); + +test('a failed claims request leaves cards unknown so they are retried, never frozen half-empty', async () => { + const source = wikimediaIdentitySource({ client: fakeClient({ failClaims: true }), profile, + enrichment: { async getBackground(id) { if (id === 9) throw new Error('throttled'); return enrichmentFor(id, id === 8 ? null : `Q${id}`); } } }); + const got = await source.lookup([7, 8, 9].map(i => ({ tokenId: String(i).repeat(64), pageId: i, title: `P${i}` }))); + assert.equal(got.get(7), undefined); // has an item, claims failed → retry + assert.equal(got.get(9), undefined); // enrichment failed → retry + assert.equal(got.get(8)?.wikidataId, null); // no Wikidata item at all → enrichment-only profile is final + assert.deepEqual(got.get(8)?.instanceOf, []); +}); -- 2.54.0 From 26ad4af02ebf550010f101ef24185b87667cc93f Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 17 Nov 2026 22:59:46 -0500 Subject: [PATCH 6/6] identity-source: WikidataEntity matches #70's shape exactly Co-Authored-By: Claude Opus 5.5 --- server/game/identity-source.ts | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/server/game/identity-source.ts b/server/game/identity-source.ts index a996ed0..f034923 100644 --- a/server/game/identity-source.ts +++ b/server/game/identity-source.ts @@ -22,7 +22,8 @@ import type { ChainArticle, IdentityFacts, IdentitySource, WikidataClass } from /** Structural slices of server/wikimedia/client.ts (#73) and shared/duel/profile.ts (#70). */ export type BackgroundRequest = { lane: 'background'; timeoutMs: number; cache?: { ttlMs: number; maxBytes?: number } }; -export type WikidataEntity = { id?: string; claims?: Record }; +export type WikidataEntity = { id?: string; claims?: Record }; export type ProfileLike = { wikidataId: string | null; description: string | null; categories: readonly string[]; instanceOf: readonly WikidataClass[]; subclassOf?: readonly WikidataClass[]; countries?: readonly string[]; years?: readonly number[]; died?: boolean; links?: readonly number[]; cites?: readonly string[]; fetchedRevisionId?: number }; -- 2.54.0