N8: 24-client network soak, and the server fixes it found

make soak runs a real server and 24 headless clients in three processes for
two whole matches in real time. Each client reaches the server through its own
UdpRelay, which adds 60 ms, +/-20 ms jitter and 2% loss each way, so ENet runs
unchanged over a bad link. SoakPlayer drives each client from what a client
knows: it drops, glides, heads for high ground, harvests and tries to build.
tools/soak_report.gd checks world hashes against the server's, errors,
unreadable packets, every client playing every match with its connection
intact, memory between matches, and the p99 tick with the network's share.
make soak-selfcheck proves a drifted client world is caught.

Server changes: --matches=N, --world-hash-every=N, tick timing that includes
after_tick, and memory in match and progress lines.

What the soak found and fixed:
- Snapshot building for 24 clients took the p99 tick to 13.8 ms, over the
  12 ms budget, against a 2.8 ms simulation. Encoded records are now shared
  between clients for the tick, unchanged records skip the field diff, and
  clients are spread over the snapshot interval by handle. p99 is 7.3 ms.
- ENet peers had a timeout limit of 0, which gives up on a peer as soon as any
  reliable message has waited the minimum timeout. Now ENet's default, 32.

Not fixed: one or two clients still lose their connection about 15 s into the
second match's drop in each run (see the issue), so the soak still fails and
N8 stays open.

Refs #39

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-12 21:56:01 -05:00
co-authored by Claude Opus 5.5
parent 65b1fe015c
commit 05c7568ec1
25 changed files with 1135 additions and 39 deletions
+28 -1
View File
@@ -251,6 +251,12 @@ changed than fits, the client's own player goes first, then the nearest, and eac
snapshot an update has waited counts as 6 m closer, so nothing waits for ever. What
didn't fit is still owed and goes in a later snapshot.
**Cost.** Building 24 snapshots is the largest part of a busy server's tick, more
than the simulation. Records are made once a tick and shared, each (entity, mask)
encoding is made once a tick and shared, an unchanged record is skipped with one
native comparison, and clients are spread over the ticks of the snapshot interval by
handle (`NetGame.snapshot_due_for()`), so each tick builds half of them.
**Input.** The client answers on channel 1 with the snapshot tick it holds and its
last few commands (`Tuning.NET_INPUT_REDUNDANCY`), oldest first. `NetGame`
(`src/server/`) offers each tick's command to `ServerLoop.submit()` once, and echoes
@@ -509,8 +515,29 @@ client is connecting, in the lobby or watching.
| Budget | Target |
|---|---|
| Server tick | Under **12 ms** with 24 players |
| Server tick | Under **12 ms** with 24 players, including the network's share (`make soak` checks the p99) |
| Client | **60 fps** on a GTX 1060-class GPU at medium settings |
| Terrain streaming | Under **8 ms** of main-thread work in any frame (measured headless; GPU upload unmeasured) |
Software rendering in CI can't verify the client budget. It needs a real GPU playtest.
## Soak testing
`make soak` (`tools/soak.sh`) is the multiplayer gate: a real server process and 24
headless clients in three processes, each client behind its own `UdpRelay`, a UDP
relay that delays, jitters and loses datagrams both ways, so ENet runs unchanged over
a bad link. `SoakPlayer` plays each client from what a client knows: it drops, glides,
heads for high ground, harvests and tries to build.
Desync is checked by hash: the server, started with `--world-hash-every=N`, logs
`NetWorld.world_hash()` on the simulation ticks from each multiple of N to the snapshot
interval, and each client hashes its own world on the snapshots at those ticks when its
world messages have caught up exactly (`world_seq`) and it has no provisional pieces.
Tick 0 is never compared: it lasts the whole lobby while the world is set up.
`tools/soak_report.gd` compares them and checks errors, unreadable packets, that every
client played every match, the p99 tick, and memory from one match to the next.
`make soak-selfcheck` tampers with one client's world to prove a drift is caught.
A client process holds back a new match's notice until it has made that client's
island, at most one a frame: a dozen clients making islands in one frame starved
their connections of service for long enough that ENet dropped them.