N8: 24-client network soak, and the server fixes it found

make soak runs a real server and 24 headless clients in three processes for
two whole matches in real time. Each client reaches the server through its own
UdpRelay, which adds 60 ms, +/-20 ms jitter and 2% loss each way, so ENet runs
unchanged over a bad link. SoakPlayer drives each client from what a client
knows: it drops, glides, heads for high ground, harvests and tries to build.
tools/soak_report.gd checks world hashes against the server's, errors,
unreadable packets, every client playing every match with its connection
intact, memory between matches, and the p99 tick with the network's share.
make soak-selfcheck proves a drifted client world is caught.

Server changes: --matches=N, --world-hash-every=N, tick timing that includes
after_tick, and memory in match and progress lines.

What the soak found and fixed:
- Snapshot building for 24 clients took the p99 tick to 13.8 ms, over the
  12 ms budget, against a 2.8 ms simulation. Encoded records are now shared
  between clients for the tick, unchanged records skip the field diff, and
  clients are spread over the snapshot interval by handle. p99 is 7.3 ms.
- ENet peers had a timeout limit of 0, which gives up on a peer as soon as any
  reliable message has waited the minimum timeout. Now ENet's default, 32.

Not fixed: one or two clients still lose their connection about 15 s into the
second match's drop in each run (see the issue), so the soak still fails and
N8 stays open.

Refs #39

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-12 21:56:01 -05:00
co-authored by Claude Opus 5.5
parent 65b1fe015c
commit 05c7568ec1
25 changed files with 1135 additions and 39 deletions
+11 -1
View File
@@ -1,7 +1,7 @@
# Tidewall build entry points. Everything runs headless with the repo-local engine.
GODOT := tools/godot.sh
.PHONY: check import parse test test-selfcheck smoke capture-test terrain-capture binaries verify-engine bot-matches match match-captures client-check client-captures discovery-check fuzz
.PHONY: check import parse test test-selfcheck smoke capture-test terrain-capture binaries verify-engine bot-matches match match-captures client-check client-captures discovery-check fuzz soak soak-selfcheck
## Everything that must pass before a push.
check: verify-engine binaries parse test test-selfcheck smoke
@@ -72,3 +72,13 @@ capture-test: import
## Renders the terrain crack check and its control in the offscreen container. Needs Docker.
terrain-capture: import
@tools/terrain_capture_check.sh
## The 24-client network soak: a real server and 24 headless clients behind a relay that
## adds latency, jitter and loss, for SOAK_MATCHES (default 2) whole matches in real
## time, about half an hour. Checks for desyncs, errors, leaks and the tick budget.
soak: import
@tools/soak.sh
## About a minute: proves the soak catches a client whose world has drifted.
soak-selfcheck: import
@SOAK_SELFCHECK=1 tools/soak.sh