Each client process now also logs the engine's object count and how many
distinct worlds and prediction grounds its clients reach, at the start of
each match's play and again a minute later (memory_late). This is for
finding the client memory growth the soak still reports (#39); the
checks are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The soak's drops under loss came from stalls: the server's island (#79) and
each client process making islands on its main thread fed ENet second-long
round-trip samples, its retransmission timer grew, and two losses of one
reliable message in the busy drop that followed reached the 15 s timeout.
- ClientGame.build_in_background makes a new match's island on a
WorkerThreadPool task and holds the packets that arrive meanwhile, then
takes them in order. The networked client and the soak's clients use it;
the soak's own one-island-a-frame holding is gone.
- NetServer.close(grace_msec) sends the shutdown notice reliably and
services the socket while clients acknowledge it. The headless server
waits up to 2 s. A notice sent once was lost under 2% loss, and that
client timed out instead of hearing the server stop.
- Tests: background island holding and ordering, freeing mid-build, a
graceful shutdown reaching four clients behind 50% loss (it fails with
the old one-shot notice), and the scripted UDP match and client capture
wait for the client's island the way a real 20 s lobby would.
The soak at 2% loss and +/-20 ms now keeps all 24 clients for both matches,
with 0 of 3,758 world hashes differing and every client told of the stop.
It still fails on client memory growth between matches in two of three
processes (17 and 20 MB), which is not yet explained, so #39 stays open.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
make soak runs a real server and 24 headless clients in three processes for
two whole matches in real time. Each client reaches the server through its own
UdpRelay, which adds 60 ms, +/-20 ms jitter and 2% loss each way, so ENet runs
unchanged over a bad link. SoakPlayer drives each client from what a client
knows: it drops, glides, heads for high ground, harvests and tries to build.
tools/soak_report.gd checks world hashes against the server's, errors,
unreadable packets, every client playing every match with its connection
intact, memory between matches, and the p99 tick with the network's share.
make soak-selfcheck proves a drifted client world is caught.
Server changes: --matches=N, --world-hash-every=N, tick timing that includes
after_tick, and memory in match and progress lines.
What the soak found and fixed:
- Snapshot building for 24 clients took the p99 tick to 13.8 ms, over the
12 ms budget, against a 2.8 ms simulation. Encoded records are now shared
between clients for the tick, unchanged records skip the field diff, and
clients are spread over the snapshot interval by handle. p99 is 7.3 ms.
- ENet peers had a timeout limit of 0, which gives up on a peer as soon as any
reliable message has waited the minimum timeout. Now ENet's default, 32.
Not fixed: one or two clients still lose their connection about 15 s into the
second match's drop in each run (see the issue), so the soak still fails and
N8 stays open.
Refs #39
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>