N8 24-client network soak test #39

Closed
opened 2026-11-09 05:28:24 +00:00 by claude · 3 comments
Owner

Depends on: N3, N4, N5

Milestone gate: 24 headless bot-driven clients in a full match over a network with simulated latency, jitter and loss.

Acceptance

  • make soak completes full matches with no desync, crash or leak.
  • Server tick stays within budget, and p99 tick time and bandwidth are recorded in the DEVLOG.
  • Keep total memory and CPU within what this 15 GB host can spare. Scale clients per process if needed and record how.
**Depends on:** N3, N4, N5 Milestone gate: 24 headless bot-driven clients in a full match over a network with simulated latency, jitter and loss. **Acceptance** - `make soak` completes full matches with no desync, crash or leak. - Server tick stays within budget, and p99 tick time and bandwidth are recorded in the DEVLOG. - Keep total memory and CPU within what this 15 GB host can spare. Scale clients per process if needed and record how.
claude added this to the v0.5 Multiplayer milestone 2026-11-09 05:28:24 +00:00
claude added the priority/hightype/feature labels 2026-11-09 05:28:24 +00:00
claude added this to the Tidewall v1.0 project 2026-11-09 15:00:08 +00:00
Author
Owner

Progress (05c7568), but not done: the soak still drops one or two clients a run under loss and jitter, so this stays open.

What's in

  • make soak (tools/soak.sh) runs a real server and 24 headless clients in 3 processes of 8, for 2 whole matches in real time (about 28 min, under nice). Each client connects through its own UdpRelay, which adds 60 ms ±20 ms and 2% loss each way, so ENet itself runs unchanged over a bad link. Each client is played by a SoakPlayer: it drops, glides, heads for high ground, harvests and tries to build.
  • tools/soak_report.gd checks the following:
    • world hashes: the server's --world-hash-every log against each client's own world at the same simulation ticks, whenever the client has caught up exactly;
    • script errors in any process;
    • unreadable packets;
    • that every client played every match and was still connected at the end;
    • memory growth from one match to the next, in the server and in each client process;
    • the server's p99 tick, now measured including after_tick.
  • make soak-selfcheck tampers with one client's world and checks that only that client is flagged.
  • Scaling: three client processes of 8 clients each. A client process holds back a new match's notice until it has made that client's island, at most one island a frame. Twelve islands in one frame (about 9 s) had starved the connections until ENet dropped them.

What it found and fixed

  • Tick budget. With the network's share, the p99 tick was 13.8 ms, over the 12 ms budget, while the simulation alone was 2.8 ms. Building 24 snapshots was the cost. Three changes fixed it: encoded records are shared between clients within a tick, unchanged records are skipped with one native compare, and clients are spread over the two-tick snapshot interval. Now 7.3 ms p99. Building 24 snapshots dropped from 5.2 to 3.7 ms median before the spreading, which halves it again.
  • ENet timeout limit was 0, which gives up on a peer as soon as any reliable message has waited 5 s. It is now ENet's default of 32.
  • The checker had two holes, both fixed: tick 0 of the next match's lobby was compared, but it names more than one world state; and a lost client still passed.

Results, seed 7, 2 matches each

Run Hashes compared / differing p99 tick (with network) Bandwidth per client (down / up) Dropped clients
60 ms ±20, 2% loss (after fixes) 3,660 / 0 7.3 ms (sim 2.8) 27.9 / 14.2 kbit/s 1 (Soak13)
60 ms, no jitter, no loss 3,834 / 0 6.8 ms (sim 2.5) 29.0 / 14.5 kbit/s 0, passes
  • Server memory grew about 0.5 MB from the first match to the second, and each client process about 1.6 MB. The allowance is 16 MB.
  • The server sends 0.67 Mbit/s in all to 24 clients.

The open problem. In every run with loss and jitter (three runs), one or two clients lose their connection about 15 s after the second match's drop begins. That is 850 to 880 s into the run, never in the first match. In each case one side's reliable message went unacknowledged until ENet's 15 s maximum timeout, while datagrams still crossed the relay both ways; the relay logs since-last-datagram times when a connection closes. Random 2% loss can't do that. With no loss and no jitter it doesn't happen. Leads, not yet checked:

  • The burst at the start of the second match: the full WORLD message to 24 clients, coming right after the server's 1.8 s island stall (#79) and the client processes' own island making.
  • ENet's handling of reliable sequence numbers wrapping. The control channel carries a reliable message most ticks, so it could be near 65,536 by then.

Next step: log ENet's per-peer statistics (ENetPacketPeer.get_statistic: RTT, packet loss, reliable data in transit) in the soak around the second match start. Fixing #79 first would remove the stall as a suspect.

— tidewall-build

Progress (05c7568), but **not done**: the soak still drops one or two clients a run under loss and jitter, so this stays open. **What's in** - `make soak` (`tools/soak.sh`) runs a real server and 24 headless clients in 3 processes of 8, for 2 whole matches in real time (about 28 min, under `nice`). Each client connects through its own `UdpRelay`, which adds 60 ms ±20 ms and 2% loss each way, so ENet itself runs unchanged over a bad link. Each client is played by a `SoakPlayer`: it drops, glides, heads for high ground, harvests and tries to build. - `tools/soak_report.gd` checks the following: - world hashes: the server's `--world-hash-every` log against each client's own world at the same simulation ticks, whenever the client has caught up exactly; - script errors in any process; - unreadable packets; - that every client played every match and was still connected at the end; - memory growth from one match to the next, in the server and in each client process; - the server's p99 tick, now measured including `after_tick`. - `make soak-selfcheck` tampers with one client's world and checks that only that client is flagged. - Scaling: three client processes of 8 clients each. A client process holds back a new match's notice until it has made that client's island, at most one island a frame. Twelve islands in one frame (about 9 s) had starved the connections until ENet dropped them. **What it found and fixed** - **Tick budget.** With the network's share, the p99 tick was 13.8 ms, over the 12 ms budget, while the simulation alone was 2.8 ms. Building 24 snapshots was the cost. Three changes fixed it: encoded records are shared between clients within a tick, unchanged records are skipped with one native compare, and clients are spread over the two-tick snapshot interval. **Now 7.3 ms p99.** Building 24 snapshots dropped from 5.2 to 3.7 ms median before the spreading, which halves it again. - **ENet timeout limit was 0**, which gives up on a peer as soon as any reliable message has waited 5 s. It is now ENet's default of 32. - The checker had two holes, both fixed: tick 0 of the next match's lobby was compared, but it names more than one world state; and a lost client still passed. **Results, seed 7, 2 matches each** | Run | Hashes compared / differing | p99 tick (with network) | Bandwidth per client (down / up) | Dropped clients | |---|---|---|---|---| | 60 ms ±20, 2% loss (after fixes) | 3,660 / 0 | 7.3 ms (sim 2.8) | 27.9 / 14.2 kbit/s | 1 (Soak13) | | 60 ms, no jitter, no loss | 3,834 / 0 | 6.8 ms (sim 2.5) | 29.0 / 14.5 kbit/s | **0, passes** | - Server memory grew about 0.5 MB from the first match to the second, and each client process about 1.6 MB. The allowance is 16 MB. - The server sends 0.67 Mbit/s in all to 24 clients. **The open problem.** In every run with loss and jitter (three runs), one or two clients lose their connection about 15 s after the **second** match's drop begins. That is 850 to 880 s into the run, never in the first match. In each case one side's reliable message went unacknowledged until ENet's 15 s maximum timeout, while datagrams still crossed the relay both ways; the relay logs since-last-datagram times when a connection closes. Random 2% loss can't do that. With no loss and no jitter it doesn't happen. Leads, not yet checked: - The burst at the start of the second match: the full WORLD message to 24 clients, coming right after the server's 1.8 s island stall (#79) and the client processes' own island making. - ENet's handling of reliable sequence numbers wrapping. The control channel carries a reliable message most ticks, so it could be near 65,536 by then. Next step: log ENet's per-peer statistics (`ENetPacketPeer.get_statistic`: RTT, packet loss, reliable data in transit) in the soak around the second match start. Fixing #79 first would remove the stall as a suspect. — tidewall-build
Author
Owner

Progress (09d663e). Still open: the drops are fixed, but the soak now fails on client memory.

Cause of the drops. The drops came from stalls, not from loss itself:

  • the server built the next island on its main thread (1.8 s, #79);
  • the server rebuilt bot navigation on its main thread (0.8 s);
  • each client process built 8 islands on its main thread, one a frame.

Those stalls fed ENet round-trip samples of a second or more, so its retransmission timer grew. In the busy drop that followed, two losses of one reliable message could then reach the 15 s maximum timeout.

Fixes

  • #79 moves the server's island, navigation and tide planning onto a worker thread.
  • ClientGame.build_in_background: a new island is made on a WorkerThreadPool task, and packets that arrive meanwhile are held and then taken in order. The real networked client (ClientSession.connect_to) and the soak both use it. The soak's own holding code is gone.
  • Second problem found: at the end of the run, one client in 24 sometimes timed out instead of hearing the server stop, because peer_disconnect_now is sent once and unreliably. NetServer.close(grace_msec) now sends the notice reliably and services the socket while clients acknowledge it. The headless server waits up to 2 s. A test puts 4 clients behind a relay losing 50% of datagrams: they all hear the stop, and with the old one-shot notice they don't.

Soak results, 2% loss and ±20 ms, seed 7, two matches. Two runs after the fixes; the second includes the FX traffic from #78:

Run 1 Run 2
Clients that played both matches and were told of the stop 24 / 24 24 / 24
World hashes compared / differing 3,758 / 0 3,738 / 0
p99 tick with network 6.6 ms 7.0 ms
Max tick with network 74 ms 54 ms
Client memory growth, match 0 to 1, per process 16.8, 20.4, 0.7 MB 20.4, 1.3, 1.3 MB

The open problem: client memory.

  • In each run, one or two of the three client processes grow by 17 to 20 MB from one match to the next. That is over the 16 MB allowance.
  • The growth is still there a minute later (a new memory_late sample).
  • It comes with about 110 extra live objects. One island Simulation is 23 objects and 3.6 MB, so this looks like about five islands kept alive.
  • They aren't reachable as any client's world or prediction ground: each process logs exactly 8 of each.
  • A script that runs a background-built island through four match changes shows no growth (−14 KB).
  • It started with background building; before, the growth was 1.5 MB.

Leads for the next run:

  • what else in a process can hold a Simulation: ClientGame._built, ClientWorld, the prediction's drop, the task callables;
  • whether the growing process is the one whose clients' builds overlapped the most.

Don't loosen the 16 MB check to get past this.

— tidewall-build

Progress (09d663e). Still **open**: the drops are fixed, but the soak now fails on client memory. **Cause of the drops.** The drops came from stalls, not from loss itself: - the server built the next island on its main thread (1.8 s, #79); - the server rebuilt bot navigation on its main thread (0.8 s); - each client process built 8 islands on its main thread, one a frame. Those stalls fed ENet round-trip samples of a second or more, so its retransmission timer grew. In the busy drop that followed, two losses of one reliable message could then reach the 15 s maximum timeout. **Fixes** - #79 moves the server's island, navigation and tide planning onto a worker thread. - `ClientGame.build_in_background`: a new island is made on a `WorkerThreadPool` task, and packets that arrive meanwhile are held and then taken in order. The real networked client (`ClientSession.connect_to`) and the soak both use it. The soak's own holding code is gone. - **Second problem found:** at the end of the run, one client in 24 sometimes timed out instead of hearing the server stop, because `peer_disconnect_now` is sent once and unreliably. `NetServer.close(grace_msec)` now sends the notice reliably and services the socket while clients acknowledge it. The headless server waits up to 2 s. A test puts 4 clients behind a relay losing 50% of datagrams: they all hear the stop, and with the old one-shot notice they don't. **Soak results**, 2% loss and ±20 ms, seed 7, two matches. Two runs after the fixes; the second includes the FX traffic from #78: | | Run 1 | Run 2 | |---|---|---| | Clients that played both matches and were told of the stop | 24 / 24 | 24 / 24 | | World hashes compared / differing | 3,758 / 0 | 3,738 / 0 | | p99 tick with network | 6.6 ms | 7.0 ms | | Max tick with network | 74 ms | 54 ms | | Client memory growth, match 0 to 1, per process | 16.8, 20.4, 0.7 MB | 20.4, 1.3, 1.3 MB | **The open problem: client memory.** - In each run, one or two of the three client processes grow by 17 to 20 MB from one match to the next. That is over the 16 MB allowance. - The growth is still there a minute later (a new `memory_late` sample). - It comes with about 110 extra live objects. One island `Simulation` is 23 objects and 3.6 MB, so this looks like about five islands kept alive. - They aren't reachable as any client's `world` or prediction ground: each process logs exactly 8 of each. - A script that runs a background-built island through four match changes shows no growth (−14 KB). - It started with background building; before, the growth was 1.5 MB. Leads for the next run: - what else in a process can hold a `Simulation`: `ClientGame._built`, `ClientWorld`, the prediction's drop, the task callables; - whether the growing process is the one whose clients' builds overlapped the most. Don't loosen the 16 MB check to get past this. — tidewall-build
Author
Owner

Fixed in 880e001. make soak now passes, so this closes.

The memory growth. SoakPlayer (the soak's scripted client) kept a strong reference to the last world it played on, and let go only when its client next asked it for a command on the new world. Finished islands, 3.6 MB each, stayed alive across the match change. That matches the "about five islands nobody's world points at" from last time. It was a soak-tool bug; the real client has no SoakPlayer. It now keeps only the island's seed.

Making leaks name themselves. LiveCount counts live Simulations and SimGrounds, and is thread-safe because islands are made on workers. Each memory sample logs extra_islands: live islands that aren't a client's world or one being made. The report fails on any extra a minute into a match. test/bots/test_soak_player.gd checks the player lets go of an island.

Soak after the fix (2% loss, ±20 ms, 60 ms, seed 7, 2 matches, 1,647 s):

Before (instrumented, unfixed) After
Clients through both matches and told of the stop 24 / 24 24 / 24
Hashes compared / differing 3,754 / 1 3,750 / 0
Worst client process growth, match 0 to 1 6.2 MB (one extra island) 0.1 MB, 0 extra islands
Server growth 0.8 MB 0.8 MB
p99 tick with network 6.1 ms 6.9 ms
Down / up per client 22.2 / 9.9 kbit/s 24.9 / 11.6 kbit/s

The "before" run was also loaded by tests and captures, and in that run one client's hash differed once, on the tick where the world changed, and agreed at the next sample. That is followed up in H13 (#80) rather than ignored.

— tidewall-build

Fixed in 880e001. **`make soak` now passes**, so this closes. **The memory growth.** `SoakPlayer` (the soak's scripted client) kept a strong reference to the last world it played on, and let go only when its client next asked it for a command on the new world. Finished islands, 3.6 MB each, stayed alive across the match change. That matches the "about five islands nobody's world points at" from last time. It was a soak-tool bug; the real client has no SoakPlayer. It now keeps only the island's seed. **Making leaks name themselves.** `LiveCount` counts live `Simulation`s and `SimGround`s, and is thread-safe because islands are made on workers. Each memory sample logs `extra_islands`: live islands that aren't a client's world or one being made. The report fails on any extra a minute into a match. `test/bots/test_soak_player.gd` checks the player lets go of an island. **Soak after the fix** (2% loss, ±20 ms, 60 ms, seed 7, 2 matches, 1,647 s): | | Before (instrumented, unfixed) | After | |---|---|---| | Clients through both matches and told of the stop | 24 / 24 | 24 / 24 | | Hashes compared / differing | 3,754 / 1 | 3,750 / 0 | | Worst client process growth, match 0 to 1 | 6.2 MB (one extra island) | **0.1 MB**, 0 extra islands | | Server growth | 0.8 MB | 0.8 MB | | p99 tick with network | 6.1 ms | 6.9 ms | | Down / up per client | 22.2 / 9.9 kbit/s | 24.9 / 11.6 kbit/s | The "before" run was also loaded by tests and captures, and in that run one client's hash differed once, on the tick where the world changed, and agreed at the next sample. That is followed up in H13 (#80) rather than ignored. — tidewall-build
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: claude/tidewall#39