H12 Server stalls about 1.8 s making the next match's island #79

Closed
opened 2026-11-13 02:26:21 +00:00 by claude · 1 comment
Owner

Found by: the N8 soak (make soak), seed 7.

When a match's results end, Match makes the next match's Simulation on the server's main thread. Generating the island takes about 1.8 s under the soak's load, so one tick takes that long (with_net_max_us 1,809,278 in the soak's stats line). Nothing is sent to anyone in the meantime. ENet survives it and the lobby that follows is 20 s, so nobody has been dropped, but everyone connected sees a freeze, and a server with a shorter lobby or more load would be closer to the 5 s minimum timeout.

Fix ideas

  • Generate the next island on a worker thread during the results phase, from the seed, which is known by then. The simulation only reads it once it's built.
  • Or generate it during the previous match's results in slices, a few rows of the heightmap a tick.

Acceptance

  • The soak's with_net_max_us stays under 50 ms across a match change.
  • The next match's world hash is unchanged by where it was generated.

— tidewall-build

**Found by:** the N8 soak (`make soak`), seed 7. When a match's results end, `Match` makes the next match's `Simulation` on the server's main thread. Generating the island takes about 1.8 s under the soak's load, so one tick takes that long (`with_net_max_us` 1,809,278 in the soak's stats line). Nothing is sent to anyone in the meantime. ENet survives it and the lobby that follows is 20 s, so nobody has been dropped, but everyone connected sees a freeze, and a server with a shorter lobby or more load would be closer to the 5 s minimum timeout. **Fix ideas** - Generate the next island on a worker thread during the results phase, from the seed, which is known by then. The simulation only reads it once it's built. - Or generate it during the previous match's results in slices, a few rows of the heightmap a tick. **Acceptance** - The soak's `with_net_max_us` stays under 50 ms across a match change. - The next match's world hash is unchanged by where it was generated. — tidewall-build
claude added this to the v0.9 Hardening milestone 2026-11-13 02:26:21 +00:00
claude added the type/bug label 2026-11-13 02:26:21 +00:00
eric closed this issue 2026-11-13 06:38:57 +00:00
Author
Owner

Fixed in ad0fd26.

What changed

  • When a match's results begin, Match builds the next match's Simulation on a WorkerThreadPool task, and takes it when the lobby opens.
  • The same task also plans the island's tide levels (Simulation.plan_tide(), about 30 ms) and runs a prepare_next hook. ServerLoop uses the hook to build the bots' NavGrid and BotWorld.
  • Every step reads only the new simulation.
  • The first soak after moving only the island showed that the bots' navigation was a second stall of 0.82 s. The next soak showed the tide plan cost 74 ms at match start. Both are now off the main thread too.

Acceptance

  • Soak, 24 clients, 2% loss, ±20 ms, seed 7, run on the final code:
    • with_net_max_us is 54,193 for the whole run.
    • The simulation's worst tick, 50.2 ms, came before tick 900, at startup and the first match's start. The progress lines never logged a higher maximum after that, so the match change at tick ~25,000 stayed under that level. Before the fix it was 1,809,278.
  • Paced server, 24 bots, real time through the results, lobby, countdown and 3 s of the next drop: worst tick 5.4 ms.
  • World hash: tests compare the background-built simulation's state hash, terrain, solids, props and tide levels with a freshly built one, and its navigation with a fresh NavGrid and BotWorld. The ten-matches-in-a-row test still matches its synchronous references.

Follow-up: the first match's start still costs up to about 50 ms on a loaded host: loot placement, plus that match's tide planned on the main thread. It's well inside a 5 s timeout, but it's worth a look in H1 (#54).

— tidewall-build

Fixed in ad0fd26. **What changed** - When a match's results begin, `Match` builds the next match's `Simulation` on a `WorkerThreadPool` task, and takes it when the lobby opens. - The same task also plans the island's tide levels (`Simulation.plan_tide()`, about 30 ms) and runs a `prepare_next` hook. `ServerLoop` uses the hook to build the bots' `NavGrid` and `BotWorld`. - Every step reads only the new simulation. - The first soak after moving only the island showed that the bots' navigation was a second stall of 0.82 s. The next soak showed the tide plan cost 74 ms at match start. Both are now off the main thread too. **Acceptance** - **Soak**, 24 clients, 2% loss, ±20 ms, seed 7, run on the final code: - `with_net_max_us` is 54,193 for the whole run. - The simulation's worst tick, 50.2 ms, came before tick 900, at startup and the first match's start. The progress lines never logged a higher maximum after that, so the match change at tick ~25,000 stayed under that level. Before the fix it was 1,809,278. - **Paced server**, 24 bots, real time through the results, lobby, countdown and 3 s of the next drop: worst tick 5.4 ms. - **World hash:** tests compare the background-built simulation's state hash, terrain, solids, props and tide levels with a freshly built one, and its navigation with a fresh `NavGrid` and `BotWorld`. The ten-matches-in-a-row test still matches its synchronous references. **Follow-up:** the first match's start still costs up to about 50 ms on a loaded host: loot placement, plus that match's tide planned on the main thread. It's well inside a 5 s timeout, but it's worth a look in H1 (#54). — tidewall-build
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: claude/tidewall#79