H4 Long-run stability #57

Closed
opened 2026-11-09 05:29:41 +00:00 by claude · 1 comment
Owner

Depends on: N8, M8

A dedicated server runs back-to-back matches across many seeds, under accelerated time where valid.

Acceptance

  • 50 consecutive seeded matches with no crash, hang or script error.
  • No memory growth trend across the run.
  • Any crash seed becomes a regression test.
**Depends on:** N8, M8 A dedicated server runs back-to-back matches across many seeds, under accelerated time where valid. **Acceptance** - 50 consecutive seeded matches with no crash, hang or script error. - No memory growth trend across the run. - Any crash seed becomes a regression test.
claude added this to the v0.9 Hardening milestone 2026-11-09 05:29:41 +00:00
claude added the priority/hightype/chore labels 2026-11-09 05:29:41 +00:00
claude added this to the Tidewall v1.0 project 2026-11-09 15:00:09 +00:00
eric closed this issue 2026-11-14 12:15:27 +00:00
Author
Owner

Done in f87d5ea.

make stability (tools/stability.sh): one dedicated server process plays 50 bot matches back to back, unthrottled, under nice, and samples its resident memory every 5 s. That's about 13 minutes on this host. StabilityReport then fails the run on any of:

  • a crash, a non-zero exit, or engine/script error lines;
  • a match over 240 s of wall time (a hang) or with no winner;
  • memory growing after the first five matches, by least squares: more than 256 KiB a match static, or 1 MiB a match resident.

First run: no crash, hang or error, but two leaks.

  1. Tick timing kept forever. ServerLoop.tick_usec appended 8 bytes a tick, which is about 21 MB a day at 30 Hz. Every progress line also copied and sorted the whole array. It's replaced by TickTimes, which gives the same nearest-rank statistics in fixed memory: a 1 µs histogram up to 65 ms, plus the rare slower ticks (up to 4,096 of them, keeping the slowest). The headless server's with-network timings use it too.
  2. Resident memory grew 28 MB a match, linearly (905 MB by match 42), while the engine's own count grew 0.2 MB. glibc gives every thread that allocates its own malloc arena, and each next island is built on a WorkerThreadPool thread.
    • Over 15 matches: no limit +28 MB/match; MALLOC_ARENA_MAX=2 +4.8 MB/match, in steps; MALLOC_ARENA_MAX=1 flat.
    • Tick times were unchanged: p50 501 vs 507 µs, p99 1635 vs 1625 µs.
    • Both server launchers now export MALLOC_ARENA_MAX="${MALLOC_ARENA_MAX:-1}". That's packaging/server/tidewall-server (Docker and systemd) and packaging/dist/tidewall-server (the tarball, and the .deb, which execs it). tools/godot.sh does the same for --server. Documented in packaging/README.md.

Result after the fixes (seed 4242, 50 matches in 756 s):

  • every match ended with a winner;
  • exit 0, 0 server errors, 0 error lines;
  • static memory +1.6 KiB a match;
  • resident memory +145 KiB a match, flat at 204.2 MB from match 26 to 49.

No seed crashed, so there was no crash seed to turn into a regression test.

Tests:

  • New tests: TickTimes checked against a sorted list (percentiles, min/max, over budget) and its memory cap; a test that the launchers set the arena limit; StabilityReport on a clean run and on each kind of failure.
  • make check (663 passed), make dist-check and make server-package-check all pass with the new launchers.

— tidewall-build

Done in f87d5ea. **`make stability`** (`tools/stability.sh`): one dedicated server process plays 50 bot matches back to back, unthrottled, under `nice`, and samples its resident memory every 5 s. That's about 13 minutes on this host. `StabilityReport` then fails the run on any of: - a crash, a non-zero exit, or engine/script error lines; - a match over 240 s of wall time (a hang) or with no winner; - memory growing after the first five matches, by least squares: more than 256 KiB a match static, or 1 MiB a match resident. **First run: no crash, hang or error, but two leaks.** 1. **Tick timing kept forever.** `ServerLoop.tick_usec` appended 8 bytes a tick, which is about 21 MB a day at 30 Hz. Every progress line also copied and sorted the whole array. It's replaced by `TickTimes`, which gives the same nearest-rank statistics in fixed memory: a 1 µs histogram up to 65 ms, plus the rare slower ticks (up to 4,096 of them, keeping the slowest). The headless server's with-network timings use it too. 2. **Resident memory grew 28 MB a match, linearly** (905 MB by match 42), while the engine's own count grew 0.2 MB. glibc gives every thread that allocates its own malloc arena, and each next island is built on a WorkerThreadPool thread. - Over 15 matches: no limit +28 MB/match; `MALLOC_ARENA_MAX=2` +4.8 MB/match, in steps; `MALLOC_ARENA_MAX=1` flat. - Tick times were unchanged: p50 501 vs 507 µs, p99 1635 vs 1625 µs. - Both server launchers now `export MALLOC_ARENA_MAX="${MALLOC_ARENA_MAX:-1}"`. That's `packaging/server/tidewall-server` (Docker and systemd) and `packaging/dist/tidewall-server` (the tarball, and the .deb, which execs it). `tools/godot.sh` does the same for `--server`. Documented in `packaging/README.md`. **Result after the fixes** (seed 4242, 50 matches in 756 s): - every match ended with a winner; - exit 0, 0 server errors, 0 error lines; - static memory +1.6 KiB a match; - resident memory +145 KiB a match, flat at 204.2 MB from match 26 to 49. No seed crashed, so there was no crash seed to turn into a regression test. **Tests:** - New tests: `TickTimes` checked against a sorted list (percentiles, min/max, over budget) and its memory cap; a test that the launchers set the arena limit; `StabilityReport` on a clean run and on each kind of failure. - `make check` (663 passed), `make dist-check` and `make server-package-check` all pass with the new launchers. — tidewall-build
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: claude/tidewall#57