A dedicated server runs back-to-back matches across many seeds, under accelerated time where valid.
Acceptance
50 consecutive seeded matches with no crash, hang or script error.
No memory growth trend across the run.
Any crash seed becomes a regression test.
**Depends on:** N8, M8
A dedicated server runs back-to-back matches across many seeds, under accelerated time where valid.
**Acceptance**
- 50 consecutive seeded matches with no crash, hang or script error.
- No memory growth trend across the run.
- Any crash seed becomes a regression test.
claude
added this to the v0.9 Hardening milestone 2026-11-09 05:29:41 +00:00
make stability (tools/stability.sh): one dedicated server process plays 50 bot matches back to back, unthrottled, under nice, and samples its resident memory every 5 s. That's about 13 minutes on this host. StabilityReport then fails the run on any of:
a crash, a non-zero exit, or engine/script error lines;
a match over 240 s of wall time (a hang) or with no winner;
memory growing after the first five matches, by least squares: more than 256 KiB a match static, or 1 MiB a match resident.
First run: no crash, hang or error, but two leaks.
Tick timing kept forever.ServerLoop.tick_usec appended 8 bytes a tick, which is about 21 MB a day at 30 Hz. Every progress line also copied and sorted the whole array. It's replaced by TickTimes, which gives the same nearest-rank statistics in fixed memory: a 1 µs histogram up to 65 ms, plus the rare slower ticks (up to 4,096 of them, keeping the slowest). The headless server's with-network timings use it too.
Resident memory grew 28 MB a match, linearly (905 MB by match 42), while the engine's own count grew 0.2 MB. glibc gives every thread that allocates its own malloc arena, and each next island is built on a WorkerThreadPool thread.
Over 15 matches: no limit +28 MB/match; MALLOC_ARENA_MAX=2 +4.8 MB/match, in steps; MALLOC_ARENA_MAX=1 flat.
Tick times were unchanged: p50 501 vs 507 µs, p99 1635 vs 1625 µs.
Both server launchers now export MALLOC_ARENA_MAX="${MALLOC_ARENA_MAX:-1}". That's packaging/server/tidewall-server (Docker and systemd) and packaging/dist/tidewall-server (the tarball, and the .deb, which execs it). tools/godot.sh does the same for --server. Documented in packaging/README.md.
Result after the fixes (seed 4242, 50 matches in 756 s):
every match ended with a winner;
exit 0, 0 server errors, 0 error lines;
static memory +1.6 KiB a match;
resident memory +145 KiB a match, flat at 204.2 MB from match 26 to 49.
No seed crashed, so there was no crash seed to turn into a regression test.
Tests:
New tests: TickTimes checked against a sorted list (percentiles, min/max, over budget) and its memory cap; a test that the launchers set the arena limit; StabilityReport on a clean run and on each kind of failure.
make check (663 passed), make dist-check and make server-package-check all pass with the new launchers.
— tidewall-build
Done in f87d5ea.
**`make stability`** (`tools/stability.sh`): one dedicated server process plays 50 bot matches back to back, unthrottled, under `nice`, and samples its resident memory every 5 s. That's about 13 minutes on this host. `StabilityReport` then fails the run on any of:
- a crash, a non-zero exit, or engine/script error lines;
- a match over 240 s of wall time (a hang) or with no winner;
- memory growing after the first five matches, by least squares: more than 256 KiB a match static, or 1 MiB a match resident.
**First run: no crash, hang or error, but two leaks.**
1. **Tick timing kept forever.** `ServerLoop.tick_usec` appended 8 bytes a tick, which is about 21 MB a day at 30 Hz. Every progress line also copied and sorted the whole array. It's replaced by `TickTimes`, which gives the same nearest-rank statistics in fixed memory: a 1 µs histogram up to 65 ms, plus the rare slower ticks (up to 4,096 of them, keeping the slowest). The headless server's with-network timings use it too.
2. **Resident memory grew 28 MB a match, linearly** (905 MB by match 42), while the engine's own count grew 0.2 MB. glibc gives every thread that allocates its own malloc arena, and each next island is built on a WorkerThreadPool thread.
- Over 15 matches: no limit +28 MB/match; `MALLOC_ARENA_MAX=2` +4.8 MB/match, in steps; `MALLOC_ARENA_MAX=1` flat.
- Tick times were unchanged: p50 501 vs 507 µs, p99 1635 vs 1625 µs.
- Both server launchers now `export MALLOC_ARENA_MAX="${MALLOC_ARENA_MAX:-1}"`. That's `packaging/server/tidewall-server` (Docker and systemd) and `packaging/dist/tidewall-server` (the tarball, and the .deb, which execs it). `tools/godot.sh` does the same for `--server`. Documented in `packaging/README.md`.
**Result after the fixes** (seed 4242, 50 matches in 756 s):
- every match ended with a winner;
- exit 0, 0 server errors, 0 error lines;
- static memory +1.6 KiB a match;
- resident memory +145 KiB a match, flat at 204.2 MB from match 26 to 49.
No seed crashed, so there was no crash seed to turn into a regression test.
**Tests:**
- New tests: `TickTimes` checked against a sorted list (percentiles, min/max, over budget) and its memory cap; a test that the launchers set the arena limit; `StabilityReport` on a clean run and on each kind of failure.
- `make check` (663 passed), `make dist-check` and `make server-package-check` all pass with the new launchers.
— tidewall-build
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Depends on: N8, M8
A dedicated server runs back-to-back matches across many seeds, under accelerated time where valid.
Acceptance
Done in
f87d5ea.make stability(tools/stability.sh): one dedicated server process plays 50 bot matches back to back, unthrottled, undernice, and samples its resident memory every 5 s. That's about 13 minutes on this host.StabilityReportthen fails the run on any of:First run: no crash, hang or error, but two leaks.
ServerLoop.tick_usecappended 8 bytes a tick, which is about 21 MB a day at 30 Hz. Every progress line also copied and sorted the whole array. It's replaced byTickTimes, which gives the same nearest-rank statistics in fixed memory: a 1 µs histogram up to 65 ms, plus the rare slower ticks (up to 4,096 of them, keeping the slowest). The headless server's with-network timings use it too.MALLOC_ARENA_MAX=2+4.8 MB/match, in steps;MALLOC_ARENA_MAX=1flat.export MALLOC_ARENA_MAX="${MALLOC_ARENA_MAX:-1}". That'spackaging/server/tidewall-server(Docker and systemd) andpackaging/dist/tidewall-server(the tarball, and the .deb, which execs it).tools/godot.shdoes the same for--server. Documented inpackaging/README.md.Result after the fixes (seed 4242, 50 matches in 756 s):
No seed crashed, so there was no crash seed to turn into a regression test.
Tests:
TickTimeschecked against a sorted list (percentiles, min/max, over budget) and its memory cap; a test that the launchers set the arena limit;StabilityReporton a clean run and on each kind of failure.make check(663 passed),make dist-checkandmake server-package-checkall pass with the new launchers.— tidewall-build