make stability (tools/stability.sh, StabilityReport) runs one dedicated
server for 50 bot matches back to back, unthrottled, sampling its resident
memory, and fails on a crash, hang, engine or script error, a match with
no winner, or memory that grows after the first five matches.
The first run had no crash, hang or error, but showed two leaks:
- ServerLoop kept every tick's duration in an array (8 bytes a tick,
about 21 MB a day at 30 Hz) and copied and sorted all of it for every
progress line. TickTimes keeps exact statistics in fixed memory: a
one-microsecond histogram up to 65 ms, plus the rare slower ticks.
- Resident memory grew about 28 MB a match (905 MB after 42 matches):
glibc gives each thread its own malloc arena, and every next island is
built on a worker thread. The server launchers (packaging/server,
packaging/dist, so also the .deb, Docker image and systemd unit) and
tools/godot.sh for --server now set MALLOC_ARENA_MAX=1 unless already
set. Tick times were unchanged (p50 501 vs 507 us, p99 1635 vs 1625).
After: 50 matches (seed 4242) in 756 s, all with a winner, exit 0, no
errors; static memory +1.6 KiB and resident +145 KiB a match, flat at
204 MB from match 26 on. No match crashed, so there is no crash seed to
turn into a regression test.
Tests: TickTimes against a sorted list, its fixed memory, the launchers'
setting, and StabilityReport. make check, dist-check and
server-package-check pass.
Closes#57
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- packaging/server/tidewall-server: a wrapper that turns SIGTERM and SIGINT
into a clean stop through the server's new --stop-file, since the engine
can't catch signals. The server tells everyone it shut down, logs its
statistics and exits 0; the wrapper kills it only after 15 s.
- --config=PATH reads options from a file (packaging/server/server.cfg);
options after it on the command line win.
- --health-port answers GET /health with JSON (phase, players, tick, uptime)
on loopback by default. HealthServer is non-blocking and bounded.
- packaging/docker: a debian:13 image that checks the engine against
tools/engine.sha512, imports the game at build time, runs as an unprivileged
user, and works with a read-only root and a small /tmp. compose.yaml is
project and container tidewall-server, read-only, no capabilities.
- packaging/systemd: a hardened unit with a dynamic user.
- make server-package-check: compose up, healthy, non-root, read-only, a real
headless client joins over UDP into the drop, then SIGTERM stops it cleanly.
This also covers H7 (graceful shutdown on SIGTERM).
Closes#38Closes#70
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>