H4: Long-run stability (make stability), and two memory leaks fixed

make stability (tools/stability.sh, StabilityReport) runs one dedicated
server for 50 bot matches back to back, unthrottled, sampling its resident
memory, and fails on a crash, hang, engine or script error, a match with
no winner, or memory that grows after the first five matches.

The first run had no crash, hang or error, but showed two leaks:
- ServerLoop kept every tick's duration in an array (8 bytes a tick,
  about 21 MB a day at 30 Hz) and copied and sorted all of it for every
  progress line. TickTimes keeps exact statistics in fixed memory: a
  one-microsecond histogram up to 65 ms, plus the rare slower ticks.
- Resident memory grew about 28 MB a match (905 MB after 42 matches):
  glibc gives each thread its own malloc arena, and every next island is
  built on a worker thread. The server launchers (packaging/server,
  packaging/dist, so also the .deb, Docker image and systemd unit) and
  tools/godot.sh for --server now set MALLOC_ARENA_MAX=1 unless already
  set. Tick times were unchanged (p50 501 vs 507 us, p99 1635 vs 1625).

After: 50 matches (seed 4242) in 756 s, all with a winner, exit 0, no
errors; static memory +1.6 KiB and resident +145 KiB a match, flat at
204 MB from match 26 on. No match crashed, so there is no crash seed to
turn into a regression test.

Tests: TickTimes against a sorted list, its fixed memory, the launchers'
setting, and StabilityReport. make check, dist-check and
server-package-check pass.

Closes #57

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-14 07:15:23 -05:00
co-authored by Claude Opus 5.5
parent 38812120df
commit f87d5eae0d
19 changed files with 486 additions and 34 deletions
+8
View File
@@ -60,6 +60,14 @@ Stopping: send SIGTERM to the wrapper (Docker, Compose and systemd all do). It c
the server's stop file, the server disconnects everyone with a "shut down" reason, logs
its statistics and exits 0. If it hasn't gone within 15 seconds the wrapper kills it.
## Memory
The launchers set `MALLOC_ARENA_MAX=1` unless it is already set. The server builds each
next island on a worker thread, and with glibc's default of an arena per thread its
resident memory grew about 28 MB a match for as long as it ran. With one arena it stays
flat, about 200 MB with 24 bots, at no measured cost to tick time. `make stability`
checks this over 50 matches.
## Ports
| Port | Protocol | What |
+5
View File
@@ -17,6 +17,11 @@ state="${TIDEWALL_STATE:-${XDG_STATE_HOME:-$HOME/.local/state}/tidewall-server}"
grace="${TIDEWALL_STOP_SECONDS:-15}"
stop_file="$state/stop"
# glibc gives each thread that allocates its own malloc arena and rarely hands the
# memory back. The server builds every next island on a worker thread, so with the
# default arenas its resident memory grew about 28 MB a match, forever (H4, #57).
# One arena keeps it flat at no measured cost to tick time.
export MALLOC_ARENA_MAX="${MALLOC_ARENA_MAX:-1}"
mkdir -p "$state/data" "$state/config" "$state/cache"
rm -f "$stop_file"
export XDG_DATA_HOME="$state/data" XDG_CONFIG_HOME="$state/config" XDG_CACHE_HOME="$state/cache"
+5
View File
@@ -22,6 +22,11 @@ state="${TIDEWALL_STATE:-${RUNTIME_DIRECTORY:-/tmp/tidewall-server}}"
grace="${TIDEWALL_STOP_SECONDS:-15}"
stop_file="$state/stop"
# glibc gives each thread that allocates its own malloc arena and rarely hands the
# memory back. The server builds every next island on a worker thread, so with the
# default arenas its resident memory grew about 28 MB a match, forever (H4, #57).
# One arena keeps it flat at no measured cost to tick time.
export MALLOC_ARENA_MAX="${MALLOC_ARENA_MAX:-1}"
mkdir -p "$state/data" "$state/config" "$state/cache"
rm -f "$stop_file"
# The engine writes its user data and caches here, so the install can stay read-only.