M9 make bot-matches fails on its default seeds #76

Closed
opened 2026-11-12 10:42:48 +00:00 by claude · 1 comment
Owner

Depends on: M5

make bot-matches (seeds 1 to 20, its default) fails its difficulty checks. M5 was verified on seeds 101 to 120, and through the make target only on 2 seeds, so the default run was never seen to pass.

Measured 2026-11-12, 20 matches each, every match finished with a winner and under 1% of deaths were drownings:

Commit Mean place easy / normal / hard Eliminations easy / normal / hard Wins Verdict
8cf164d (before M8) 12.74 / 12.98 / 11.78 98 / 195 / 167 not recorded fails both checks
1d6a24a (M8, per-bot random streams) 12.89 / 11.67 / 12.93 118 / 160 / 178 easy 0, normal 9, hard 11 fails "easy places worst" by 0.04

So this isn't caused by M8; the checks are too noisy for 20 matches. With 8 bots a level and 20 matches, the standard error of a level's mean place is about 0.5, and the levels differ by about 1. Easy bots clearly do worst by wins and eliminations in both runs; hard and normal aren't reliably separated by anything measured.

Acceptance

  • The difficulty check uses measures that hold on at least three different 20-seed runs (for example wins and eliminations for easy against the rest, and the duel test for hard against easy), or the batch is large enough for the measure used.
  • make bot-matches passes on its default seeds, and the numbers are recorded here.
  • If hard doesn't beat normal by any measure in whole matches, say so and pass it to H3 (#56) rather than tuning the check until it passes.
**Depends on:** M5 `make bot-matches` (seeds 1 to 20, its default) fails its difficulty checks. M5 was verified on seeds 101 to 120, and through the make target only on 2 seeds, so the default run was never seen to pass. Measured 2026-11-12, 20 matches each, every match finished with a winner and under 1% of deaths were drownings: | Commit | Mean place easy / normal / hard | Eliminations easy / normal / hard | Wins | Verdict | |---|---|---|---|---| | 8cf164d (before M8) | 12.74 / 12.98 / 11.78 | 98 / 195 / 167 | not recorded | fails both checks | | 1d6a24a (M8, per-bot random streams) | 12.89 / 11.67 / 12.93 | 118 / 160 / 178 | easy 0, normal 9, hard 11 | fails "easy places worst" by 0.04 | So this isn't caused by M8; the checks are too noisy for 20 matches. With 8 bots a level and 20 matches, the standard error of a level's mean place is about 0.5, and the levels differ by about 1. Easy bots clearly do worst by wins and eliminations in both runs; hard and normal aren't reliably separated by anything measured. **Acceptance** - The difficulty check uses measures that hold on at least three different 20-seed runs (for example wins and eliminations for easy against the rest, and the duel test for hard against easy), or the batch is large enough for the measure used. - `make bot-matches` passes on its default seeds, and the numbers are recorded here. - If hard doesn't beat normal by any measure in whole matches, say so and pass it to H3 (#56) rather than tuning the check until it passes.
claude added this to the v0.4 Match Loop and Bots milestone 2026-11-12 10:42:48 +00:00
claude added the priority/hightype/bug labels 2026-11-12 10:42:48 +00:00
eric closed this issue 2026-11-12 13:26:07 +00:00
Author
Owner

Fixed in 881f449.

Measured on the current code (per-bot random streams), three batches of 20 matches. Every match finished with a winner, none by the time limit.

Seeds Wins easy / normal / hard Eliminations easy / normal / hard Mean place easy / normal / hard Drowned
1 to 20 0 / 9 / 11 118 / 160 / 178 12.89 / 11.67 / 12.93 0.9%
21 to 40 4 / 9 / 7 140 / 167 / 149 12.69 / 12.18 / 12.63 1.1%
101 to 120 2 / 6 / 12 136 / 143 / 180 12.53 / 12.36 / 12.61 0.2%

What holds on all three (and on the two earlier batches in this issue's table, for eliminations): easy bots win less than an even share (6.7 of 20) and take the fewest eliminations. So that is what make bot-matches now checks. With fewer than 20 matches the difficulty checks are skipped and the output says so, since they mean nothing at that size.

What doesn't hold: mean place doesn't separate the levels at all, and hard is not reliably ahead of normal by any whole-match measure. Hard took fewer eliminations than normal on seeds 21 to 40 and fewer wins too, and its mean place was worse than normal's in all three batches. Hard against easy is still covered by the duel test (test_harder_bots_win_most_duels_against_easier_ones). Passed to #56 (H3) rather than tuned away.

Tested: make bot-matches on its default seeds passes (exit 0, numbers as the first row). make check passes, 384 tests.

Honest margin: the wins check is safe (easy won 6 of 60). The eliminations check held five times out of five but by as little as 140 against 149, so it could fail on some other seed range; the default seeds are deterministic and pass.

Fixed in 881f449. **Measured** on the current code (per-bot random streams), three batches of 20 matches. Every match finished with a winner, none by the time limit. | Seeds | Wins easy / normal / hard | Eliminations easy / normal / hard | Mean place easy / normal / hard | Drowned | |---|---|---|---|---| | 1 to 20 | 0 / 9 / 11 | 118 / 160 / 178 | 12.89 / 11.67 / 12.93 | 0.9% | | 21 to 40 | 4 / 9 / 7 | 140 / 167 / 149 | 12.69 / 12.18 / 12.63 | 1.1% | | 101 to 120 | 2 / 6 / 12 | 136 / 143 / 180 | 12.53 / 12.36 / 12.61 | 0.2% | **What holds on all three** (and on the two earlier batches in this issue's table, for eliminations): easy bots win less than an even share (6.7 of 20) and take the fewest eliminations. So that is what `make bot-matches` now checks. With fewer than 20 matches the difficulty checks are skipped and the output says so, since they mean nothing at that size. **What doesn't hold:** mean place doesn't separate the levels at all, and hard is not reliably ahead of normal by any whole-match measure. Hard took fewer eliminations than normal on seeds 21 to 40 and fewer wins too, and its mean place was worse than normal's in all three batches. Hard against easy is still covered by the duel test (`test_harder_bots_win_most_duels_against_easier_ones`). Passed to #56 (H3) rather than tuned away. **Tested:** `make bot-matches` on its default seeds passes (exit 0, numbers as the first row). `make check` passes, 384 tests. **Honest margin:** the wins check is safe (easy won 6 of 60). The eliminations check held five times out of five but by as little as 140 against 149, so it could fail on some other seed range; the default seeds are deterministic and pass.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: claude/tidewall#76