make bot-matches (seeds 1 to 20, its default) fails its difficulty checks. M5 was verified on seeds 101 to 120, and through the make target only on 2 seeds, so the default run was never seen to pass.
Measured 2026-11-12, 20 matches each, every match finished with a winner and under 1% of deaths were drownings:
So this isn't caused by M8; the checks are too noisy for 20 matches. With 8 bots a level and 20 matches, the standard error of a level's mean place is about 0.5, and the levels differ by about 1. Easy bots clearly do worst by wins and eliminations in both runs; hard and normal aren't reliably separated by anything measured.
Acceptance
The difficulty check uses measures that hold on at least three different 20-seed runs (for example wins and eliminations for easy against the rest, and the duel test for hard against easy), or the batch is large enough for the measure used.
make bot-matches passes on its default seeds, and the numbers are recorded here.
If hard doesn't beat normal by any measure in whole matches, say so and pass it to H3 (#56) rather than tuning the check until it passes.
**Depends on:** M5
`make bot-matches` (seeds 1 to 20, its default) fails its difficulty checks. M5 was verified on seeds 101 to 120, and through the make target only on 2 seeds, so the default run was never seen to pass.
Measured 2026-11-12, 20 matches each, every match finished with a winner and under 1% of deaths were drownings:
| Commit | Mean place easy / normal / hard | Eliminations easy / normal / hard | Wins | Verdict |
|---|---|---|---|---|
| 8cf164d (before M8) | 12.74 / 12.98 / 11.78 | 98 / 195 / 167 | not recorded | fails both checks |
| 1d6a24a (M8, per-bot random streams) | 12.89 / 11.67 / 12.93 | 118 / 160 / 178 | easy 0, normal 9, hard 11 | fails "easy places worst" by 0.04 |
So this isn't caused by M8; the checks are too noisy for 20 matches. With 8 bots a level and 20 matches, the standard error of a level's mean place is about 0.5, and the levels differ by about 1. Easy bots clearly do worst by wins and eliminations in both runs; hard and normal aren't reliably separated by anything measured.
**Acceptance**
- The difficulty check uses measures that hold on at least three different 20-seed runs (for example wins and eliminations for easy against the rest, and the duel test for hard against easy), or the batch is large enough for the measure used.
- `make bot-matches` passes on its default seeds, and the numbers are recorded here.
- If hard doesn't beat normal by any measure in whole matches, say so and pass it to H3 (#56) rather than tuning the check until it passes.
claude
added this to the v0.4 Match Loop and Bots milestone 2026-11-12 10:42:48 +00:00
Measured on the current code (per-bot random streams), three batches of 20 matches. Every match finished with a winner, none by the time limit.
Seeds
Wins easy / normal / hard
Eliminations easy / normal / hard
Mean place easy / normal / hard
Drowned
1 to 20
0 / 9 / 11
118 / 160 / 178
12.89 / 11.67 / 12.93
0.9%
21 to 40
4 / 9 / 7
140 / 167 / 149
12.69 / 12.18 / 12.63
1.1%
101 to 120
2 / 6 / 12
136 / 143 / 180
12.53 / 12.36 / 12.61
0.2%
What holds on all three (and on the two earlier batches in this issue's table, for eliminations): easy bots win less than an even share (6.7 of 20) and take the fewest eliminations. So that is what make bot-matches now checks. With fewer than 20 matches the difficulty checks are skipped and the output says so, since they mean nothing at that size.
What doesn't hold: mean place doesn't separate the levels at all, and hard is not reliably ahead of normal by any whole-match measure. Hard took fewer eliminations than normal on seeds 21 to 40 and fewer wins too, and its mean place was worse than normal's in all three batches. Hard against easy is still covered by the duel test (test_harder_bots_win_most_duels_against_easier_ones). Passed to #56 (H3) rather than tuned away.
Tested:make bot-matches on its default seeds passes (exit 0, numbers as the first row). make check passes, 384 tests.
Honest margin: the wins check is safe (easy won 6 of 60). The eliminations check held five times out of five but by as little as 140 against 149, so it could fail on some other seed range; the default seeds are deterministic and pass.
Fixed in 881f449.
**Measured** on the current code (per-bot random streams), three batches of 20 matches. Every match finished with a winner, none by the time limit.
| Seeds | Wins easy / normal / hard | Eliminations easy / normal / hard | Mean place easy / normal / hard | Drowned |
|---|---|---|---|---|
| 1 to 20 | 0 / 9 / 11 | 118 / 160 / 178 | 12.89 / 11.67 / 12.93 | 0.9% |
| 21 to 40 | 4 / 9 / 7 | 140 / 167 / 149 | 12.69 / 12.18 / 12.63 | 1.1% |
| 101 to 120 | 2 / 6 / 12 | 136 / 143 / 180 | 12.53 / 12.36 / 12.61 | 0.2% |
**What holds on all three** (and on the two earlier batches in this issue's table, for eliminations): easy bots win less than an even share (6.7 of 20) and take the fewest eliminations. So that is what `make bot-matches` now checks. With fewer than 20 matches the difficulty checks are skipped and the output says so, since they mean nothing at that size.
**What doesn't hold:** mean place doesn't separate the levels at all, and hard is not reliably ahead of normal by any whole-match measure. Hard took fewer eliminations than normal on seeds 21 to 40 and fewer wins too, and its mean place was worse than normal's in all three batches. Hard against easy is still covered by the duel test (`test_harder_bots_win_most_duels_against_easier_ones`). Passed to #56 (H3) rather than tuned away.
**Tested:** `make bot-matches` on its default seeds passes (exit 0, numbers as the first row). `make check` passes, 384 tests.
**Honest margin:** the wins check is safe (easy won 6 of 60). The eliminations check held five times out of five but by as little as 140 against 149, so it could fail on some other seed range; the default seeds are deterministic and pass.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Depends on: M5
make bot-matches(seeds 1 to 20, its default) fails its difficulty checks. M5 was verified on seeds 101 to 120, and through the make target only on 2 seeds, so the default run was never seen to pass.Measured 2026-11-12, 20 matches each, every match finished with a winner and under 1% of deaths were drownings:
8cf164d(before M8)1d6a24a(M8, per-bot random streams)So this isn't caused by M8; the checks are too noisy for 20 matches. With 8 bots a level and 20 matches, the standard error of a level's mean place is about 0.5, and the levels differ by about 1. Easy bots clearly do worst by wins and eliminations in both runs; hard and normal aren't reliably separated by anything measured.
Acceptance
make bot-matchespasses on its default seeds, and the numbers are recorded here.Fixed in
881f449.Measured on the current code (per-bot random streams), three batches of 20 matches. Every match finished with a winner, none by the time limit.
What holds on all three (and on the two earlier batches in this issue's table, for eliminations): easy bots win less than an even share (6.7 of 20) and take the fewest eliminations. So that is what
make bot-matchesnow checks. With fewer than 20 matches the difficulty checks are skipped and the output says so, since they mean nothing at that size.What doesn't hold: mean place doesn't separate the levels at all, and hard is not reliably ahead of normal by any whole-match measure. Hard took fewer eliminations than normal on seeds 21 to 40 and fewer wins too, and its mean place was worse than normal's in all three batches. Hard against easy is still covered by the duel test (
test_harder_bots_win_most_duels_against_easier_ones). Passed to #56 (H3) rather than tuned away.Tested:
make bot-matcheson its default seeds passes (exit 0, numbers as the first row).make checkpasses, 384 tests.Honest margin: the wins check is safe (easy won 6 of 60). The eliminations check held five times out of five but by as little as 140 against 149, so it could fail on some other seed range; the default seeds are deterministic and pass.