H3 Balance pass from bot match batches #56

Closed
opened 2026-11-09 05:29:41 +00:00 by claude · 3 comments
Owner

Depends on: M5, B8

Run batches of bot matches, report on match length, weapon win rates, cause of death and tide deaths, and tune the data table.

Acceptance

  • make balance produces a report across at least 200 matches.
  • Median match length is 12 to 18 minutes, and no weapon dominates beyond the limit set in the GDD.
  • Findings and the changes they led to are in docs/BALANCE.md.
**Depends on:** M5, B8 Run batches of bot matches, report on match length, weapon win rates, cause of death and tide deaths, and tune the data table. **Acceptance** - `make balance` produces a report across at least 200 matches. - Median match length is 12 to 18 minutes, and no weapon dominates beyond the limit set in the GDD. - Findings and the changes they led to are in `docs/BALANCE.md`.
claude added this to the v0.9 Hardening milestone 2026-11-09 05:29:41 +00:00
claude added the priority/hightype/chore labels 2026-11-09 05:29:41 +00:00
claude added this to the Tidewall v1.0 project 2026-11-09 15:00:09 +00:00
Author
Owner

Data for this pass, from M8's make match runs (one scripted player and 23 mixed bots, 10 seeds): matches ended between 3:18 and 9:16, and the tide's schedule is 18 minutes. On seed 2 the sea had only reached 18 m (phase 3 of 7) at the end, and the last three players wandered for three minutes without meeting. So the tide rarely decides a bot match. Things to look at here: time to kill and ammunition, how fast bots find each other, and whether the early holds (150 s and 100 s) are too long.

make match saves a command log, and MatchReplay can stop a recorded match at any tick, which may help when looking at a particular match.

Data for this pass, from M8's `make match` runs (one scripted player and 23 mixed bots, 10 seeds): matches ended between 3:18 and 9:16, and the tide's schedule is 18 minutes. On seed 2 the sea had only reached 18 m (phase 3 of 7) at the end, and the last three players wandered for three minutes without meeting. So the tide rarely decides a bot match. Things to look at here: time to kill and ammunition, how fast bots find each other, and whether the early holds (150 s and 100 s) are too long. `make match` saves a command log, and `MatchReplay` can stop a recorded match at any tick, which may help when looking at a particular match.
Author
Owner

From #76: over three 20-match batches (seeds 1 to 20, 21 to 40, 101 to 120), hard bots are not reliably better than normal ones in whole matches. Wins 30 against 24 and eliminations 507 against 470 in total, but normal led on both in one batch of the three, and hard's mean place was worse than normal's in all three (12.93, 12.63, 12.61 against 11.67, 12.18, 12.36). A guess, not traced: hard bots pick more fights and fall earlier. Easy is clearly worst by wins (6 of 60) and eliminations. The balance pass should decide whether hard needs more than aim and reaction time, for example better judgement about when to disengage.

From #76: over three 20-match batches (seeds 1 to 20, 21 to 40, 101 to 120), hard bots are not reliably better than normal ones in whole matches. Wins 30 against 24 and eliminations 507 against 470 in total, but normal led on both in one batch of the three, and hard's mean place was worse than normal's in all three (12.93, 12.63, 12.61 against 11.67, 12.18, 12.36). A guess, not traced: hard bots pick more fights and fall earlier. Easy is clearly worst by wins (6 of 60) and eliminations. The balance pass should decide whether hard needs more than aim and reaction time, for example better judgement about when to disengage.
eric closed this issue 2026-11-14 11:28:20 +00:00
Author
Owner

Done in 3881212. The full write-up is in docs/BALANCE.md.

make balance (tools/balance.sh) plays 200 bot-only matches (seeds 1001–1200) in four processes under nice, about 15 minutes on this host, and writes build/balance/report.txt. The report covers match-length quantiles, players standing every two minutes, causes of death and the sea's share, eliminations by weapon (boathook included) and wins, eliminations and mean place by difficulty. It fails if there are fewer than 200 matches, a match has no winner, the median is outside 12–18 min, or one weapon has more than 35% of weapon eliminations. The targets are BALANCE_* in Tuning, and the GDD now has a Balance section that sets the 35% limit.

Result: passes.

Before (30 matches) After (200 matches)
Median match 5.0 min 13.1 min (p10 9.2, p90 14.6, max 16.4)
Standing at 2 / 6 / 10 min 3.4 / 0.7 / 0.0 6.4 / 3.6 / 2.4
Top weapon boathook 41% rivet repeater 31.7%
Falls to the sea 0.4% 2.1%
Wins easy / normal / hard 2 / 14 / 14 10 / 42 / 148

What I found by tracing single matches, and the change each one led to:

  • Bug: bots leapt from the barge on a fixed timer and came down hundreds of metres short of their landing spot. NavBot._ride now waits until the spot is in gliding reach, or until the barge starts drawing away from it.
  • Landing brawls: unarmed bots brawled anyone within 6 m. Now they only swing back at whoever hit them. Bots also spread out their landing spots.
  • Bots hunted the whole island:
    • They pick out new opponents only within 30 m (BOT_ENGAGE_METRES), but answer anyone who shoots them.
    • With nothing to do they roam within 60 m and hold ground for 45 s. This was the biggest single change to match length.
  • No disengaging:
    • An unarmed bot now runs from a gun it can see.
    • A hurt bot with something to heal with breaks off at retreat health: normal 35, hard 50, easy never. This also settles the hard-against-normal question from #76.
  • Aim and weapons: aim error widened. Rivet repeater damage cut from 17 to 15.
  • Tide:
    • Lower early levels: 6/14/24/34/46/58/75 m.
    • Longer holds: 150/130/120/110/90/60/45 s.
    • Every match now ends within 17.75 min.
  • Tried and dropped: doubling LOOT_MIN_POI_WEAPONS. It made no clear difference; the leap bug was the real cause.

Tests:

  • make check: 657 passed.
  • make bot-matches: ok. 20/20 matches had a winner, 2.6% drowned, easy won 3 of 20.
  • make match on seed 29: ok, the scripted player harvested 45 and built 2.
  • New tests cover BalanceReport, retaliation-only brawling, the engage range, running from a gun, retreating by difficulty, holding ground and the tide's total length.
  • make match and the scripted-player test move from seed 2 to seed 29. On seed 2 the scripted player now dies in a landing fight.
  • Tests that relied on old bot luck now set up what they check. Idle bots are pinned in place, the offline-client test brings the player down directly, and the replication test keeps the scripted player alive until it has built and breaks one prop itself.

Still open (noted in BALANCE.md):

  • The landing is still bloody: about 18 of 24 bots fall in the first two minutes.
  • The whaler's longgun and harpoon gun rarely get the eliminations.
  • Hard bots now win three quarters of matches.
  • Bot-only matches aren't people, so R6 is the real check. I'll add the looting follow-up to #74.

— tidewall-build

Done in 3881212. The full write-up is in `docs/BALANCE.md`. **`make balance`** (`tools/balance.sh`) plays 200 bot-only matches (seeds 1001–1200) in four processes under `nice`, about 15 minutes on this host, and writes `build/balance/report.txt`. The report covers match-length quantiles, players standing every two minutes, causes of death and the sea's share, eliminations by weapon (boathook included) and wins, eliminations and mean place by difficulty. It fails if there are fewer than 200 matches, a match has no winner, the median is outside 12–18 min, or one weapon has more than 35% of weapon eliminations. The targets are `BALANCE_*` in Tuning, and the GDD now has a Balance section that sets the 35% limit. **Result: passes.** | | Before (30 matches) | After (200 matches) | |---|---|---| | Median match | 5.0 min | **13.1 min** (p10 9.2, p90 14.6, max 16.4) | | Standing at 2 / 6 / 10 min | 3.4 / 0.7 / 0.0 | 6.4 / 3.6 / 2.4 | | Top weapon | boathook 41% | rivet repeater 31.7% | | Falls to the sea | 0.4% | 2.1% | | Wins easy / normal / hard | 2 / 14 / 14 | 10 / 42 / 148 | **What I found by tracing single matches, and the change each one led to:** - **Bug:** bots leapt from the barge on a fixed timer and came down hundreds of metres short of their landing spot. `NavBot._ride` now waits until the spot is in gliding reach, or until the barge starts drawing away from it. - **Landing brawls:** unarmed bots brawled anyone within 6 m. Now they only swing back at whoever hit them. Bots also spread out their landing spots. - **Bots hunted the whole island:** - They pick out new opponents only within 30 m (`BOT_ENGAGE_METRES`), but answer anyone who shoots them. - With nothing to do they roam within 60 m and hold ground for 45 s. This was the biggest single change to match length. - **No disengaging:** - An unarmed bot now runs from a gun it can see. - A hurt bot with something to heal with breaks off at `retreat` health: normal 35, hard 50, easy never. This also settles the hard-against-normal question from #76. - **Aim and weapons:** aim error widened. Rivet repeater damage cut from 17 to 15. - **Tide:** - Lower early levels: 6/14/24/34/46/58/75 m. - Longer holds: 150/130/120/110/90/60/45 s. - Every match now ends within 17.75 min. - **Tried and dropped:** doubling `LOOT_MIN_POI_WEAPONS`. It made no clear difference; the leap bug was the real cause. **Tests:** - `make check`: 657 passed. - `make bot-matches`: ok. 20/20 matches had a winner, 2.6% drowned, easy won 3 of 20. - `make match` on seed 29: ok, the scripted player harvested 45 and built 2. - New tests cover `BalanceReport`, retaliation-only brawling, the engage range, running from a gun, retreating by difficulty, holding ground and the tide's total length. - `make match` and the scripted-player test move from seed 2 to seed 29. On seed 2 the scripted player now dies in a landing fight. - Tests that relied on old bot luck now set up what they check. Idle bots are pinned in place, the offline-client test brings the player down directly, and the replication test keeps the scripted player alive until it has built and breaks one prop itself. **Still open (noted in BALANCE.md):** - The landing is still bloody: about 18 of 24 bots fall in the first two minutes. - The whaler's longgun and harpoon gun rarely get the eliminations. - Hard bots now win three quarters of matches. - Bot-only matches aren't people, so R6 is the real check. I'll add the looting follow-up to #74. — tidewall-build
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: claude/tidewall#56