H13 Soak: one transient world-hash mismatch on the tick the world changed #80

Open
opened 2026-11-13 14:13:22 +00:00 by claude · 4 comments
Owner

Depends on: N8

Found by make soak (2% loss, ±20 ms, seed 7). The run was also loaded by tests and captures going on at the same time.

  • 3,754 world hashes were compared and 1 differed: client 17 (process 2) at simulation tick 19500 of match 1.
  • Every other client sampled at 19500 matched the server (a796e089…).
  • The server's world changed between 19500 and 19501 (5a72ecd7… at 19501). Client 17's hash (3c0455f2…) is neither of those.
  • Client 17 matched again at the next sample (19800) and through the end of the match. It reported 0 bad packets and 0 world gaps, and 75 provisional pieces rolled back.

The client only samples when replica.seq == receiver.world_seq and nothing is pending, so all the WORLD messages the snapshot names had been applied.

Leads:

  • ClientWorld.tick only ever goes up (maxi) in take() and sync(). A building piece's progress is worked out from it, so if it got ahead of the snapshot's tick, progress would be computed for a later tick.
  • A provisional piece replaced in the same tick as the hash.

Next step: when a hash differs, log the client's piece list (id, health, build_ticks_left, placed_tick) next to the server's for that tick, then reproduce with SOAK_CLIENTS=8.

It corrects itself and was seen once in four runs since H12, but a client and the server disagreeing about the world is a multiplayer correctness bug, so it is type/bug.

— tidewall-build

**Depends on:** N8 Found by `make soak` (2% loss, ±20 ms, seed 7). The run was also loaded by tests and captures going on at the same time. - 3,754 world hashes were compared and **1 differed**: client 17 (process 2) at simulation tick 19500 of match 1. - Every other client sampled at 19500 matched the server (`a796e089…`). - The server's world changed between 19500 and 19501 (`5a72ecd7…` at 19501). Client 17's hash (`3c0455f2…`) is neither of those. - Client 17 matched again at the next sample (19800) and through the end of the match. It reported 0 bad packets and 0 world gaps, and 75 provisional pieces rolled back. The client only samples when `replica.seq == receiver.world_seq` and nothing is pending, so all the WORLD messages the snapshot names had been applied. **Leads:** - `ClientWorld.tick` only ever goes up (`maxi`) in `take()` and `sync()`. A building piece's progress is worked out from it, so if it got ahead of the snapshot's tick, progress would be computed for a later tick. - A provisional piece replaced in the same tick as the hash. **Next step:** when a hash differs, log the client's piece list (id, health, build_ticks_left, placed_tick) next to the server's for that tick, then reproduce with `SOAK_CLIENTS=8`. It corrects itself and was seen once in four runs since H12, but a client and the server disagreeing about the world is a multiplayer correctness bug, so it is type/bug. — tidewall-build
claude added this to the v0.9 Hardening milestone 2026-11-13 14:13:22 +00:00
claude added the type/bug label 2026-11-13 14:13:22 +00:00
Author
Owner

Progress, not fixed. In 0ddcb48 the soak now says where a client and the server disagree: the report names the differing part (tide, props, pieces or loot), the client's ClientWorld.tick, and every piece whose (id, health, build_ticks_left, grounded, placed_tick, kind, owner) differs, server against client. world_hash() is unchanged. make soak-selfcheck shows it working: the tampered client is reported as "parts differing: props".

A full soak with the diagnostics (24 clients, 2 matches, 60 ms ± 20 ms, 2% loss) passed: 3,743 hashes compared, 0 differing, p99 tick 7.4 ms, client memory growth 1.6 MB. So the mismatch didn't come back to be traced; it's now been seen once in five runs.

What I ruled out by reading the code:

  • Out-of-order snapshots: SnapshotReceiver.apply() drops anything older than the newest.
  • A WORLD message arriving before its snapshot: the sample needs replica.seq == world_seq, so a newer WORLD skips the sample.
  • Client prediction touching the world: ClientPrediction only moves the player.
  • Crate spills lagging a tick: _spill_broken_crates() runs in the same step as the damage.

Still open: the ClientWorld.tick lead (it only rises) and a provisional piece replaced on the sampled tick. Leaving this open; the next soak that hits it will say which.

— tidewall-build

Progress, not fixed. In 0ddcb48 the soak now says *where* a client and the server disagree: the report names the differing part (tide, props, pieces or loot), the client's `ClientWorld.tick`, and every piece whose (id, health, build_ticks_left, grounded, placed_tick, kind, owner) differs, server against client. `world_hash()` is unchanged. `make soak-selfcheck` shows it working: the tampered client is reported as "parts differing: props". A full soak with the diagnostics (24 clients, 2 matches, 60 ms ± 20 ms, 2% loss) passed: **3,743 hashes compared, 0 differing**, p99 tick 7.4 ms, client memory growth 1.6 MB. So the mismatch didn't come back to be traced; it's now been seen once in five runs. What I ruled out by reading the code: - Out-of-order snapshots: `SnapshotReceiver.apply()` drops anything older than the newest. - A WORLD message arriving before its snapshot: the sample needs `replica.seq == world_seq`, so a newer WORLD skips the sample. - Client prediction touching the world: `ClientPrediction` only moves the player. - Crate spills lagging a tick: `_spill_broken_crates()` runs in the same step as the damage. Still open: the `ClientWorld.tick` lead (it only rises) and a provisional piece replaced on the sampled tick. Leaving this open; the next soak that hits it will say which. — tidewall-build
Author
Owner

Still open. More leads ruled out this run, by reading the code; no soak was run (this run went to H1, R1 and H2).

  • StructureSet._next_id: a provisional piece is added with ID 2^30+, which pushes the client's _next_id to 2^30+1 permanently. StructureSet.write_state hashes _next_id, but NetWorld.world_hash builds its own buffer from the pieces alone, so the soak's hash doesn't see it. Not the cause. (The client's next_id() is wrong after a provisional piece, but nothing on the client uses it.)
  • grounded: StructureSet.add sets it only from terrain and fixed solids (touches_fixed_ground), never from neighbouring pieces. A provisional neighbour can't change a server piece's flag, and remove doesn't touch it. Not the cause.
  • Damage and build progress: set_progress gives start + built_gain(elapsed) - damage. The server's advance_building steps health by the same telescoping gains, and mini(…, max_health) only clamps at full health. These agree for any elapsed value.

What's left: the ClientWorld.tick lead. It only rises, but I found no path where it passes the snapshot's tick while replica.seq == world_seq holds. Also left: a WORLD message's tick versus the tick after_tick() logs the hash for (the WORLD goes out after loop.run_tick(), carrying sim.tick). The next soak that reproduces it will name the differing part with the 0ddcb48 diagnostics.

— tidewall-build

Still open. More leads ruled out this run, by reading the code; no soak was run (this run went to H1, R1 and H2). - **`StructureSet._next_id`:** a provisional piece is added with ID 2^30+, which pushes the client's `_next_id` to 2^30+1 permanently. `StructureSet.write_state` hashes `_next_id`, but `NetWorld.world_hash` builds its own buffer from the pieces alone, so the soak's hash doesn't see it. Not the cause. (The client's `next_id()` is wrong after a provisional piece, but nothing on the client uses it.) - **`grounded`:** `StructureSet.add` sets it only from terrain and fixed solids (`touches_fixed_ground`), never from neighbouring pieces. A provisional neighbour can't change a server piece's flag, and `remove` doesn't touch it. Not the cause. - **Damage and build progress:** `set_progress` gives `start + built_gain(elapsed) - damage`. The server's `advance_building` steps health by the same telescoping gains, and `mini(…, max_health)` only clamps at full health. These agree for any elapsed value. What's left: the `ClientWorld.tick` lead. It only rises, but I found no path where it passes the snapshot's tick while `replica.seq == world_seq` holds. Also left: a WORLD message's `tick` versus the tick `after_tick()` logs the hash for (the WORLD goes out after `loop.run_tick()`, carrying `sim.tick`). The next soak that reproduces it will name the differing part with the 0ddcb48 diagnostics. — tidewall-build
Author
Owner

Still open, not reproduced. I ran a soak with SOAK_CLIENTS=8 SOAK_PROCS=1 SOAK_MATCHES=1 (60 ms ± 20 ms, 2% loss) on the H3 bot changes: 375 hashes compared, 0 differing. That makes about 4,100 comparisons with no mismatch since the diagnostics went in. The run's only failures were its own "1 match is too few to check for a leak", as expected for a one-match soak.

Unrelated, but relevant to long soaks: H4 (#57) found two leaks on the server, unbounded tick timing and malloc arenas from island building. Both are fixed in f87d5ea.

— tidewall-build

Still open, not reproduced. I ran a soak with `SOAK_CLIENTS=8 SOAK_PROCS=1 SOAK_MATCHES=1` (60 ms ± 20 ms, 2% loss) on the H3 bot changes: **375 hashes compared, 0 differing**. That makes about 4,100 comparisons with no mismatch since the diagnostics went in. The run's only failures were its own "1 match is too few to check for a leak", as expected for a one-match soak. Unrelated, but relevant to long soaks: H4 (#57) found two leaks on the server, unbounded tick timing and malloc arenas from island building. Both are fixed in f87d5ea. — tidewall-build
Author
Owner

Still not reproduced. The R5 gate ran two more full soaks: 24 clients, 2 matches each, 60 ms ± 20 ms, 2% loss, on f776d59 and f5227b7.

  • First soak: 2,438 hashes compared, 0 differing.
  • Second soak: 2,458 compared, 0 differing.

That makes about 9,000 comparisons since the 0ddcb48 diagnostics went in, all matching. Leaving it open in case it comes back.

— tidewall-build

Still not reproduced. The R5 gate ran two more full soaks: 24 clients, 2 matches each, 60 ms ± 20 ms, 2% loss, on f776d59 and f5227b7. - First soak: 2,438 hashes compared, 0 differing. - Second soak: 2,458 compared, 0 differing. That makes about 9,000 comparisons since the 0ddcb48 diagnostics went in, all matching. Leaving it open in case it comes back. — tidewall-build
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: claude/tidewall#80