Found by make soak (2% loss, ±20 ms, seed 7). The run was also loaded by tests and captures going on at the same time.
3,754 world hashes were compared and 1 differed: client 17 (process 2) at simulation tick 19500 of match 1.
Every other client sampled at 19500 matched the server (a796e089…).
The server's world changed between 19500 and 19501 (5a72ecd7… at 19501). Client 17's hash (3c0455f2…) is neither of those.
Client 17 matched again at the next sample (19800) and through the end of the match. It reported 0 bad packets and 0 world gaps, and 75 provisional pieces rolled back.
The client only samples when replica.seq == receiver.world_seq and nothing is pending, so all the WORLD messages the snapshot names had been applied.
Leads:
ClientWorld.tick only ever goes up (maxi) in take() and sync(). A building piece's progress is worked out from it, so if it got ahead of the snapshot's tick, progress would be computed for a later tick.
A provisional piece replaced in the same tick as the hash.
Next step: when a hash differs, log the client's piece list (id, health, build_ticks_left, placed_tick) next to the server's for that tick, then reproduce with SOAK_CLIENTS=8.
It corrects itself and was seen once in four runs since H12, but a client and the server disagreeing about the world is a multiplayer correctness bug, so it is type/bug.
— tidewall-build
**Depends on:** N8
Found by `make soak` (2% loss, ±20 ms, seed 7). The run was also loaded by tests and captures going on at the same time.
- 3,754 world hashes were compared and **1 differed**: client 17 (process 2) at simulation tick 19500 of match 1.
- Every other client sampled at 19500 matched the server (`a796e089…`).
- The server's world changed between 19500 and 19501 (`5a72ecd7…` at 19501). Client 17's hash (`3c0455f2…`) is neither of those.
- Client 17 matched again at the next sample (19800) and through the end of the match. It reported 0 bad packets and 0 world gaps, and 75 provisional pieces rolled back.
The client only samples when `replica.seq == receiver.world_seq` and nothing is pending, so all the WORLD messages the snapshot names had been applied.
**Leads:**
- `ClientWorld.tick` only ever goes up (`maxi`) in `take()` and `sync()`. A building piece's progress is worked out from it, so if it got ahead of the snapshot's tick, progress would be computed for a later tick.
- A provisional piece replaced in the same tick as the hash.
**Next step:** when a hash differs, log the client's piece list (id, health, build_ticks_left, placed_tick) next to the server's for that tick, then reproduce with `SOAK_CLIENTS=8`.
It corrects itself and was seen once in four runs since H12, but a client and the server disagreeing about the world is a multiplayer correctness bug, so it is type/bug.
— tidewall-build
claude
added this to the v0.9 Hardening milestone 2026-11-13 14:13:22 +00:00
claude
added the type/bug label 2026-11-13 14:13:22 +00:00
Progress, not fixed. In 0ddcb48 the soak now says where a client and the server disagree: the report names the differing part (tide, props, pieces or loot), the client's ClientWorld.tick, and every piece whose (id, health, build_ticks_left, grounded, placed_tick, kind, owner) differs, server against client. world_hash() is unchanged. make soak-selfcheck shows it working: the tampered client is reported as "parts differing: props".
A full soak with the diagnostics (24 clients, 2 matches, 60 ms ± 20 ms, 2% loss) passed: 3,743 hashes compared, 0 differing, p99 tick 7.4 ms, client memory growth 1.6 MB. So the mismatch didn't come back to be traced; it's now been seen once in five runs.
What I ruled out by reading the code:
Out-of-order snapshots: SnapshotReceiver.apply() drops anything older than the newest.
A WORLD message arriving before its snapshot: the sample needs replica.seq == world_seq, so a newer WORLD skips the sample.
Client prediction touching the world: ClientPrediction only moves the player.
Crate spills lagging a tick: _spill_broken_crates() runs in the same step as the damage.
Still open: the ClientWorld.tick lead (it only rises) and a provisional piece replaced on the sampled tick. Leaving this open; the next soak that hits it will say which.
— tidewall-build
Progress, not fixed. In 0ddcb48 the soak now says *where* a client and the server disagree: the report names the differing part (tide, props, pieces or loot), the client's `ClientWorld.tick`, and every piece whose (id, health, build_ticks_left, grounded, placed_tick, kind, owner) differs, server against client. `world_hash()` is unchanged. `make soak-selfcheck` shows it working: the tampered client is reported as "parts differing: props".
A full soak with the diagnostics (24 clients, 2 matches, 60 ms ± 20 ms, 2% loss) passed: **3,743 hashes compared, 0 differing**, p99 tick 7.4 ms, client memory growth 1.6 MB. So the mismatch didn't come back to be traced; it's now been seen once in five runs.
What I ruled out by reading the code:
- Out-of-order snapshots: `SnapshotReceiver.apply()` drops anything older than the newest.
- A WORLD message arriving before its snapshot: the sample needs `replica.seq == world_seq`, so a newer WORLD skips the sample.
- Client prediction touching the world: `ClientPrediction` only moves the player.
- Crate spills lagging a tick: `_spill_broken_crates()` runs in the same step as the damage.
Still open: the `ClientWorld.tick` lead (it only rises) and a provisional piece replaced on the sampled tick. Leaving this open; the next soak that hits it will say which.
— tidewall-build
Still open. More leads ruled out this run, by reading the code; no soak was run (this run went to H1, R1 and H2).
StructureSet._next_id: a provisional piece is added with ID 2^30+, which pushes the client's _next_id to 2^30+1 permanently. StructureSet.write_state hashes _next_id, but NetWorld.world_hash builds its own buffer from the pieces alone, so the soak's hash doesn't see it. Not the cause. (The client's next_id() is wrong after a provisional piece, but nothing on the client uses it.)
grounded:StructureSet.add sets it only from terrain and fixed solids (touches_fixed_ground), never from neighbouring pieces. A provisional neighbour can't change a server piece's flag, and remove doesn't touch it. Not the cause.
Damage and build progress:set_progress gives start + built_gain(elapsed) - damage. The server's advance_building steps health by the same telescoping gains, and mini(…, max_health) only clamps at full health. These agree for any elapsed value.
What's left: the ClientWorld.tick lead. It only rises, but I found no path where it passes the snapshot's tick while replica.seq == world_seq holds. Also left: a WORLD message's tick versus the tick after_tick() logs the hash for (the WORLD goes out after loop.run_tick(), carrying sim.tick). The next soak that reproduces it will name the differing part with the 0ddcb48 diagnostics.
— tidewall-build
Still open. More leads ruled out this run, by reading the code; no soak was run (this run went to H1, R1 and H2).
- **`StructureSet._next_id`:** a provisional piece is added with ID 2^30+, which pushes the client's `_next_id` to 2^30+1 permanently. `StructureSet.write_state` hashes `_next_id`, but `NetWorld.world_hash` builds its own buffer from the pieces alone, so the soak's hash doesn't see it. Not the cause. (The client's `next_id()` is wrong after a provisional piece, but nothing on the client uses it.)
- **`grounded`:** `StructureSet.add` sets it only from terrain and fixed solids (`touches_fixed_ground`), never from neighbouring pieces. A provisional neighbour can't change a server piece's flag, and `remove` doesn't touch it. Not the cause.
- **Damage and build progress:** `set_progress` gives `start + built_gain(elapsed) - damage`. The server's `advance_building` steps health by the same telescoping gains, and `mini(…, max_health)` only clamps at full health. These agree for any elapsed value.
What's left: the `ClientWorld.tick` lead. It only rises, but I found no path where it passes the snapshot's tick while `replica.seq == world_seq` holds. Also left: a WORLD message's `tick` versus the tick `after_tick()` logs the hash for (the WORLD goes out after `loop.run_tick()`, carrying `sim.tick`). The next soak that reproduces it will name the differing part with the 0ddcb48 diagnostics.
— tidewall-build
Still open, not reproduced. I ran a soak with SOAK_CLIENTS=8 SOAK_PROCS=1 SOAK_MATCHES=1 (60 ms ± 20 ms, 2% loss) on the H3 bot changes: 375 hashes compared, 0 differing. That makes about 4,100 comparisons with no mismatch since the diagnostics went in. The run's only failures were its own "1 match is too few to check for a leak", as expected for a one-match soak.
Unrelated, but relevant to long soaks: H4 (#57) found two leaks on the server, unbounded tick timing and malloc arenas from island building. Both are fixed in f87d5ea.
— tidewall-build
Still open, not reproduced. I ran a soak with `SOAK_CLIENTS=8 SOAK_PROCS=1 SOAK_MATCHES=1` (60 ms ± 20 ms, 2% loss) on the H3 bot changes: **375 hashes compared, 0 differing**. That makes about 4,100 comparisons with no mismatch since the diagnostics went in. The run's only failures were its own "1 match is too few to check for a leak", as expected for a one-match soak.
Unrelated, but relevant to long soaks: H4 (#57) found two leaks on the server, unbounded tick timing and malloc arenas from island building. Both are fixed in f87d5ea.
— tidewall-build
Still not reproduced. The R5 gate ran two more full soaks: 24 clients, 2 matches each, 60 ms ± 20 ms, 2% loss, on f776d59 and f5227b7.
First soak: 2,438 hashes compared, 0 differing.
Second soak: 2,458 compared, 0 differing.
That makes about 9,000 comparisons since the 0ddcb48 diagnostics went in, all matching. Leaving it open in case it comes back.
— tidewall-build
Still not reproduced. The R5 gate ran two more full soaks: 24 clients, 2 matches each, 60 ms ± 20 ms, 2% loss, on f776d59 and f5227b7.
- First soak: 2,438 hashes compared, 0 differing.
- Second soak: 2,458 compared, 0 differing.
That makes about 9,000 comparisons since the 0ddcb48 diagnostics went in, all matching. Leaving it open in case it comes back.
— tidewall-build
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Depends on: N8
Found by
make soak(2% loss, ±20 ms, seed 7). The run was also loaded by tests and captures going on at the same time.a796e089…).5a72ecd7…at 19501). Client 17's hash (3c0455f2…) is neither of those.The client only samples when
replica.seq == receiver.world_seqand nothing is pending, so all the WORLD messages the snapshot names had been applied.Leads:
ClientWorld.tickonly ever goes up (maxi) intake()andsync(). A building piece's progress is worked out from it, so if it got ahead of the snapshot's tick, progress would be computed for a later tick.Next step: when a hash differs, log the client's piece list (id, health, build_ticks_left, placed_tick) next to the server's for that tick, then reproduce with
SOAK_CLIENTS=8.It corrects itself and was seen once in four runs since H12, but a client and the server disagreeing about the world is a multiplayer correctness bug, so it is type/bug.
— tidewall-build
Progress, not fixed. In
0ddcb48the soak now says where a client and the server disagree: the report names the differing part (tide, props, pieces or loot), the client'sClientWorld.tick, and every piece whose (id, health, build_ticks_left, grounded, placed_tick, kind, owner) differs, server against client.world_hash()is unchanged.make soak-selfcheckshows it working: the tampered client is reported as "parts differing: props".A full soak with the diagnostics (24 clients, 2 matches, 60 ms ± 20 ms, 2% loss) passed: 3,743 hashes compared, 0 differing, p99 tick 7.4 ms, client memory growth 1.6 MB. So the mismatch didn't come back to be traced; it's now been seen once in five runs.
What I ruled out by reading the code:
SnapshotReceiver.apply()drops anything older than the newest.replica.seq == world_seq, so a newer WORLD skips the sample.ClientPredictiononly moves the player._spill_broken_crates()runs in the same step as the damage.Still open: the
ClientWorld.ticklead (it only rises) and a provisional piece replaced on the sampled tick. Leaving this open; the next soak that hits it will say which.— tidewall-build
Still open. More leads ruled out this run, by reading the code; no soak was run (this run went to H1, R1 and H2).
StructureSet._next_id: a provisional piece is added with ID 2^30+, which pushes the client's_next_idto 2^30+1 permanently.StructureSet.write_statehashes_next_id, butNetWorld.world_hashbuilds its own buffer from the pieces alone, so the soak's hash doesn't see it. Not the cause. (The client'snext_id()is wrong after a provisional piece, but nothing on the client uses it.)grounded:StructureSet.addsets it only from terrain and fixed solids (touches_fixed_ground), never from neighbouring pieces. A provisional neighbour can't change a server piece's flag, andremovedoesn't touch it. Not the cause.set_progressgivesstart + built_gain(elapsed) - damage. The server'sadvance_buildingsteps health by the same telescoping gains, andmini(…, max_health)only clamps at full health. These agree for any elapsed value.What's left: the
ClientWorld.ticklead. It only rises, but I found no path where it passes the snapshot's tick whilereplica.seq == world_seqholds. Also left: a WORLD message'stickversus the tickafter_tick()logs the hash for (the WORLD goes out afterloop.run_tick(), carryingsim.tick). The next soak that reproduces it will name the differing part with the0ddcb48diagnostics.— tidewall-build
Still open, not reproduced. I ran a soak with
SOAK_CLIENTS=8 SOAK_PROCS=1 SOAK_MATCHES=1(60 ms ± 20 ms, 2% loss) on the H3 bot changes: 375 hashes compared, 0 differing. That makes about 4,100 comparisons with no mismatch since the diagnostics went in. The run's only failures were its own "1 match is too few to check for a leak", as expected for a one-match soak.Unrelated, but relevant to long soaks: H4 (#57) found two leaks on the server, unbounded tick timing and malloc arenas from island building. Both are fixed in
f87d5ea.— tidewall-build
Still not reproduced. The R5 gate ran two more full soaks: 24 clients, 2 matches each, 60 ms ± 20 ms, 2% loss, on
f776d59andf5227b7.That makes about 9,000 comparisons since the
0ddcb48diagnostics went in, all matching. Leaving it open in case it comes back.— tidewall-build