Skip to content

transport: add turmoil support - #88

Open
jttdd wants to merge 4 commits into
kezhuw:masterfrom
jttdd:turmoil-support
Open

jttdd wants to merge 4 commits into
kezhuw:masterfrom
jttdd:turmoil-support

Conversation

@jttdd

@jttdd jttdd commented May 7, 2026

Copy link
Copy Markdown

this PR adds optional turmoil socket support to be able to run the client within a turmoil simulation.

I’d like to use this client in deterministic simulation tests, where I can exercise connection loss, partitions, delayed traffic, retries, and session behavior without relying on timing-sensitive real sockets or external ZooKeeper infrastructure. Turmoil is a good fit for that kind of coverage, and this change keeps the support small and opt-in while preserving the crate’s existing async-runtime-agnostic shape.

@jttdd

jttdd commented May 28, 2026

Copy link
Copy Markdown
Author

@kezhuw any thoughts on this? something you'd be willing to land?

@jttdd

jttdd commented Sep 25, 2026

Copy link
Copy Markdown
Author

@kezhuw bumping here, wanted to know your thoughts on this

@kezhuw kezhuw closed this Oct 20, 2026
@kezhuw kezhuw reopened this Oct 20, 2026

@kezhuw kezhuw left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! Looks good to me. I left some inline comments about the details.

Comment thread Cargo.toml Outdated
tokio = ["asyncs/tokio"]
smol = ["asyncs/smol"]
async-global-executor = ["asyncs/async-global-executor"]
turmoil = ["dep:turmoil", "dep:tokio-util", "tokio"]

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test fails due to unused async_net in case of turmoil. Does rust have way to opt out dependency in case of feature ?

If not, we probably have to use it in net module to circumvent this. I think it is ok for turmoil support.

Comment thread src/lib.rs Outdated
//! * `sasl`: Toggle SASL support.
//! * `sasl-gssapi`: Toggle only GSSAPI SASL support. This relies on binding package `libgssapi-sys`.
//! * `sasl-digest-md5`: Toggle only DIGEST-MD5 SASL support.
//! * `turmoil`: Toggle simulated networking support via `turmoil`.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about move this to a separated section "Network simulation" for turmoil ?

Comment thread src/lib.rs Outdated
//! * `async-global-executor`: Toggle support for [async-global-executor](https://docs.rs/async-global-executor).
//!
//! `turmoil` currently builds on the `tokio` runtime support and adapts simulated sockets into the
//! crate's existing `futures::io` transport layer.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

adapts simulated sockets into the crate's existing futures::io transport layer

Is this an implmentation detail ?

We should point out that Client must be constructed on tokio tasks for turmoil to function.

jttdd and others added 3 commits October 21, 2026 14:27
CI fails on `fmt` and `build`, and `coverage` would fail next:

- build: with `turmoil`, `async-net` goes unused and
  `-Dunused-crate-dependencies` rejects it. Cargo features are additive,
  so the feature can't drop the non-optional dependency it replaces;
  mark it used in `net` instead.
- fmt: split a test import as nightly rustfmt wants.
- coverage: `--all-features` turns on `turmoil`, under which the client
  connects only inside a turmoil simulation, so the real ZooKeeper and
  TLS tests panic. Run them with every feature but `turmoil`, and run
  the turmoil tests in their own step so they still run in CI. The
  feature matrix runs the whole suite per feature set, so `turmoil`
  can't join it without gating every real-socket test. `make test`
  follows suit.

Move the `turmoil` docs to their own "Network simulation" section and
replace the implementation detail with what callers must do: construct
`Client` on tokio tasks of a turmoil host, and seed fastrand, which
turmoil's seed doesn't reach, for reproducible endpoint shuffle and
reconnect jitter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The turmoil feature moves sockets, TLS and DNS into the simulation, but
every timer and the clock stay on async-io and std wall time. Turmoil
advances simulated time far faster than wall time, so connect timeouts,
reconnect backoff, pings and session expiry fire at the wrong simulated
moment, or not at all within the simulated duration, and the client's
view of elapsed time drifts from the simulation's. That defeats
simulating retries and session behavior.

Add a `time` module next to `net` with `Timer` and `now`. Under
`turmoil` they run on tokio's clock, which turmoil advances: `Timer`
mirrors the never/once/periodic `async_io::Timer` API the call sites
use, including the first periodic tick being one period out, and `now`
converts tokio's `Instant` back to `std`. Otherwise they are
`async_io::Timer` and `Instant::now`, so default builds are unchanged.
Switch the session, connection, endpoint and deadline call sites to it.
`async-io` then goes unused under `turmoil`, so mark it used like
`async-net`.

Under `turmoil` the session loops' `select!` is `biased`: an unbiased
`select!` picks the branch to poll first from the address of its
futures, which differs between runs, so the same seed could take
different branches. The connect `select!` is already biased.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jttdd

jttdd commented Oct 21, 2026

Copy link
Copy Markdown
Author

Thank you! Looks good to me. I left some inline comments about the details.

@kezhuw thanks for taking a look, addressing your comments. going to refresh this a bit, there are the timer apis we also need to abstract for turmoil + also going to rebase off latest.

@jttdd
jttdd force-pushed the turmoil-support branch 2 times, most recently from 2f5173a to 6937882 Compare October 21, 2026 14:46
The session fails pending requests, notifies watchers of session state
and resends watches on reconnect by walking hashbrown maps. hashbrown's
default hasher (foldhash) seeds each map from addresses and other
per-run state that a simulation's seed doesn't reach, so after a
disconnect the same seed could wake callers and watchers in a different
order, and the run diverge from there.

Under `turmoil` the session's maps hash with std's fixed-key SipHash,
so the same inserts traverse in the same order on every run. Otherwise
they keep hashbrown's default hasher, so default builds are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants