docs: clarify digest contracts accuracy and memory costs

Author: Paul Meng

CHANGELOG.md

@@ -2,6 +2,32 @@
 
 All notable changes to this project will be documented in this file.
 
+## [Unreleased]
+
+### Fixed
+
+- Preserve the requested compression size and sort centroids in `TDigest::new`; reject zero compression sizes.
+- Keep compression within `max_size` even when fractional centroid weights and the stored count differ by roundoff.
+- Make quantile interpolation monotonic across uneven centroid gaps, with finite results for extreme values and subnormal weights.
+- Interpolate ranks for single-centroid digests that represent varying values and avoid overflow across extreme endpoints.
+- Keep centroid means and trimmed means finite when weighted sums overflow; compute `mean()` from retained data if its sum overflows.
+- Preserve centroids when adding zero sum and zero weight, including empty accumulators.
+- Make centroid equality consistent with its total ordering, including weights and signed zeros.
+- Reject invalid serde states before they can cause query or merge panics, preserving the existing field layout for valid digests.
+- Restrict debug-assertion tests to debug builds so the release test suite can run.
+
+### Performance
+
+- Allocate the ingestion buffer on first use and reuse it across flushes.
+- Bound compression-vector reservations by the available input size.
+- Avoid building cumulative weights for empty or single-element bulk quantile queries.
+
+### Documentation and validation
+
+- Clarify memory usage, flush requirements, serialization limits, query semantics, and distribution-dependent accuracy.
+- Add constructor, numeric, and serde regression tests; extend fuzzing to buffered ingestion and interior quantiles.
+- Cover optimized tests and Rust 1.62 `no_std` configurations in CI.
+
 ## [1.0.0] - 2026-07-12
 
 ### Added

CLAUDE.md

@@ -10,28 +10,45 @@ This is `tdigest`, a Rust implementation of the t-digest data structure for accu
 cargo build                  # Build with default features
 cargo build --all-features   # Build with serde support
 cargo test --all-features    # Run all tests including serde round-trip
+cargo test --release --all-features # Test optimized builds without debug assertions
 cargo test --features serde  # Test the preferred serde feature
 cargo test --features use_serde # Test the compatibility alias
 cargo fmt --all -- --check   # Check formatting
 cargo check --all-features   # Type-check only
 cargo check --no-default-features # Verify no_std + alloc
+cargo check --no-default-features --features serde # Verify serde with alloc
+cargo clippy --all-features --all-targets -- -D warnings
+RUSTDOCFLAGS="-D warnings" cargo doc --all-features --no-deps
+cargo bench                  # Criterion performance benchmarks
+cargo run --release --example accuracy # Deterministic accuracy comparison
 ```
 
 ## Project structure
 
-- `src/lib.rs` -- entire library: `Centroid`, `TDigest`, all methods, and all tests
+- `src/lib.rs` -- entire library: `Centroid`, `TDigest`, all methods, unit tests, and property tests
+- `tests/` -- integration regression tests for public API behavior
 - `Cargo.toml` -- package metadata, MSRV 1.62, default `std`, optional `serde`, and compatibility `use_serde` features
 - `docs/architecture.md` -- algorithm and architecture documentation
+- `benches/tdigest.rs` -- deterministic Criterion ingestion, merge, and query benchmarks
+- `examples/accuracy.rs` -- exact versus estimated quantiles across six distributions
+- `fuzz/` -- standalone cargo-fuzz package; see `fuzz/README.md` for nightly commands
 - `rustfmt.toml` -- max_width = 120
 
 ## Code conventions
 
-- Single-file library (`src/lib.rs`), tests are inline `#[cfg(test)]` at the bottom
-- Immutable API: data methods (`merge_sorted`, `merge_unsorted`) take `&self` and return a new `TDigest`
-- `Option<f64>` for fallible queries (`min`, `max`, `mean`, `estimate_quantile`) -- never NaN sentinels
+- Single-file library (`src/lib.rs`), with inline `#[cfg(test)]` tests and public API regression tests in `tests/`
+- Batch API: `merge_sorted` and `merge_unsorted` take `&self` and return a new `TDigest`; the sorted method requires ascending input
+- Mutable API: `push`, `extend_values`, and `Extend<f64>` buffer data; summary statistics update immediately
+- Call `flush` before centroid queries, immutable merges, or serialization; `FromIterator<f64>` flushes before returning
+- `Option<f64>` for queries that require samples (`min`, `max`, `mean`, `estimate_quantile`, `estimate_rank`, `trimmed_mean`); `quantiles` returns a vector of optional estimates
 - `#[must_use]` on methods returning new values
 - `#[non_exhaustive]` on public structs
-- `debug_assert` for invariant checks (NaN rejection, min/max presence)
+- Require a positive compression size in constructors; `max_size` is a compression target, not an exact byte budget
+- `debug_assert` for ingestion contracts (finite values) and state invariants; these checks are absent in release builds
+- Serde skips the pending buffer; never assume serialization flushes or rejects pending data
+- Deserialization validates centroid order, weights, extrema, and count consistency; nonempty sums may be non-finite after valid ingestion
+- Compression's final bucket absorbs remaining weight so count roundoff cannot exceed `max_size`
+- Keep library code compatible with Rust 1.62 without `std`; verify this configuration separately from all-features builds
 - No external dependencies by default; serde is behind the `serde` feature, with `use_serde` as a deprecated alias
 - The library is `no_std` and uses `alloc`; the default `std` feature preserves existing behavior
 - Formatting: rustfmt with max_width=120

README.md

@@ -12,9 +12,9 @@ This implementation follows Facebook's [folly TDigest](https://github.com/facebo
 
 ## Features
 
-- **Accurate quantile estimation** with bounded error, especially at the tails (p99, p999)
+- **Approximate quantiles** with finer resolution at the tails (p99, p99.9)
 - **Mergeable** -- combine digests computed on different machines or threads
-- **Compact** -- fixed memory footprint regardless of input size
+- **Compact** -- retained state scales with the compression factor, rather than the number of samples
 - **No dependencies** by default (optional `serde` support behind a feature flag)
 - **`no_std` + `alloc` support**, including serde without its `std` feature
 
@@ -88,6 +88,10 @@ let unsorted = vec![5.0, 3.0, 1.0, 4.0, 2.0];
 let t = t.merge_unsorted(unsorted);
 ```
 
+`merge_sorted` requires ascending input. Use `merge_unsorted` when the order is
+unknown. Both methods borrow the existing digest and return a new one; keep the
+returned value to retain the inserted data.
+
 For streaming ingestion, use the buffered mutable API and flush before a
 centroid-based query:
 
@@ -102,6 +106,22 @@ t.flush();
 assert_eq!(t.estimate_quantile(0.5), Some(3.0));
 ```
 
+`push`, `extend_values`, and the standard `Extend<f64>` implementation update
+`count`, `sum`, `mean`, `min`, `max`, and `is_empty` immediately. Call `flush`
+before `estimate_quantile`, `quantiles`, `estimate_rank`, `trimmed_mean`,
+`centroids`, immutable merges, or serialization. Automatic compression happens
+when the buffer fills, so a partially filled buffer still needs an explicit flush.
+
+Collecting an iterator creates a digest with the default compression factor and
+flushes it before returning:
+
+```rust
+use tdigest::TDigest;
+
+let t: TDigest = [5.0, 1.0, 3.0].into_iter().collect();
+assert_eq!(t.estimate_quantile(0.5), Some(3.0));
+```
+
 ### Querying
 
 ```rust
@@ -117,18 +137,32 @@ let max    = t.max();                    // Some(5.0)
 let mean   = t.mean();                   // Some(3.0)
 let count  = t.count();                  // 5.0
 let sum    = t.sum();                    // 15.0
+
+// Estimate several quantiles in one call; input order is preserved
+let percentiles = t.quantiles(&[0.5, 0.9, 0.99]);
+
+// Approximate CDF and the mean of the middle 80% of the distribution
+let rank = t.estimate_rank(3.0);
+let middle_mean = t.trimmed_mean(0.1, 0.9);
 ```
 
+Quantiles at or below 0 return the minimum; quantiles at or above 1 return the
+maximum. `trimmed_mean` clamps its bounds to `[0, 1]` and returns `None` for an
+empty interval or NaN bound. Rank and trimmed-mean queries also return `None`
+for empty digests. These queries interpolate compressed data and need not match
+an exact sample percentile or empirical CDF.
+
 ### Merging digests (distributed / parallel use)
 
 ```rust
 use tdigest::TDigest;
 
 // Build digests independently (e.g. on different threads)
+let data: Vec<f64> = (1..=10_000).map(f64::from).collect();
 let mut digests = Vec::new();
 for chunk in data.chunks(1000) {
     let t = TDigest::new_with_size(100);
-    let t = t.merge_sorted(chunk.to_vec());
+    let t = t.merge_unsorted(chunk.to_vec());
     digests.push(t);
 }
 
@@ -137,6 +171,11 @@ let combined = TDigest::merge_digests(digests);
 let p99 = combined.estimate_quantile(0.99);
 ```
 
+Flush any digests built with mutable ingestion before merging. The result uses
+the largest input `max_size`; an empty input vector produces an empty digest
+with the default size of 100. Repeated merging can change estimates, so measure
+accuracy with the merge pattern used by your application.
+
 ### Serialization (serde)
 
 Enable the `serde` feature, then use any serde-compatible format. The deprecated
@@ -152,18 +191,55 @@ let json = serde_json::to_string(&t).unwrap();
 let restored: TDigest = serde_json::from_str(&json).unwrap();
 ```
 
-## Compression factor
+To run this JSON example with exact finite-float round trips, add
+`serde_json = { version = "1.0", features = ["float_roundtrip"] }` to your
+dependencies. Without that feature, JSON parsing can introduce small rounding
+differences in centroid values.
 
-The `max_size` parameter controls the trade-off between accuracy and memory:
+The pending ingestion buffer is omitted from serialization. Always call `flush`
+after mutable ingestion: serializing without it succeeds but loses pending
+samples while retaining their summary statistics. Deserialization validates
+centroid ordering, weights, extrema, and count consistency, and rejects
+inconsistent payloads such as an omitted buffer that leaves a count mismatch.
+The 1.0 format uses `Option<f64>` for `min` and `max`; data written by older
+versions may need migration, depending on the serialization format.
 
-| `max_size` | Memory (approx.) | Accuracy |
-|------------|-------------------|----------|
-| 50         | ~1.2 KB           | Good for rough estimates |
-| 100        | ~2.4 KB           | Good default |
-| 200        | ~4.8 KB           | High accuracy |
-| 500        | ~12 KB            | Very high accuracy |
+## Compression factor
 
-Larger values produce more centroids, giving better accuracy at the cost of more memory and slightly slower merges. A value of 100 is sufficient for most use cases.
+The positive `max_size` parameter controls the trade-off between accuracy and
+memory; zero is rejected. The default is 100.
+
+| `max_size` | Centroid payload (approx.) | Full ingestion buffer |
+|------------|----------------------------|-----------------------|
+| 50         | 0.8 KB                     | 2 KB                  |
+| 100        | 1.6 KB                     | 4 KB                  |
+| 200        | 3.2 KB                     | 8 KB                  |
+| 500        | 8 KB                       | 20 KB                 |
+
+Each centroid holds two `f64` values (16 bytes); the streaming buffer holds up to
+`5 * max_size` values (8 bytes each), allocated on the first mutable insertion
+and reused after flushing. Batch-only digests do not allocate this buffer.
+These are payload estimates in decimal KB, excluding the digest struct, spare
+vector capacity, allocator overhead, input vectors, and temporary storage during
+merges. `max_size` is a compression target, not an exact byte budget.
+
+Larger values generally improve accuracy at the cost of more memory and merge
+work. Start with the default of 100, then measure on your own distribution.
+
+## Input and numeric limits
+
+Inserted values must be finite. NaN and positive or negative infinity violate
+the ingestion contract: debug builds reject them, while release-build estimates
+are unspecified. NaN also violates the contract for `estimate_quantile`,
+`quantiles`, and `estimate_rank`; `trimmed_mean` instead returns `None` for a NaN
+bound.
+
+Counts and sums use `f64`. Integer counts are exact only through `2^53`, and a
+sum of finite values can overflow to infinity or NaN. `mean` normally computes
+`sum / count`; when the sum is non-finite, it falls back to an approximate
+weighted mean of the centroids and pending values. This fallback takes linear
+time in the retained state. `sum` remains non-finite: formats such as JSON cannot
+round-trip that value. Scale inputs when you need a finite sum or JSON storage.
 
 ## Minimum Supported Rust Version (MSRV)
 
@@ -187,6 +263,10 @@ quantiles across deterministic uniform, normal, lognormal, exponential, bimodal,
 and adversarial distributions. See [architecture and internals](docs/architecture.md#measured-accuracy)
 for a summary of the measured error profile.
 
+The crate does not provide a distribution-independent error bound. Tail
+resolution improves with compression size, but skew, repeated values, gaps in
+the distribution, batch boundaries, and merge order can affect estimates.
+
 ## Benchmarks
 
 Run `cargo bench` to measure sorted and unsorted batch ingestion, incremental