NexusQuant
Near-lossless KV cache quantization. Training-free. One line of code.
E8 lattice quantization applied to the KV cache after prefill. No training, no calibration data, no model modifications.
Live demo: huggingface.co/spaces/jmarquex/nexusquant-demo
Headline: on Llama-3.1-8B-Instruct at 4K (n=30, one harness and one needle set), at iso-bpe (2.125 bpe for both once the fp16 scale overhead is counted), NexusQuant K2V2 pb=0 holds 30/30 NIAH while a 2-bit TurboQuant cache drops to 0/30 (NQ-vs-TQ McNemar b=30, c=0; FP16-vs-TQ floor b=29; nq_tq_h2h_llamainst.json). The TurboQuant side is our own reimplementation on our harness, so a faithfulness gap to the reference code cannot be excluded. Near-lossless without eviction: K3V2 pb=0 costs +0.43% PPL on Mistral-Inst-v0.3 (n=161; nq_mistral_subbpe_frontier.json). Validated on 9 models; NIAH recall holds through 32K context (24-25/25 per cell; nq_32k_only.json).
The earlier PyPI/HuggingFace figure of +0.276% PPL on Mistral-7B-v0.1 (n=60, KIVI-iso) is committed in rank116_kivi_iso_bpe.json. The 5.83x compression figure is measured on Mistral-Inst-v0.3 (nq_e8_bpwmatrix.json). PPL deltas above were measured with the 0.5.x quantizer; 0.6.0 corrects the E8 tie-break, so published deltas are conservative for current installs.
pip install "nexusquant-kv[hf]" # with HuggingFace transformerscompress_kv_cache (quant-only) needs transformers >= 4.46. nexusquant_evict needs transformers >= 5.0 and torch >= 2.4 (uses DynamicLayer hooks).
Prefill all but the last token, compress the cache, then generate.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, DynamicCache
from nexusquant import compress_kv_cache
model = AutoModelForCausalLM.from_pretrained(
"mistralai/Mistral-7B-v0.1", torch_dtype=torch.float16, device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")
inputs = tokenizer("The KV cache stores key and value tensors for each attention layer.", return_tensors="pt").to(model.device)
cache = DynamicCache()
with torch.no_grad():
out = model(inputs["input_ids"][:, :-1], use_cache=True, past_key_values=cache)
compressed = compress_kv_cache(out.past_key_values, mode="quant_only", rope_base=getattr(model.config, "rope_theta", 10000.0))
with torch.no_grad():
print(tokenizer.decode(model.generate(inputs["input_ids"], past_key_values=compressed, max_new_tokens=200, do_sample=False)[0], skip_special_tokens=True))With eviction (higher compression, needs transformers >= 5.0):
from nexusquant import nexusquant_evict
with nexusquant_evict(model, quality="balanced") as nq:
output = model.generate(inputs["input_ids"], max_new_tokens=512)A DynamicCache subclass (E8QuantizedCache) and a kvpress adapter (E8Press) are also available. Full argument reference is in the docstrings.
| Config | bpe (honest) | PPL delta vs FP16 |
|---|---|---|
| FP16 | 16.000 | 0.000% |
| K3V2 pb=0 | 2.625 | +0.276% |
| K2V2 pb=0 | 2.125 | +0.96% |
bpe = (key_bits + value_bits)/2 + 0.125 (the per-head fp16 scale). Source: rank116_kivi_iso_bpe.json.
| Context | FP16 | K3V2 pb=0 | K2V2 pb=0 |
|---|---|---|---|
| 12K | 25/25 | 25/25 | 25/25 |
| 32K | 25/25 | 25/25 | 24/25 |
Sources: nq_32k_only.json, nq_32k_modal_partial.json.
TurboQuant and Block-GTQ are our own reimplementations on our harness; a faithfulness gap to each reference code cannot be excluded.
| Config | bpe | NIAH hits | Notes |
|---|---|---|---|
| FP16 | 16.0 | 29/30 | 1 natural model failure |
| TurboQuant bw2 | 2.125 | 0/30 | Complete retrieval collapse |
| Block-GTQ K2V2 | 2.0 | 29/30 | Calibrated, pre-RoPE K-alloc (non-iso) |
| NexusQuant K2V2 pb=0 | 2.125 | 30/30 | Calibration-free, post-RoPE E8 |
NQ-vs-TQ McNemar b=30, c=0 (30 discordant pairs, all favor NexusQuant). Sources: nq_tq_h2h_llamainst.json, nq_blockgtq_h2h_4k.json.
Quality correlates with KV head count. Models with 4+ KV heads are safe at K3V2 (Qwen family needs protect_boundary=2).
| Model | K3V2 pb=0 PPL delta | Notes |
|---|---|---|
| Gemma-2-2b | +0.21% | global-only (SWA at fp16) |
| Mistral-7B-v0.1 | +0.276% | main benchmark |
| Yi-6B | +0.35% | 2K prefix |
| Mistral-Inst-v0.3 | +0.33% | 1K-prefix |
| Llama-3.1-Inst | +0.64% | pass rope_scaling; K4V2 recommended |
| Mixtral-8x7B-Inst | +0.63% | NF4-weights baseline, n=80 (nq_mixtral_stream.json) |
| Qwen3-8B | +1.130% | strict FP16-weights (nq_q3fp16_iso_strict_fp16_weights.json) |
| Qwen2.5-1.5B | catastrophic | 2 KV heads; +5.04% is a 35% token-removal cell with protect_boundary=1, a different config from this pb=0 column; not recommended |
Gemma4 (SWA): use compress_layers="global_only". Encoder-decoder, GPT-NeoX/J, and GGUF are not supported.
- (optional) Importance-score tokens and evict the lowest, keeping BOS + a recent sliding window
- Remove RoPE from keys so they share a common subspace
- Hadamard-rotate to spread outliers across dimensions
- Quantize each 8-float group to the nearest E8 lattice point (asymmetric: 3-bit keys + 2-bit values)
- (optional) Boundary-protect first/last N layers at FP16 (mandatory for Qwen-family)
The E8 lattice is the proven densest sphere packing in 8 dimensions and the best known lattice quantizer in 8D (normalized second moment 0.0717 vs 0.0833 for scalar rounding), so the average quantization error per 8-float group is lower than scalar round-to-nearest at the same bit budget. Sources: Viazovska, arXiv:1603.04246 (packing); Agrell and Allen, arXiv:2202.09605 (quantizer, Table I).
The long-form report is broken into six focused, topic-separated papers.
- Method -
paper/nexusquant-method.pdf: the E8 lattice quantizer, the six-stage pipeline, and the quantizer ablations (E8 vs uniform vs Lloyd-Max). - Architectures -
paper/nexusquant-architectures.pdf: does it ruin the language model? Cross-architecture PPL (nine models), prefix-length scaling, mixture-of-experts, multi-turn KV reuse. - Retrieval -
paper/nexusquant-retrieval.pdf: does it remember facts? The 2-bit iso-bpe wedge, FP16 NIAH dead zones, RULER, LongBench, and the eviction cliff. - Downstream tasks -
paper/nexusquant-downstream.pdf: function-call accuracy, HumanEval pass@1, multi-document QA, and GSM8K/MATH reasoning under compression. - Competitors -
paper/nexusquant-competitors.pdf: iso-protocol head-to-heads against KIVI, CommVQ, and AQUA-KV, the non-iso Block-GTQ system comparison, and why the KVTC row was retracted. - System and negative results -
paper/nexusquant-system.pdf: honest byte accounting, rANS entropy coding, latency, memory, the K/V bit grid, and the twelve approaches that did not work.
- Eviction hurts factual recall: NIAH degrades past 35% eviction (PPL hides this). Use quant-only when factual accuracy matters.
- Qwen-family models catastrophically fail without
protect_boundary=2. Always test your model. - E8 quantization is CPU-bound; the Triton GPU kernel is written but not yet latency-benchmarked.
nexusquant_evictdoes not support batch size > 1 or multi-turn chat; vLLM runtime integration is stubbed (NotImplementedError).
The work is broken into six focused papers. Cite the one(s) you draw from:
@techreport{nexusquant_method_2026,
author = {Marques, Jo\~{a}o},
title = {{NexusQuant}: Training-Free {E8} Lattice Vector Quantization for {KV} Cache Compression},
year = {2026},
url = {https://github.com/jagmarques/nexusquant/blob/main/paper/nexusquant-method.pdf},
}
@techreport{nexusquant_arch_2026,
author = {Marques, Jo\~{a}o},
title = {{NexusQuant} Cross-Architecture Perplexity: Does {KV} Cache Compression Preserve Language Modeling?},
year = {2026},
url = {https://github.com/jagmarques/nexusquant/blob/main/paper/nexusquant-architectures.pdf},
}
@techreport{nexusquant_retrieval_2026,
author = {Marques, Jo\~{a}o},
title = {{NexusQuant} Factual Retrieval: Does {KV} Cache Compression Preserve Needle Recall?},
year = {2026},
url = {https://github.com/jagmarques/nexusquant/blob/main/paper/nexusquant-retrieval.pdf},
}
@techreport{nexusquant_downstream_2026,
author = {Marques, Jo\~{a}o},
title = {Downstream Task Retention Under {KV} Compression: Function Calls, Code, {Q\&A}, and Reasoning under the {E8} Lattice},
year = {2026},
url = {https://github.com/jagmarques/nexusquant/blob/main/paper/nexusquant-downstream.pdf},
}
@techreport{nexusquant_competitors_2026,
author = {Marques, Jo\~{a}o},
title = {Calibration-Free {E8} {KV} Cache Quantization versus Competitors: Iso-Protocol Head-to-Head Comparisons},
year = {2026},
url = {https://github.com/jagmarques/nexusquant/blob/main/paper/nexusquant-competitors.pdf},
}
@techreport{nexusquant_system_2026,
author = {Marques, Jo\~{a}o},
title = {{NexusQuant}: System Cost and Negative Results for Training-Free {E8} {KV} Cache Compression},
year = {2026},
url = {https://github.com/jagmarques/nexusquant/blob/main/paper/nexusquant-system.pdf},
}Apache 2.0. See LICENSE.