Loop-level continuous batching for recurrent language models.
Documentation · Getting Started · Supported Models · Developer Guide · Releases · Community · CDB Paper
- 2026/11 — v0.2.0: core runtime refactoring around Stage and Loop, wavefront chunked prefill (WCPB), and experimental token-level decode pipelining. See the changelog for model additions, migration notes, and validation scope.
vllm-rlt is a standalone inference and serving engine for recurrent language models. It schedules individual recurrent loops, allowing requests at different depths to share a batch instead of waiting for an entire cohort to finish. Its engine organization is inspired by vLLM, but it runs independently and does not require vLLM to be installed.
- Stage/Loop architecture. A stage owns deployment and resources; a loop is a stage-local execution and scheduling unit. Scheduler selects work and applies results; Executor submits execution. Prefill and decode share this runtime structure.
- WCPB prefill. Wavefront chunked prefill exposes loop boundaries so prompt chunks can advance at different recurrent depths. Compatible non-speculative prefill and decode work can share a batch.
- Decode pipeline. Experimental token-level self-speculation advances different tokens at different loop depths while retaining hidden states. Asynchronous execution separates device progress from CPU output delivery.
Original conceptual animation, not a benchmark or a claim that every mode can be combined. Static illustration · Animation source · Architecture · WCPB · Decode pipeline.
Prefill and decode provide complementary pipelined execution paths, connected through separate PD stages and KV transfer in supported configurations. PD is currently same-host CUDA with supported Triton/FA4 paths; it is not a claim of universal multi-node or model support. Speculative steps are not included in mixed prefill/decode batches.
- Depth-aware paged KV cache: LAST_EXITED and SHARED layouts, with optional prefix reuse, incremental allocation, priorities and preemption.
- Adaptive computation: fixed-depth inference or Ouro's trained early-exit gate, with per-request loop bounds. Changing exit policy can change quality.
- Configurable execution: Torch reference execution, GPU attention backends, CUDA Graphs, and optional asynchronous/speculative paths. An optional native CUDA SiLU-and-Mul operator accelerates the Ouro activation path, with a Torch fallback; it is not a general model-speedup claim.
- Offline and online use: Python API, CLI, and a streaming OpenAI-compatible completions endpoint. This is not a chat-completions API.
Important
v0.2.0 is a research/evaluation milestone. Default execution is synchronous; advanced modes are opt-in and have model-specific constraints. Token-pipeline speculation is experimental. Historical serving measurements have unresolved cross-configuration output-equivalence differences, so they are diagnostic, not equal-quality speedup or production-readiness claims.
Requires Python 3.10+ and PyTorch 2.5+. The commands below install
v0.2.0 for NVIDIA CUDA on Linux with compatible CUDA-enabled PyTorch.
For Ascend NPU on main, follow the PyTorch and Torch-NPU installation steps.
# Activate your Python environment first.
git clone --branch v0.2.0 --depth 1 https://github.com/InferMatrixLab/vllm-rlt.git
cd vllm-rlt
uv pip install -e .
vllm-rlt-serve --helpThis installs text, serving and Linux Triton dependencies. Standard isolated builds use the Torch activation fallback; optional native CUDA compilation requires a compatible toolchain. Model weights are downloaded separately. No PyPI publication is implied.
- Installation, first request, Python API, and CLI
- HTTP serving and streaming. Use a trusted network or secure gateway. The server has no built-in authentication or TLS.
- Supported models and Ouro-Thinking recipe
- Configuration and compatibility
- Native CUDA build
- Migration from v0.1.0
Packed prefill uses ExecutionConfig.packed_prefill and the
--packed-prefill / --no-packed-prefill flags. Input transfer is controlled
independently by use_uva; see the selection rules.
FP8 KV defaults to per-vector scales on Torch/Triton. Experimental native FA4 FP8,
including Dory sliding attention, uses flash-attn-4==4.0.0b39 and calibrates
a fixed scale at startup. See the
hardware and calibration requirements.
| Checkpoint | Implementation | Platforms | Documented validation scope |
|---|---|---|---|
| Ouro-1.4B | Native Ouro | CUDA Ascend |
Ascend BF16 full prefill with eager Python execution, paged KV and optional fixed-depth async scheduling. Arbitrary chunked prefill remains unresolved. |
| Ouro-2.6B | Shared Ouro | CUDA | Shares the Ouro implementation. Separate quality and performance qualification remains open. |
| Ouro-1.4B-Thinking | Reuses Ouro | CUDA | Bounded BF16, fixed-four-loop synchronous basic integration |
| Ouro-2.6B-Thinking | Reuses Ouro | CUDA | Bounded BF16, fixed-four-loop synchronous basic integration |
| Nanbeige4.2-3B | Native Nanbeige | CUDA | Fixed two-loop inference. CUDA PD validation |
| Huginn-0125 | Native Huginn | CUDA | Synchronous fixed-depth, LAST_EXITED KV, CUDA Graph decode core and coda |
| NVIDIA-Nemotron-Labs-Dory1.0-4B | Native Dory | CUDA | Synchronous fixed-loop execution, boundary KV, per-loop and last-loop KV modes, Triton/Torch sliding-window masking, BF16 real-checkpoint agreement at 1/2/4 loops on A100; long context, Graphs/async and reclamation not qualified |
The broader model roadmap is in RFC #69.
Huginn uses native checkpoint loading and depth-aware KV for its recurrent core, with separate prelude and coda KV. It supports synchronous fixed-depth inference and CUDA Graphs for the recurrent core and decode coda.
Huginn does not currently support WCPB/mixed staged prefill, asynchronous execution, self-speculation, or PD. Feature availability is not equivalent to validation of every model/backend/workload combination.
Dory runs the native normalized-GPT stack with attention-bearing input and
output stages, so its boundary stages own their own KV. Synchronous
fixed-loop execution is supported for both per_loop and last_loop KV
modes. Sliding-window masking exists only in the Torch reference attention
backend, and WCPB, CUDA Graphs, asynchronous execution, self-speculation,
prefix reuse and PD are rejected. See the
Dory baseline notes for the exact qualified scope.
Start with the documentation index, then explore:
| Topic | Guide |
|---|---|
| Runtime and process boundaries | Unified architecture |
| Chunk/depth scheduling | WCPB prefill |
| Asynchronous block execution and token pipelines | Self-speculation |
| Cache ownership and lifetime | KV walkthrough |
| Evaluation and measurement | Accuracy, profiling |
Performance evidence: the October 2026 serving report records Ouro-1.4B BF16 on B300/FA4, workloads, one- versus two-GPU configurations, metric definitions and limitations. Compare matched hardware, workload and quality; do not interpret unequal resource budgets or differing output lengths as isolated runtime speedups.
For the current architecture and ownership boundaries, see the unified architecture guide. Ongoing model and feature work is tracked in the model roadmap. Roadmap proposals are not released capabilities.
We welcome systems, model, evaluation and documentation contributions. Read CONTRIBUTING.md and the Developer Must-Read before implementation. Include reproducible configurations, correctness evidence and the scope of any measurements. For larger changes, start with a design issue.
For the CDB method that informs this project, please cite the original paper: Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching (PDF).
@misc{schwethelm2026continuousdepthbatching,
title = {Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching},
author = {Kristian Schwethelm and Daniel R\"{u}ckert and Georgios Kaissis},
year = {2026},
eprint = {2608.09444},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2608.09444}
}The paper's measurements are separate from the vllm-rlt performance baselines linked above.
Ask questions, report reproducible problems or discuss designs in GitHub issues. You can also join the existing vllm-rlt developer group through WeChat:
vllm-rlt builds on the published Ouro architecture and Continuous Depth Batching. Its engine and Python API organization are inspired by vLLM. See NOTICE for upstream model attribution.
Apache License 2.0. Model checkpoints retain their own licenses and usage conditions.
