Rust Streaming ASR WebSocket Server

A high-performance WebSocket streaming ASR server built with Rust, implementing async I/O + parallel CPU architecture for speech recognition.

Architecture

This server follows the optimal pattern for streaming ASR:

Async WebSocket Layer: Uses Tokio and Axum to handle thousands of concurrent WebSocket connections efficiently
CPU-Bound ASR Processing: Dedicated thread pool using Rayon for parallel speech recognition inference
Non-Blocking Design: Audio processing is offloaded to spawn_blocking threads, keeping the async runtime free for I/O

Features

High Performance: Async I/O for WebSocket handling + parallel threads for CPU-intensive ASR
Configurable: TOML-based configuration system
WebSocket Streaming: Real-time audio streaming and transcription results
Sherpa-ONNX Integration: Real sherpa-onnx with Estonian-English streaming zipformer model
Whisper Support: OpenAI Whisper Large-v3 and Turbo models for offline transcription
Parakeet EOU Support: NVIDIA Parakeet Realtime EOU 120M for low-latency English streaming
FastConformer Support: NVIDIA FastConformer streaming models (80ms/1040ms latency)
Parakeet TDT Support: NVIDIA Parakeet TDT 0.6B-v3 model for 25 European languages
Automatic Punctuation: CT-Transformer punctuation model adds commas, periods, and capitalization
Multi-Language: Estonian, English, Russian, Ukrainian, and 21 more European languages
Multi-Format Audio: Support for Opus, FLAC, WAV, MP3, and AAC audio formats
Safe & Fast: Built with Rust for memory safety and performance
Production Ready: Powers the ASR backend at tekstiks.ee

Quick Start

Prerequisites

Install sherpa-onnx CLI tools (required for Parakeet TDT model):

# Download latest sherpa-onnx
wget https://github.com/k2-fsa/sherpa-onnx/releases/download/v1.12.14/sherpa-onnx-v1.12.14-linux-x64-static.tar.bz2
tar xf sherpa-onnx-v1.12.14-linux-x64-static.tar.bz2

# Install to user bin (no sudo required)
cp sherpa-onnx-v1.12.14-linux-x64-static/bin/sherpa-onnx-offline ~/.cargo/bin/

# Or install system-wide (requires sudo)
sudo cp sherpa-onnx-v1.12.14-linux-x64-static/bin/sherpa-onnx-offline /usr/local/bin/

Note: The Parakeet TDT model (25 European languages) requires the sherpa-onnx-offline CLI binary. Estonian and English streaming models work without this.

Running the Server

Start the server:
```
cargo run
```
Connect via WebSocket: The server runs on ws://127.0.0.1:8080/ws

Offline Transcription (Whisper)

For batch transcription of audio files using Whisper models:

Convert audio to WAV format (if needed):

ffmpeg -i audio.opus -ar 16000 -ac 1 audio.wav

Run transcription:

# Ensure library path is set
export LD_LIBRARY_PATH="/path/to/sherpa/lib:$LD_LIBRARY_PATH"

# Run opus transcription binary
cargo run --bin transcribe_opus -- input.opus

WebSocket Protocol

Client Messages

Start Session:
```
{
  "type": "start",
  "sample_rate": 16000,
  "format": "pcm",
  "language": "en"
}
```
Available language values:
- en - English (Parakeet EOU 120M, low-latency streaming)
- et - Estonian/English (Zipformer bilingual)
- fastconformer_en_1040ms - English (FastConformer 1040ms latency, high accuracy)
- parakeet_tdt_v3 - Auto-detect (25 European languages)

Send Audio:

{
  "type": "audio",
  "data": "base64-encoded-audio-data"
}

Stop Session:
```
{
  "type": "stop"
}
```
Ping:
```
{
  "type": "ping"
}
```

Server Messages

Ready:

{
  "type": "ready",
  "session_id": "uuid-here"
}

Transcript:

{
  "type": "transcript",
  "session_id": "uuid-here",
  "text": "transcribed text",
  "is_final": false,
  "confidence": 0.95
}

Error:

{
  "type": "error",
  "message": "error description"
}

Real-Time Speaker Diarization

Optional per-session speaker diarization with known-speaker identification, compatible with offline-asr rosters. Enable it in config.toml ([diarization] enabled = true) and opt in per session:

{ "type": "start", "language": "en", "diarize": true }

Audio is VAD-segmented, each speech window is embedded with a ReDimNet2 ONNX model, and clusters are re-computed every few seconds — so speaker tags are revised retroactively as more audio arrives. Cluster centroids are matched against a roster.json produced by offline-asr enroll to auto-suggest names.

Additional server messages when diarization is active:

Speaker segment (provisional, sent as speech is detected):

{
  "type": "speaker_segment",
  "session_id": "uuid",
  "segment_id": 4,
  "start_time": 7.3,
  "end_time": 8.9,
  "cluster": "S2",
  "name": "mari",
  "name_confidence": 0.84
}

start_time/end_time are seconds relative to the start of the session's audio. name/name_confidence are omitted when no roster speaker matches.

Speakers updated (whenever re-clustering changes any tag or name; a full snapshot — the client should replace its segment→speaker mapping):

{
  "type": "speakers_updated",
  "session_id": "uuid",
  "revision": 3,
  "segments": [ { "segment_id": 0, "cluster": "S1" }, ... ],
  "clusters": [ { "cluster": "S1", "name": "mari", "name_confidence": 0.84 },
                { "cluster": "S2" } ]
}

revision increases strictly; apply snapshots in order. A final update is always sent during session finalization.

transcript messages additionally carry audio_time (seconds of audio ingested when the result was emitted) so the UI can align transcript text with speaker segments.

Setup:

Export the embedding model (one-time, in offline-asr): python scripts/export_redimnet2_onnx.py -o <this repo>/models/speaker/redimnet2-b6-vb2-vox2-lm.onnx The script verifies torch↔ONNX equivalence (cosine > 0.999), so embeddings and rosters are interchangeable between the two systems. The roster's model_version must match [diarization] model_version exactly.
Optionally enroll known speakers with offline-asr enroll and point [speaker_id] roster_path at the resulting roster.json.
Calibrate the clustering threshold on your own recordings if needed: cargo run --release --bin diarize_wav -- meeting.wav --sweep (measured on 16 kHz test data: same-speaker cosine distance ≈ 0.1–0.2, cross-speaker ≈ 0.5; the default cut is 0.4).

Model Setup

IMPORTANT: ASR models are NOT included in this repository and must be downloaded separately.

Quick Setup

Download TTS models (included in Git LFS):
```
git lfs install
git lfs pull
```

Download ASR models (must be obtained separately):

The following models are required but NOT included in the repository:

Estonian-English Streaming Model

# Download Estonian-English zipformer model
# This model must be obtained from TalTechNLP or other sources
# Place files in: models/streaming-zipformer-large.et-en/
# Required files: encoder.onnx, decoder.onnx, joiner.onnx, tokens.txt

English Streaming Model

# Download English zipformer model from sherpa-onnx releases
wget https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-streaming-zipformer-en-2023-06-26.tar.bz2
tar xvf sherpa-onnx-streaming-zipformer-en-2023-06-26.tar.bz2
mv sherpa-onnx-streaming-zipformer-en-2023-06-26 models/streaming-zipformer-en-2023-06-26

Parakeet TDT v3 (25 European Languages)

# Download Parakeet TDT model
wget https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8.tar.bz2
tar xvf sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8.tar.bz2
mv sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8 models/

Included Models (Git LFS)

The following models ARE included via Git LFS:

TTS Model: Estonian TTS (models/tts/vits-coqui-et-cv/model.onnx)
VAD Model: Silero VAD (models/silero_vad.onnx) - currently disabled

Model Directory Structure

After setup, your models directory should look like:

models/
├── streaming-zipformer-large.et-en/          # Estonian-English (obtain separately)
│   ├── encoder.onnx
│   ├── decoder.onnx
│   ├── joiner.onnx
│   └── tokens.txt
├── parakeet-realtime-eou-120m-v1-onnx/       # Parakeet EOU (English streaming)
│   └── ...
├── fastconformer-en-80ms-transducer/         # FastConformer 80ms (with punctuation)
│   ├── encoder.onnx
│   ├── decoder.onnx
│   ├── joiner.onnx
│   └── tokens.txt
├── fastconformer-en-1040ms-transducer/       # FastConformer 1040ms (high accuracy)
│   ├── encoder.onnx
│   ├── decoder.onnx
│   ├── joiner.onnx
│   └── tokens.txt
├── sherpa-onnx-punct-ct-transformer-zh-en-vocab272727-2024-04-12/  # Punctuation model
│   └── model.onnx
├── sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8/  # Parakeet TDT (25 languages)
│   ├── encoder.int8.onnx
│   ├── decoder.int8.onnx
│   ├── joiner.int8.onnx
│   └── tokens.txt
├── tts/
│   └── vits-coqui-et-cv/                     # Estonian TTS (Git LFS)
│       ├── model.onnx
│       └── tokens.txt
└── silero_vad.onnx                           # VAD (Git LFS)

Whisper Models (Offline Recognition)

For offline transcription with OpenAI Whisper models:

# Download Whisper Large-v3 model
wget https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-whisper-large-v3.tar.bz2
tar xvf sherpa-onnx-whisper-large-v3.tar.bz2
mv sherpa-onnx-whisper-large-v3 models/whisper-large-v3

# Download Whisper Turbo model
wget https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-whisper-turbo.tar.bz2
tar xvf sherpa-onnx-whisper-turbo.tar.bz2
mv sherpa-onnx-whisper-turbo models/whisper-turbo

Note: Whisper models in sherpa-onnx process only the first 30 seconds of audio due to architectural limitations.

Punctuation Model

The server supports automatic punctuation restoration using a CT-Transformer model. This adds commas, periods, question marks, and proper capitalization to ASR output.

# Download CT-Transformer punctuation model (bilingual EN/ZH, ~294MB)
wget https://github.com/k2-fsa/sherpa-onnx/releases/download/punctuation-models/sherpa-onnx-punct-ct-transformer-zh-en-vocab272727-2024-04-12.tar.bz2
tar xvf sherpa-onnx-punct-ct-transformer-zh-en-vocab272727-2024-04-12.tar.bz2
mv sherpa-onnx-punct-ct-transformer-zh-en-vocab272727-2024-04-12 models/

Note: The English-only model (sherpa-onnx-online-punct-en-2024-08-06) uses a different architecture (CNN-BiLSTM) that is not compatible with the current sherpa-rs bindings. Use the CT-Transformer model instead.

Transducer Models (RNN-T)

Transducer models (also known as RNN-T or RNNT) are streaming ASR models that use an encoder-decoder-joiner architecture. They provide real-time transcription with good accuracy.

Model Types

The server supports two transducer model types:

Transducer - Standard sherpa-onnx transducer models (zipformer, FastConformer)
NemoTransducer - NVIDIA NeMo transducer models (Parakeet TDT) - requires CLI subprocess

Configuration

Transducer models require three ONNX files: encoder, decoder, and joiner. Configure them in config.toml:

[asr.models.my_transducer_model]
model_path = "./models/my-model/encoder.onnx"      # Primary path (encoder)
tokens_path = "./models/my-model/tokens.txt"
encoder_path = "./models/my-model/encoder.onnx"    # Explicit encoder path
decoder_path = "./models/my-model/decoder.onnx"    # RNN decoder
joiner_path = "./models/my-model/joiner.onnx"      # Joiner network
name = "My Transducer Model"
language_code = "en"
model_type = "Transducer"
sample_rate = 16000
punctuation = true  # Enable punctuation post-processing

Available Transducer Models

Estonian-English Zipformer (model_type = "Transducer"):

[asr.models.et]
model_path = "./models/streaming-zipformer-large.et-en/encoder.onnx"
tokens_path = "./models/streaming-zipformer-large.et-en/tokens.txt"
name = "Estonian/English Zipformer"
language_code = "et"
model_type = "Transducer"

FastConformer 1040ms (model_type = "Transducer"):

High accuracy, 1040ms latency
No built-in punctuation - enable punctuation = true for post-processing

[asr.models.fastconformer_transducer_en_1040ms]
model_path = "./models/fastconformer-en-1040ms-transducer/encoder.onnx"
tokens_path = "./models/fastconformer-en-1040ms-transducer/tokens.txt"
encoder_path = "./models/fastconformer-en-1040ms-transducer/encoder.onnx"
decoder_path = "./models/fastconformer-en-1040ms-transducer/decoder.onnx"
joiner_path = "./models/fastconformer-en-1040ms-transducer/joiner.onnx"
name = "FastConformer English Streaming 1040ms (RNNT)"
language_code = "en"
model_type = "Transducer"
sample_rate = 16000
punctuation = true

FastConformer 80ms with Punctuation (model_type = "Transducer"):

Low latency (80ms)
Has built-in punctuation & capitalization (_pc variant)

[asr.models.fastconformer_transducer_en_80ms]
model_path = "./models/fastconformer-en-80ms-transducer/encoder.onnx"
tokens_path = "./models/fastconformer-en-80ms-transducer/tokens.txt"
encoder_path = "./models/fastconformer-en-80ms-transducer/encoder.onnx"
decoder_path = "./models/fastconformer-en-80ms-transducer/decoder.onnx"
joiner_path = "./models/fastconformer-en-80ms-transducer/joiner.onnx"
name = "FastConformer English Streaming 80ms (RNNT)"
language_code = "en"
model_type = "Transducer"
sample_rate = 16000
punctuation = false  # Model has native punctuation

Parakeet TDT v3 (model_type = "NemoTransducer"):

25 European languages with auto-detection
Requires sherpa-onnx-offline CLI binary (see Prerequisites)
Uses offline/pseudo-streaming mode

[asr.models.parakeet_tdt_v3]
model_path = "./models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8/encoder.int8.onnx"
tokens_path = "./models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8/tokens.txt"
encoder_path = "./models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8/encoder.int8.onnx"
decoder_path = "./models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8/decoder.int8.onnx"
joiner_path = "./models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8/joiner.int8.onnx"
name = "Parakeet TDT v3 (25 European languages)"
language_code = "auto"
model_type = "NemoTransducer"
sample_rate = 16000

Converting NeMo Models to ONNX

To convert NVIDIA NeMo models to sherpa-onnx format:

# Install dependencies
pip install nemo_toolkit[asr] onnx

# Use the export script (for FastConformer transducer)
python scripts/export_fastconformer_rnnt.py

# Or use sherpa-onnx's official conversion scripts
# See: https://github.com/k2-fsa/sherpa-onnx/blob/master/scripts/nemo/

The conversion produces three files:

encoder.onnx - Audio feature encoder
decoder.onnx - RNN decoder (prediction network)
joiner.onnx - Joint network combining encoder and decoder outputs

FastConformer Models

NVIDIA FastConformer streaming models with different latency configurations:

# The 80ms model with punctuation is included in models/fastconformer-en-80ms-transducer/
# (converted from nvidia/stt_en_fastconformer_hybrid_medium_streaming_80ms_pc)

# The 1040ms model (higher accuracy, no built-in punctuation) is included in:
# models/fastconformer-en-1040ms-transducer/
# (converted from nvidia/stt_en_fastconformer_hybrid_large_streaming_1040ms)

Note: The 1040ms model doesn't have built-in punctuation support (NVIDIA doesn't offer a _pc variant for 1040ms). Enable the punctuation post-processor in config to add punctuation.

Parakeet EOU Model

NVIDIA Parakeet Realtime EOU 120M for low-latency English streaming:

# Download from HuggingFace or use the included model
# Place in: models/parakeet-realtime-eou-120m-v1-onnx/

Configuration

Create a config.toml file or set ASR_CONFIG environment variable:

bind_address = "127.0.0.1:8081"

[asr]
default_language = "en"
provider = "cpu"
num_threads = 4
sample_rate = 16000

# Punctuation post-processing (adds commas, periods, capitalization)
[asr.punctuation]
enabled = true
model_path = "./models/sherpa-onnx-punct-ct-transformer-zh-en-vocab272727-2024-04-12/model.onnx"
num_threads = 2

# Pseudo-streaming for offline models (Whisper, NeMo)
[asr.pseudo_streaming]
enabled = true
chunk_duration_seconds = 1.6
total_buffer_seconds = 4.0

# Estonian/English streaming model
[asr.models.et]
model_path = "./models/streaming-zipformer-large.et-en/encoder.onnx"
tokens_path = "./models/streaming-zipformer-large.et-en/tokens.txt"
name = "Estonian/English Zipformer"
language_code = "et"
model_type = "Transducer"

# English Parakeet EOU (low-latency streaming)
[asr.models.en]
model_path = "./models/parakeet-realtime-eou-120m-v1-onnx"
name = "Parakeet Realtime EOU 120M v1 (English)"
language_code = "en"
model_type = "ParakeetEOU"
sample_rate = 16000

# English FastConformer 1040ms (high accuracy streaming)
[asr.models.fastconformer_en_1040ms]
model_path = "./models/fastconformer-en-1040ms-transducer/encoder.onnx"
tokens_path = "./models/fastconformer-en-1040ms-transducer/tokens.txt"
name = "FastConformer English Streaming 1040ms"
language_code = "en"
model_type = "Transducer"
sample_rate = 16000

[audio]
buffer_size = 8192
max_chunk_size = 4096
silence_threshold = 0.01
min_speech_duration_ms = 500

[processing]
max_concurrent_sessions = 100
processing_threads = 4
session_timeout_seconds = 300

Project Structure

src/
├── main.rs              # Axum server setup and routing
├── lib.rs               # Library exports
├── config.rs            # Configuration management (models, punctuation, pseudo-streaming)
├── asr.rs               # ASR service with thread pool and punctuation integration
├── parakeet_backend.rs  # Parakeet EOU model backend
└── websocket.rs         # WebSocket message handling

Sherpa-ONNX Integration

Real sherpa-onnx is now integrated!

The server uses:

Model: Estonian-English streaming zipformer from TalTechNLP
Dependency: Git version of sherpa-rs (bypasses checksum issues)
Architecture: Transducer model with encoder, decoder, and joiner
Languages: Estonian and English speech recognition

Troubleshooting sherpa-rs

If you encounter checksum errors with the published sherpa-rs crate:

# Use git dependency instead of crates.io version
sherpa-rs = { git = "https://github.com/thewh1teagle/sherpa-rs.git", features = ["download-binaries"] }

Library Linking Issues (sherpa-onnx)

If you encounter runtime errors like libsherpa-onnx-c-api.so => not found or fatal runtime error: Rust cannot catch foreign exceptions, this indicates that the sherpa-onnx C library is not found at runtime.

Solution:

Locate the sherpa-onnx library: Find where the library was installed. Common locations:

# Check if you have a working installation (e.g., from desktop-asr project)
find /home/$USER -name "libsherpa-onnx-c-api.so" 2>/dev/null

Set library path before running:

export LD_LIBRARY_PATH="/path/to/sherpa/lib:$LD_LIBRARY_PATH"
cargo run

For permanent solution, copy to system location:

sudo cp /path/to/libsherpa-onnx-c-api.so /usr/local/lib/
sudo ldconfig

Example with working desktop-asr setup:

export LD_LIBRARY_PATH="/home/aivo/dev2/desktop-asr/dist/lib:$LD_LIBRARY_PATH"
cargo run

Note: This issue typically occurs after a cargo clean rebuild or when sherpa-rs downloads/builds new binaries but doesn't place them in the system library path.

Development

cargo check - Type check
cargo test - Run tests
cargo run - Start server
RUST_LOG=debug cargo run - Debug logging

Testing

Start the server: cargo run
Open the test client: Open client_test.html in your browser
Test the pipeline:
- Click "Connect" to establish WebSocket connection
- Click "Start Session" to begin ASR session
- Click "Send Test Audio Data" to test transcription
- Watch real-time results in the output panel

Performance Notes

Concurrent Connections: Handles thousands of WebSocket connections
CPU Utilization: Dedicated thread pool prevents blocking async runtime
Memory Efficient: Rust's zero-cost abstractions and careful memory management
Model Size: ~720MB Estonian-English model with high accuracy

License

MIT License

Name		Name	Last commit message	Last commit date
models		models
scripts		scripts
src		src
tests		tests
.gitattributes		.gitattributes
.gitignore		.gitignore
.gitmodules		.gitmodules
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
README.md		README.md
client_test.html		client_test.html
config.toml		config.toml
ecosystem.config.js		ecosystem.config.js
rust-asr-server.service		rust-asr-server.service

Latest commit

History

Folders and files

Repository files navigation

Rust Streaming ASR WebSocket Server

Architecture

Features

Quick Start

Prerequisites

Running the Server

Offline Transcription (Whisper)

WebSocket Protocol

Client Messages

Server Messages

Real-Time Speaker Diarization

Model Setup

Quick Setup

Estonian-English Streaming Model

English Streaming Model

Parakeet TDT v3 (25 European Languages)

Included Models (Git LFS)

Model Directory Structure

Whisper Models (Offline Recognition)

Punctuation Model

Transducer Models (RNN-T)

Model Types

Configuration

Available Transducer Models

Converting NeMo Models to ONNX

FastConformer Models

Parakeet EOU Model

Configuration

Project Structure

Sherpa-ONNX Integration

Troubleshooting sherpa-rs

Library Linking Issues (sherpa-onnx)

Development

Testing

Performance Notes

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages