SparkMoE

Run giant local models on modest hardware.

SparkMoE is an independent, CPU-first fork of llama.cpp that can keep standard routed MoE expert tensors on a GGUF file and load only the experts selected by the router into bounded per-layer RAM caches.

This is v0.4.2-alpha, refreshed onto llama.cpp stable v0.3.0. The Windows regression suite includes synthetic full-logit equivalence, real two-shard GGUF, HTTP/Desktop controls and exact Ling-3.0-tiny Q4_K_M logits with matching CPU kernels and microbatch. Qwen3.8-Flash-Next now has a bounded PLE adapter, a separate native MTP sidecar path and tiny-fixture parity tests; full-model memory and speed validation is still pending. See supported models. This remains an alpha rather than a performance or memory-fit guarantee.

What it does

• Leaves non-routed weights on the normal llama.cpp path.
• Indexes routed expert tensor offsets across single or split GGUF files.
• Loads selected experts exactly, without zero weights, expert substitution, or approximation.
• Keeps bounded per-layer LRU by default, with optional hybrid global budgeting and runtime resize.
• Offers validated Windows/Linux direct I/O with buffered fallback, shared demand-priority workers, per-expert compute/I/O overlap and temporal prefetch.
• Exposes real cache/I/O statistics through C, llama-server, and llama-bench.
• Supports built-in MTP draft contexts together with bounded expert paging.
• Supports a compatible separate MTP GGUF and bounded PLE page cache for Qwen3.8-Flash-Next.
• Uses hardware-aware safe defaults and an automatic loading mode that falls back to standard mmap for dense or incompatible models.
• Keeps all SparkMoE controls in a dedicated page inside the original llama.cpp Web UI: Router limits, model profiles, loading, expert paging, context, MTP, RAM, logs, and diagnostics.
• Runs the launcher and every managed model process without a console window on Windows, while preserving complete per-session logs under the user's local application data.

It does not promise that every model or quantization fits in 16 GB. Advanced hybrid/overlap/prefetch modes require CPU-only inference. Buffered per-layer paging remains the default; direct I/O and speculative prefetch are not always faster. Explicit mode validates routed expert tensor names, shapes, strides, shard bounds, and active-expert capacity before inference. See engine measurements and refresh report.

Hardware

• Windows 10/11 x64 or Linux x86-64
• CPU with a supported public build variant
• SSD strongly recommended; HDD latency is normally unsuitable
• Enough RAM for dense/shared weights, context/KV cache, runtime overhead, and the configured expert cache

Quick start

Permanent Windows installation:

1. Use sparkmoe-0.4.2-alpha-windows-x64-avx2-setup.exe from your local artifacts/release build or a published release.
2. Run the installer. It installs for the current user, creates an uninstall entry, and can create desktop and Start menu shortcuts.
3. Run SparkMoE from the Start menu.
4. While SparkMoE is running, its notification-area icon provides Open, Router status, Start, Stop, and Exit actions without a command window.

Portable Windows package:

1. Extract the ZIP to a writable folder.
2. Run SparkMoE.exe.
3. The llama.cpp Web UI opens directly. Open SparkMoE in its sidebar and choose the folder that contains your GGUF models.
4. Select a model using the normal llama.cpp model picker or the SparkMoE model profile selector.
5. Keep Automatic loading unless you are testing a specific mode. Context length, MTP, loading, cache, logs, and diagnostics are all under SparkMoE.
6. Use the SparkMoE icon in the Windows notification area to reopen the UI, inspect Router state, start or stop the Router, or exit the complete application.

Dense and other non-MoE GGUF models use the normal llama.cpp path and are fully supported by the launcher. models-max is user configurable: use 1 for large models, a higher value for small models that fit together, or 0 for unlimited Router instances.

Build from source on Windows:

git clone https://github.com/menachem-dadon/SparkMoE.git
cd SparkMoE
pwsh scripts/build-windows.ps1 -Variant avx2 -RunTests
.\build-windows-avx2\bin\SparkMoE.exe

Use -Variant generic for older x64 CPUs without AVX2. Linux users can run ./scripts/build-linux.sh avx2 and then ./build-linux-avx2/bin/SparkMoE. See the platform build guides for dependencies and troubleshooting.

CLI:

llama-cli.exe -m C:\models\qwen.gguf --moe-paging explicit --moe-cache-mib 1536 --moe-io-threads 3 -p "Hello"

Server with context and MTP:

llama-server.exe -m C:\models\qwen.gguf --host 127.0.0.1 --port 8080 --parallel 1 --ctx-size 16384 --moe-paging explicit --moe-cache-mib 1536 --moe-io-threads 3 --spec-type draft-mtp --spec-draft-n-max 3

Qwen3.8-Flash-Next uses --model-draft for its separate compatible MTP sidecar. See the Qwen3.8 implementation and validation guide.

OpenAI-compatible request:

$body = @{ model = "local"; messages = @(@{ role = "user"; content = "Hello" }) } | ConvertTo-Json -Depth 4
Invoke-RestMethod http://127.0.0.1:8080/v1/chat/completions -Method Post -ContentType application/json -Body $body

The Router loads the requested model with its saved SparkMoE process profile. Generation settings remain request-specific. A SmartiAI request can disable thinking without reloading the model by sending reasoning_effort = "none" and chat_template_kwargs = @{ enable_thinking = $false }.

Paging modes:

• off: normal resident loading.
• os: normal mmap behavior and operating-system paging.
• explicit: SparkMoE expert slot caches and file reads.
• auto: chooses explicit when a cache/slot limit is supplied, otherwise OS paging. It is the SparkMoE profile default.

Run sparkmoe-iobench.exe --format json to measure the buffered file path. OS caching can affect the result.

Documentation

• Hebrew README
• User guide
• MoE paging
• Architecture
• Project map
• Licensing map
• Supported models
• API
• Windows build
• Future WebView2 desktop integration plan
• Linux build
• Benchmarking
• Troubleshooting

Privacy and security

SparkMoE has no telemetry, analytics, crash upload, or automatic model download. The launcher binds to 127.0.0.1. If you expose llama-server beyond localhost, configure its API key and network controls. See SECURITY.md.

Upstream and licensing

This repository contains a pinned llama.cpp snapshot plus SparkMoE changes. It does not automatically contain upstream features added after the commit recorded in UPSTREAM_BASE.txt. See the project map for the exact integration and synchronization model.

Upstream llama.cpp code and changes made directly in upstream-derived files remain under MIT. Original SparkMoE components are licensed under PolyForm Noncommercial 1.0.0. See LICENSING.md, LICENSE-SPARKMOE-NONCOMMERCIAL.md, THIRD_PARTY_NOTICES.md, and the model license guidance.

SparkMoE is an independent community project and is not affiliated with or endorsed by Qwen, Alibaba Cloud, Google, Gemma, ggml-org, or the Colibri authors.