VM Migration Compression Benchmark: Intel QAT vs CPU

Comprehensive performance analysis of Intel Quick Assist Technology (QAT) hardware acceleration vs CPU-based compression for VM live migration workloads.

Executive Summary

TL;DR: For VM migration, use CPU compression. QAT is 13.3x slower.

Metric	CPU (zstd)	QAT (async)	Winner
Throughput (64GB)	11.47 GB/s	0.86 GB/s	CPU (13.3x)
Migration Time	5.58s	74.5s	CPU (68.9s faster)
CPU Cores Used	4 cores	0.87 cores	QAT (saves 3 cores)
Compression Ratio	1.77x	0.95x (expands!)	CPU (1.86x better)
Complexity	Simple	Very complex	CPU

Verdict: On a 128-core system, saving 3 cores is not worth 68.9 seconds of extra VM downtime per migration.

Background

Motivation

VM live migration requires compressing memory pages quickly to minimize downtime. Modern CPUs offer:

Software compression (zstd): Mature, optimized, adaptive algorithms
Hardware acceleration (Intel QAT): Dedicated compression engines

Question: Does QAT hardware acceleration outperform CPU software compression for VM memory?

Expected vs Reality

Expected: Intel QAT = 20 CPUs worth of performance
Reality: Intel QAT = 0.3 CPUs worth of performance (for this workload)

Why the gap? Realistic VM memory is nearly incompressible, and QAT's DEFLATE algorithm takes 33.64ms per 32MB chunk regardless of compressibility, while CPU zstd detects incompressible data in 50µs and switches to a fast-path.

Hardware Setup

Test System

Server: Dell PowerEdge (Granite Rapids platform)

CPU: Intel Xeon 6776P (128 cores, Granite Rapids)
Memory: DDR5 @ 8800 MT/s
QAT: Intel 420xx integrated (17 acceleration engines)
OS: Ubuntu 24.04 LTS, Kernel 6.8.0-110

QAT Configuration

Hardware:

Device: Intel 420xx (PCI 0000:01:00.0)
Engines: 17 acceleration engines (9 for compression)
VF Instances: 4 DC instances via qatmgr
Firmware: qat_420xx.bin

Software:

qatlib: v26.02 (built from source)
Driver: qat_420xx (in-tree kernel driver)
Configuration: dc mode (compression only)

Benchmark Results

64GB Realistic VM Memory Data

Characteristics: Random data, compression ratio 0.95x (expands!)

Tool	Implementation	Throughput	Time	CPU Usage
compress_cpu	zstd level 3, 4 threads	11.47 GB/s	5.58s	4 cores
qat_compress_sync	QAT synchronous	0.41 GB/s	157.7s	0.4 cores
qat_compress_async	QAT async pipeline (depth 32)	0.86 GB/s	74.5s	0.87 cores
qat_compress_multi	QAT 4 instances parallel	0.82 GB/s	78.5s	0.87 cores

Winner: CPU by 13.3x

4GB Highly Compressible Data

Characteristics: All zeros, compression ratio 81.6x

Tool	Throughput	Time
compress_cpu	9.90 GB/s	0.40s
qat_compress_async	1.85 GB/s	2.16s

Winner: CPU by 5.4x

Detailed Performance Breakdown

Per 32MB chunk timing (realistic data):

Phase	CPU (zstd)	QAT
Disk I/O	4.0 ms (80%)	4.0 ms (10%)
Compression	0.01 ms (0.2%)	33.64 ms (84%) ⚠️
Overhead	0.02 ms (0.4%)	1.6 ms (4%)
Write output	1.0 ms (20%)	1.0 ms (2%)
Total	5.06 ms	40.09 ms

Bottleneck:

CPU: Disk I/O (compression is nearly free)
QAT: Hardware DEFLATE engine (33.64ms wait time)

Root Cause Analysis

Why QAT is 13.3x Slower

1. Incompressible Data Detection (336x difference!)

CPU (zstd):

Check compressibility: 50-100 µs
If incompressible:
    Fast-path memcpy: 10 µs (3-4 GB/s)
Total: ~100 µs

QAT (DEFLATE):

Always run full DEFLATE: 33,640 µs
No early abort
No incompressible detection
Total: 33,640 µs

Result: CPU is 336x faster at handling incompressible data.

2. Hardware Architecture Limitations

QAT per-instance throughput:

Compressible data: 3.8 GB/s (8.41ms per 32MB)
Incompressible data: 0.95 GB/s (33.64ms per 32MB)

CPU per-core throughput:

Both data types: 2.87 GB/s (adaptive algorithm)

Result: Single CPU core > Single QAT VF instance

3. Memory Efficiency

CPU: 1 copy (disk → memory, process in-place)
QAT: 4 copies (disk → mem → DMA → QAT → DMA → output)

Result: 4x more memory operations for QAT

4. Parallelism

CPU: True 4-way parallelism (independent threads on separate cores)
QAT: Limited parallelism (4 VF instances share 17 engines)

Result: CPU achieves 4x speedup, QAT achieves <1.1x

Measured Hardware Performance

Diagnostic tool results:

Realistic data (32MB chunks, 100 samples):
  QAT compression time: 33.63 ms (avg)
  Hardware rate: 951 MB/s = 0.93 GB/s
  Variation: 33.56 - 33.71 ms (very consistent)

Compressible data (zeros, 32MB chunks):
  QAT compression time: 8.41 ms (avg)
  Hardware rate: 3,805 MB/s = 3.72 GB/s
  Variation: 8.33 - 8.48 ms (very consistent)

Conclusion: QAT hardware is working correctly at its design limits. The issue is fundamental to DEFLATE on incompressible data.

Software Stack Comparison

CPU (Simple, 4 Layers)

Application (compress_cpu.c)
    ↓
libzstd.so (40K lines, pure software)
    ↓
Kernel VFS (standard Linux)
    ↓
CPU cores + SIMD (AVX-512)

Features:

Direct memory access (zero-copy)
Incompressible detection (50µs)
SIMD acceleration (AVX-512)
Simple threading model

QAT (Complex, 6+ Layers)

Application (qat_compress_async.c)
    ↓
libqat.so + libusdm.so (110K lines)
    ↓
qatmgr daemon (VF allocation)
    ↓
qat_420xx.ko + vfio-pci (50K lines)
    ↓
Firmware (qat_420xx.bin)
    ↓
QAT 420xx hardware (17 engines)

Overhead:

DMA buffer allocation (VFIO constraints)
4 memory copies
PCIe transfers (~900µs)
Polling/callback mechanism
Complex async pipeline

See: SOFTWARE_STACK_COMPARISON.md for detailed analysis.

Repository Structure

vm-migration-benchmark/
├── README.md                           # This file
├── LICENSE                             # MIT License
│
├── tools/                              # Benchmark tools
│   ├── compress_cpu.c                  # CPU compression (zstd)
│   ├── qat_compress_bench.c            # QAT synchronous
│   ├── qat_compress_async.c            # QAT async pipeline ⭐
│   ├── qat_compress_multi.c            # QAT multi-instance
│   ├── qat_diagnose.c                  # Detailed timing tool
│   ├── test_qat_instances.c            # Instance detection
│   └── Makefile                        # Build all tools
│
├── data/                               # Test datasets
│   ├── test_4G.bin                     # 4GB realistic data
│   ├── test_4G_zeros.bin               # 4GB compressible
│   ├── test_64G.bin                    # 64GB realistic data
│   └── test_64G_zeros.bin              # 64GB compressible
│
├── scripts/                            # Automation scripts
│   ├── setup_qat.sh                    # QAT configuration
│   ├── run_benchmarks.sh               # Run all tests
│   └── generate_test_data.sh           # Create test files
│
├── results/                            # Benchmark outputs
│   ├── cpu_results.json
│   ├── qat_async_results.json
│   └── comparison.csv
│
└── docs/                               # Detailed documentation
    ├── QAT_SUCCESS_REPORT.md           # Initial QAT setup
    ├── QAT_INVESTIGATION_REPORT.md     # Root cause analysis
    ├── QAT_PERFORMANCE_ANALYSIS.md     # Detailed measurements
    ├── QAT_ASYNC_BENCHMARK_RESULTS.md  # Async optimization
    ├── QAT_FINAL_RESULTS.md            # Final verdict
    ├── SOFTWARE_STACK_COMPARISON.md    # Stack analysis
    ├── CODE_FLOW_COMPARISON.txt        # Code execution flow
    └── STACK_COMPARISON_VISUAL.txt     # Visual diagrams

Building and Running

Prerequisites

For CPU benchmarks:

sudo apt-get install libzstd-dev build-essential

For QAT benchmarks:

# Build qatlib from source (required for Granite Rapids)
git clone https://github.com/intel/qatlib.git
cd qatlib
./autogen.sh
./configure --prefix=/usr/local
make -j$(nproc)
sudo make install
sudo ldconfig

# Configure QAT device
sudo sh -c "echo dc > /sys/devices/pci*/*/qat/cfg_services"
sudo sh -c "echo up > /sys/devices/pci*/*/qat/state"
sudo systemctl restart qat.service

Build Tools

cd tools
make all

# Or individually:
gcc -o compress_cpu compress_cpu.c -lzstd -lpthread -O3
gcc -o qat_compress_async qat_compress_async.c \
    -I/usr/local/include/qat -L/usr/local/lib \
    -lqat -lusdm -lpthread -O3

Generate Test Data

cd data
# 4GB realistic (random data)
dd if=/dev/urandom of=test_4G.bin bs=1M count=4096

# 4GB compressible (zeros)
dd if=/dev/zero of=test_4G_zeros.bin bs=1M count=4096

# 64GB realistic
dd if=/dev/urandom of=test_64G.bin bs=1M count=65536

# 64GB compressible
dd if=/dev/zero of=test_64G_zeros.bin bs=1M count=65536

Run Benchmarks

CPU compression:

./tools/compress_cpu data/test_64G.bin /tmp/cpu_output.zst 3 4

QAT async compression:

export LD_LIBRARY_PATH=/usr/local/lib
./tools/qat_compress_async data/test_64G.bin /tmp/qat_output.qat 3

QAT diagnostics:

export LD_LIBRARY_PATH=/usr/local/lib
./tools/qat_diagnose data/test_64G.bin

Automated Testing

./scripts/run_benchmarks.sh
# Outputs: results/comparison.csv

Key Findings

1. QAT is Working Correctly

✅ Hardware is functional
✅ Firmware loaded properly
✅ 4 DC instances available
✅ Async pipeline implemented optimally
✅ Achieving 92% of hardware maximum (0.86 / 0.93 GB/s)

The issue is not the implementation - it's the fundamental use case mismatch.

2. Incompressible Data is the Problem

Compression ratio results:

Realistic VM memory: 0.95x (data expands by 5%)
Zeros (compressible): 81.6x (excellent compression)

QAT performance:

Realistic: 0.86 GB/s (13.3x slower than CPU)
Zeros: 1.85 GB/s (5.4x slower than CPU)

Even with highly compressible data, CPU still wins.

3. CPU zstd is Exceptionally Well Optimized

Key optimizations:

Entropy sampling (detects incompressible in 50µs)
Fast-path for incompressible data (3-4 GB/s memcpy)
SIMD acceleration (AVX-512)
Adaptive algorithm selection
Zero-copy processing

QAT has none of these.

4. Wrong Expectations

Where "QAT = 20 CPUs" comes from:

Aggregate throughput (all 16 VFs with compressible data)
Crypto + compression pipelines (QAT excels here)
Comparison to old/slow CPUs
Marketing benchmarks with ideal data

For single-stream VM memory migration:

QAT = 0.3 CPUs worth of performance
Not competitive with modern CPU compression

5. When QAT Would Actually Help

QAT is beneficial when ALL of these apply:

✅ Highly compressible data (>10x ratio)
✅ Crypto + compression pipeline
✅ Many concurrent streams (50+)
✅ CPU extremely constrained (<5 cores)
✅ DEFLATE compatibility required

VM migration meets 0 of these 5 criteria.

Documentation

Reports (in `docs/`)

QAT_SUCCESS_REPORT.md
- Initial QAT hardware setup
- Building qatlib v26.02 from source
- Verification with Intel samples
QAT_PERFORMANCE_ANALYSIS.md
- Detailed timing measurements
- Root cause investigation
- Per-chunk breakdown
QAT_ASYNC_BENCHMARK_RESULTS.md
- Async pipeline optimization
- Multi-instance testing
- Configuration tuning
QAT_FINAL_RESULTS.md
- SRIOV reconfiguration attempts
- Final performance numbers
- Comprehensive conclusion
SOFTWARE_STACK_COMPARISON.md
- Complete stack analysis
- Layer-by-layer comparison
- Overhead breakdown
CODE_FLOW_COMPARISON.txt
- Step-by-step execution paths
- Function call traces
- Timing annotations

Key Metrics

Throughput:

CPU: 11.47 GB/s
QAT: 0.86 GB/s
Ratio: 13.3x in favor of CPU

Latency (64GB migration):

CPU: 5.58 seconds
QAT: 74.5 seconds
Difference: 68.9 seconds extra downtime

CPU Efficiency:

CPU: 0.35 core-sec/GB
QAT: 1.01 core-sec/GB
CPU is 2.9x more efficient per GB

Conclusion

For VM Live Migration: Use CPU Compression

Reasons:

13.3x faster throughput (11.47 vs 0.86 GB/s)
68.9 seconds less downtime per 64GB migration
Better compression (1.77x vs 0.95x ratio)
Simpler stack (no special drivers, firmware, daemons)
More CPU efficient per GB (0.35 vs 1.01 core-sec/GB)

On a 128-core system:

Using 4 cores for 5.58 seconds = 0.17% of total capacity
Not worth 68.9 seconds of extra VM downtime

QAT Recommendation

Consider QAT only if:

Need crypto + compression (AES-GCM + DEFLATE together)
Many concurrent streams (100+ simultaneous operations)
Data is highly compressible (logs, backups, database dumps)
CPU is critically constrained (embedded, edge devices)
DEFLATE compatibility is mandatory

For VM migration: All conditions are NOT met.

Performance Summary Table

Workload	CPU	QAT	Winner	Gap
64GB realistic	11.47 GB/s	0.86 GB/s	CPU	13.3x
4GB compressible	9.90 GB/s	1.85 GB/s	CPU	5.4x
Compression ratio (realistic)	1.77x	0.95x	CPU	1.86x better
Compression ratio (zeros)	32,753x	81.6x	CPU	401x better
CPU cores used	4	0.87	QAT	Saves 3 cores
Core-seconds per GB	0.35	1.01	CPU	2.9x efficient
Migration time (64GB)	5.58s	74.5s	CPU	68.9s faster
Code complexity	Simple	Complex	CPU	Much simpler
Setup difficulty	Easy	Hard	CPU	apt install vs build from source

Overall winner: CPU by every metric except core count

License

MIT License - See LICENSE file for details.

Contributing

This is a research/benchmarking project. Contributions welcome:

Additional benchmark scenarios
Different QAT configurations
Alternative compression algorithms
Performance optimizations

Citation

If you use this benchmark in your research, please cite:

VM Migration Compression Benchmark: Intel QAT vs CPU
https://github.com/[username]/vm-migration-benchmark
Date: 2026-07-09

Contact

For questions or issues, please open a GitHub issue.

Last Updated: 2026-07-09
Test System: Intel Xeon 6776P (Granite Rapids) with QAT 420xx
QAT Version: qatlib v26.02
Conclusion: Use CPU compression for VM migration

Name		Name	Last commit message	Last commit date
configs		configs
docs		docs
scripts		scripts
tools		tools
.gitignore		.gitignore
GITHUB_SETUP.md		GITHUB_SETUP.md
LICENSE		LICENSE
QUICKSTART.md		QUICKSTART.md
README.md		README.md
code_flow_comparison.txt		code_flow_comparison.txt

Latest commit

History

Folders and files

Repository files navigation