A small machine-learning framework built from scratch in C++20 and SYCL, with a working GPT-2 Small inference pipeline.
tinyml is an educational project for understanding how neural-network operations map to GPU kernels. It keeps tensor storage, device memory, matrix multiplication, neural-network layers, attention, Transformer blocks, and model inference explicit rather than hiding them behind a large machine-learning framework.
The project uses Intel oneAPI icpx, SYCL, and an Intel integrated GPU. GPT-2 Small provides a real pretrained model against which the implementation can be tested.
The current implementation supports text tokenization, pretrained-weight loading, the full 12-block GPT-2 Small forward pass, next-token prediction, and autoregressive text generation. The complete model has also been compared numerically against a reference implementation using the same checkpoint.
The main inference pipeline is operational.
| Component | Status |
|---|---|
| USM device memory and tensor storage | Implemented |
| Basic tensor operations | Implemented |
| Naive and tiled matrix multiplication | Implemented |
| Linear layers and activation functions | Implemented |
| Serial and parallel softmax | Implemented |
| Serial and parallel LayerNorm | Implemented |
| Unfused scaled dot-product attention | Implemented |
| Fused online-softmax attention | Implemented |
| Multi-head and causal attention | Implemented |
| Transformer block | Implemented and tested against a CPU reference |
| GPT-2 token and positional embeddings | Implemented |
| GPT-2 pretrained-weight export and loading | Implemented |
| Assembly of all 12 GPT-2 Transformer blocks | Implemented |
| Final LayerNorm and language-model output projection | Implemented |
| C++ GPT-2 tokenization and decoding | Implemented using tokenizers-cpp |
| Autoregressive text generation | Implemented |
| Numerical comparison against Hugging Face GPT-2 | Tested on multiple prompts |
| KV-cache optimization | Not implemented |
| Sampling strategies such as temperature or top-k | Not implemented |
| Automatic differentiation and training | Not implemented |
The primary objective is correctness and understanding. Performance optimization follows once the computation can be validated against a trusted reference.
The complete model has been compared against Hugging Face GPT-2 using the same local checkpoint, the same input token IDs, and CPU reference inference.
The test exports the reference model's final-position logits and compares them with the corresponding logits generated by the SYCL implementation.
Each comparison covers all 50,257 vocabulary logits, not just the highest-scoring token.
| Input | Input tokens | Maximum absolute error | RMSE | Top-1 match | Top-10 overlap |
|---|---|---|---|---|---|
Hello |
1 | 0.0003052 | 0.0002148 | Yes | 10/10 |
The capital of Finland is |
5 | 0.0001907 | 0.0000452 | Yes | 10/10 |
Nice to meet you . My name is cris. What's your name? |
15 | 0.0001984 | 0.0000434 | Yes | 10/10 |
The quick brown fox jumps over the lazy dog. This is a longer test of the model. |
19 | 0.0003357 | 0.0000854 | Yes | 10/10 |
For these tested inputs, the SYCL logits closely match the reference logits. The observed maximum absolute errors are between approximately (1.9 \times 10^{-4}) and (3.4 \times 10^{-4}). The top-10 predicted token IDs match exactly in every listed case.
This is strong evidence that the complete forward pass reproduces the reference implementation for the tested inputs. It is not proof of correctness for every possible sequence length, input, batch size, or device configuration.
A single pre-LayerNorm Transformer block was separately validated against CPU reference calculations at GPT-2 Small dimensions:
B = 2
L = 1024
d_model = 768
heads = 12
d_head = 64
FF = 3072
dtype = float32
The test compares intermediate tensors and the final block output, including LayerNorm, Q/K/V projections, causal attention, output projection, residual connections, feed-forward projections, and GELU.
All comparisons passed with an absolute tolerance of 5e-4. In the reported run, the maximum absolute error was approximately 4.77e-7 for the second LayerNorm and 7.45e-8 for the final output.
The block test and the full-model logit comparison validate different aspects of the implementation. Both are useful: the block test isolates individual operations, while the full-model test checks the assembled inference pipeline.
The project is organized around several layers of abstraction:
- Device memory: Owns allocations on the GPU.
- Tensors: Store multidimensional shapes and device pointers.
- Operations: Implement mathematical computations using SYCL kernels.
- Neural-network layers: Compose operations into linear transformations, embeddings, and normalization.
- Attention and Transformer blocks: Combine these layers into the architecture used by GPT-2.
- GPT-2 model: Loads pretrained parameters and assembles the complete neural network.
- Text generation: Converts a text prompt into token IDs, runs inference, and repeatedly predicts new tokens.
The lower layers are designed to be reusable. The GPT-2-specific code uses them to reproduce a particular pretrained model.
See docs/ARCHITECTURE_AND_TUTORIAL.md for a beginner-oriented explanation of SYCL, tensors, attention, Transformer blocks, GPT-2, and the source-code execution path.
The pretrained checkpoint targets the following architecture:
Vocabulary size = 50257
Context length = 1024
Hidden dimension = 768
Attention heads = 12
Dimension per head = 64
Transformer blocks = 12
Feed-forward dimension = 3072
Data type = float32
At a high level, the model performs the following computation:
Text prompt
|
v
Tokenizer
|
v
Token IDs [B, L]
|
v
Token embeddings + positional embeddings
|
v
12 Transformer blocks
|
v
Final LayerNorm
|
v
Language-model output projection
|
v
Logits [B, L, 50257]
|
v
Select the next token
|
v
Append token to the prompt and repeat
The output projection uses the token-embedding matrix as the output weight matrix, following GPT-2's weight-tying design.
The current generation loop uses greedy decoding: it selects the token with the largest logit. It recomputes the model for the full context at every generation step. A KV cache and more advanced decoding strategies remain possible future improvements.
GPT-2 Small is a pretrained next-token prediction model, not an instruction-tuned conversational assistant. Repetitive or unhelpful generated text does not by itself indicate a numerical implementation error.
DeviceBuffer<T> provides RAII-style ownership of SYCL USM device allocations using sycl::malloc_device. Owned memory is released automatically, and accidental shallow copies are prevented.
Tensor<T> stores a multidimensional shape, element count, and device pointer.
Example:
tinyml::Tensor<float> X(queue, {32, 128});This creates a tensor with shape [32, 128], containing 4,096 floating-point elements in device memory.
Memory transfers between CPU and GPU are explicit. This is useful for learning how data movement affects the performance and correctness of GPU programs.
Operations are implemented in include/tinyml/ops.hpp. The current implementation includes:
- Host-to-device and device-to-host copies
- Elementwise arithmetic and addition
- Fill and scalar multiplication
- Bias addition
- ReLU
- Exact GELU and tanh-approximate GELU
- Serial and parallel softmax
- Serial and parallel LayerNorm
- Affine parallel LayerNorm
- Naive matrix multiplication
- Tiled matrix multiplication
- Tiled matrix multiplication with a transposed second operand
- Scaled tiled multiplication with a transposed second operand
- Tiled online-softmax weighted-value products
- Fused scaled dot-product attention using online softmax
- Multi-head attention
- Strided multi-head attention on
[B, L, d_model]tensors - Causal strided multi-head attention
Operations return sycl::event objects where applicable. Event dependencies express the ordering between asynchronous GPU operations.
Several matrix-multiplication kernels are available, including naive and tiled implementations. The tiled implementations use SYCL local memory to reuse data within work-groups.
The transposed operation computes
[ C = AB^T ]
for A with shape [M, K], B with shape [N, K], and C with shape [M, N]. It avoids creating a physical transposed copy of B.
A scaled variant computes
[ C = \frac{AB^T}{\sqrt{K}}. ]
This operation is useful for attention scores when A = Q, B = K, and K is the attention head dimension.
These kernels are learning and optimization exercises, not replacements for tuned BLAS libraries.
Linear<T> implements a fully connected layer:
[ Y = XW + b ]
where X contains the input features, W contains the weights, and b contains the bias.
The layer supports Transformer-style 3D tensors such as [B, L, in_features]. The contiguous tensor can be treated as a [B*L, in_features] matrix for multiplication without physically rearranging the data.
The project also implements ReLU, exact GELU, tanh-approximate GELU, softmax, and affine LayerNorm.
The project contains multiple scaled dot-product attention implementations to expose their mathematical and memory-access trade-offs.
The standard operation is
[ \operatorname{Attention}(Q,K,V) = \operatorname{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V. ]
The unfused path materializes the score and softmax-weight matrices. The online-softmax implementations maintain the running maximum, normalization factor, and weighted-value accumulator as tiles are processed, avoiding the need to materialize the complete softmax-weight matrix.
The fused implementation is inspired by the blockwise online-softmax approach used by memory-efficient algorithms such as FlashAttention. It is an educational implementation, not a reproduction of an optimized FlashAttention kernel.
Multi-head attention operates on projected queries, keys, and values. The strided implementation accesses individual heads directly within [B, L, d_model] tensors without physically rearranging them to [B, H, L, d_head].
The causal attention kernel restricts each query to positions at or before itself, preventing a token from attending to future positions.
TransformerBlock<T> implements a pre-LayerNorm Transformer block.
Its data path is:
Input
|
v
LayerNorm
|
v
Q / K / V projections
|
v
Causal multi-head attention
|
v
Attention output projection
|
v
Residual addition
|
v
LayerNorm
|
v
Feed-forward projection: 768 -> 3072
|
v
GELU
|
v
Feed-forward projection: 3072 -> 768
|
v
Residual addition
|
v
Output
The block owns its attention projections, feed-forward layers, and learned LayerNorm parameters. Its forward() method composes the operations through SYCL event dependencies.
GPT-2 uses two embedding tables:
- Token embeddings: map each token ID to a vector of 768 values.
- Positional embeddings: map each sequence position to a vector of 768 values.
The two vectors are added before the first Transformer block.
The original checkpoint is stored at models/gpt2/model.safetensors. Export scripts create binary parameter files for the C++ implementation, including the token embeddings, positional embeddings, each Transformer block's parameters, and the final LayerNorm parameters.
The model loader loads these parameters and assembles all 12 blocks. The language-model output projection uses the token-embedding weights, consistent with GPT-2's weight tying.
GPT2Tokenizer is a C++ wrapper around mlc-ai/tokenizers-cpp. It loads the GPT-2 tokenizer.json file and supports encoding text to token IDs, decoding token IDs to text, and querying the vocabulary size.
The tokenizer is separate from the neural network. It converts the text representation into the integer inputs expected by the model and converts generated token IDs back into text.
src/gpt2_generate.cpp implements the generation loop.
At each step, it:
- Takes the current token IDs.
- Copies them to device memory.
- Runs
GPT2Model::forward()using SYCL. - Copies the final sequence position's logits back to the CPU.
- Selects the token with the largest logit.
- Appends the token to the context.
- Repeats until the requested number of tokens is generated or the end-of-text token is produced.
The implementation uses greedy decoding. It does not currently use a KV cache, temperature sampling, top-k sampling, or a repetition penalty.
The following is a representative structure of the current project:
tinyml/
├── CMakeLists.txt
├── README.md
├── docs/
│ └── ARCHITECTURE_AND_TUTORIAL.md
├── include/
│ └── tinyml/
│ ├── device_buffer.hpp
│ ├── tensor.hpp
│ ├── ops.hpp
│ ├── linear.hpp
│ ├── embedding.hpp
│ ├── gpt2_embedding.hpp
│ ├── gpt2_layer_norm.hpp
│ ├── gpt2_weights.hpp
│ ├── gpt2_block_weights.hpp
│ ├── gpt2_model.hpp
│ ├── gpt2_tokenizer.hpp
│ ├── transformer_block.hpp
│ └── flash_attention.hpp
├── src/
│ ├── main.cpp
│ ├── gpt2_generate.cpp
│ └── gpt2_dump_logits.cpp
├── tools/
│ ├── export_gpt2_embeddings.py
│ ├── export_gpt2_block.py
│ ├── export_gpt2_final_norm.py
│ ├── reference_gpt2_block.py
│ ├── reference_gpt2_logits.py
│ ├── compare_gpt2_logits.py
│ └── query_gpt2.py
├── models/
│ └── gpt2/
│ ├── model.safetensors
│ ├── tokenizer/
│ │ ├── tokenizer.json
│ │ └── tokenizer_config.json
│ ├── token_embeddings.bin
│ ├── position_embeddings.bin
│ ├── final_ln_gamma.bin
│ ├── final_ln_beta.bin
│ ├── block0/
│ ├── ...
│ └── block11/
├── external/
│ └── tokenizers-cpp/
├── tests/
│ ├── test_core.cpp
│ ├── test_nn.cpp
│ ├── test_sdpa.cpp
│ ├── test_attention.cpp
│ ├── test_flash_attention.cpp
│ ├── test_transformer_block.cpp
│ ├── test_embedding.cpp
│ ├── test_gpt2_embedding.cpp
│ ├── test_gpt2_embedding_validation.cpp
│ ├── test_gpt2_block.cpp
│ ├── test_gpt2_model.cpp
│ ├── test_gpt2_weights.cpp
│ ├── benchmark_ops.cpp
│ └── benchmark_flash_attention.cpp
└── build/
The list is representative. The exact contents of a checkout may differ as development continues. Generated model files and build artifacts should not be assumed to be source-controlled.
The project currently uses:
- C++20
- Intel oneAPI
icpx - SYCL with the
spir64target - An available SYCL GPU device
- CMake 3.20 or newer
- The
tokenizers-cppdependency inexternal/tokenizers-cpp
Python is only required for the export scripts, the Hugging Face tokenizer wrapper, and numerical reference tests. The direct C++ inference executable does not require Python during generation.
For numerical reference testing, the Python environment also needs NumPy, PyTorch, Transformers, and safetensors.
Configure the main project:
cmake -S . -B build \
-DCMAKE_CXX_COMPILER=icpx \
-DTINYML_BUILD_TESTS=ON \
-DTINYML_BUILD_GPT2_WEIGHTS_TEST=ON \
-DTINYML_BUILD_GPT2_BLOCK_TEST=ON \
-DTINYML_BUILD_GPT2_MODEL_TEST=ON \
-DTINYML_BUILD_GPT2_GENERATE=ON \
-DTINYML_BUILD_GPT2_DUMP_LOGITS=ONBuild the GPT-2 text-generation executable:
cmake --build build --target gpt2_generate -j2Run inference from the repository root:
./build/gpt2_generate --max-new-tokens 20 "The capital of Finland is"The executable uses the C++ tokenizer, runs model inference through SYCL, and decodes the generated tokens in C++.
The model parameters and tokenizer are loaded through paths relative to the repository root, so run the executable from that directory.
cmake --build build --target gpt2_dump_logits -j2This executable reads token IDs from build/reference_input_ids.txt, runs the SYCL model, and writes the final-position logits to build/sycl_logits.bin.
Build the available targets:
cmake --build build --target \
test_core \
test_nn \
test_sdpa \
test_attention \
test_flash_attention \
test_transformer_block \
test_embedding \
test_gpt2_embedding \
test_gpt2_embedding_validation \
test_gpt2_weights \
test_gpt2_block \
test_gpt2_model \
-j2Run the relevant tests:
./build/test_core
./build/test_nn
./build/test_sdpa
./build/test_attention
./build/test_flash_attention
./build/test_transformer_block
./build/test_embedding
./build/test_gpt2_embedding
./build/test_gpt2_embedding_validation
./build/test_gpt2_weights
./build/test_gpt2_block
./build/test_gpt2_modelThese commands require the corresponding targets to have been enabled during configuration. A target that is disabled or not defined in the current CMake configuration must be configured or added before it can be built.
The Python wrapper uses Hugging Face's tokenizer to turn text into token IDs, then passes those IDs to the C++ executable. Model inference is still performed by the C++/SYCL implementation.
source .venv/bin/activate
python tools/query_gpt2.py --max-new-tokens 20 "The capital of Finland is"This is useful for checking that both tokenizer paths pass equivalent token IDs to the same model. Python is not performing neural-network inference in this workflow.
The local Hugging Face checkpoint can be used to generate reference logits. This requires PyTorch and the Python reference dependencies.
Generate the reference logits:
python tools/reference_gpt2_logits.py "The capital of Finland is"Export the SYCL logits for the same input token IDs:
./build/gpt2_dump_logitsCompare all 50,257 vocabulary logits:
python tools/compare_gpt2_logits.pyThe scripts use:
build/reference_input_ids.txt
build/reference_logits.bin
build/sycl_logits.bin
The comparison reports maximum absolute error, mean absolute error, RMSE, centered-logit error, correlation, and agreement between the top predictions.
Running a new reference command overwrites the reference files. Run the diagnostic and comparison before changing to another prompt.
For the current reference script, use a prompt that fits GPT-2's 1,024-token context limit.
Performance benchmarks are separate from correctness tests. The benchmark programs explore execution time for operations such as GELU, tiled matrix multiplication, and attention.
The project currently contains both unfused and online-softmax attention implementations. The unfused path materializes intermediate score and probability matrices, while the fused implementation avoids storing the complete softmax-weight matrix.
The implementations remain educational kernels rather than highly optimized production equivalents. The next optimization opportunities include reducing intermediate global-memory traffic, improving work-group mapping, keeping more partial output state in registers or private storage, introducing a KV cache for generation, and tuning kernels after correctness is established.
- C++20 and SYCL
- Explicit USM device-memory ownership
- Explicit host/device transfers
- Small, understandable tensor and operation abstractions
- GPU kernels separated from neural-network layers
- Reusable Transformer components
- Correctness checks against CPU reference calculations
- Full-model validation against a trusted reference
- Separate correctness testing and performance benchmarking
- Optimization only after the computation can be measured and validated
- Implement device memory management and tensor storage
- Implement basic tensor operations and matrix multiplication
- Implement Linear layers and activation functions
- Implement serial and parallel softmax
- Implement serial and parallel LayerNorm
- Implement scaled dot-product and causal multi-head attention
- Implement online-softmax attention
- Implement a reusable Transformer block
- Validate the Transformer block at GPT-2 Small dimensions
- Implement the GPT-2 tokenizer wrapper
- Implement token and learned positional embeddings
- Export and load pretrained GPT-2 parameters
- Assemble the complete 12-block GPT-2 Small model
- Implement final LayerNorm and the language-model output projection
- Implement autoregressive text generation
- Add a C++ text-prompt interface
- Add a Python-tokenizer-to-C++-inference diagnostic path
- Compare full-model logits against Hugging Face GPT-2 for multiple inputs
- Expand full-model validation across more prompts and context lengths
- Add more randomized inputs and multiple seeds to the Transformer-block tests
- Expand shape coverage for batch size, sequence length, embedding dimension, and feed-forward width
- Profile end-to-end inference and generation time
- Implement a KV cache to avoid recomputing the entire context at every generation step
- Explore temperature sampling, top-k/top-p sampling, and repetition penalties
- Profile and optimize matrix multiplication and attention bottlenecks
- Reduce global-memory traffic in fused attention
- Explore keeping attention output fragments in registers or private storage
- Explore a Triton version of selected kernels
- Explore portability to other SYCL implementations
- Consider automatic differentiation and training as longer-term extensions
tinyml now runs pretrained GPT-2 Small inference through a C++/SYCL implementation on an Intel GPU. The full forward pass has been compared numerically against the local Hugging Face checkpoint on several inputs, with very small logit errors and matching top-10 predictions in the reported tests.