Skip to content
csccvaPublic

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

24 Commits

Folders and files

Repository files navigation

tinyml

A small machine-learning framework built from scratch in C++20 and SYCL, with a working GPT-2 Small inference pipeline.

tinyml is an educational project for understanding how neural-network operations map to GPU kernels. It keeps tensor storage, device memory, matrix multiplication, neural-network layers, attention, Transformer blocks, and model inference explicit rather than hiding them behind a large machine-learning framework.

The project uses Intel oneAPI icpx, SYCL, and an Intel integrated GPU. GPT-2 Small provides a real pretrained model against which the implementation can be tested.

The current implementation supports text tokenization, pretrained-weight loading, the full 12-block GPT-2 Small forward pass, next-token prediction, and autoregressive text generation. The complete model has also been compared numerically against a reference implementation using the same checkpoint.

Current status

The main inference pipeline is operational.

Component Status
USM device memory and tensor storage Implemented
Basic tensor operations Implemented
Naive and tiled matrix multiplication Implemented
Linear layers and activation functions Implemented
Serial and parallel softmax Implemented
Serial and parallel LayerNorm Implemented
Unfused scaled dot-product attention Implemented
Fused online-softmax attention Implemented
Multi-head and causal attention Implemented
Transformer block Implemented and tested against a CPU reference
GPT-2 token and positional embeddings Implemented
GPT-2 pretrained-weight export and loading Implemented
Assembly of all 12 GPT-2 Transformer blocks Implemented
Final LayerNorm and language-model output projection Implemented
C++ GPT-2 tokenization and decoding Implemented using tokenizers-cpp
Autoregressive text generation Implemented
Numerical comparison against Hugging Face GPT-2 Tested on multiple prompts
KV-cache optimization Not implemented
Sampling strategies such as temperature or top-k Not implemented
Automatic differentiation and training Not implemented

The primary objective is correctness and understanding. Performance optimization follows once the computation can be validated against a trusted reference.

Numerical validation

The complete model has been compared against Hugging Face GPT-2 using the same local checkpoint, the same input token IDs, and CPU reference inference.

The test exports the reference model's final-position logits and compares them with the corresponding logits generated by the SYCL implementation.

Each comparison covers all 50,257 vocabulary logits, not just the highest-scoring token.

Input Input tokens Maximum absolute error RMSE Top-1 match Top-10 overlap
Hello 1 0.0003052 0.0002148 Yes 10/10
The capital of Finland is 5 0.0001907 0.0000452 Yes 10/10
Nice to meet you . My name is cris. What's your name? 15 0.0001984 0.0000434 Yes 10/10
The quick brown fox jumps over the lazy dog. This is a longer test of the model. 19 0.0003357 0.0000854 Yes 10/10

For these tested inputs, the SYCL logits closely match the reference logits. The observed maximum absolute errors are between approximately (1.9 \times 10^{-4}) and (3.4 \times 10^{-4}). The top-10 predicted token IDs match exactly in every listed case.

This is strong evidence that the complete forward pass reproduces the reference implementation for the tested inputs. It is not proof of correctness for every possible sequence length, input, batch size, or device configuration.

Transformer-block validation

A single pre-LayerNorm Transformer block was separately validated against CPU reference calculations at GPT-2 Small dimensions:

B       = 2
L       = 1024
d_model = 768
heads   = 12
d_head  = 64
FF      = 3072
dtype   = float32

The test compares intermediate tensors and the final block output, including LayerNorm, Q/K/V projections, causal attention, output projection, residual connections, feed-forward projections, and GELU.

All comparisons passed with an absolute tolerance of 5e-4. In the reported run, the maximum absolute error was approximately 4.77e-7 for the second LayerNorm and 7.45e-8 for the final output.

The block test and the full-model logit comparison validate different aspects of the implementation. Both are useful: the block test isolates individual operations, while the full-model test checks the assembled inference pipeline.

Understanding the project

The project is organized around several layers of abstraction:

  1. Device memory: Owns allocations on the GPU.
  2. Tensors: Store multidimensional shapes and device pointers.
  3. Operations: Implement mathematical computations using SYCL kernels.
  4. Neural-network layers: Compose operations into linear transformations, embeddings, and normalization.
  5. Attention and Transformer blocks: Combine these layers into the architecture used by GPT-2.
  6. GPT-2 model: Loads pretrained parameters and assembles the complete neural network.
  7. Text generation: Converts a text prompt into token IDs, runs inference, and repeatedly predicts new tokens.

The lower layers are designed to be reusable. The GPT-2-specific code uses them to reproduce a particular pretrained model.

See docs/ARCHITECTURE_AND_TUTORIAL.md for a beginner-oriented explanation of SYCL, tensors, attention, Transformer blocks, GPT-2, and the source-code execution path.

GPT-2 Small architecture

The pretrained checkpoint targets the following architecture:

Vocabulary size        = 50257
Context length         = 1024
Hidden dimension       = 768
Attention heads        = 12
Dimension per head     = 64
Transformer blocks     = 12
Feed-forward dimension = 3072
Data type              = float32

At a high level, the model performs the following computation:

Text prompt
    |
    v
Tokenizer
    |
    v
Token IDs [B, L]
    |
    v
Token embeddings + positional embeddings
    |
    v
12 Transformer blocks
    |
    v
Final LayerNorm
    |
    v
Language-model output projection
    |
    v
Logits [B, L, 50257]
    |
    v
Select the next token
    |
    v
Append token to the prompt and repeat

The output projection uses the token-embedding matrix as the output weight matrix, following GPT-2's weight-tying design.

The current generation loop uses greedy decoding: it selects the token with the largest logit. It recomputes the model for the full context at every generation step. A KV cache and more advanced decoding strategies remain possible future improvements.

GPT-2 Small is a pretrained next-token prediction model, not an instruction-tuned conversational assistant. Repetitive or unhelpful generated text does not by itself indicate a numerical implementation error.

Implemented functionality

Device memory and tensors

DeviceBuffer<T> provides RAII-style ownership of SYCL USM device allocations using sycl::malloc_device. Owned memory is released automatically, and accidental shallow copies are prevented.

Tensor<T> stores a multidimensional shape, element count, and device pointer.

Example:

tinyml::Tensor<float> X(queue, {32, 128});

This creates a tensor with shape [32, 128], containing 4,096 floating-point elements in device memory.

Memory transfers between CPU and GPU are explicit. This is useful for learning how data movement affects the performance and correctness of GPU programs.

Tensor operations

Operations are implemented in include/tinyml/ops.hpp. The current implementation includes:

  • Host-to-device and device-to-host copies
  • Elementwise arithmetic and addition
  • Fill and scalar multiplication
  • Bias addition
  • ReLU
  • Exact GELU and tanh-approximate GELU
  • Serial and parallel softmax
  • Serial and parallel LayerNorm
  • Affine parallel LayerNorm
  • Naive matrix multiplication
  • Tiled matrix multiplication
  • Tiled matrix multiplication with a transposed second operand
  • Scaled tiled multiplication with a transposed second operand
  • Tiled online-softmax weighted-value products
  • Fused scaled dot-product attention using online softmax
  • Multi-head attention
  • Strided multi-head attention on [B, L, d_model] tensors
  • Causal strided multi-head attention

Operations return sycl::event objects where applicable. Event dependencies express the ordering between asynchronous GPU operations.

Matrix multiplication

Several matrix-multiplication kernels are available, including naive and tiled implementations. The tiled implementations use SYCL local memory to reuse data within work-groups.

The transposed operation computes

[ C = AB^T ]

for A with shape [M, K], B with shape [N, K], and C with shape [M, N]. It avoids creating a physical transposed copy of B.

A scaled variant computes

[ C = \frac{AB^T}{\sqrt{K}}. ]

This operation is useful for attention scores when A = Q, B = K, and K is the attention head dimension.

These kernels are learning and optimization exercises, not replacements for tuned BLAS libraries.

Neural-network layers

Linear<T> implements a fully connected layer:

[ Y = XW + b ]

where X contains the input features, W contains the weights, and b contains the bias.

The layer supports Transformer-style 3D tensors such as [B, L, in_features]. The contiguous tensor can be treated as a [B*L, in_features] matrix for multiplication without physically rearranging the data.

The project also implements ReLU, exact GELU, tanh-approximate GELU, softmax, and affine LayerNorm.

Attention

The project contains multiple scaled dot-product attention implementations to expose their mathematical and memory-access trade-offs.

The standard operation is

[ \operatorname{Attention}(Q,K,V) = \operatorname{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V. ]

The unfused path materializes the score and softmax-weight matrices. The online-softmax implementations maintain the running maximum, normalization factor, and weighted-value accumulator as tiles are processed, avoiding the need to materialize the complete softmax-weight matrix.

The fused implementation is inspired by the blockwise online-softmax approach used by memory-efficient algorithms such as FlashAttention. It is an educational implementation, not a reproduction of an optimized FlashAttention kernel.

Multi-head attention operates on projected queries, keys, and values. The strided implementation accesses individual heads directly within [B, L, d_model] tensors without physically rearranging them to [B, H, L, d_head].

The causal attention kernel restricts each query to positions at or before itself, preventing a token from attending to future positions.

Transformer block

TransformerBlock<T> implements a pre-LayerNorm Transformer block.

Its data path is:

Input
  |
  v
LayerNorm
  |
  v
Q / K / V projections
  |
  v
Causal multi-head attention
  |
  v
Attention output projection
  |
  v
Residual addition
  |
  v
LayerNorm
  |
  v
Feed-forward projection: 768 -> 3072
  |
  v
GELU
  |
  v
Feed-forward projection: 3072 -> 768
  |
  v
Residual addition
  |
  v
Output

The block owns its attention projections, feed-forward layers, and learned LayerNorm parameters. Its forward() method composes the operations through SYCL event dependencies.

GPT-2 embeddings and weights

GPT-2 uses two embedding tables:

  • Token embeddings: map each token ID to a vector of 768 values.
  • Positional embeddings: map each sequence position to a vector of 768 values.

The two vectors are added before the first Transformer block.

The original checkpoint is stored at models/gpt2/model.safetensors. Export scripts create binary parameter files for the C++ implementation, including the token embeddings, positional embeddings, each Transformer block's parameters, and the final LayerNorm parameters.

The model loader loads these parameters and assembles all 12 blocks. The language-model output projection uses the token-embedding weights, consistent with GPT-2's weight tying.

Tokenization

GPT2Tokenizer is a C++ wrapper around mlc-ai/tokenizers-cpp. It loads the GPT-2 tokenizer.json file and supports encoding text to token IDs, decoding token IDs to text, and querying the vocabulary size.

The tokenizer is separate from the neural network. It converts the text representation into the integer inputs expected by the model and converts generated token IDs back into text.

Autoregressive generation

src/gpt2_generate.cpp implements the generation loop.

At each step, it:

  1. Takes the current token IDs.
  2. Copies them to device memory.
  3. Runs GPT2Model::forward() using SYCL.
  4. Copies the final sequence position's logits back to the CPU.
  5. Selects the token with the largest logit.
  6. Appends the token to the context.
  7. Repeats until the requested number of tokens is generated or the end-of-text token is produced.

The implementation uses greedy decoding. It does not currently use a KV cache, temperature sampling, top-k sampling, or a repetition penalty.

Repository structure

The following is a representative structure of the current project:

tinyml/
├── CMakeLists.txt
├── README.md
├── docs/
│   └── ARCHITECTURE_AND_TUTORIAL.md
├── include/
│   └── tinyml/
│       ├── device_buffer.hpp
│       ├── tensor.hpp
│       ├── ops.hpp
│       ├── linear.hpp
│       ├── embedding.hpp
│       ├── gpt2_embedding.hpp
│       ├── gpt2_layer_norm.hpp
│       ├── gpt2_weights.hpp
│       ├── gpt2_block_weights.hpp
│       ├── gpt2_model.hpp
│       ├── gpt2_tokenizer.hpp
│       ├── transformer_block.hpp
│       └── flash_attention.hpp
├── src/
│   ├── main.cpp
│   ├── gpt2_generate.cpp
│   └── gpt2_dump_logits.cpp
├── tools/
│   ├── export_gpt2_embeddings.py
│   ├── export_gpt2_block.py
│   ├── export_gpt2_final_norm.py
│   ├── reference_gpt2_block.py
│   ├── reference_gpt2_logits.py
│   ├── compare_gpt2_logits.py
│   └── query_gpt2.py
├── models/
│   └── gpt2/
│       ├── model.safetensors
│       ├── tokenizer/
│       │   ├── tokenizer.json
│       │   └── tokenizer_config.json
│       ├── token_embeddings.bin
│       ├── position_embeddings.bin
│       ├── final_ln_gamma.bin
│       ├── final_ln_beta.bin
│       ├── block0/
│       ├── ...
│       └── block11/
├── external/
│   └── tokenizers-cpp/
├── tests/
│   ├── test_core.cpp
│   ├── test_nn.cpp
│   ├── test_sdpa.cpp
│   ├── test_attention.cpp
│   ├── test_flash_attention.cpp
│   ├── test_transformer_block.cpp
│   ├── test_embedding.cpp
│   ├── test_gpt2_embedding.cpp
│   ├── test_gpt2_embedding_validation.cpp
│   ├── test_gpt2_block.cpp
│   ├── test_gpt2_model.cpp
│   ├── test_gpt2_weights.cpp
│   ├── benchmark_ops.cpp
│   └── benchmark_flash_attention.cpp
└── build/

The list is representative. The exact contents of a checkout may differ as development continues. Generated model files and build artifacts should not be assumed to be source-controlled.

Requirements

The project currently uses:

  • C++20
  • Intel oneAPI icpx
  • SYCL with the spir64 target
  • An available SYCL GPU device
  • CMake 3.20 or newer
  • The tokenizers-cpp dependency in external/tokenizers-cpp

Python is only required for the export scripts, the Hugging Face tokenizer wrapper, and numerical reference tests. The direct C++ inference executable does not require Python during generation.

For numerical reference testing, the Python environment also needs NumPy, PyTorch, Transformers, and safetensors.

Building

Configure the main project:

cmake -S . -B build \
  -DCMAKE_CXX_COMPILER=icpx \
  -DTINYML_BUILD_TESTS=ON \
  -DTINYML_BUILD_GPT2_WEIGHTS_TEST=ON \
  -DTINYML_BUILD_GPT2_BLOCK_TEST=ON \
  -DTINYML_BUILD_GPT2_MODEL_TEST=ON \
  -DTINYML_BUILD_GPT2_GENERATE=ON \
  -DTINYML_BUILD_GPT2_DUMP_LOGITS=ON

Build the GPT-2 text-generation executable:

cmake --build build --target gpt2_generate -j2

Run inference from the repository root:

./build/gpt2_generate --max-new-tokens 20 "The capital of Finland is"

The executable uses the C++ tokenizer, runs model inference through SYCL, and decodes the generated tokens in C++.

The model parameters and tokenizer are loaded through paths relative to the repository root, so run the executable from that directory.

Building the diagnostic executable

cmake --build build --target gpt2_dump_logits -j2

This executable reads token IDs from build/reference_input_ids.txt, runs the SYCL model, and writes the final-position logits to build/sycl_logits.bin.

Building the correctness tests

Build the available targets:

cmake --build build --target \
  test_core \
  test_nn \
  test_sdpa \
  test_attention \
  test_flash_attention \
  test_transformer_block \
  test_embedding \
  test_gpt2_embedding \
  test_gpt2_embedding_validation \
  test_gpt2_weights \
  test_gpt2_block \
  test_gpt2_model \
  -j2

Run the relevant tests:

./build/test_core
./build/test_nn
./build/test_sdpa
./build/test_attention
./build/test_flash_attention
./build/test_transformer_block
./build/test_embedding
./build/test_gpt2_embedding
./build/test_gpt2_embedding_validation
./build/test_gpt2_weights
./build/test_gpt2_block
./build/test_gpt2_model

These commands require the corresponding targets to have been enabled during configuration. A target that is disabled or not defined in the current CMake configuration must be configured or added before it can be built.

Python query wrapper

The Python wrapper uses Hugging Face's tokenizer to turn text into token IDs, then passes those IDs to the C++ executable. Model inference is still performed by the C++/SYCL implementation.

source .venv/bin/activate
python tools/query_gpt2.py --max-new-tokens 20 "The capital of Finland is"

This is useful for checking that both tokenizer paths pass equivalent token IDs to the same model. Python is not performing neural-network inference in this workflow.

Numerical reference comparison

The local Hugging Face checkpoint can be used to generate reference logits. This requires PyTorch and the Python reference dependencies.

Generate the reference logits:

python tools/reference_gpt2_logits.py "The capital of Finland is"

Export the SYCL logits for the same input token IDs:

./build/gpt2_dump_logits

Compare all 50,257 vocabulary logits:

python tools/compare_gpt2_logits.py

The scripts use:

build/reference_input_ids.txt
build/reference_logits.bin
build/sycl_logits.bin

The comparison reports maximum absolute error, mean absolute error, RMSE, centered-logit error, correlation, and agreement between the top predictions.

Running a new reference command overwrites the reference files. Run the diagnostic and comparison before changing to another prompt.

For the current reference script, use a prompt that fits GPT-2's 1,024-token context limit.

Performance experiments

Performance benchmarks are separate from correctness tests. The benchmark programs explore execution time for operations such as GELU, tiled matrix multiplication, and attention.

The project currently contains both unfused and online-softmax attention implementations. The unfused path materializes intermediate score and probability matrices, while the fused implementation avoids storing the complete softmax-weight matrix.

The implementations remain educational kernels rather than highly optimized production equivalents. The next optimization opportunities include reducing intermediate global-memory traffic, improving work-group mapping, keeping more partial output state in registers or private storage, introducing a KV cache for generation, and tuning kernels after correctness is established.

Design principles

  • C++20 and SYCL
  • Explicit USM device-memory ownership
  • Explicit host/device transfers
  • Small, understandable tensor and operation abstractions
  • GPU kernels separated from neural-network layers
  • Reusable Transformer components
  • Correctness checks against CPU reference calculations
  • Full-model validation against a trusted reference
  • Separate correctness testing and performance benchmarking
  • Optimization only after the computation can be measured and validated

Roadmap

Inference and numerical validation

  • Implement device memory management and tensor storage
  • Implement basic tensor operations and matrix multiplication
  • Implement Linear layers and activation functions
  • Implement serial and parallel softmax
  • Implement serial and parallel LayerNorm
  • Implement scaled dot-product and causal multi-head attention
  • Implement online-softmax attention
  • Implement a reusable Transformer block
  • Validate the Transformer block at GPT-2 Small dimensions
  • Implement the GPT-2 tokenizer wrapper
  • Implement token and learned positional embeddings
  • Export and load pretrained GPT-2 parameters
  • Assemble the complete 12-block GPT-2 Small model
  • Implement final LayerNorm and the language-model output projection
  • Implement autoregressive text generation
  • Add a C++ text-prompt interface
  • Add a Python-tokenizer-to-C++-inference diagnostic path
  • Compare full-model logits against Hugging Face GPT-2 for multiple inputs

Correctness and performance

  • Expand full-model validation across more prompts and context lengths
  • Add more randomized inputs and multiple seeds to the Transformer-block tests
  • Expand shape coverage for batch size, sequence length, embedding dimension, and feed-forward width
  • Profile end-to-end inference and generation time
  • Implement a KV cache to avoid recomputing the entire context at every generation step
  • Explore temperature sampling, top-k/top-p sampling, and repetition penalties
  • Profile and optimize matrix multiplication and attention bottlenecks
  • Reduce global-memory traffic in fused attention
  • Explore keeping attention output fragments in registers or private storage
  • Explore a Triton version of selected kernels
  • Explore portability to other SYCL implementations
  • Consider automatic differentiation and training as longer-term extensions

Status summary

tinyml now runs pretrained GPT-2 Small inference through a C++/SYCL implementation on an Intel GPU. The full forward pass has been compared numerically against the local Hugging Face checkpoint on several inputs, with very small logit errors and matching top-10 predictions in the reported tests.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages