An educational implementation of neural-network operations and GPT-2 inference using PyTorch and custom Triton kernels.
The project targets Intel GPUs through the Triton XPU backend. Its primary goal is to explore GPU programming, understand the operations behind Transformer models, and progressively implement GPT-2 inference using custom GPU kernels.
- PyTorch XPU detects Intel Graphics (device ID 0x7d55) under WSL2.
- Custom Triton kernels execute on the Intel GPU.
- PyTorch reference implementations are used for numerical validation.
- Elementwise operations: addition, subtraction, multiplication and division.
- Scalar operations: addition and multiplication.
- Sum reduction.
- Maximum reduction.
- Matrix multiplication (GEMM).
- Linear projection with bias.
- GELU-new activation.
- Layer normalization.
- Stable row-wise softmax.
- Tensor indexing, reshape and transpose.
- Single-head attention.
- Flash Attention using tiling and online softmax.
- Multi-head causal attention.
- Argmax.
- Token and positional embedding lookup.
- Assemble the GPT-2 decoder block.
- Implement the complete GPT-2 Small forward pass.
- Implement autoregressive text generation.
- Validate numerical agreement with a reference implementation.
The initial implementation targets GPT-2 Small:
Parameter Value
Vocabulary size 50,257 Context length 1,024 Hidden dimension 768 Number of decoder blocks 12 Attention heads 12 Head dimension 64 MLP intermediate dimension 3,072
Initial implementation constraints:
- Inference only; no training or backpropagation.
- FP32 precision.
- Greedy decoding.
- PyTorch reference implementation for correctness validation.
- Custom Triton kernels for GPU operations.
- KV cache and performance optimizations will be introduced in later stages.
The project has been tested with:
- Python 3.12.3
- PyTorch 2.7.1+XPU
- Triton 3.3.1
- Pytest 9.2.0
- Intel Graphics (device ID 0x7d55)
- Ubuntu 24.04 under WSL2
These are the verified test versions, not a guarantee that other configurations behave identically.
Run the complete test suite:
python -m pytest -vLatest validation:
- 77 tests passed.
- No failing tests.
- Execution time: 27.09 seconds.
Tests compare custom Triton kernel outputs against independent PyTorch reference implementations.
tinyml-triton/
├── docs/
│ ├── ARCHITECTURE.md
│ ├── OPERATIONS.md
│ └── VALIDATION.md
├── benchmarks/
│ └── benchmark_matmul.py
├── src/
│ └── tinyml_triton/
│ └── kernels/
│ ├── elementwise.py
│ ├── reduction.py
│ └── matmul.py
├── tests/
├── pyproject.toml
├── README.md
└── LICENSE
Operations are implemented incrementally, following this workflow:
- Understand the mathematical operation.
- Implement the Triton kernel.
- Validate against an independent PyTorch reference.
- Test different tensor dimensions and edge cases.
- Benchmark when relevant.
- Document the implementation and validation.
- Commit the completed milestone.
Correctness takes priority over performance optimization. Individual operations are validated before being assembled into larger Transformer components.
This project is released under the MIT License. See LICENSE.