Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,66 @@ against, and semantic-release manages only the **patch** component.

## [Unreleased]

### Added

- **MTL5 pinned to v5.11.0, and `accumulator='i32'`**
([#88](https://github.com/stillwater-sc/mtl5-python/issues/88) phase 2). The
narrow integer element types phase 1 registered can now compute:

```python
a = np.random.randint(0, 256, 4096, dtype=np.uint8) # activations
w = np.random.randint(-128, 128, 4096, dtype=np.int8) # weights
mtl5.mixed.dot(a, w, accumulator="i32")
```

MTL5 v5.11.0 routes `dot<int32_t>` over 8- and 16-bit operands onto the
hardware widening multiply-accumulate — `vpmaddwd` / `vpdpbusd` on x86,
`SMLAL` / `SDOT` on NEON — and an int32 accumulator is what those
instructions accumulate into.

**Every signedness pairing is accepted, in either order.** `u8 × i8` is VNNI's
native shape on x86 and what quantized inference is written in; ARM implements
the symmetric pairings first. A dot product is symmetric, so MTL5 swaps the
operands onto whichever form the machine has. The kernel below
(`simd::reduce_dot_widen`) rejects `(int8, uint8)`, but that restriction is the
kernel's and re-exposing it would refuse a call the library can serve.

**The overflow contract is part of the API, not a footnote.** Products are
always exact; the sum wraps, and how soon depends on operand *magnitude*
rather than vector length — roughly `2^(31-2b)` terms at `b` bits. Measured at
full range: one `i16 × i16` product uses 2³⁰ of the int32 range, so **two**
already overflow it, while `i8 × i8` holds **131071** terms. That five order
of magnitude gap is why the quantized-inference instructions are 8-bit. The
wrap is two's complement and therefore bit-identical across lane counts,
backends and thread partitions — reproducible, but still wrapping. It is
stated in the docstring and asserted in the tests rather than left to be
discovered.

Two guardrails, both deliberate. `accumulator=` is **required** for these
dtypes, unlike every other: omitting it means element precision, which is
exactly the wrapping phase 1 refused to expose, and silently redefining `None`
for three dtypes alone would be worse than asking. And `result='element'` is
**refused**, since rounding an int32 sum back to an 8-bit element
re-introduces the wrap the accumulator exists to avoid.

`norm` and `frobenius_norm` remain unregistered for these types: `two_norm`
takes `sqrt` of the accumulated sum, and the API has no way to say
"accumulate in int32, deliver a real square root".

`mtl5.dtypes()` is deliberately **unchanged**. Its contract is the set
`convert()` can target — the suite parametrizes over it and converts into
every entry — and `convert()` cannot target an integer: re-quantizing reals
into 8 bits is a quantization scheme (scale, zero point, rounding mode), not
a cast, and a naive version would silently clip everything outside
[-128, 127]. `mtl5.mixed.accumulators('i8')` answers for them instead.

The v5.11.0 pin is what makes this sound rather than merely available.
`batch<int32_t>` does not exist before it, so the kernels are absent — and
`detail/wrapping_arithmetic.hpp` is new in it, without which the generic
integer loops are **UB** on overflow rather than the documented modular wrap.
For these operand widths overflow is the normal regime, not an edge case.


### Added

- **Narrow integer element types: `i8`, `i16`, `u8`**
Expand Down
17 changes: 13 additions & 4 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -100,17 +100,26 @@ endif()
# regression on the refactor path, with no diagnostic.
# v5.10.0 -- mtl5_module.cpp calls mtl::util::build_isa_list() (MTL5 #443),
# which does not exist before this tag.
# v5.11.0 -- mtl5_mixed_precision.cpp offers accumulator='i32', which is the
# widening integer dot: int32 accumulation over 8- and 16-bit
# operands, on vpmaddwd / vpdpbusd / SDOT. Two things arrive with
# this tag and neither is optional. batch<int32_t> does not exist
# before it, so the kernels are simply absent. And
# detail/wrapping_arithmetic.hpp is new here: without it the
# generic integer loops are UB on overflow rather than the
# documented two's-complement wrap, and overflow is the NORMAL
# regime for these operand widths, not an edge case.
#
# The version argument to find_package is what applies that floor to the OTHER
# build path. A bare find_package(MTL5 QUIET) accepts *any* system-installed
# MTL5 and skips the FetchContent block entirely, so a developer with 5.7.x
# installed would silently build against it and get ~140 template errors deep
# inside norms.hpp instead of a version message. MTL5 ships its ConfigVersion
# with COMPATIBILITY SameMajorVersion, under which 5.10.0 rejects an installed
# 5.9.x and accepts 5.10.0 or 5.11.0 -- exactly the floor we want. (It also
# with COMPATIBILITY SameMajorVersion, under which 5.11.0 rejects an installed
# 5.10.x and accepts 5.11.0 or 5.12.0 -- exactly the floor we want. (It also
# rejects a 6.x install, which is correct: an MTL5 major bump is the manual
# intervention case in the version policy, not something to absorb silently.)
find_package(MTL5 5.10.0 QUIET)
find_package(MTL5 5.11.0 QUIET)
if(NOT MTL5_FOUND)
include(FetchContent)
# Suppress MTL5's own tests, examples, and install targets
Expand All @@ -120,7 +129,7 @@ if(NOT MTL5_FOUND)
FetchContent_Declare(
mtl5
GIT_REPOSITORY https://github.com/stillwater-sc/mtl5.git
GIT_TAG v5.10.0
GIT_TAG v5.11.0
GIT_SHALLOW TRUE
EXCLUDE_FROM_ALL
)
Expand Down
33 changes: 33 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,39 @@ quires have known upstream limitations documented in
`accumulator=` is available on `dot`, `norm` (ord=2), `frobenius_norm`,
`matvec` and `matmul`.

### Integer operands: `accumulator="i32"`

The same idea with integers, which is where the hardware is. `int8` and `int16`
operands accumulated in `int32` map onto the widening multiply-accumulate —
`vpmaddwd` / `vpdpbusd` on x86, `SMLAL` / `SDOT` on NEON:

```python
a = np.random.randint(0, 256, 4096, dtype=np.uint8) # activations
w = np.random.randint(-128, 128, 4096, dtype=np.int8) # weights
mtl5.mixed.dot(a, w, accumulator="i32")
```

`u8 × i8` is VNNI's native pairing on x86; ARM implements the symmetric ones
first. Either order works — a dot product is symmetric, so MTL5 swaps the
operands onto whatever the machine has.

**The sum wraps, and sooner than vector length suggests.** Products are always
exact, but headroom goes as operand *magnitude*: about `2^(31-2b)` terms at `b`
bits. Measured at full range, one `int16 × int16` product uses 2³⁰ of the int32
range so **two** overflow it, while `int8 × int8` holds **131071** terms. That
gap is why quantized inference is 8-bit. The wrap is two's complement, hence
bit-identical across lane counts, backends and thread counts.

`accumulator="i32"` is required for these dtypes — the default is element
precision, and an 8-bit accumulator overflows almost immediately. Build the
arrays with NumPy: `convert()` does not target integers, because re-quantizing
reals into 8 bits is a quantization scheme rather than a cast.

A released wheel is built at the x86-64 baseline, so it gets int8's **bandwidth**
win — one byte per element against float64's eight — but not the VNNI
instruction. `mtl5.build_info()["build_isa"]` reports which you have; build with
`-C cmake.define.MTL5_NATIVE_ARCH=ON` to reach it.

### Iterative refinement

Factor cheaply in a low precision, then recover accuracy with a residual formed
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ build-backend = "scikit_build_core.build"

[project]
name = "mtl5"
version = "5.10.3"
version = "5.11.0"
description = "Python bindings for MTL5 — NumPy/SciPy/JAX/PyTorch interop with hardware accelerator dispatch"
readme = "README.md"
license = {text = "MIT"}
Expand Down
Loading