Skip to content

feat(kernel): NUMA topology, per-node memory and scheduling - #598

Merged
lexoliu merged 46 commits into
devfrom
feat/516-numa-topology
Nov 13, 2026
Merged

lexoliu merged 46 commits into
devfrom
feat/516-numa-topology

Conversation

@lexoliu

@lexoliu lexoliu commented Nov 11, 2026 •

Copy link
Copy Markdown
Owner

The kernel learns which node each processor and each stretch of memory belongs to and places memory and tasks by it.

  • Description. hal gains the NUMA contract: memory regions carry their node and a NumaTopology (nodes, processors per node, a distance matrix) reaches the kernel through the platform. firmware/ reads it from a device tree (numa-node-id, distance-map) and from the ACPI SRAT and SLIT; riscv and aarch64 use the first, x86 and aarch64 under --acpi the second. A machine that publishes neither is one node. The ACPI reader needs SRAT and SLIT accessors the acpi crate gets wrong, so the workspace's acpi dependency moves to the fork's helios branch, which carries that fix (Fix SRAT entry layout and SLIT distance lookup rust-osdev/acpi#381) beside the notify handler processor hotplug needs (aml: deliver Notify to the host through Handler::handle_notify rust-osdev/acpi#380).
  • Memory. One user frame pool per node. An allocation takes from the home node of the instance it is for, falls back through the other nodes in distance order, and a frame freed from any processor goes back to the node it came from. Per-processor kernel structures are backed from their processor's node. The stats sample reports user memory per node.
  • Scheduling. One ready queue per node and a home node per instance; a processor serves its own queue, then its node's, then steals from the other nodes nearest first.
  • Tools. helios-inspector vm --numa-nodes N [--numa-distance D] presents a NUMA machine on every architecture.

Three repairs a multi-node boot needed ride along, each under its own issue: riscv read only the first devicetree memory node (#561); riscv PLIC discovery stopped at the first PLIC and every hart's contexts were opened from the boot hart, which left a second socket's interrupts masked after an OpenSBI reset (#562, #863); and KernelPhysFrameAllocator, which nothing used, is removed rather than ported.

On aarch64 the device tree Limine hands over cannot carry the topology: Limine 12 deletes every /memory node from it, and the numa-node-id of each with them. The backend therefore reads NUMA from the firmware's own tree, found through the UEFI configuration table's DTB entry (aarch64/src/platform/efi.rs, Limine's EFI system table request) and accepted only where the memory map says the firmware or the loader owns the range. A tree that names nodes on processors and none on memory is refused rather than booted as one node.

Processor lifecycle: a wake goes only to an online processor, nearest node first; a departing processor leaves its node's queue in place and wakes a successor, which steals from it; the frames it cached drain into their own nodes' pools; and a processor that returns keeps the node the topology gave it. Every per-processor and per-node structure is sized by the possible processors, so a hot-added processor lands in the node its SRAT entry names; --numa-nodes composes with --max-processors when the nodes divide it and is refused otherwise.

Concurrency: a node's pool is one IrqSafeMutex taken from any processor; the allocating processor's index chooses the node and nothing else, so a foreign free takes the owning node's lock like a local one. The topology is immutable after boot.

Evidence

  • Two-node boots in every smoke lane, each asserting the NUMA topology nodes=2 line against what QEMU published: riscv64 and aarch64 from the device tree, x86-64 and aarch64 from ACPI (run 43367028877; smoke-aarch64 prints NUMA topology nodes=2 distances=10,20;20,10 for both its boots).
  • a_cross_processor_free_returns_the_frame_to_its_own_node and its neighbours in kernel/src/memory/user.rs drive a two-node plan for node-local placement and distance-ordered fallback; firmware/tests/acpi.rs and the devicetree tests cover the readers.
  • Single-node paired run 43337651395 (suite job 131610999071, Intel Xeon Platinum 8370C, noise floor 4.4%) against a226af3 and its rebuild, warm medians in ms: instance-startup-100 49.0 vs 48.1 / 47.0, spawn-wait 60.8 vs 59.3 / 59.0, sched-tasks 106 vs 108 / 106, pipe-pingpong 163 vs 158 / 161, hostcall-loop 192 vs 194 / 191, tcp-throughput 47.7 vs 47.4 / 45.6, cpython-json 87.9 vs 87.9 / 87.3, aot-curl 247 vs 246 / 242, control quickjs-loop 20.4 vs 20.2 / 19.9. No row leaves the baseline pair's own spread except spawn-wait, by 2.5% inside overlapping intervals ([60.5, 66.9] vs [59.2, 61.3]).
  • The head with the processor lifecycle merged, paired run 43367031679 (suite job 131704066573, gate job 131737631838, AMD EPYC 7763, noise floor 13.2%) against 7f56e9b and its rebuild: every compared row unchanged, sched-tasks 100 vs 99.5 / 99.4, spawn-wait 56.2 vs 55.2 / 54.7, pipe-pingpong 153 vs 151 / 155, hostcall-loop 204 vs 203 / 229, instance-startup-100 44.2 vs 44.0 / 42.7, tcp-throughput 39.1 vs 40.0 / 39.4, control quickjs-loop 21.9 vs 22.1 / 22.2. Its gate reports incomplete evidence: fs-smallfiles had a rejected cell on that host, so that row rests on the first run alone.

No speed claim is made for placement: an emulated topology on a one-node host has uniform memory latency. That measurement is what #516 stays open for.

Fixes #561
Fixes #562
Fixes #863
Part of #516

`Fdt::memory()` returns the first `/memory` node only, so a tree with
one node per range lost all but the first.

Fixes #561
Discovery stopped at the first compatible PLIC node; a hart wired to a
later one had no interrupt route.

Fixes #562
Its only production reference was a PhantomData; UserMemoryPool is the
frame allocator. Drops buddy_system_allocator with it.

Part of #516
…eaches the pool

hal gains numa (NodeId, NodeDistance, MemoryAffinity, NumaTopology,
SingleNode); MemoryRegion is a struct with its node, tagged by
usable_node_regions, which refuses usable memory no affinity range
covers. Platform no longer carries regions: a backend primes the pool
with prime_bootstrap_allocator(regions, &numa). The kernel builds an
immutable Locality from the description and prints it at boot. Every
backend reports a single node; the pool refuses more until it is split
per node.

Part of #516
A new device-library crate turns firmware tables into `hal::numa`
values before the heap exists. This is the devicetree reader:
`numa-node-id` on cpu and memory nodes, and `/distance-map`. A
description is validated whole at construction and refused with the
entry it broke on. No backend uses it yet.

Part of #516
# Conflicts:
#	Cargo.toml
#	kernel/src/lib.rs
The user pool is a buddy pool per node with a frame list per
(processor, node). An allocation walks the executing processor's
distance order: its cached frame of a node, that node's pool, then a
drain of every processor's list for the node, before the next node; a
free returns the frame to its own node. Compaction takes a replacement
frame from the old frame's node. The boot split hands out the bootstrap
processor's node first, which the backend names because the processor
register is not seeded that early (#765).
A processor runs its own node's tasks first and steals from the other
nodes in distance order; a push wakes a parked processor of the target
node before one further away. An instance's tasks are queued on the
node of the processor that ran its first turn, provisional until then
on the node that registered it.
# Conflicts:
#	Cargo.toml
#	kernel/src/memory/mod.rs
Both backends hand the bootstrap allocator a DeviceTreeNuma instead of
one node. riscv maps a processor to its hart id; aarch64 maps it to the
affinity bits of its MPIDR, taken from the Limine MP response in the
order the processor table is built in. The aarch64 ACPI path still
describes one node.
helios:system/stats memory gains a nodes list: each node's seeded and
available bytes and the allocations its processors had served from
another node. The inspector's memory panel draws one line per node on a
machine with more than one.
vm gains --numa-nodes and --numa-distance (numa_nodes and numa_distance
in the config file). The machine is symmetric: one socket and one
memory backend per node, so the count must divide --smp and --memory,
and a combination that describes no machine is refused by name.
PerProcessor<T> holds one cache-padded block per processor, carved from a
per-node arena that grows by one-way lends from that node's memory. The
frame slab and reserve shards, owner cells, fiber-stack counters, the
executor's per-processor queues and task arenas, the network shards, and
the x86 and aarch64 processor slots and stacks move onto it. A machine
with one node keeps its behaviour; the task arenas leave the kernel heap
and are reported as per_processor_bytes at boot.
Both device-tree lanes gain one boot with --numa-nodes 2 that asserts the
node count, the distance map and each node's processors from the kernel's
boot lines. The x86-64 boot joins with the ACPI reader.
# Conflicts:
#	kernel/src/network/service/mod.rs
`-numa cpu,socket-id` is refused by the RISC-V `virt` machine, whose
processors carry only `core-id`. `-numa node,cpus=` takes processor
indexes and is accepted by every machine the inspector boots.
# Conflicts:
#	Cargo.toml
#	aarch64/src/vmm.rs
#	hosted/src/swap_tests.rs
#	kernel/src/exec/executor.rs
#	kernel/src/lib.rs
#	kernel/src/memory/mod.rs
…rries the kernel log

The riscv64 boot UART is the debugger's RPC line unless the machine has a vsock device, so the two-node step found no NUMA lines in debug-serial.log although the guest booted. The step now runs after the vsock transport is set up and uses it.
# Conflicts:
#	hal/src/pmm.rs
#	inspector/src/vm.rs
#	kernel/src/instance.rs
#	kernel/src/memory/balloon.rs
#	kernel/src/memory/pmm.rs
#	kernel/src/memory/user.rs
A machine may carry several PLICs: QEMU's virt gives each socket its
own and wires the virtio-mmio devices to the second. Every device was
enabled on the bootstrap hart's controller, so on such a machine no
virtio interrupt arrived.

A device's interrupt now resolves to its interrupt parent as the
devicetree specification walks it, each PLIC is an interrupt domain
taken by a hart with a supervisor-external context there, and a
granted device's mask and unmask act on its own controller.
# Conflicts:
#	hosted/src/lib.rs
#	kernel/src/exec/executor.rs
#	kernel/src/exec/mod.rs
#	x86/src/lib.rs
# Conflicts:
#	AGENTS.md
#	inspector/src/vm.rs
The NUMA pool made it the eighth parameter; the handler already carries
the offset, TSC base and timebase the function unpacked into one.
The bootstrap hart enabled every device's source on the taking hart's
context during bring-up. The SBI firmware resets a hart's PLIC contexts
when it starts the hart (OpenSBI's plic_warm_irqchip_init disables every
source and raises the threshold to 7), so on a PLIC taken by a secondary
hart those writes were lost: a two-socket virt machine showed the virtio
sources pending in the second PLIC with its enable words zero and its
thresholds at 7. Each hart now enables its domains' sources and opens
their thresholds when it builds its runtime.
AcpiNuma answers hal's NumaTopology from the SRAT and SLIT before the heap exists: processors keyed by APIC id on x86 and by ACPI processor UID through the MADT's GICC entries on aarch64, the SLIT validated as Linux's slit_valid does and refused instead of ignored. x86 and the aarch64 ACPI boot build it in place of SingleNode. acpi moves to 7.0.0-pre1 from the fork branch that carries the SRAT layout and SLIT lookup fixes. smoke-x86-64 and smoke-aarch64 --acpi boot two nodes.
# Conflicts:
#	Cargo.lock
#	aarch64/Cargo.toml
#	aarch64/src/lib.rs
#	x86/Cargo.toml
#	x86/src/lib.rs
# Conflicts:
#	.github/workflows/ci.yml
#	AGENTS.md
#	Cargo.lock
#	Cargo.toml
#	Justfile
#	aarch64/Cargo.toml
#	aarch64/src/platform/acpi.rs
#	kernel/src/lib.rs
#	kernel/src/wasmtime_adapter/component_host/mod.rs
#	riscv/src/net/mod.rs
#	x86/Cargo.toml
#	x86/src/smp.rs
Limine hands over a copy of the firmware's tree with every memory node
deleted, so the tree the backend parses names nodes on its processors
and no memory at all. The NUMA description now comes from the tree the
UEFI configuration table points at, reached through the EFI system
table request and read where the memory map shows the direct map covers
it. The reader refuses a tree with node ids and no memory node instead
of answering with an empty memory affinity.
# Conflicts:
#	Cargo.lock
#	aarch64/Cargo.toml
#	aarch64/src/lib.rs
#	aarch64/src/platform/mod.rs
#	kernel/src/exec/executor.rs
#	kernel/src/memory/frame_slab.rs
#	kernel/src/memory/mod.rs
#	kernel/src/memory/user.rs
#	x86/src/smp.rs
@lexoliu lexoliu changed the title kernel: NUMA topology, first stage (riscv repairs, dead allocator removal) feat(kernel): NUMA topology, per-node memory and scheduling Nov 13, 2026
@lexoliu
lexoliu marked this pull request as ready for review November 13, 2026 05:17
# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	aarch64/Cargo.toml
#	inspector/src/vm.rs
#	x86/Cargo.toml
#	x86/src/lib.rs
#	x86/src/smp.rs
# Conflicts:
#	kernel/src/network/service/shard.rs
#	kernel/src/test_support.rs
# Conflicts:
#	kernel/benches/data_structures.rs
#	kernel/src/lib.rs
# Conflicts:
#	kernel/src/network/service/mod.rs
@lexoliu
lexoliu merged commit 6d3dd71 into dev Nov 13, 2026
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant