Repository navigation
feat(kernel): NUMA topology, per-node memory and scheduling - #598
Merged
Merged
Conversation
`Fdt::memory()` returns the first `/memory` node only, so a tree with one node per range lost all but the first. Fixes #561
Discovery stopped at the first compatible PLIC node; a hart wired to a later one had no interrupt route. Fixes #562
Its only production reference was a PhantomData; UserMemoryPool is the frame allocator. Drops buddy_system_allocator with it. Part of #516
…e removed allocator Part of #516
…eaches the pool hal gains numa (NodeId, NodeDistance, MemoryAffinity, NumaTopology, SingleNode); MemoryRegion is a struct with its node, tagged by usable_node_regions, which refuses usable memory no affinity range covers. Platform no longer carries regions: a backend primes the pool with prime_bootstrap_allocator(regions, &numa). The kernel builds an immutable Locality from the description and prints it at boot. Every backend reports a single node; the pool refuses more until it is split per node. Part of #516
A new device-library crate turns firmware tables into `hal::numa` values before the heap exists. This is the devicetree reader: `numa-node-id` on cpu and memory nodes, and `/distance-map`. A description is validated whole at construction and refused with the entry it broke on. No backend uses it yet. Part of #516
# Conflicts: # Cargo.toml # kernel/src/lib.rs
The user pool is a buddy pool per node with a frame list per (processor, node). An allocation walks the executing processor's distance order: its cached frame of a node, that node's pool, then a drain of every processor's list for the node, before the next node; a free returns the frame to its own node. Compaction takes a replacement frame from the old frame's node. The boot split hands out the bootstrap processor's node first, which the backend names because the processor register is not seeded that early (#765).
A processor runs its own node's tasks first and steals from the other nodes in distance order; a push wakes a parked processor of the target node before one further away. An instance's tasks are queued on the node of the processor that ran its first turn, provisional until then on the node that registered it.
# Conflicts: # Cargo.toml # kernel/src/memory/mod.rs
Both backends hand the bootstrap allocator a DeviceTreeNuma instead of one node. riscv maps a processor to its hart id; aarch64 maps it to the affinity bits of its MPIDR, taken from the Limine MP response in the order the processor table is built in. The aarch64 ACPI path still describes one node.
helios:system/stats memory gains a nodes list: each node's seeded and available bytes and the allocations its processors had served from another node. The inspector's memory panel draws one line per node on a machine with more than one.
vm gains --numa-nodes and --numa-distance (numa_nodes and numa_distance in the config file). The machine is symmetric: one socket and one memory backend per node, so the count must divide --smp and --memory, and a combination that describes no machine is refused by name.
PerProcessor<T> holds one cache-padded block per processor, carved from a per-node arena that grows by one-way lends from that node's memory. The frame slab and reserve shards, owner cells, fiber-stack counters, the executor's per-processor queues and task arenas, the network shards, and the x86 and aarch64 processor slots and stacks move onto it. A machine with one node keeps its behaviour; the task arenas leave the kernel heap and are reported as per_processor_bytes at boot.
Both device-tree lanes gain one boot with --numa-nodes 2 that asserts the node count, the distance map and each node's processors from the kernel's boot lines. The x86-64 boot joins with the ACPI reader.
# Conflicts: # kernel/src/network/service/mod.rs
`-numa cpu,socket-id` is refused by the RISC-V `virt` machine, whose processors carry only `core-id`. `-numa node,cpus=` takes processor indexes and is accepted by every machine the inspector boots.
# Conflicts: # Cargo.toml # aarch64/src/vmm.rs # hosted/src/swap_tests.rs # kernel/src/exec/executor.rs # kernel/src/lib.rs # kernel/src/memory/mod.rs
…rries the kernel log The riscv64 boot UART is the debugger's RPC line unless the machine has a vsock device, so the two-node step found no NUMA lines in debug-serial.log although the guest booted. The step now runs after the vsock transport is set up and uses it.
# Conflicts: # hal/src/pmm.rs # inspector/src/vm.rs # kernel/src/instance.rs # kernel/src/memory/balloon.rs # kernel/src/memory/pmm.rs # kernel/src/memory/user.rs
A machine may carry several PLICs: QEMU's virt gives each socket its own and wires the virtio-mmio devices to the second. Every device was enabled on the bootstrap hart's controller, so on such a machine no virtio interrupt arrived. A device's interrupt now resolves to its interrupt parent as the devicetree specification walks it, each PLIC is an interrupt domain taken by a hart with a supervisor-external context there, and a granted device's mask and unmask act on its own controller.
# Conflicts: # hosted/src/lib.rs # kernel/src/exec/executor.rs # kernel/src/exec/mod.rs # x86/src/lib.rs
# Conflicts: # AGENTS.md # inspector/src/vm.rs
The NUMA pool made it the eighth parameter; the handler already carries the offset, TSC base and timebase the function unpacked into one.
The bootstrap hart enabled every device's source on the taking hart's context during bring-up. The SBI firmware resets a hart's PLIC contexts when it starts the hart (OpenSBI's plic_warm_irqchip_init disables every source and raises the threshold to 7), so on a PLIC taken by a secondary hart those writes were lost: a two-socket virt machine showed the virtio sources pending in the second PLIC with its enable words zero and its thresholds at 7. Each hart now enables its domains' sources and opens their thresholds when it builds its runtime.
AcpiNuma answers hal's NumaTopology from the SRAT and SLIT before the heap exists: processors keyed by APIC id on x86 and by ACPI processor UID through the MADT's GICC entries on aarch64, the SLIT validated as Linux's slit_valid does and refused instead of ignored. x86 and the aarch64 ACPI boot build it in place of SingleNode. acpi moves to 7.0.0-pre1 from the fork branch that carries the SRAT layout and SLIT lookup fixes. smoke-x86-64 and smoke-aarch64 --acpi boot two nodes.
# Conflicts: # Cargo.lock # aarch64/Cargo.toml # aarch64/src/lib.rs # x86/Cargo.toml # x86/src/lib.rs
# Conflicts: # .github/workflows/ci.yml # AGENTS.md # Cargo.lock # Cargo.toml # Justfile # aarch64/Cargo.toml # aarch64/src/platform/acpi.rs # kernel/src/lib.rs # kernel/src/wasmtime_adapter/component_host/mod.rs # riscv/src/net/mod.rs # x86/Cargo.toml # x86/src/smp.rs
Limine hands over a copy of the firmware's tree with every memory node deleted, so the tree the backend parses names nodes on its processors and no memory at all. The NUMA description now comes from the tree the UEFI configuration table points at, reached through the EFI system table request and read where the memory map shows the direct map covers it. The reader refuses a tree with node ids and no memory node instead of answering with an empty memory affinity.
# Conflicts: # Cargo.lock # aarch64/Cargo.toml # aarch64/src/lib.rs # aarch64/src/platform/mod.rs # kernel/src/exec/executor.rs # kernel/src/memory/frame_slab.rs # kernel/src/memory/mod.rs # kernel/src/memory/user.rs # x86/src/smp.rs
lexoliu
marked this pull request as ready for review
November 13, 2026 05:17
# Conflicts: # Cargo.lock # Cargo.toml # aarch64/Cargo.toml # inspector/src/vm.rs # x86/Cargo.toml # x86/src/lib.rs # x86/src/smp.rs
# Conflicts: # kernel/src/network/service/shard.rs # kernel/src/test_support.rs
# Conflicts: # kernel/benches/data_structures.rs # kernel/src/lib.rs
# Conflicts: # kernel/src/network/service/mod.rs
# Conflicts: # Cargo.toml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The kernel learns which node each processor and each stretch of memory belongs to and places memory and tasks by it.
halgains the NUMA contract: memory regions carry their node and aNumaTopology(nodes, processors per node, a distance matrix) reaches the kernel through the platform.firmware/reads it from a device tree (numa-node-id,distance-map) and from the ACPI SRAT and SLIT; riscv and aarch64 use the first, x86 and aarch64 under--acpithe second. A machine that publishes neither is one node. The ACPI reader needs SRAT and SLIT accessors theacpicrate gets wrong, so the workspace'sacpidependency moves to the fork'sheliosbranch, which carries that fix (Fix SRAT entry layout and SLIT distance lookup rust-osdev/acpi#381) beside the notify handler processor hotplug needs (aml: deliver Notify to the host through Handler::handle_notify rust-osdev/acpi#380).helios-inspector vm --numa-nodes N [--numa-distance D]presents a NUMA machine on every architecture.Three repairs a multi-node boot needed ride along, each under its own issue: riscv read only the first devicetree
memorynode (#561); riscv PLIC discovery stopped at the first PLIC and every hart's contexts were opened from the boot hart, which left a second socket's interrupts masked after an OpenSBI reset (#562, #863); andKernelPhysFrameAllocator, which nothing used, is removed rather than ported.On aarch64 the device tree Limine hands over cannot carry the topology: Limine 12 deletes every
/memorynode from it, and thenuma-node-idof each with them. The backend therefore reads NUMA from the firmware's own tree, found through the UEFI configuration table's DTB entry (aarch64/src/platform/efi.rs, Limine's EFI system table request) and accepted only where the memory map says the firmware or the loader owns the range. A tree that names nodes on processors and none on memory is refused rather than booted as one node.Processor lifecycle: a wake goes only to an online processor, nearest node first; a departing processor leaves its node's queue in place and wakes a successor, which steals from it; the frames it cached drain into their own nodes' pools; and a processor that returns keeps the node the topology gave it. Every per-processor and per-node structure is sized by the possible processors, so a hot-added processor lands in the node its SRAT entry names;
--numa-nodescomposes with--max-processorswhen the nodes divide it and is refused otherwise.Concurrency: a node's pool is one
IrqSafeMutextaken from any processor; the allocating processor's index chooses the node and nothing else, so a foreign free takes the owning node's lock like a local one. The topology is immutable after boot.Evidence
NUMA topology nodes=2line against what QEMU published: riscv64 and aarch64 from the device tree, x86-64 and aarch64 from ACPI (run 43367028877; smoke-aarch64 printsNUMA topology nodes=2 distances=10,20;20,10for both its boots).a_cross_processor_free_returns_the_frame_to_its_own_nodeand its neighbours inkernel/src/memory/user.rsdrive a two-node plan for node-local placement and distance-ordered fallback;firmware/tests/acpi.rsand the devicetree tests cover the readers.instance-startup-10049.0 vs 48.1 / 47.0,spawn-wait60.8 vs 59.3 / 59.0,sched-tasks106 vs 108 / 106,pipe-pingpong163 vs 158 / 161,hostcall-loop192 vs 194 / 191,tcp-throughput47.7 vs 47.4 / 45.6,cpython-json87.9 vs 87.9 / 87.3,aot-curl247 vs 246 / 242, controlquickjs-loop20.4 vs 20.2 / 19.9. No row leaves the baseline pair's own spread exceptspawn-wait, by 2.5% inside overlapping intervals ([60.5, 66.9] vs [59.2, 61.3]).sched-tasks100 vs 99.5 / 99.4,spawn-wait56.2 vs 55.2 / 54.7,pipe-pingpong153 vs 151 / 155,hostcall-loop204 vs 203 / 229,instance-startup-10044.2 vs 44.0 / 42.7,tcp-throughput39.1 vs 40.0 / 39.4, controlquickjs-loop21.9 vs 22.1 / 22.2. Its gate reports incomplete evidence:fs-smallfileshad a rejected cell on that host, so that row rests on the first run alone.No speed claim is made for placement: an emulated topology on a one-node host has uniform memory latency. That measurement is what #516 stays open for.
Fixes #561
Fixes #562
Fixes #863
Part of #516