recentish Alpine don't love >64 vCPUs

[iximeow]

I noticed that a VM on a racklette seemed to get stuck at booting 'Linux virt' when I gave it a bunch (140) vCPUs. if you look at per-processor load, a bunch of vCPUs end up sitting at 100% in guest code, but it never really makes progress towards booting.

I've narrowed this down to a Linux change somewhere between 6.12.31 (Alpine 3.22.0) and 6.12.38 (Alpine 3.22.1) but the changelogs for 6.12.33 to 6.12.36 are ... voluminous.

this is very easily reproducible on any helios system and propolis-standalone, you really do just need to set cpus = 66 or whatever. I was using 80 as a nice round number. I'm probably holding Linux wrong but I couldn't quickly get log lines out to the serial port while it's booting, so I'm not sure where it gets stuck. bit I did notice that all vCPUs get busy for a short period early in boot that seems to line up with smpboot: x86: Booting SMP configuration:, a line about all the CPUs, and then smp: Brought up 1 node, 64 CPUs.

I basically expect that this is #940 in some weird way, but it'd be good to be sure what this is or isn't.

[iximeow]

later update here, Ubuntu 26.04 does not seem afflicted by .. whatever is happening here .. though it also has a moment of busying every vCPU over the course of booting. if this is directly from Linux, that suggests that whatever's in the way of booting is either fixed by 7.0 or part of a backport to 6.12 maybe?

[iximeow]

somewhere between Alpine 3.22.0 and 3.22.1 they started setting CONFIG_NUMA and a bunch of other stuff in the kernel config. grep NUMA /media/nvme0n1/boot/config-6.12.31-0-virt turns up # CONFIG_NUMA is not set whereas in 3.22.1 with config-6.12.38-0-virt you'll find CONFIG_NUMA=y and more importantly CONFIG_NODES_SHIFT=6.

then, among other things, struct node *node_devices[MAX_NUMNODES] is only 64 entries, and indexing it with any ol NUMA node ID means >64 nodes will index past the buffer and into random kernel data. great. this kills the kernel. it's as shrimple as that.

I was missing logs before because while I'd set verbose debug ignore_loglevel I had not set console=ttyS0 so it was very verbosely logging to nowhere. once you actually have logs going somewhere useful you'll find a kernel panic trying to alloc for perf counters as part of setting up the NMI watchdog, and if you force your way past that by turning off the NMI watchdog there's a slightly more clear failure later on when setting up sysfs for the cpu. either way, yep, this is #940 and NUMA-aware Linuxes probably do not like this state of affaris at all.