Build without the CUDA toolkit: declare the NVML entry points nvfd uses - #15
Conversation
nvfd only needs NVML, and NVML ships with every NVIDIA driver as libnvidia-ml.so.1. The only reason the CUDA toolkit was required was nvml.h, and installing the toolkit through a distribution package (nvidia-cuda-toolkit on Debian/Ubuntu) can replace or pin the driver on the host. On Ubuntu 22.04 the packaged header is also CUDA 11.5, which predates nvmlDeviceSetFanControlPolicy, so the #ifdef in fan.c silently compiled the auto-restore path out. - Add include/nvml_api.h declaring the 17 NVML entry points nvfd calls, with the enum values and struct layouts from nvml.h. Signatures are part of the versioned ABI (_v2 suffixes), so this is stable. - Link by SONAME (-l:libnvidia-ml.so.1) so no libnvidia-ml.so dev symlink is needed either. Compile with -Werror=implicit-function-declaration so an undeclared NVML call is a build error, not a silent implicit int. - nvmlDeviceSetFanControlPolicy is always compiled in and its result checked. Minimum driver is R520: the first branch exporting every symbol (Set*FanSpeed_v2 are R515; SetFanControlPolicy is 520.61.05, backported to 515.105.01). - install.sh: stop installing nvidia-cuda-toolkit. Preflight libnvidia-ml.so.1 via ldconfig before installing anything, build, probe NVML with the fresh build/nvfd list (which also runs the v1.x migration), and only then stop the old service, install and start, so an NVML failure leaves the host untouched instead of half-upgraded. The link directory comes from the same ldconfig line, so the linker searches where ld.so actually resolves the library. - CI: build without the toolkit, and link against a stub libnvidia-ml.so.1 built from tests/nvml_stub.c, which defines every declared entry point with the same signature. This exercises the -l: link line and the symbol set, and fails to compile if nvml_api.h drifts.
There was a problem hiding this comment.
Thanks, this is a good change and I'd like to take it. I checked nvml_api.h against the nvml.h that ships with driver 595 and everything matches (names, _v2 suffixes, enum values, struct layouts). Built and ran it here on a 5090 without any CUDA headers, works fine.
One thing before merging: plain make (which the README still documents) fails on Debian, because libnvidia-ml1 puts the library in /usr/lib/x86_64-linux-gnu/nvidia/current/, which ld.so knows about but ld doesn't. install.sh is fine since it passes -L. Could you move the ldconfig -p lookup into the Makefile so both paths work? Something like:
NVML_LIBDIR ?= $(shell /sbin/ldconfig -p 2>/dev/null | awk '/libnvidia-ml\.so\.1 \(libc6,/{print $$NF; exit}' | xargs -r dirname)
LDFLAGS += $(if $(NVML_LIBDIR),-L$(NVML_LIBDIR))Two small things, take or leave: nvmlFanControlPolicy_t is unsigned int + #defines in nvml.h rather than an enum, and \(libc6 also matches i386 entries on machines with 32-bit driver libs (\(libc6, doesn't).
Nice side effect of running nvfd list before make install: the old curve file actually gets migrated now, which it didn't before.
|
Moved NVML library-directory discovery into the Makefile, with an override and 64-bit |
2 participants
This one is about the install experience. The only reason nvfd needs the CUDA toolkit is
nvml.h, and on Debian/Ubuntuinstall.shgets it by installingnvidia-cuda-toolkit— which is a few gigabytes and, more importantly, can pull a different driver stack in behind your back. On my box (Ubuntu 22.04, driver 595 from the graphics-drivers PPA) I really didn't want apt touching the driver. And even if you accept that, 22.04's packaged toolkit is CUDA 11.5, whose header predatesnvmlDeviceSetFanControlPolicy, so the#ifdefinfan.cquietly compiled the auto-restore path out.NVML itself ships with every driver as
libnvidia-ml.so.1, and its ABI is stable and versioned by symbol name — that's what the_v2suffixes are for. So this PR declares the 17 entry points nvfd actually calls ininclude/nvml_api.h(same signatures, enum values and struct layouts asnvml.h) and drops the<nvml.h>include. The Makefile links by SONAME (-l:libnvidia-ml.so.1) so the unversioned.sodev symlink isn't needed either, and adds-Werror=implicit-function-declarationso a call to something not declared in that header is a build error rather than a silent implicitint.nvmlDeviceSetFanControlPolicyis now always compiled in, with its return value checked.A few things I changed in
install.shwhile I was in there:nvidia-cuda-toolkit. It checks forlibnvidia-ml.so.1withldconfig -pbefore installing anything, and feeds that directory to the linker so it looks whereld.soactually resolves the library.build/nvfd list) happen before the old service is stopped or anything is copied into/usr/local. Previously the probe ran aftermake installwith|| true, so an NVML failure went unnoticed; if it had failed loudly at that point it would have left a half-upgraded host. Now a failure leaves the machine exactly as it was.For CI, the runner obviously has no driver, so
tests/nvml_stub.cdefines every declared entry point with the same signature and CI builds it into a stublibnvidia-ml.so.1and links nvfd against it. That exercises the link line and the full symbol set instead of just compiling objects, and the stub fails to compile ifnvml_api.hever drifts.Minimum driver ends up being R520 — the first branch whose
libnvidia-ml.so.1exports everything here (Set*FanSpeed_v2are R515;SetFanControlPolicyarrived in 520.61.05 and was backported to 515.105.01). The README says so.Verified the 17 symbols resolve in driver 595.84's
libnvidia-ml.so.1withnm -D, and the daemon has been running against it on an RTX 6000 Ada. If you'd rather keepnvml.has an option I can make the header switchable, but I suspect the simpler story is better here.