fix: protect the hardware when a fan speed stops making sense - #19
Merged
Conversation
An audit of everything nvfd can leave the GPU doing turned up three ways
the fans can end up too slow for the die temperature, none of which any
existing code path notices.
Thermal failsafe. A curve holds its last point's speed at every
temperature above it, and manual mode never reads temperature at all, so
`nvfd 0 manual 30` pins a loaded card at 30% forever. Before every write
the daemon now raises - never lowers - the computed speed to 100% once
the GPU reaches a few degrees below its own throttle point, read from
the driver via nvmlDeviceGetTemperatureThreshold (87 C on an RTX 5090:
GPU_MAX 90 less a 3 C margin). It releases 5 C lower so it cannot
chatter at the boundary, and both edges are logged. The decision itself
is fan_failsafe_speed() in speed.c with unit tests; main.c keeps the
probing and the logging.
Falling curves. curve_load validated ranges and duplicates but never
shape, so {"30":100,"80":30} loaded clean and made the fan slow down as
the die heated - positive feedback that runs away to the hardware's own
throttle point. curve_write now refuses to save a curve that falls, from
the CLI and the editor alike. An existing one still loads, because a
daemon that will not start is worse than a badly shaped curve, but it is
logged once per start.
Fans after an unclean exit. NVML manual fan policy outlives the process
that set it. The daemon resets on every exit it can see, but not on
SIGKILL, the OOM killer or a segfault, and not once StartLimitBurst
gives up - so a GPU could sit pinned at 30% until the next reboot. Adds
a fans-only `nvfd reset-fans` verb and wires it to ExecStopPost=, which
systemd runs however the process died. `nvfd auto` is not usable there:
it would rewrite the operator's saved modes. WatchdogSignal=SIGTERM
likewise makes a watchdog kill land on the clean-shutdown path.
Also, three smaller things the audit turned up:
- manual_speed() logged its clamp warning every poll, 17k journal lines
a day for one misconfigured GPU. It now warns once per distinct value.
- utils/nvfd-fan-control.sh trapped only INT and TERM, so a `set -e`
abort - an unguarded nvidia-smi during a driver reset, say - skipped
cleanup and left every GPU in curve mode with no supervisor. It now
traps EXIT too, guards the temperature read, and no longer unlinks the
lock file it still holds, which had let a second instance take a fresh
inode and run concurrently.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I went looking for anything that could shorten the life of the card, and found three ways the fans can end up too slow for the temperature.
A curve holds its last point's speed at everything above it, and manual mode never looks at temperature at all, so
nvfd 0 manual 30keeps a loaded card at 30% forever. The daemon now pushes the speed up to 100% — never down — once the GPU gets within a few degrees of throttling itself. The threshold comes from the driver rather than being hardcoded; on a 5090 that lands at 87 °C. It backs off 5 °C lower so it can't chatter at the boundary.A falling curve like
{"30":100,"80":30}used to load fine and make the fan slow down as the card heated up, which feeds back on itself. Saving one is now refused, from the CLI and the editor. An existing one still loads — a daemon that won't start is worse than a bad curve — but it gets logged once per start.NVML's manual fan policy outlives the process that set it, and #17 only covers the exits the daemon can see. SIGKILL, the OOM killer, a segfault, or the start limit giving up would all leave a GPU pinned until the next reboot. So there's now a
nvfd reset-fansverb wired toExecStopPost=, which systemd runs however the process died. It only touches the fans —nvfd autowould rewrite the saved per-GPU modes, which isn't what you want after a crash.WatchdogSignal=SIGTERMalso makes a watchdog kill go through the normal shutdown path instead of SIGABRT, which nothing catches.Two smaller ones while I was in there. The clamp warning was logging every poll, about 17k journal lines a day for one bad config — now once per distinct value. And
utils/nvfd-fan-control.shonly trapped INT and TERM, so aset -eabort (annvidia-smihiccup during a driver reset, say) skipped cleanup and left every GPU in curve mode with nothing supervising it; it also deleted the lock file it was still holding, which let a second copy run alongside it.Tested in an Ubuntu 24.04 / systemd 255 container with the real device nodes against a 5090: the failsafe arms at 87 °C,
reset-fansexits 0 and leaves config.json byte-identical,kill -9on the main PID triggers the ExecStopPost reset,nvfd curve 40 20is refused with the reason, an existing falling curve still starts the daemon and warns once, and Type=notify, the watchdog,systemctl reloadand a clean stop all behave as before.make checkcovers the new failsafe logic.One thing I deliberately left out. With the driver in charge this card sits at 0% fan at 35 °C; under curve or manual mode
FAN_SPEED_MIN=30means it never drops below 30%, because the floor is applied right at the NVML call. On a box that mostly idles that's something like 8,700 extra fan-hours a year. Restoring the zero-RPM window means handing the fans back to the driver below a threshold and reclaiming them above it, which is a behaviour change rather than a fix — happy to do it separately if you want it.